You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "caozj1011 (via GitHub)" <gi...@apache.org> on 2023/09/22 07:35:54 UTC
[GitHub] [iotdb] caozj1011 opened a new pull request, #11197: Optimize merge selection process and increase disk space availability logic
caozj1011 opened a new pull request, #11197:
URL: https://github.com/apache/iotdb/pull/11197
## Description
This PR mainly has the following upgrades:
1. Added the selection criteria for merging tasks in the space. When the availability of disk space is insufficient, TsFile containing mods file is preferentially selected.
2. Compaction adds the CompactionTaskType attribute to a task and compacts the task to `NORMAL` and `MOD_SETTLE`. NORMAL is the default value. When the disk size of the mod file exceeds 50M or the disk space availability is insufficient, the mods file sorting task is preferred. In this case, the task type is `MOD_SETTLE`. When a task is executed, the system determines the disk space availability. If the task type is NORMAL and the disk space availability is insufficient, the task is discarded from the priority queue to avoid unexpected errors.
3. Optimized the return value of the `selectInnerSpaceTask()` method. The previous return value was not convenient to wrap the CompactionTask object, so we made the logic of this method more pure, the purpose is to return a list of tasks that can be executed, which can be put directly into the task queue or directly executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] caozj1011 commented on a diff in pull request #11197: Optimize merge selection process and increase disk space availability logic
Posted by "caozj1011 (via GitHub)" <gi...@apache.org>.
caozj1011 commented on code in PR #11197:
URL: https://github.com/apache/iotdb/pull/11197#discussion_r1334078355
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/CrossSpaceCompactionTask.java:
##########
@@ -366,6 +366,11 @@ public boolean checkValidAndSetMerging() {
resetCompactionCandidateStatusForAllSourceFiles();
return false;
}
+ if (compactionTaskType == CompactionTaskType.NORMAL && !CompactionUtils.isDiskHaveSpace()) {
Review Comment:
good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #11197: Optimize merge selection process and increase disk space availability logic
Posted by "xingtanzjr (via GitHub)" <gi...@apache.org>.
xingtanzjr commented on code in PR #11197:
URL: https://github.com/apache/iotdb/pull/11197#discussion_r1334023257
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/CrossSpaceCompactionTask.java:
##########
@@ -366,6 +366,11 @@ public boolean checkValidAndSetMerging() {
resetCompactionCandidateStatusForAllSourceFiles();
return false;
}
+ if (compactionTaskType == CompactionTaskType.NORMAL && !CompactionUtils.isDiskHaveSpace()) {
Review Comment:
Maybe we can make this judgement as a method of parent class to get rid of duplicated codes in InnerSpaceCompaction
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #11197: Optimize merge selection process and increase disk space availability logic
Posted by "xingtanzjr (via GitHub)" <gi...@apache.org>.
xingtanzjr commented on code in PR #11197:
URL: https://github.com/apache/iotdb/pull/11197#discussion_r1334024755
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/CompactionUtils.java:
##########
@@ -466,4 +472,29 @@ public static void deleteSourceTsFileAndUpdateFileMetrics(
FileMetrics.getInstance().deleteFile(fileSizes, seq, fileNames);
}
}
+
+ public static boolean isDiskHaveSpace() {
Review Comment:
Maybe rename it to `isDiskHasSpace` or using another description ? 😄
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/utils/CompactionUtils.java:
##########
@@ -466,4 +472,29 @@ public static void deleteSourceTsFileAndUpdateFileMetrics(
FileMetrics.getInstance().deleteFile(fileSizes, seq, fileNames);
}
}
+
+ public static boolean isDiskHaveSpace() {
Review Comment:
Maybe rename it to `isDiskHasSpace` or using another description ? 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] caozj1011 commented on a diff in pull request #11197: Optimize merge selection process and increase disk space availability logic
Posted by "caozj1011 (via GitHub)" <gi...@apache.org>.
caozj1011 commented on code in PR #11197:
URL: https://github.com/apache/iotdb/pull/11197#discussion_r1334078794
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/CrossSpaceCompactionTask.java:
##########
@@ -366,6 +366,11 @@ public boolean checkValidAndSetMerging() {
resetCompactionCandidateStatusForAllSourceFiles();
return false;
}
+ if (compactionTaskType == CompactionTaskType.NORMAL && !CompactionUtils.isDiskHaveSpace()) {
Review Comment:
good idea
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/CrossSpaceCompactionTask.java:
##########
@@ -366,6 +366,11 @@ public boolean checkValidAndSetMerging() {
resetCompactionCandidateStatusForAllSourceFiles();
return false;
}
+ if (compactionTaskType == CompactionTaskType.NORMAL && !CompactionUtils.isDiskHaveSpace()) {
Review Comment:
good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #11197: Optimize merge selection process and increase disk space availability logic
Posted by "xingtanzjr (via GitHub)" <gi...@apache.org>.
xingtanzjr commented on code in PR #11197:
URL: https://github.com/apache/iotdb/pull/11197#discussion_r1334021803
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/execute/task/CrossSpaceCompactionTask.java:
##########
@@ -366,6 +366,11 @@ public boolean checkValidAndSetMerging() {
resetCompactionCandidateStatusForAllSourceFiles();
return false;
}
+ if (compactionTaskType == CompactionTaskType.NORMAL && !CompactionUtils.isDiskHaveSpace()) {
+ LOGGER.debug(
Review Comment:
using INFO is OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [iotdb] xingtanzjr merged pull request #11197: Optimize merge selection process and increase disk space availability logic
Posted by "xingtanzjr (via GitHub)" <gi...@apache.org>.
xingtanzjr merged PR #11197:
URL: https://github.com/apache/iotdb/pull/11197
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org