You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@iotdb.apache.org by "shuwenwei (via GitHub)" <gi...@apache.org> on 2023/07/19 10:16:38 UTC

[GitHub] [iotdb] shuwenwei opened a new pull request, #10601: Fix a bug of cross space selection

shuwenwei opened a new pull request, #10601:
URL: https://github.com/apache/iotdb/pull/10601

   ## Description
   If an unsequence file is generated by a load operation, some devices in it may not overlap with any files in sequence space, these devices' startTime may larger than the latest endTime of it in sequence space. When performing a cross-space compaction, the selected sequence file may not contain these devices, causing these devices to be allocated to the first target file when performing the merge. 
   This PR ensures that if there have some sequence files before the time range of the unsequence device , the last sequence file will also be selected.
   <img width="472" alt="截屏2023-07-19 18 15 04" src="https://github.com/apache/iotdb/assets/55970239/6e15d5b7-7453-4bf2-91cf-a228a0f43495">
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [iotdb] shuwenwei commented on a diff in pull request #10601: Fix a bug of cross space selection

Posted by "shuwenwei (via GitHub)" <gi...@apache.org>.
shuwenwei commented on code in PR #10601:
URL: https://github.com/apache/iotdb/pull/10601#discussion_r1270374914


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/utils/CrossSpaceCompactionCandidate.java:
##########
@@ -113,15 +116,39 @@ private boolean prepareNextSplit() throws IOException {
           }
           nextUnseqFileHasOverlap = true;
           // if this condition is satisfied, all subsequent seq files is unnecessary to check
+          atLeastOneSelectedSeqFileContainsCurrentDevice = true;
           break;
         } else if (unseqDeviceInfo.startTime <= seqDeviceInfo.endTime) {
           if (!seqFile.selected) {
             ret.add(seqFile);
             seqFile.markAsSelected();
           }
+          atLeastOneSelectedSeqFileContainsCurrentDevice = true;
           nextUnseqFileHasOverlap = true;
+        } else if (!atLeastOneSelectedSeqFileContainsCurrentDevice) {
+          lastSeqFileWhoseDeviceTimeBeforeCurrentUnseqDevice = seqFile;
         }
       }
+      // If an unsequence file is generated by a load operation or some
+      // files in sequence space is deleted, some devices in it may not overlap
+      // with any files in sequence space, these devices' startTime may larger
+      // than the latest endTime of it in sequence space.
+      // When performing a cross-space compaction, the selected sequence file may
+      // not contain these devices, causing these devices to be allocated to the
+      // first target file when performing the compaction.
+      // Here ensures that if there have some sequence files before the time range
+      // of the unsequence device , the last sequence file will also be selected unless
+      // it is unclosed.
+      boolean seqSpaceHasCurrentUnseqDeviceButNotOverlap =
+          (!atLeastOneSelectedSeqFileContainsCurrentDevice
+              && lastSeqFileWhoseDeviceTimeBeforeCurrentUnseqDevice != null);
+      if (!seqSpaceHasCurrentUnseqDeviceButNotOverlap
+          || lastSeqFileWhoseDeviceTimeBeforeCurrentUnseqDevice.unsealed()

Review Comment:
   If there is a unclosed sequence file which contains this device and it is not overlapped with the unsequence file, it will become this lastSeqFileWhoseDeviceTimeBeforeCurrentUnseqDevice. However, a unclosed file should not be selected in any case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [iotdb] xingtanzjr commented on a diff in pull request #10601: Fix a bug of cross space selection

Posted by "xingtanzjr (via GitHub)" <gi...@apache.org>.
xingtanzjr commented on code in PR #10601:
URL: https://github.com/apache/iotdb/pull/10601#discussion_r1270352394


##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/storageengine/dataregion/compaction/selector/utils/CrossSpaceCompactionCandidate.java:
##########
@@ -113,15 +116,39 @@ private boolean prepareNextSplit() throws IOException {
           }
           nextUnseqFileHasOverlap = true;
           // if this condition is satisfied, all subsequent seq files is unnecessary to check
+          atLeastOneSelectedSeqFileContainsCurrentDevice = true;
           break;
         } else if (unseqDeviceInfo.startTime <= seqDeviceInfo.endTime) {
           if (!seqFile.selected) {
             ret.add(seqFile);
             seqFile.markAsSelected();
           }
+          atLeastOneSelectedSeqFileContainsCurrentDevice = true;
           nextUnseqFileHasOverlap = true;
+        } else if (!atLeastOneSelectedSeqFileContainsCurrentDevice) {
+          lastSeqFileWhoseDeviceTimeBeforeCurrentUnseqDevice = seqFile;
         }
       }
+      // If an unsequence file is generated by a load operation or some
+      // files in sequence space is deleted, some devices in it may not overlap
+      // with any files in sequence space, these devices' startTime may larger
+      // than the latest endTime of it in sequence space.
+      // When performing a cross-space compaction, the selected sequence file may
+      // not contain these devices, causing these devices to be allocated to the
+      // first target file when performing the compaction.
+      // Here ensures that if there have some sequence files before the time range
+      // of the unsequence device , the last sequence file will also be selected unless
+      // it is unclosed.
+      boolean seqSpaceHasCurrentUnseqDeviceButNotOverlap =
+          (!atLeastOneSelectedSeqFileContainsCurrentDevice
+              && lastSeqFileWhoseDeviceTimeBeforeCurrentUnseqDevice != null);
+      if (!seqSpaceHasCurrentUnseqDeviceButNotOverlap
+          || lastSeqFileWhoseDeviceTimeBeforeCurrentUnseqDevice.unsealed()

Review Comment:
   which scenario does this line `lastSeqFileWhoseDeviceTimeBeforeCurrentUnseqDevice.unsealed()` avoid ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [iotdb] xingtanzjr merged pull request #10601: Fix a bug of cross space selection

Posted by "xingtanzjr (via GitHub)" <gi...@apache.org>.
xingtanzjr merged PR #10601:
URL: https://github.com/apache/iotdb/pull/10601


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@iotdb.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org