Re: [PR] [FLINK-31238] [WIP] Use IngestDB to speed up Rocksdb rescaling recovery (rebased) [flink]

["mayuehappy (via GitHub)" <gi...@apache.org>]

mayuehappy commented on code in PR #24031:
URL: https://github.com/apache/flink/pull/24031#discussion_r1481169952


##########
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/restore/RocksDBIncrementalRestoreOperation.java:
##########
@@ -539,13 +552,14 @@ private void rescaleClipIngestDB(
             }
 
             // If there are temporary instances for which we detected overlap in the buffer or
-            // if we have remaining handles to restore, we will insert by copy with from temporary
-            // instances to base DB.
+            // if we have remaining unopened handles to restore, we will insert by copy via
+            // temporary instances to base DB.

Review Comment:
   In the old code, we called `choseTheBestStateHandleForInitial` to choose the best state handle to init the initial db .Because if we use TheBestHandle as a temporary DB instead of the main DB, we may need to write a lot of data when copying. Can we maintain this logic int the new code ?
   When choosing db to export, we prioritize `TheBestHandle` to ensure that we will not copy TheBestHandle during the subsequent copying phase. Can this ensure that there is no regression compared to the old code
   
   



##########
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/restore/RocksDBIncrementalRestoreOperation.java:
##########
@@ -488,6 +495,8 @@ private void rescaleClipIngestDB(
                 List<ColumnFamilyHandle> tmpColumnFamilyHandles =
                         tmpRestoreDBInfo.columnFamilyHandles;
 
+                // Check if the data in all SST files referenced in the handle is within the
+                // proclaimed key-groups range of the handle.
                 if (RocksDBIncrementalCheckpointUtils.isSstDataInKeyGroupRange(

Review Comment:
   Here we need to check
   Is key in proclaimed range or is there overlap between the checking stateHandles?
   For example, the range of proclaimed is [1,5] [6,10] but the actual range is [1,7] [8,9]. Should it be possible to export in this case?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org