Re: [PR] Fix realtime table deletion races with segment creation [pinot]

["Jackie-Jiang (via GitHub)" <gi...@apache.org>]

Jackie-Jiang commented on code in PR #19468:
URL: https://github.com/apache/pinot/pull/19468#discussion_r4087665815


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/realtime/RealtimeTableDataManager.java:
##########
@@ -542,7 +550,29 @@ private void doAddConsumingSegment(String segmentName)
       return;
     }
     IndexLoadingConfig indexLoadingConfig = fetchIndexLoadingConfig();
-    handleSegmentPreload(zkMetadata, indexLoadingConfig);
+    LLCSegmentName llcSegmentName = new LLCSegmentName(segmentName);
+    int partitionGroupId = llcSegmentName.getPartitionGroupId();
+    PartitionUpsertMetadataManager partitionUpsertMetadataManager;
+    PartitionDedupMetadataManager partitionDedupMetadataManager;
+    synchronized (_segmentDataManagerMap) {

Review Comment:
   An admitted CONSUMING callback is not included in BaseTableDataManager.shutDown()'s wait. It can pass this check, pause, let deleteTable() finish and a same-name table start, then resume through FileUtils.deleteQuietly(_indexDir/segmentName) and the MmapMemoryManager constructor before the later shutdown check. LLC names use minute precision and a recreated table starts its partition sequence at 0, so a same-minute recreation can reuse the name and the old callback can delete the new consumer's files. Could shutdown drain admitted consuming callbacks before releasing the table lifecycle lock? A regression test should pause the old callback before filesystem cleanup while the replacement uses the shared data directory.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org