You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "J-HowHuang (via GitHub)" <gi...@apache.org> on 2026/04/20 21:33:16 UTC

Re: [PR] Fix parallel segment reload race on IndexLoadingConfig tier; reuse configs per tier and IndexLoadingConfig.copy() [pinot]

J-HowHuang commented on code in PR #18174:
URL: https://github.com/apache/pinot/pull/18174#discussion_r3113798227


##########
pinot-core/src/main/java/org/apache/pinot/core/data/manager/BaseTableDataManager.java:
##########
@@ -926,17 +926,18 @@ protected void setZkOperationTimeIfAvailable(ImmutableSegment segment, @Nullable
     }
   }
 
-  private void reloadSegments(List<SegmentDataManager> segmentDataManagers, IndexLoadingConfig indexLoadingConfig,
+  private void reloadSegmentDataManagersInParallel(List<SegmentDataManager> segmentDataManagers,
       boolean forceDownload, String reloadJobId)
       throws Exception {
+    IndexLoadingConfig indexLoadingConfigTemplate = fetchIndexLoadingConfig();
     List<String> failedSegments = new ArrayList<>();
     AtomicReference<Throwable> sampleException = new AtomicReference<>();
     CompletableFuture.allOf(segmentDataManagers.stream().map(segmentDataManager -> CompletableFuture.runAsync(() -> {
       String segmentName = segmentDataManager.getSegmentName();
       try {
         _segmentReloadSemaphore.acquire(segmentName, _logger);
         try {
-          reloadSegment(segmentDataManager, indexLoadingConfig, forceDownload);
+          reloadSegment(segmentDataManager, indexLoadingConfigTemplate.copy(), forceDownload);

Review Comment:
   > Since offline reload mutates IndexLoadingConfig (tier, setTableDataDir, etc.), so the same instance can’t be used concurrently without coordination.
   
   I don't get this. Once you make a copy of index loading config per tier, you can then set tier and table data dir on the copy for each tier, and there's no need to mutate them within each reload task, no? Synchronizing the reload operation here is a significant performance degradation, which is definitely not preferred.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org