You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "leixm (via GitHub)" <gi...@apache.org> on 2026/08/06 03:33:57 UTC

[PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

leixm opened a new pull request, #3652:
URL: https://github.com/apache/celeborn/pull/3652

   ### What changes were proposed in this pull request?
   This PR makes partition file sorting asynchronous in the worker's FetchHandler, so that open stream requests no longer block RPC threads while waiting for sort completion.
   
   Key changes:
   
   PartitionFilesSorter: Replaced the blocking Thread.sleep polling loop in getSortedFileInfo with a callback-based async model using FileResolvedCallback and FileSortedCallback. When a sort is initiated, the caller's callback is invoked upon completion instead of blocking. Also introduced pendingSortCallbacks to correctly handle concurrent readers — when multiple open stream requests arrive for the same file being sorted, all their callbacks are notified upon sort completion.
   
   FetchHandler: Refactored the open stream handling into three clear responsibilities:
   
   openReduceStreamAsync — decides whether sorting is needed and dispatches to PartitionFilesSorter asynchronously or invokes the callback directly.
   registerAndHandleStream — pure stream registration logic (no sorting), called from the async callback with the resolved FileInfo.
   Both single PbOpenStream and batch PbOpenStreamList handlers now use the async model. The batch handler uses AtomicInteger completion counting and an ordered Array[PbStreamHandlerOpt] to aggregate results and send a single response when all files are resolved.
   FileResolvedCallback (new interface): Defines onSuccess(FileInfo) / onFailure(Throwable) for async file resolution notification.
   
   
   ### Why are the changes needed?
   Currently, getSortedFileInfo blocks the calling RPC thread with a Thread.sleep(50) polling loop (up to 220s timeout) while waiting for partition file sorting to complete. This ties up Netty RPC threads, which are a shared and limited resource. Under high concurrency — especially when many reduce tasks open streams simultaneously and trigger sorting — this can exhaust the RPC thread pool and cause timeouts or stalls for unrelated requests.
   
   By making the sort wait asynchronous via callbacks, RPC threads are released immediately after dispatching the sort request, significantly improving worker responsiveness and throughput under concurrent open stream workloads.
   
   
   ### Does this PR resolve a correctness bug?
   
   No.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. This is an internal implementation change. The RPC protocol and behavior remain unchanged.
   
   ### How was this patch tested?
   Existing UTs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3652: [CELEBORN-2065] Worker should support wait partition sort asynchronously
URL: https://github.com/apache/celeborn/pull/3652


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-4235012939

   Pretty similar to something I was trying to do with https://github.com/apache/celeborn/pull/3593. +1 on getting this in.
   
   I have one concern here though, it seems like we're making the sorting thread write and flush the response. This can mean that the sorting threads, supposed to purely handle sorting tasks are now assigned a new response serialization and flush responsibilities.
   
   Should we instead make sure that the response is instead sent by some other set of thread (preferrably the eventloop thread)? We can create a dedicate set of threads for responding to clients perhaps?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-5727457605

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "github-actions[bot] (via GitHub) via issues" <is...@celeborn.apache.org>.
github-actions[bot] commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-6735639103

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-4369690864

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-5452401244

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on code in PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#discussion_r4322138403


##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionFilesSorter.java:
##########
@@ -804,13 +870,18 @@ public void sort() {
       } catch (Exception e) {
         logger.error(
             "Sorting shuffle file for " + fileId + " " + originFilePath + " failed, detail: ", e);
+        success = false;
+        fileSortedCallback.onFailure(e);

Review Comment:
   [P1] Make failure completion atomic with waiter registration. This invokes the failure callback before fileId is removed from sorting. notifyPendingSortCallbacks removes the pending list and then invokes its callbacks; a concurrent getSortedFileInfo call during that notification still sees sorting.contains(fileId), creates a new pending list, and is never notified when the finally block clears sorting. This can leave an entire batch response waiting and retain the callback. Please transition the per-file state and detach its waiters under the same lock used for registration, then notify outside the lock, keeping each sort attempt's waiters separate. A deterministic test can pause one pending reader's failure callback and register another reader before releasing it.



##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionFilesSorter.java:
##########
@@ -243,73 +253,121 @@ public FileInfo getSortedFileInfo(
 
       String sortedFilePath = Utils.getSortedFilePath(diskFileInfo.getFilePath());
       String indexFilePath = Utils.getIndexFilePath(diskFileInfo.getFilePath());
-      boolean fileSorting = true;
       synchronized (sorting) {
         if (sorted.contains(fileId)) {
-          fileSorting = false;
-        } else if (!sorting.contains(fileId)) {
           try {
-            FileSorter fileSorter = new FileSorter(diskFileInfo, fileId, shuffleKey);
+            FileInfo sortedFileInfo =
+                resolve(
+                    shuffleKey,
+                    fileId,
+                    userIdentifier,
+                    sortedFilePath,
+                    indexFilePath,
+                    startMapIndex,
+                    endMapIndex);
+            fileResolvedCallback.onSuccess(sortedFileInfo);
+          } catch (Throwable e) {
+            fileResolvedCallback.onFailure(e);
+          }
+          return null;
+        } else if (sorting.contains(fileId)) {
+          FileResolvedCallback pendingCallback =
+              new FileResolvedCallback() {
+                @Override
+                public void onSuccess(FileInfo ignored) {
+                  try {
+                    FileInfo sortedFileInfo =
+                        resolve(
+                            shuffleKey,
+                            fileId,
+                            userIdentifier,
+                            sortedFilePath,
+                            indexFilePath,
+                            startMapIndex,
+                            endMapIndex);
+                    fileResolvedCallback.onSuccess(sortedFileInfo);
+                  } catch (Throwable e) {
+                    fileResolvedCallback.onFailure(e);
+                  }
+                }
+
+                @Override
+                public void onFailure(Throwable e) {
+                  fileResolvedCallback.onFailure(e);
+                }
+              };
+          pendingSortCallbacks
+              .computeIfAbsent(fileId, k -> new CopyOnWriteArrayList<>())
+              .add(pendingCallback);
+        } else {
+          FileSortedCallback fileSortedCallback =
+              new FileSortedCallback() {
+                @Override
+                public void onSuccess() {
+                  try {
+                    FileInfo sortedFileInfo =
+                        resolve(
+                            shuffleKey,
+                            fileId,
+                            userIdentifier,
+                            sortedFilePath,
+                            indexFilePath,
+                            startMapIndex,
+                            endMapIndex);
+                    fileResolvedCallback.onSuccess(sortedFileInfo);
+                  } catch (Throwable e) {
+                    fileResolvedCallback.onFailure(e);
+                  } finally {
+                    notifyPendingSortCallbacks(fileId, null);
+                  }
+                }
+
+                @Override
+                public void onFailure(Throwable e) {
+                  try {
+                    fileResolvedCallback.onFailure(e);
+                  } finally {
+                    notifyPendingSortCallbacks(fileId, e);
+                  }
+                }
+              };
+          try {
+            FileSorter fileSorter =
+                new FileSorter(diskFileInfo, fileId, shuffleKey, fileSortedCallback);
             sorting.add(fileId);
             logger.debug(
                 "Adding sorter to sort queue shuffle key {}, file name {}", shuffleKey, fileName);
             shuffleSortTaskDeque.put(fileSorter);

Review Comment:
   [P1] Preserve the configured sort timeout in the async path. Removing the polling loop also removes the only enforcement of celeborn.worker.sortPartition.timeout; sortTimeout is now only initialized. If sorting waits for prefetch memory or stalls in I/O, neither completion callback is delivered, and pending callbacks remain retained even after the client RPC times out. I reproduced this with prefetch enabled, sortMemoryReady() held false, and a 100ms sort timeout: the reader still had no callback after 2 seconds. Please add an independent deadline for each reader with exactly-once completion, remove expired waiters, and cancel the deadline on normal completion. A reader timing out must not fail the shared sort for other readers. Cover queued/stalled sorting and completion racing with or arriving after the deadline.



##########
worker/src/main/scala/org/apache/celeborn/service/deploy/worker/FetchHandler.scala:
##########
@@ -143,36 +145,92 @@ class FetchHandler(
         val startIndices = openStreamList.getStartIndexList
         val endIndices = openStreamList.getEndIndexList
         val readLocalFlags = openStreamList.getReadLocalShuffleList
-        val pbOpenStreamListResponse = PbOpenStreamListResponse.newBuilder()
         checkAuth(client, Utils.splitShuffleKey(shuffleKey)._1)
         val openStreamRequestId = Utils.makeOpenStreamRequestId(
           shuffleKey,
           client.getChannel.id().toString,
           rpcRequest.requestId)
         workerSource.startTimer(WorkerSource.OPEN_STREAM_TIME, openStreamRequestId)
-        try {
-          0 until files.size() foreach { idx =>
-            val pbStreamHandlerOpt = handleReduceOpenStreamInternal(
-              client,
+        val totalFiles = files.size()
+        val results = new Array[PbStreamHandlerOpt](totalFiles)
+        val completedCount = new AtomicInteger(0)
+        val replied = new AtomicBoolean(false)
+
+        def trySendBatchResponse(): Unit = {
+          if (completedCount.get() >= totalFiles && replied.compareAndSet(false, true)) {
+            val pbOpenStreamListResponse = PbOpenStreamListResponse.newBuilder()
+            results.foreach(pbOpenStreamListResponse.addStreamHandlerOpt)
+            workerSource.stopTimer(WorkerSource.OPEN_STREAM_TIME, openStreamRequestId)
+            client.getChannel.writeAndFlush(new RpcResponse(
+              rpcRequest.requestId,
+              new NioManagedBuffer(new TransportMessage(
+                MessageType.BATCH_OPEN_STREAM_RESPONSE,
+                pbOpenStreamListResponse.build().toByteArray).toByteBuffer)))
+          }
+        }
+
+        0 until totalFiles foreach { idx =>

Review Comment:
   [P2] Complete an empty batch as well. trySendBatchResponse is only called from per-file completion paths. With zero file names this loop never runs, so the valid empty PbOpenStreamList receives no response and its OPEN_STREAM_TIME timer is not stopped. Previously this returned an empty PbOpenStreamListResponse. Please invoke the completion check after dispatching the files, or handle totalFiles == 0 explicitly, and add an empty-batch regression test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "zaynt4606 (via GitHub)" <gi...@apache.org>.
zaynt4606 commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-6285118844

   @leixm GitHub currently reports merge conflicts with `main`. Please update this branch against the latest `main` and resolve the conflicts. In `FetchHandlerSuiteJ`, please preserve both the batch tests from this PR and `testFetchChunkForUnregisteredStream`, then rerun the affected tests.
   
   I'd like to keep this PR moving forward. The correctness findings and suggested next steps are in [my review](https://github.com/apache/celeborn/pull/3652#pullrequestreview-5581549994); happy to continue reviewing the updated version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-5077906120

   This issue was closed because it has been staled for 10 days with no activity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-4967295907

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-4240954253

   > Pretty similar to something I was trying to do with #3593. +1 on getting this in.
   > 
   > I have one concern here though, it seems like we're making the sorting thread write and flush the response. This can mean that the sorting threads, supposed to purely handle sorting tasks are now assigned a new response serialization and flush responsibilities.
   > 
   > Should we instead make sure that the response is instead sent by some other set of thread (preferrably the eventloop thread)? We can create a dedicate set of threads for responding to clients perhaps?
   
   The sort thread only needs to call `client.getChannel.writeAndFlush(new RpcResponse(xxx))`, which is a very lightweight step. The serialization of the `RpcResponse` object and the flushing at the network layer will be handled by Netty's event loop thread.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-5200443189

   ## [Codecov](https://app.codecov.io/gh/apache/celeborn/pull/3652?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 58.51%. Comparing base ([`b4cb5a0`](https://app.codecov.io/gh/apache/celeborn/commit/b4cb5a0b1ac097d33baf8dded1b5be2afd0578a4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`93b0879`](https://app.codecov.io/gh/apache/celeborn/commit/93b0879887446ba844ae60d8b1110284f6be2cbf?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 98 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##               main    #3652      +/-   ##
   ============================================
   - Coverage     66.91%   58.51%   -8.40%     
   - Complexity        0      229     +229     
   ============================================
     Files           358      399      +41     
     Lines         21986    28047    +6061     
     Branches       1946     2738     +792     
   ============================================
   + Hits          14710    16409    +1699     
   - Misses         6262    10439    +4177     
   - Partials       1014     1199     +185     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/celeborn/pull/3652?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-4212182565

   @SteNicholas  @RexXiong  Can you help review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-4240965389

   > client.getChannel.writeAndFlush
   
   Inside the `client.getChannel.writeAndFlush` method:
   
   if (executor.inEventLoop()) {
   // Execute write and flush operations synchronously
   } else {
   // Add to the eventExecutor's task queue
   safeExecute(executor, task, promise, m, !flush)
   }


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2065] Worker should support wait partition sort asynchronously [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3652:
URL: https://github.com/apache/celeborn/pull/3652#issuecomment-6077610666

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org