You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "saurabhd336 (via GitHub)" <gi...@apache.org> on 2026/02/03 06:24:44 UTC

[PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

saurabhd336 opened a new pull request, #3593:
URL: https://github.com/apache/celeborn/pull/3593

   
   
   ### What changes were proposed in this pull request?
   During openStream calls to workers, the worker netty threads can get blocked for `sortTimeout` milliseconds. This could degrade other worker RPC calls. Ideally, if we introduce an async reply mechanism, we can free the netty threads thereby ensuring other RPC calls are not impacted.
   
   
   ### Why are the changes needed?
   Avoid blocking worker netty threads during heavy sorting openStream calls.
   
   ### Does this PR resolve a correctness bug?
   No
   
   ### Does this PR introduce _any_ user-facing change?
   Set worker config "celeborn.worker.async.openStream.enabled" to move to async reply mechanism for openStream calls.
   
   
   
   ### How was this patch tested?
   TODO
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3593:
URL: https://github.com/apache/celeborn/pull/3593#issuecomment-4231137321

   This issue was closed because it has been staled for 10 days with no activity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3593:
URL: https://github.com/apache/celeborn/pull/3593#issuecomment-3957699035

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3593:
URL: https://github.com/apache/celeborn/pull/3593#issuecomment-4458621854

   This issue was closed because it has been staled for 10 days with no activity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on PR #3593:
URL: https://github.com/apache/celeborn/pull/3593#issuecomment-3958575435

   Hey @zaynt4606 thanks for reviewing this. Currently occupied with some other threads, let me revisit this PR this week.
   
   > Currently, every request is returned via the EventLoop thread, which might incur significant overhead when handling a large number of requests with long sort times.
   My idea was to simply make sure the eventloop thread isn't block (i.e. avoid the thread.sleep()), with the current changes the response is still sent by the eventloop threads, but the wait is eliminated. I can check if we can / should use a seperate threadpool for responding


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3593:
URL: https://github.com/apache/celeborn/pull/3593#issuecomment-4377927043

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3593: [WIP] Support async response to open stream calls to avoid blocking netty threads
URL: https://github.com/apache/celeborn/pull/3593


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3593: [WIP] Support async response to open stream calls to avoid blocking netty threads
URL: https://github.com/apache/celeborn/pull/3593


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3593:
URL: https://github.com/apache/celeborn/pull/3593#issuecomment-4168528643

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on code in PR #3593:
URL: https://github.com/apache/celeborn/pull/3593#discussion_r2757501772


##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionFilesSorter.java:
##########
@@ -313,6 +322,93 @@ public FileInfo getSortedFileInfo(
     }
   }
 
+  // Async variant: returns a future that completes with sorted FileInfo without blocking Netty threads.
+  public CompletableFuture<FileInfo> getSortedFileInfoAsync(
+      String shuffleKey,
+      String fileName,
+      FileInfo fileInfo,
+      int startMapIndex,
+      int endMapIndex,
+      Executor executor) {
+    if (fileInfo instanceof MemoryFileInfo) {
+      return CompletableFuture.completedFuture(getSortedMemoryFileInfo(
+              (MemoryFileInfo) fileInfo,
+              startMapIndex,
+              endMapIndex));
+    } else {
+      DiskFileInfo diskFileInfo = ((DiskFileInfo) fileInfo);
+      String fileId = shuffleKey + "-" + fileName;
+      Set<String> sorted =
+          sortedShuffleFiles.computeIfAbsent(shuffleKey, v -> ConcurrentHashMap.newKeySet());
+      Set<String> sorting =
+          sortingShuffleFiles.computeIfAbsent(shuffleKey, v -> ConcurrentHashMap.newKeySet());
+
+      String sortedFilePath = Utils.getSortedFilePath(diskFileInfo.getFilePath());
+      String indexFilePath = Utils.getIndexFilePath(diskFileInfo.getFilePath());
+
+      synchronized (sorting) {
+        if (!sorted.contains(fileId) && !sorting.contains(fileId)) {
+          try {
+            FileSorter fileSorter = new FileSorter(diskFileInfo, fileId, shuffleKey);
+            sorting.add(fileId);
+            logger.debug(
+                "Adding sorter to sort queue (async) shuffle key {}, file name {}",
+                shuffleKey,
+                fileName);
+            shuffleSortTaskDeque.put(fileSorter);
+          } catch (InterruptedException e) {
+            CompletableFuture<FileInfo> failed = new CompletableFuture<>();
+            failed.completeExceptionally(
+                new IOException(
+                    "Sort scheduler thread is interrupted means worker is shutting down.", e));
+            return failed;
+          } catch (IOException e) {
+            CompletableFuture<FileInfo> failed = new CompletableFuture<>();
+            failed.completeExceptionally(new IOException("File sorter access DFS failed.", e));
+            return failed;
+          }
+        }
+      }
+
+      if (sorted.contains(fileId)) {
+        try {
+          return CompletableFuture.completedFuture(
+              resolve(

Review Comment:
   This should already be running on the caller thread i.e. netty thread



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [WIP] Support async response to open stream calls to avoid blocking netty threads [celeborn]

Posted by "saurabhd336 (via GitHub)" <gi...@apache.org>.
saurabhd336 commented on PR #3593:
URL: https://github.com/apache/celeborn/pull/3593#issuecomment-4249909869

   @zaynt4606 Think we can close this in lieu of https://github.com/apache/celeborn/pull/3652, which seems to be solving the same problem and is already being reviewed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org