You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "RexXiong (via GitHub)" <gi...@apache.org> on 2024/08/12 04:28:45 UTC
[PR] [CELEBORN-1558]Fix the incorrect decrement of pendingWrites in handlePushMergeData [celeborn]
RexXiong opened a new pull request, #2677:
URL: https://github.com/apache/celeborn/pull/2677
…PushMergeData.
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
1. Fix the incorrect decrement of pendingWrites for FileWriter
2. Improve some logs about hardSplit/ExceptionLogs
### Why are the changes needed?
There are multiple file writers that write data in handlePushMergeData. If the previous FileWriter has already been closed, the next decrementPendingWrites will use an incorrect FileWriter.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GA & manual test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1558] Fix the incorrect decrement of pendingWrites in handlePushMergeData [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #2677:
URL: https://github.com/apache/celeborn/pull/2677#discussion_r1713262326
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionDataWriter.java:
##########
@@ -566,7 +566,7 @@ protected void waitOnNoPending(AtomicInteger counter) throws IOException {
waitTime -= WAIT_INTERVAL_MS;
}
if (counter.get() > 0) {
- IOException ioe = new IOException("Wait pending actions timeout.");
+ IOException ioe = new IOException("Wait pending actions timeout, Counter: " + counter.get());
Review Comment:
Pending actions have two types: numPendingWrites and numPendingFlushes, So I use counter here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1558]Fix the incorrect decrement of pendingWrites in handlePushMergeData [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on code in PR #2677:
URL: https://github.com/apache/celeborn/pull/2677#discussion_r1713192171
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/PartitionDataWriter.java:
##########
@@ -566,7 +566,7 @@ protected void waitOnNoPending(AtomicInteger counter) throws IOException {
waitTime -= WAIT_INTERVAL_MS;
}
if (counter.get() > 0) {
- IOException ioe = new IOException("Wait pending actions timeout.");
+ IOException ioe = new IOException("Wait pending actions timeout, Counter: " + counter.get());
Review Comment:
```suggestion
IOException ioe = new IOException("Wait pending actions timeout, pending flushes count: " + counter.get());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1558] Fix the incorrect decrement of pendingWrites in handlePushMergeData [celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture closed pull request #2677: [CELEBORN-1558] Fix the incorrect decrement of pendingWrites in handlePushMergeData
URL: https://github.com/apache/celeborn/pull/2677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1558]Fix the incorrect decrement of pendingWrites in handlePushMergeData [celeborn]
Posted by "mridulm (via GitHub)" <gi...@apache.org>.
mridulm commented on PR #2677:
URL: https://github.com/apache/celeborn/pull/2677#issuecomment-2283155697
+CC @akpatnam25
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org