You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "cfmcgrady (via GitHub)" <gi...@apache.org> on 2023/10/24 07:06:26 UTC
[PR] [CELEBORN-1082] Fixing partition sorter task failures due to duplicate sorting [incubator-celeborn]
cfmcgrady opened a new pull request, #2031:
URL: https://github.com/apache/incubator-celeborn/pull/2031
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
As title
### Why are the changes needed?
Recently, while testing on the main branch, we discovered that the partition sorter task might fail with a `NoSuchFileException`, leading to the entire job's failure. Upon further investigation, we identified that the root cause of this issue is the potential addition of the same sorting task to the sorter queue multiple times.
```
23/10/22 01:02:15,334 DEBUG [worker-file-sorter-execute-9530] PartitionFilesSorter: sort complete for application_1653035898918_4284043-9975 /data1/celeborn/worker/celeborn-worker/shuffle_data/application_1653035898918_4284043/9975/0-0-0
...
23/10/22 01:02:15,335 ERROR [worker-file-sorter-execute-9532] PartitionFilesSorter: Sorting shuffle file for application_1653035898918_4284043-9975-0-0-0 /data1/celeborn/worker/cele
born-worker/shuffle_data/application_1653035898918_4284043/9975/0-0-0 failed, detail:
java.nio.file.NoSuchFileException: /data1/celeborn/worker/celeborn-worker/shuffle_data/application_1653035898918_4284043/9975/0-0-0
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
at java.nio.channels.FileChannel.open(FileChannel.java:287)
at java.nio.channels.FileChannel.open(FileChannel.java:335)
at org.apache.celeborn.common.util.FileChannelUtils.openReadableFileChannel(FileChannelUtils.java:33)
at org.apache.celeborn.service.deploy.worker.storage.PartitionFilesSorter$FileSorter.initializeFiles(PartitionFilesSorter.java:641)
at org.apache.celeborn.service.deploy.worker.storage.PartitionFilesSorter$FileSorter.sort(PartitionFilesSorter.java:559)
at org.apache.celeborn.service.deploy.worker.storage.PartitionFilesSorter.lambda$null$0(PartitionFilesSorter.java:146)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
...
```
Before this PR, there was a scenario where sorter tasks for the same `fileId` could arrive after being removed from the `sorting` state, and they could be mistakenly added to the sorter queue. To address this, we moved the code block that checks the `fileId`'s status in `sorted` inside the `synchronized (sorting)` block. This change ensures that tasks are not added to the sorter queue multiple times because if a `fileId`'s sorter task has already completed and its status has been removed from `sorting`, it will definitely be present in `sorted`. This behavior is consistent with how it worked prior to version 0.3.0.
### Does this PR introduce _any_ user-facing change?
No, only bug fix
### How was this patch tested?
Pass GA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1082] Fixing partition sorter task failures due to duplicate sorting [incubator-celeborn]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #2031:
URL: https://github.com/apache/incubator-celeborn/pull/2031#issuecomment-1776650282
the bug was introduced in https://github.com/apache/incubator-celeborn/pull/1932
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1082] Fixing partition sorter task failures due to duplicate sorting [incubator-celeborn]
Posted by "cfmcgrady (via GitHub)" <gi...@apache.org>.
cfmcgrady commented on PR #2031:
URL: https://github.com/apache/incubator-celeborn/pull/2031#issuecomment-1776650685
cc @waitinfuture @FMX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1082] Fixing partition sorter task failures due to duplicate sorting [incubator-celeborn]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2031:
URL: https://github.com/apache/incubator-celeborn/pull/2031#issuecomment-1776658623
## [Codecov](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2031?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#2031](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2031?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (fd86199) into [main](https://app.codecov.io/gh/apache/incubator-celeborn/commit/6dc30d39765f6312579856d421b5f89e26f95a7c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (6dc30d3) will **increase** coverage by `0.01%`.
> Report is 2 commits behind head on main.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #2031 +/- ##
==========================================
+ Coverage 46.90% 46.90% +0.01%
==========================================
Files 165 165
Lines 10531 10535 +4
Branches 959 958 -1
==========================================
+ Hits 4938 4940 +2
- Misses 5271 5273 +2
Partials 322 322
```
[see 6 files with indirect coverage changes](https://app.codecov.io/gh/apache/incubator-celeborn/pull/2031/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1082] Fixing partition sorter task failures due to duplicate sorting [incubator-celeborn]
Posted by "waitinfuture (via GitHub)" <gi...@apache.org>.
waitinfuture closed pull request #2031: [CELEBORN-1082] Fixing partition sorter task failures due to duplicate sorting
URL: https://github.com/apache/incubator-celeborn/pull/2031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org