You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "reswqa (via GitHub)" <gi...@apache.org> on 2024/11/06 03:12:19 UTC
[PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
reswqa opened a new pull request, #2886:
URL: https://github.com/apache/celeborn/pull/2886
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Fix storageFetcherPool concurrent problem
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
No need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on code in PR #2886:
URL: https://github.com/apache/celeborn/pull/2886#discussion_r1830459333
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/CreditStreamManager.java:
##########
@@ -44,7 +44,8 @@ public class CreditStreamManager {
private final AtomicLong nextStreamId;
private final ConcurrentHashMap<Long, StreamState> streams;
private final ConcurrentHashMap<FileInfo, MapPartitionData> activeMapPartitions;
- private final HashMap<String, ExecutorService> storageFetcherPool = new HashMap<>();
+ private final ConcurrentHashMap<String, ExecutorService> storageFetcherPool =
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #2886:
URL: https://github.com/apache/celeborn/pull/2886#issuecomment-2458765463
## [Codecov](https://app.codecov.io/gh/apache/celeborn/pull/2886?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 32.35%. Comparing base [(`165e914`)](https://app.codecov.io/gh/apache/celeborn/commit/165e914b9b747164b240c8a68896f3b38e67434d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`c372969`)](https://app.codecov.io/gh/apache/celeborn/commit/c3729695f58c795ddc8030293c03c8914a02f57e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 13 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #2886 +/- ##
==========================================
+ Coverage 32.35% 32.35% +0.01%
==========================================
Files 329 329
Lines 19511 19538 +27
Branches 1752 1759 +7
==========================================
+ Hits 6311 6320 +9
- Misses 12855 12873 +18
Partials 345 345
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/celeborn/pull/2886?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas closed pull request #2886: [CELEBORN-1693] Fix storageFetcherPool concurrent problem
URL: https://github.com/apache/celeborn/pull/2886
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on code in PR #2886:
URL: https://github.com/apache/celeborn/pull/2886#discussion_r1830437240
##########
worker/src/main/java/org/apache/celeborn/service/deploy/worker/storage/CreditStreamManager.java:
##########
@@ -44,7 +44,8 @@ public class CreditStreamManager {
private final AtomicLong nextStreamId;
private final ConcurrentHashMap<Long, StreamState> streams;
private final ConcurrentHashMap<FileInfo, MapPartitionData> activeMapPartitions;
- private final HashMap<String, ExecutorService> storageFetcherPool = new HashMap<>();
+ private final ConcurrentHashMap<String, ExecutorService> storageFetcherPool =
Review Comment:
use JavaUtils.newConcurrentHashMap instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #2886:
URL: https://github.com/apache/celeborn/pull/2886#issuecomment-2461240330
CC @SteNicholas
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #2886:
URL: https://github.com/apache/celeborn/pull/2886#issuecomment-2459621251
This surely improved the potential concurrent issue of the storage fetch pool. But are there any known issues related to this point?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on PR #2886:
URL: https://github.com/apache/celeborn/pull/2886#issuecomment-2461211453
> > This surely improved the potential concurrent issue of the storage fetch pool. But are there any known issues related to this point?
>
> There may be duplicate thread pools created, pls describe this @reswqa see: 
Thanks, update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
Posted by "SteNicholas (via GitHub)" <gi...@apache.org>.
SteNicholas commented on PR #2886:
URL: https://github.com/apache/celeborn/pull/2886#issuecomment-2461386018
Merged to main(v0.6.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1693] Fix storageFetcherPool concurrent problem [celeborn]
Posted by "RexXiong (via GitHub)" <gi...@apache.org>.
RexXiong commented on PR #2886:
URL: https://github.com/apache/celeborn/pull/2886#issuecomment-2459883693
> This surely improved the potential concurrent issue of the storage fetch pool. But are there any known issues related to this point?
There may be duplicate thread pools created, pls describe this @reswqa see:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org