You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "mcdull-zhang (via GitHub)" <gi...@apache.org> on 2025/12/25 08:03:27 UTC
[PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]
mcdull-zhang opened a new pull request, #3572:
URL: https://github.com/apache/celeborn/pull/3572
### What changes were proposed in this pull request?
Improve performance by using memory swapping instead of memory copying.
### Why are the changes needed?
Improve performance
### Does this PR resolve a correctness bug?
NO
### Does this PR introduce _any_ user-facing change?
NO
### How was this patch tested?
Existing Unit Tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3572: [CELEBORN-2241] Memory swap instead of memory copy
URL: https://github.com/apache/celeborn/pull/3572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-3691192011
## [Codecov](https://app.codecov.io/gh/apache/celeborn/pull/3572?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 67.21%. Comparing base ([`ffff5bb`](https://app.codecov.io/gh/apache/celeborn/commit/ffff5bb94efd7c3a2c6c51aebccfd3c11b5d5198?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ef3c308`](https://app.codecov.io/gh/apache/celeborn/commit/ef3c308d859c5a8a2f23f9529548c8d4599f6429?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 16 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #3572 +/- ##
==========================================
+ Coverage 67.05% 67.21% +0.17%
==========================================
Files 357 357
Lines 21779 21847 +68
Branches 1930 1938 +8
==========================================
+ Hits 14602 14683 +81
+ Misses 6160 6144 -16
- Partials 1017 1020 +3
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/celeborn/pull/3572?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-3957699242
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]
Posted by "mcdull-zhang (via GitHub)" <gi...@apache.org>.
mcdull-zhang commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-3708854935
> SortBasedShuffleWriter HashBasedShuffleWriter
> Before 615.44W ms
> After 442.47W ms
> You mean that the running time dropped from 6000 seconds to 4000 seconds, that's very impressive.
@FMX It is the sum of shuffle write block time for 22 queries, not the end-to-end query duration;
And this is the total effect of multiple PRs;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-4168528808
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-4231137362
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-3707555594
SortBasedShuffleWriter | HashBasedShuffleWriter
-- | --
Before | 615.44W ms | 502.31W ms
After | 442.47W ms | 434.77W ms
You mean that the running time dropped from 6000 seconds to 4000 seconds, that's very impressive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org