You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "mcdull-zhang (via GitHub)" <gi...@apache.org> on 2025/12/25 08:03:27 UTC

[PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]

mcdull-zhang opened a new pull request, #3572:
URL: https://github.com/apache/celeborn/pull/3572

   ### What changes were proposed in this pull request?
   Improve performance by using memory swapping instead of memory copying.
   
   
   ### Why are the changes needed?
   Improve performance
   
   
   ### Does this PR resolve a correctness bug?
   NO
   
   ### Does this PR introduce _any_ user-facing change?
   NO
   
   
   ### How was this patch tested?
   Existing Unit Tests.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3572: [CELEBORN-2241] Memory swap instead of memory copy
URL: https://github.com/apache/celeborn/pull/3572


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-3691192011

   ## [Codecov](https://app.codecov.io/gh/apache/celeborn/pull/3572?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 67.21%. Comparing base ([`ffff5bb`](https://app.codecov.io/gh/apache/celeborn/commit/ffff5bb94efd7c3a2c6c51aebccfd3c11b5d5198?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`ef3c308`](https://app.codecov.io/gh/apache/celeborn/commit/ef3c308d859c5a8a2f23f9529548c8d4599f6429?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 16 commits behind head on main.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main    #3572      +/-   ##
   ==========================================
   + Coverage   67.05%   67.21%   +0.17%     
   ==========================================
     Files         357      357              
     Lines       21779    21847      +68     
     Branches     1930     1938       +8     
   ==========================================
   + Hits        14602    14683      +81     
   + Misses       6160     6144      -16     
   - Partials     1017     1020       +3     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/celeborn/pull/3572?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-3957699242

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]

Posted by "mcdull-zhang (via GitHub)" <gi...@apache.org>.
mcdull-zhang commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-3708854935

   > SortBasedShuffleWriter	HashBasedShuffleWriter
   > Before	615.44W ms
   > After	442.47W ms
   > You mean that the running time dropped from 6000 seconds to 4000 seconds, that's very impressive.
   
   @FMX It is the sum of shuffle write block time for 22 queries, not the end-to-end query duration;
   And this is the total effect of multiple PRs;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-4168528808

   This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-4231137362

   This issue was closed because it has been staled for 10 days with no activity.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [CELEBORN-2241] Memory swap instead of memory copy [celeborn]

Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #3572:
URL: https://github.com/apache/celeborn/pull/3572#issuecomment-3707555594

   
   SortBasedShuffleWriter | HashBasedShuffleWriter
   -- | --
   Before | 615.44W ms | 502.31W ms
   After | 442.47W ms | 434.77W ms
   
   You mean that the running time dropped from 6000 seconds to 4000 seconds, that's very impressive.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org