You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "ErikFang (via GitHub)" <gi...@apache.org> on 2026/02/13 16:27:38 UTC
[PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
ErikFang opened a new pull request, #3260:
URL: https://github.com/apache/celeborn/pull/3260
### What changes were proposed in this pull request?
enhance celeborn to
1. allow shuffle client to write to one partition with multiple partition locations parallelly
2. allow lifecycle manager to detect parallel level for each partition by partition splitting frequency
CIP-20: Dynamically optimize shuffle write parallelism
https://docs.google.com/document/d/1CqFswIOP5nR8Cy2THo8tELOwVxUf0ZP_8pDjaYj2HGc/edit?usp=sharing
### Why are the changes needed?
TBD
### Does this PR introduce _any_ user-facing change?
TBD
### How was this patch tested?
TBD
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-3717848404
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3260: [WIP] [CIP-20] Dynamically adjust partition write parallelism
URL: https://github.com/apache/celeborn/pull/3260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-3092167258
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-3765060483
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-4175664047
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-3317580029
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "yew1eb (via GitHub)" <gi...@apache.org>.
yew1eb commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-5585499533
Hi @ErikFang @mridulm @turboFei @RexXiong — I've been working on the same problem and have submitted CIP-24 (#3843), which shares the goal of adaptive partition write parallelism but takes a different approach to the control loop (LifecycleManager measures actual fill time per location rather than estimating from event counts, and the active set is propagated through the existing revive RPC). Your earlier work on CIP-20 was a valuable reference for this design.
The proposal is under discussion on dev@: https://lists.apache.org/thread/fvy5jor59mxqp15v8xq2d5xw64yb71gs — would love to hear your thoughts there or on the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-4010371294
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-3173777887
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-3213558507
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3260: [WIP] Dynamically adjust partition write parallelism
URL: https://github.com/apache/celeborn/pull/3260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-4235191199
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-2994045054
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-3364781638
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3260: [WIP] [CIP-20] Dynamically adjust partition write parallelism
URL: https://github.com/apache/celeborn/pull/3260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] [CIP-20] Dynamically adjust partition write parallelism [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3260: [WIP] [CIP-20] Dynamically adjust partition write parallelism
URL: https://github.com/apache/celeborn/pull/3260
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Dynamically adjust partition write parallelism [celeborn]
Posted by "ErikFang (via GitHub)" <gi...@apache.org>.
ErikFang commented on PR #3260:
URL: https://github.com/apache/celeborn/pull/3260#issuecomment-3015454804
CIP document updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org