You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "qqu0127 (via GitHub)" <gi...@apache.org> on 2023/05/01 15:39:54 UTC
[GitHub] [helix] qqu0127 opened a new issue, #2468: Hard constraint scope expansion to avoid instance overload during state transition in WAGED (N+1 scenario)
qqu0127 opened a new issue, #2468:
URL: https://github.com/apache/helix/issues/2468
**Is your feature request related to a problem? Please describe.**
The WAGED rebalancer assigns partitions to instance only if all the hard constraints are satisfied. However, the calculation doesn't account for the temporary state during state transition. The result is measured at the eventual state after all state transitions complete. The "N+1" scenario is when on one `instance_0`, p1 is moving out while p3 is moving in, p2 stays at instance_0. The beginning state `[p1, p2]`, and ending state `[p2, p3]` all satisfies the hard constraint, while the temporary state [p1, p2, p3] may not.
Note that this will expand the definition of hard constraint in WAGED.
**Describe the solution you'd like**
An ideal implementation should account for the "limit" during state transition so that a set of configurable limits or constraints are always honored.
The feature doesn't need to and should not be tied to WAGED, it should be made as a universal feature that applies to any rebalancer. (The issue is raised from WAGED applications because it's the only place such constraints are defined, but the underlying causes would apply to all)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] xyuanlu closed issue #2468: Hard constraint scope expansion to avoid instance overload during state transition in WAGED (N+1 scenario)
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu closed issue #2468: Hard constraint scope expansion to avoid instance overload during state transition in WAGED (N+1 scenario)
URL: https://github.com/apache/helix/issues/2468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on issue #2468: Hard constraint scope expansion to avoid instance overload during state transition in WAGED (N+1 scenario)
Posted by "jiajunwang (via GitHub)" <gi...@apache.org>.
jiajunwang commented on issue #2468:
URL: https://github.com/apache/helix/issues/2468#issuecomment-1535598451
This is an interesting problem. Is it possible for you to write up an one-pager to elaborate your proposal? I feel we'd better not rush to coding before the high level direction is clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] qqu0127 commented on issue #2468: Hard constraint scope expansion to avoid instance overload during state transition in WAGED (N+1 scenario)
Posted by "qqu0127 (via GitHub)" <gi...@apache.org>.
qqu0127 commented on issue #2468:
URL: https://github.com/apache/helix/issues/2468#issuecomment-1536429930
@jiajunwang Please refer to https://github.com/apache/helix/wiki/WAGED-rebalancer-Hard-Constraint-Scope-Expansion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on issue #2468: Hard constraint scope expansion to avoid instance overload during state transition in WAGED (N+1 scenario)
Posted by "jiajunwang (via GitHub)" <gi...@apache.org>.
jiajunwang commented on issue #2468:
URL: https://github.com/apache/helix/issues/2468#issuecomment-1548217591
@qqu0127 thanks a lot for the writeup. I like the straightforward way the problems and proposals are elaborated. The github wiki is not easy for us to comment, if we are looking for more external comments, I would suggest creating a Google Doc first and then convert to wiki when the idea is converged. cc @desaikomal
Given that saying, here're my thoughts:
1. Highlevel idea makes sense. And we should do it.
2. Kind of extending the topic a little bit. If we believe the "capacity" of the node can precisely predict it's real capability of hosting workload, do we still need evenness? Does it make sense if we just add partitions to a node until the workload weight is about to exceed "capacity"? What I'm trying to say is that, if users are using this weight/capacity system more seriously, we shall consider simplify the design instead of adding more layers to it. The thing is that, although each design makes great sense, if we implement all of them in one system, the system becomes too complicated and unmanagible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] desaikomal commented on issue #2468: Hard constraint scope expansion to avoid instance overload during state transition in WAGED (N+1 scenario)
Posted by "desaikomal (via GitHub)" <gi...@apache.org>.
desaikomal commented on issue #2468:
URL: https://github.com/apache/helix/issues/2468#issuecomment-1548464387
I couldn't find a way to share Google doc from my official LinkedIn account. Here is the write up in Google Doc: (komald@gmail.com/desaimom)
https://docs.google.com/document/d/1uxCqSU0xriUTdZIyxVoEvPMvywRgHCANatUlR0YQojk/edit#
Please note: I have added a new proposal - Approach-2.
@jiajunwang @NealSun96 @qqu0127 @junkaixue - Please review and get back to us soon as there are few consumers waiting for this fix. Thanks,
komal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org