You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by "somandal (via GitHub)" <gi...@apache.org> on 2025/04/15 22:42:43 UTC
[I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
somandal opened a new issue, #3021:
URL: https://github.com/apache/helix/issues/3021
**Is your feature request related to a problem? Please describe.**
We use CUSTOMIZED mode and today there does not seem to be any fairness in terms of how the Helix state transition messages are sent out to the cluster across different resources.
For example, say I have two resources, `TableA` and `TableB` that share the same instances. If I update `TableA` with a very large number of IdealState changes (say 10K), then updates to `TableB` land up getting queued behind `TableA` and no progress is made until `TableA`'s state transitions complete.
This leads to starvation for `TableB`. Ideally resources should not affect each other in a Helix cluster.
**Describe the solution you'd like**
Ideally the Helix state transitions should be scheduled fairly across resources in a cluster to prevent the above starvation scenario.
**Additional context**
I know WAGED solves some of these problems, but moving to WAGED is not an option for us. Are ther any other solutions that can be leveraged?
cc @junkaixue @zpinto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2828478544
@xyuanlu we haven't had a chance to try this out yet. Once we get a chance I'll let you know. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2828351523
@somandal May I ask does setting the throttling config work? :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2807705020
Helix support a fine grained level of message throttling that allow user to config state transition message limit per cluster/instance/resource/partition.
It is in cluster config
```
"STATE_TRANSITION_THROTTLE_CONFIGS": [
"{\"THROTTLE_SCOPE\":\"RESOURCE\",\"MAX_PARTITION_IN_TRANSITION\":\"20\",\"REBALANCE_TYPE\":\"LOAD_BALANCE\"}"
],
```
If we set resource level state transition throttle, all resources get at ma 20 state transitions and TableB wont starve.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2807712001
got it, thanks @xyuanlu I'll look into this
what are the different `REBALANCE_TYPE` values that are relevant for CUSTOMIZED mode?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2895819027
And there are other rebalance types like ANY and NONE. When should I be using them?
-> ANY included both load rebalance and recovery rebalance.
NONE is an error value should not be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2895805767
@xyuanlu Let's say, In our use case `MESSAGE_CONSTRAINT` is set to be throttled to certain value and if STATE_TRANSITION_THROTTLE_CONFIGS are also being set, does the `STATE_TRANSITION_THROTTLE_CONFIGS` not kick in or will it be overriden by the STATE_TRANSITION constraint value being set as I am not able to throttle it based on the below configs?
```
"STATE_TRANSITION_THROTTLE_CONFIGS": [
"{\"THROTTLE_SCOPE\":\"RESOURCE\",\"MAX_PARTITION_IN_TRANSITION\":\"20\",\"REBALANCE_TYPE\":\"LOAD_BALANCE\"}"
],
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2899814293
@deepthi912 is the behavior the same if you set that config for REBALANCE_TYPE: ANY?
I'm just wondering if the other state transition messages were added due to the other REBALANCE_TYPE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2807716177
REBALANCE_TYPE meaning it is recovery rebalance or load balance
recovery rebalabce -> currently have less than min active replica, or missing top state
load rebalance -> meet the requirement of min replica, just need to converge the current state to ideal state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2899929942
@somandal Yes I tried out ANY, RECOVERY_REBALANCE and LOAD_BALANCE REBALANCE_TYPE types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2899137091
So, this is how the pending message map is looking like on the Helix Throttling side when I enable it:
Here instead of giving only one message for this Resource fake_table18_REALTIME, it is giving out 10 messages when following throttling is set, I am unsure if I am setting anything wrong or need to tweek some other configs to achieve only max 1 transition message to be allowed for this resource:
{"THROTTLE_SCOPE":"RESOURCE","MAX_PARTITION_IN_TRANSITION":"1","REBALANCE_TYPE":"LOAD_BALANCE"}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "deepthi912 (via GitHub)" <gi...@apache.org>.
deepthi912 commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2899232508
@xyuanlu I don't think the throttling configs are helping us out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
Re: [I] Request for Fair State Transition Scheduling Across Multiple Resources [helix]
Posted by "xyuanlu (via GitHub)" <gi...@apache.org>.
xyuanlu commented on issue #3021:
URL: https://github.com/apache/helix/issues/3021#issuecomment-2895824402
If both MESSAGE_CONSTRAINT and STATE_TRANSITION_THROTTLE_CONFIGS are set, they are both honored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org