You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@helix.apache.org by GitBox <gi...@apache.org> on 2021/10/04 22:56:36 UTC
[GitHub] [helix] jackjlli opened a new issue #1889: Prioritizing Helix tasks in Helix state model
jackjlli opened a new issue #1889:
URL: https://github.com/apache/helix/issues/1889
**Is your feature request related to a problem? Please describe.**
Currently Helix tasks are picked up by the participant based on the inqueue time. While there is some scenario that the tasks which are queued later need to be picked up first (due to some constraints like disk usage, etc).
**Describe the solution you'd like**
Helix should be able to provide the ability to prioritize the tasks based on some weighted value.
**Additional context**
E.g. there is a customized Helix state model called "SegmentOnlineOfflineStateModel" in Pinot:
https://github.com/apache/pinot/blob/a5f3dc507e6441baca35dae5bbfad122356683b6/pinot-server/src/main/java/org/apache/pinot/server/starter/helix/SegmentOnlineOfflineStateModelFactory.java
The state transition "OFFLINE->ONLINE" downloads new segment to local disk, and the one "OFFLINE->DROPPED" deletes segment from local disk.
While we notice that the state transition "OFFLINE->ONLINE" always comes before "OFFLINE->DROPPED", which makes the pinot server busy downloading new segments and then fills up the full disk.
E.g. there are 20 threads in the pool but it seems those threads will pick up the task from the queue in FIFO order.
```
[Running, pool size = 20, active threads = 20, queued tasks = 558, completed tasks = 1538]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jiajunwang commented on issue #1889: Prioritizing Helix tasks in Helix state model
Posted by GitBox <gi...@apache.org>.
jiajunwang commented on issue #1889:
URL: https://github.com/apache/helix/issues/1889#issuecomment-938280857
Let me clarify first. The Helix tasks mentioned here are the state transition tasks instead of the Task Framework tasks, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] jackjlli commented on issue #1889: Prioritizing Helix tasks in Helix state model
Posted by GitBox <gi...@apache.org>.
jackjlli commented on issue #1889:
URL: https://github.com/apache/helix/issues/1889#issuecomment-942762187
@jiajunwang Correct, this only denotes the state transition tasks, not the task framework task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org
[GitHub] [helix] mcvsubbu commented on issue #1889: Prioritizing Helix tasks in Helix state model
Posted by GitBox <gi...@apache.org>.
mcvsubbu commented on issue #1889:
URL: https://github.com/apache/helix/issues/1889#issuecomment-933922299
@junkaixue and @jiajunwang which release of helix can we possibly get this in? What may be the time frame? thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@helix.apache.org
For additional commands, e-mail: reviews-help@helix.apache.org