You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "jakubmatyszewski (via GitHub)" <gi...@apache.org> on 2023/08/14 08:24:14 UTC
[I] Assign worker category spec per task (druid)
jakubmatyszewski opened a new issue, #14809:
URL: https://github.com/apache/druid/issues/14809
### Description
I've recently started using MiddleManager categories and Overlords `WorkerCategorySpec` for [worker select strategy](https://druid.apache.org/docs/latest/configuration/#worker-select-strategy). As I understand currently it only supports picking MM category based on **task type**. I wonder if it's possible or it would be feasible feature to request, so that category spec would be assigned **per task**?
### Motivation
I was thinking on separating big batch ingestions from other types of ingestions, so that I could have dedicated Middlemanager instances processing such workloads. Currently I can only select by task type, but even though separating kafka ingestions from batch ingestions gets me a step closer to what I'm aiming for, I wish I could separate small batch ingestions from batch ingestions we run monthly that are exponentially bigger, so this tasks would be processed by scaled Middlemanagers dedicated only for that purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Assign worker category spec per task (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14809:
URL: https://github.com/apache/druid/issues/14809#issuecomment-2752822931
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Assign worker category spec per task (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14809:
URL: https://github.com/apache/druid/issues/14809#issuecomment-2121471302
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Assign worker category spec per task (druid)
Posted by "jakubmatyszewski (via GitHub)" <gi...@apache.org>.
jakubmatyszewski commented on issue #14809:
URL: https://github.com/apache/druid/issues/14809#issuecomment-2122254940
@a2l007, no, we settled with some workaround and I think I won't be able to come back to this idea any time soon, but your plan sounds in fact like something that would solve some of my problems. Thanks for the mention
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Assign worker category spec per task (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe commented on issue #14809:
URL: https://github.com/apache/druid/issues/14809#issuecomment-1677520605
+1
seems useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Assign worker category spec per task (druid)
Posted by "a2l007 (via GitHub)" <gi...@apache.org>.
a2l007 commented on issue #14809:
URL: https://github.com/apache/druid/issues/14809#issuecomment-2121696273
@jakubmatyszewski Are you planning to proceed with this feature contribution? I'm planning to implement something similar where the select strategy could be assigned at a datasource level, which might work for your usecase as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Assign worker category spec per task (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14809:
URL: https://github.com/apache/druid/issues/14809#issuecomment-2683583928
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Assign worker category spec per task (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14809: Assign worker category spec per task
URL: https://github.com/apache/druid/issues/14809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org