You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@druid.apache.org by Yue Zhang <zh...@163.com> on 2021/02/26 08:06:09 UTC

Contribute a new Community extensions : Launch Peon Pods Based on K8s

Hi Druid, 

I’d like to contribute a new Community Extension named druid-kubernetes-middlemanager-extensions. Proposal Link: https://github.com/apache/druid/issues/10824 and PR link : https://github.com/apache/druid/pull/10910

When enable this feature, Druid can launch Peon as a pod to do data ingestion. The advantages are as follows:

Cost saving: there is no need to let MiddleManager take up a lot of resources in advance, and just require resources whenever it will use. Now we can use a single 2Cores and 2Gi Memory middelmanager pod to control Dozens or even hundreds peon pods.
Improve resource utilization: Now different kinds of tasks can use different configs including CPU resources and Memory resources if necessary.

      If you are interested in this feature,  please let me know. Looking forward to your reply.

Best wishes,

Yue Zhang

Re: Contribute a new Community extensions : Launch Peon Pods Based on K8s

Posted by Gian Merlino <gi...@apache.org>.

Hey Yue,

Very interesting idea. I am not a kubernetes expert, but this seems like a
neat concept. I guess the idea is only one MM would be needed? (Or maybe a
handful, if one can't manage every pod?) If so, great. Hopefully someone
that is more of a kubernetes expert will be able to chime in on the PR.

On Fri, Feb 26, 2021 at 12:46 AM Yue Zhang <zh...@163.com> wrote:

> Hi Druid,
>
> I’d like to contribute a new Community Extension named
> druid-kubernetes-middlemanager-extensions. Proposal Link:
> https://github.com/apache/druid/issues/10824 and PR link :
> https://github.com/apache/druid/pull/10910
>
> When enable this feature, Druid can launch Peon as a pod to do data
> ingestion. The advantages are as follows:
>
> Cost saving: there is no need to let MiddleManager take up a lot of
> resources in advance, and just require resources whenever it will use. Now
> we can use a single 2Cores and 2Gi Memory middelmanager pod to control
> Dozens or even hundreds peon pods.
> Improve resource utilization: Now different kinds of tasks can use
> different configs including CPU resources and Memory resources if necessary.
>
>       If you are interested in this feature,  please let me know. Looking
> forward to your reply.
>
> Best wishes,
>
> Yue Zhang
>
>
>