You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by bo zhaobo <bz...@gmail.com> on 2022/07/07 01:15:13 UTC
[DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal
Hi, all.
I would like to raise a discussion in Flink dev ML about Support Customized
Kubernetes Schedulers.
Currentlly, Kubernetes becomes more and more polular for Flink Cluster
deployment, and its ability is better, especially, it supports customized
scheduling.
Essentially, in high-performance workloads, we need to apply new scheduling
policies for meeting the new requirements. And now Flink native Kubernetes
solution is using Kubernetes default scheduler to work with all scenarios,
the default scheduling policy might be difficult to apply in some extreme
cases, so
we need to improve the Flink Kubernetes for coupling those Kubernetes
customized schedulers with Flink native Kubernetes, provides a way for Flink
administrators or users to use/apply their Flink Clusters on Kubernetes
more flexibility.
The proposal will introduce the customized K8S schdulers plugin mechanism
and a reference implementation 'Volcano' in Flink. More details see [1].
Looking forward to your feedback.
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-250%3A+Support+Customized+Kubernetes+Schedulers+Proposal
Thanks,
BR
Re: [DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal
Posted by Shqiprim Bunjaku <sh...@gmail.com>.
+1 (non binding)
On Tue, Jul 12, 2022 at 6:50 AM Kelu Tao <le...@apache.org> wrote:
> +1 for this proposal.
>
> Volcano has show its strength in some specified cases, such as ML. And its
> schedule ability will be enhancement for kubernetes default scheduler.
>
> Look forward to see the integration between Volcano and Flink.
>
> Thanks.
>
> On 2022/07/08 00:25:45 bo zhaobo wrote:
> > Thanks Yang Wang.
> >
> > <Just to remind that the flink-kubernetes-volcano-*.jar will be an
> optional
> > jar located in the $FLINK_HOME/opt. Users who want to try the volcano
> > scheduler need
> > to copy it to the plugins directory.r.>
> >
> > --- >>
> > A: Yeah, the said separated jar pkg won't be loaded by default. But when
> > users copy the jar pkg from $FLINK_HOME/opt/flink-kubernetes-*.jar to
> > plugins directory,
> > then the customized scheduler will be load and the functionality will
> > be enabled. That would be helpful that different users want different
> > functions if more and more
> > schedulers will be introduced in the future.
> >
> > Thanks,
> >
> > BR
> >
> > Bo Zhao
> >
> >
> >
> > Yang Wang <da...@gmail.com> 于2022年7月7日周四 12:08写道：
> >
> > > Thanks zhaobo for starting the discussion and preparing the FLIP.
> > >
> > > The customized Kubernetes Schedulers support will be very helpful for
> the
> > > users who still hesitates to migrate the Flink workloads from YARN to
> > > Kubernetes.
> > > Now leveraging the ability of customized K8s scheduler, many advanced
> > > scheduling features(e.g. priority scheduling, dynamic resource sharing,
> > > etc.) could be
> > > introduced to make the streaming/batch jobs run more smoothly in a
> shared
> > > K8s cluster.
> > >
> > > Just to remind that the flink-kubernetes-volcano-*.jar will be an
> optional
> > > jar located in the $FLINK_HOME/opt. Users who want to try the volcano
> > > scheduler need
> > > to copy it to the plugins directory.
> > >
> > >
> > > Best,
> > > Yang
> > >
> > > bo zhaobo <bz...@gmail.com> 于2022年7月7日周四 09:16写道：
> > >
> > > > Hi, all.
> > > >
> > > > I would like to raise a discussion in Flink dev ML about Support
> > > Customized
> > > > Kubernetes Schedulers.
> > > > Currentlly, Kubernetes becomes more and more polular for Flink
> Cluster
> > > > deployment, and its ability is better, especially, it supports
> > > customized
> > > > scheduling.
> > > > Essentially, in high-performance workloads, we need to apply new
> > > scheduling
> > > > policies for meeting the new requirements. And now Flink native
> > > Kubernetes
> > > > solution is using Kubernetes default scheduler to work with all
> > > scenarios,
> > > > the default scheduling policy might be difficult to apply in some
> extreme
> > > > cases, so
> > > > we need to improve the Flink Kubernetes for coupling those Kubernetes
> > > > customized schedulers with Flink native Kubernetes, provides a way
> for
> > > > Flink
> > > > administrators or users to use/apply their Flink Clusters on
> Kubernetes
> > > > more flexibility.
> > > >
> > > > The proposal will introduce the customized K8S schdulers plugin
> mechanism
> > > > and a reference implementation 'Volcano' in Flink. More details see
> [1].
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > > [1]
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-250%3A+Support+Customized+Kubernetes+Schedulers+Proposal
> > > >
> > > > Thanks,
> > > > BR
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal
Posted by Kelu Tao <le...@apache.org>.
+1 for this proposal.
Volcano has show its strength in some specified cases, such as ML. And its schedule ability will be enhancement for kubernetes default scheduler.
Look forward to see the integration between Volcano and Flink.
Thanks.
On 2022/07/08 00:25:45 bo zhaobo wrote:
> Thanks Yang Wang.
>
> <Just to remind that the flink-kubernetes-volcano-*.jar will be an optional
> jar located in the $FLINK_HOME/opt. Users who want to try the volcano
> scheduler need
> to copy it to the plugins directory.r.>
>
> --- >>
> A: Yeah, the said separated jar pkg won't be loaded by default. But when
> users copy the jar pkg from $FLINK_HOME/opt/flink-kubernetes-*.jar to
> plugins directory,
> then the customized scheduler will be load and the functionality will
> be enabled. That would be helpful that different users want different
> functions if more and more
> schedulers will be introduced in the future.
>
> Thanks,
>
> BR
>
> Bo Zhao
>
>
>
> Yang Wang <da...@gmail.com> 于2022年7月7日周四 12:08写道：
>
> > Thanks zhaobo for starting the discussion and preparing the FLIP.
> >
> > The customized Kubernetes Schedulers support will be very helpful for the
> > users who still hesitates to migrate the Flink workloads from YARN to
> > Kubernetes.
> > Now leveraging the ability of customized K8s scheduler, many advanced
> > scheduling features(e.g. priority scheduling, dynamic resource sharing,
> > etc.) could be
> > introduced to make the streaming/batch jobs run more smoothly in a shared
> > K8s cluster.
> >
> > Just to remind that the flink-kubernetes-volcano-*.jar will be an optional
> > jar located in the $FLINK_HOME/opt. Users who want to try the volcano
> > scheduler need
> > to copy it to the plugins directory.
> >
> >
> > Best,
> > Yang
> >
> > bo zhaobo <bz...@gmail.com> 于2022年7月7日周四 09:16写道：
> >
> > > Hi, all.
> > >
> > > I would like to raise a discussion in Flink dev ML about Support
> > Customized
> > > Kubernetes Schedulers.
> > > Currentlly, Kubernetes becomes more and more polular for Flink Cluster
> > > deployment, and its ability is better, especially, it supports
> > customized
> > > scheduling.
> > > Essentially, in high-performance workloads, we need to apply new
> > scheduling
> > > policies for meeting the new requirements. And now Flink native
> > Kubernetes
> > > solution is using Kubernetes default scheduler to work with all
> > scenarios,
> > > the default scheduling policy might be difficult to apply in some extreme
> > > cases, so
> > > we need to improve the Flink Kubernetes for coupling those Kubernetes
> > > customized schedulers with Flink native Kubernetes, provides a way for
> > > Flink
> > > administrators or users to use/apply their Flink Clusters on Kubernetes
> > > more flexibility.
> > >
> > > The proposal will introduce the customized K8S schdulers plugin mechanism
> > > and a reference implementation 'Volcano' in Flink. More details see [1].
> > >
> > > Looking forward to your feedback.
> > >
> > > [1]
> > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-250%3A+Support+Customized+Kubernetes+Schedulers+Proposal
> > >
> > > Thanks,
> > > BR
> > >
> >
>
Re: [DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal
Posted by bo zhaobo <bz...@gmail.com>.
Thanks Yang Wang.
<Just to remind that the flink-kubernetes-volcano-*.jar will be an optional
jar located in the $FLINK_HOME/opt. Users who want to try the volcano
scheduler need
to copy it to the plugins directory.r.>
--- >>
A: Yeah, the said separated jar pkg won't be loaded by default. But when
users copy the jar pkg from $FLINK_HOME/opt/flink-kubernetes-*.jar to
plugins directory,
then the customized scheduler will be load and the functionality will
be enabled. That would be helpful that different users want different
functions if more and more
schedulers will be introduced in the future.
Thanks,
BR
Bo Zhao
Yang Wang <da...@gmail.com> 于2022年7月7日周四 12:08写道：
> Thanks zhaobo for starting the discussion and preparing the FLIP.
>
> The customized Kubernetes Schedulers support will be very helpful for the
> users who still hesitates to migrate the Flink workloads from YARN to
> Kubernetes.
> Now leveraging the ability of customized K8s scheduler, many advanced
> scheduling features(e.g. priority scheduling, dynamic resource sharing,
> etc.) could be
> introduced to make the streaming/batch jobs run more smoothly in a shared
> K8s cluster.
>
> Just to remind that the flink-kubernetes-volcano-*.jar will be an optional
> jar located in the $FLINK_HOME/opt. Users who want to try the volcano
> scheduler need
> to copy it to the plugins directory.
>
>
> Best,
> Yang
>
> bo zhaobo <bz...@gmail.com> 于2022年7月7日周四 09:16写道：
>
> > Hi, all.
> >
> > I would like to raise a discussion in Flink dev ML about Support
> Customized
> > Kubernetes Schedulers.
> > Currentlly, Kubernetes becomes more and more polular for Flink Cluster
> > deployment, and its ability is better, especially, it supports
> customized
> > scheduling.
> > Essentially, in high-performance workloads, we need to apply new
> scheduling
> > policies for meeting the new requirements. And now Flink native
> Kubernetes
> > solution is using Kubernetes default scheduler to work with all
> scenarios,
> > the default scheduling policy might be difficult to apply in some extreme
> > cases, so
> > we need to improve the Flink Kubernetes for coupling those Kubernetes
> > customized schedulers with Flink native Kubernetes, provides a way for
> > Flink
> > administrators or users to use/apply their Flink Clusters on Kubernetes
> > more flexibility.
> >
> > The proposal will introduce the customized K8S schdulers plugin mechanism
> > and a reference implementation 'Volcano' in Flink. More details see [1].
> >
> > Looking forward to your feedback.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-250%3A+Support+Customized+Kubernetes+Schedulers+Proposal
> >
> > Thanks,
> > BR
> >
>
Re: [DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal
Posted by Yang Wang <da...@gmail.com>.
Thanks zhaobo for starting the discussion and preparing the FLIP.
The customized Kubernetes Schedulers support will be very helpful for the
users who still hesitates to migrate the Flink workloads from YARN to
Kubernetes.
Now leveraging the ability of customized K8s scheduler, many advanced
scheduling features(e.g. priority scheduling, dynamic resource sharing,
etc.) could be
introduced to make the streaming/batch jobs run more smoothly in a shared
K8s cluster.
Just to remind that the flink-kubernetes-volcano-*.jar will be an optional
jar located in the $FLINK_HOME/opt. Users who want to try the volcano
scheduler need
to copy it to the plugins directory.
Best,
Yang
bo zhaobo <bz...@gmail.com> 于2022年7月7日周四 09:16写道：
> Hi, all.
>
> I would like to raise a discussion in Flink dev ML about Support Customized
> Kubernetes Schedulers.
> Currentlly, Kubernetes becomes more and more polular for Flink Cluster
> deployment, and its ability is better, especially, it supports customized
> scheduling.
> Essentially, in high-performance workloads, we need to apply new scheduling
> policies for meeting the new requirements. And now Flink native Kubernetes
> solution is using Kubernetes default scheduler to work with all scenarios,
> the default scheduling policy might be difficult to apply in some extreme
> cases, so
> we need to improve the Flink Kubernetes for coupling those Kubernetes
> customized schedulers with Flink native Kubernetes, provides a way for
> Flink
> administrators or users to use/apply their Flink Clusters on Kubernetes
> more flexibility.
>
> The proposal will introduce the customized K8S schdulers plugin mechanism
> and a reference implementation 'Volcano' in Flink. More details see [1].
>
> Looking forward to your feedback.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-250%3A+Support+Customized+Kubernetes+Schedulers+Proposal
>
> Thanks,
> BR
>
Re: [DISCUSS] FLIP-250: Support Customized Kubernetes Schedulers Proposal
Posted by William Wang <wa...@gmail.com>.
Thanks zhaobo for the proposal.
We've met with a lot of big data users who are trying to migrate their
business to flink on kubernetes to improve job performance and cluster
utilization.
Unfortunately so far kubernetes lacks batch scheduling capabilities
compared to traditional batch systems like yarn, slurm and so forth.
The customized kubernetes scheduler support will fill these gaps and speed
up the migration of flink on kubernetes for users.
Best,
William
bo zhaobo <bz...@gmail.com> 于2022年7月7日周四 09:15写道：
> Hi, all.
>
> I would like to raise a discussion in Flink dev ML about Support Customized
> Kubernetes Schedulers.
> Currentlly, Kubernetes becomes more and more polular for Flink Cluster
> deployment, and its ability is better, especially, it supports customized
> scheduling.
> Essentially, in high-performance workloads, we need to apply new scheduling
> policies for meeting the new requirements. And now Flink native Kubernetes
> solution is using Kubernetes default scheduler to work with all scenarios,
> the default scheduling policy might be difficult to apply in some extreme
> cases, so
> we need to improve the Flink Kubernetes for coupling those Kubernetes
> customized schedulers with Flink native Kubernetes, provides a way for
> Flink
> administrators or users to use/apply their Flink Clusters on Kubernetes
> more flexibility.
>
> The proposal will introduce the customized K8S schdulers plugin mechanism
> and a reference implementation 'Volcano' in Flink. More details see [1].
>
> Looking forward to your feedback.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-250%3A+Support+Customized+Kubernetes+Schedulers+Proposal
>
> Thanks,
> BR
>