You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Madan D via user <us...@flink.apache.org> on 2023/01/14 00:13:41 UTC
Flink reactive mode for application clusters on AWS EMR - Auto Scaling
Hello Team,I would like to understand auto scaling on EMR using either reactive mode or adaptive scheduler with custom or managed scaling.Can some one help me on this.
Regards,Madan
Re: Flink reactive mode for application clusters on AWS EMR - Auto Scaling
Posted by Gyula Fóra <gy...@gmail.com>.
I am not aware of any autoscaler/operator like functionality that works on
EMR/YARN at the moment.
It is probably easier to migrate to Kubernetes, or improve the Flink
Kubernetes Operator itself to be able to manage YARN clusters than to
improve all these things for yarn directly :)
Gyula
On Sun, Jan 15, 2023 at 5:27 AM Madan D <ma...@yahoo.com.au> wrote:
> Hi Gyula/Team,
>
> Thanks for your response. I see that the above one works on Kubernetes,
> but all our applications are currently running on EMR Yarn and looking to
> integrate with EMR-managed scaling with adaptive scheduler.
> Do we have any features or operators which we can use to run on Yarn
> similar to Flink Kubernetes Operator which can perform auto-scaling.
>
>
> Regards,
> Madan
>
> On Friday, 13 January 2023 at 09:37:11 pm GMT-8, Gyula Fóra <
> gyula.fora@gmail.com> wrote:
>
>
> Hi Madan,
>
> With reactive mode you need to build a completely custom auto scaling
> logic, it can work but it takes considerable effort.
>
> Instead I recommend using the Flink Kubernetes Operator which now contains
> the first version of an actual autoscaler module that collects metrics and
> scales Flink jobs .
>
>
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/
>
> The operator autoscaler uses Kubernetes native integration and works more
> or less out of the box.
>
> Cheers
> Gyula
>
> On Sat, 14 Jan 2023 at 01:16, Madan D via user <us...@flink.apache.org>
> wrote:
>
> Hello Team,
> I would like to understand auto scaling on EMR using either reactive mode
> or adaptive scheduler with custom or managed scaling.
> Can some one help me on this.
>
>
> Regards,
> Madan
>
>
>
>
Re: Flink reactive mode for application clusters on AWS EMR - Auto Scaling
Posted by Madan D via user <us...@flink.apache.org>.
Hi Gyula/Team,
Thanks for your response. I see that the above one works on Kubernetes, but all our applications are currently running on EMR Yarn and looking to integrate with EMR-managed scaling with adaptive scheduler.
Do we have any features or operators which we can use to run on Yarn similar to Flink Kubernetes Operator which can perform auto-scaling.
Regards,Madan
On Friday, 13 January 2023 at 09:37:11 pm GMT-8, Gyula Fóra <gy...@gmail.com> wrote:
Hi Madan,
With reactive mode you need to build a completely custom auto scaling logic, it can work but it takes considerable effort.
Instead I recommend using the Flink Kubernetes Operator which now contains the first version of an actual autoscaler module that collects metrics and scales Flink jobs .
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/
The operator autoscaler uses Kubernetes native integration and works more or less out of the box.
Cheers Gyula
On Sat, 14 Jan 2023 at 01:16, Madan D via user <us...@flink.apache.org> wrote:
Hello Team,I would like to understand auto scaling on EMR using either reactive mode or adaptive scheduler with custom or managed scaling.Can some one help me on this.
Regards,Madan
Re: Flink reactive mode for application clusters on AWS EMR - Auto Scaling
Posted by Gyula Fóra <gy...@gmail.com>.
Hi Madan,
With reactive mode you need to build a completely custom auto scaling
logic, it can work but it takes considerable effort.
Instead I recommend using the Flink Kubernetes Operator which now contains
the first version of an actual autoscaler module that collects metrics and
scales Flink jobs .
https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/
The operator autoscaler uses Kubernetes native integration and works more
or less out of the box.
Cheers
Gyula
On Sat, 14 Jan 2023 at 01:16, Madan D via user <us...@flink.apache.org>
wrote:
> Hello Team,
> I would like to understand auto scaling on EMR using either reactive mode
> or adaptive scheduler with custom or managed scaling.
> Can some one help me on this.
>
>
> Regards,
> Madan
>
>
>
>