You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Madan D via user <us...@flink.apache.org> on 2022/12/10 21:02:35 UTC

Flink adaptive Scheduler enabling EMR Auto Scaling

Hello All,
We are running our application on EMR and we would like to implement elastic scaling(since reactive mode works only in standalone mode) for flink using EMR managed scaling where we are trying to use adaptive scheduler to make sure job will not be failed when resources are not available but we are seeing high EC2 instances scaling up and job is using only 1 or 2 slots our of available slots. And also jobs is going down due to resources not availability not always though.
Any ideas in implementing auto scaling flink on EMR.
Command which I am running:
flink run -t yarn-per-job   -m yarn-cluster  -ys 6  -Dyarn.application.name=kafka-to-pubsub-e2e-test1  -Djobmanager.scheduler=adaptive



Regards,Madan