You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Peter Voß <pv...@datameer.com> on 2015/09/09 11:45:03 UTC
Adjusting number of YARN containers
Hi,
I have started a Flink YARN session using yarn-session.sh and the configuration of number of YARN container seems to be pretty static.
Is it possible to have Flink adjust the number of containers depending on the actual workload. E.g. stop containers that are idle for too long and start them up again when more jobs get submitted?
Or are there any plans to have more dynamic resource management?
Thanks,
—Peter
Re: Adjusting number of YARN containers
Posted by Robert Metzger <rm...@apache.org>.
Hi,
Currently, Flink does not support automatic scaling of the YARN containers.
There are certainly plans to add this feature:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/add-some-new-api-to-the-scheduler-in-the-job-manager-tp7153p7448.html
Adding an API for manually starting and stopping containers is something we
can do very easily, something dynamic is a bit harder.
The feature is something I was always interested to work on, but nobody
really asked for it. But it seems that more and more people are asking for
it.
If this is a requirement for you to use Flink, I'm certainly willing to add
the feature into Flink.
Best,
Robert
On Wed, Sep 9, 2015 at 11:45 AM, Peter Voß <pv...@datameer.com> wrote:
> Hi,
>
> I have started a Flink YARN session using yarn-session.sh and the
> configuration of number of YARN container seems to be pretty static.
>
> Is it possible to have Flink adjust the number of containers depending on
> the actual workload. E.g. stop containers that are idle for too long and
> start them up again when more jobs get submitted?
> Or are there any plans to have more dynamic resource management?
>
> Thanks,
> —Peter