You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Peter Voß <pv...@datameer.com> on 2015/09/09 11:45:03 UTC

Adjusting number of YARN containers

Hi,

I have started a Flink YARN session using yarn-session.sh and the configuration of number of YARN container seems to be pretty static.

Is it possible to have Flink adjust the number of containers depending on the actual workload. E.g. stop containers that are idle for too long and start them up again when more jobs get submitted?
Or are there any plans to have more dynamic resource management?

Thanks,
—Peter

Re: Adjusting number of YARN containers

Posted by Robert Metzger <rm...@apache.org>.
Hi,

Currently, Flink does not support automatic scaling of the YARN containers.
There are certainly plans to add this feature:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/add-some-new-api-to-the-scheduler-in-the-job-manager-tp7153p7448.html

Adding an API for manually starting and stopping containers is something we
can do very easily, something dynamic is a bit harder.
The feature is something I was always interested to work on, but nobody
really asked for it. But it seems that more and more people are asking for
it.
If this is a requirement for you to use Flink, I'm certainly willing to add
the feature into Flink.

Best,
Robert




On Wed, Sep 9, 2015 at 11:45 AM, Peter Voß <pv...@datameer.com> wrote:

> Hi,
>
> I have started a Flink YARN session using yarn-session.sh and the
> configuration of number of YARN container seems to be pretty static.
>
> Is it possible to have Flink adjust the number of containers depending on
> the actual workload. E.g. stop containers that are idle for too long and
> start them up again when more jobs get submitted?
> Or are there any plans to have more dynamic resource management?
>
> Thanks,
> —Peter