You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Meghashyam Sandeep V <vr...@gmail.com> on 2017/02/02 00:39:35 UTC

Support for Auto scaling

Hi Guys,

I currently run flink 1.1.4 streaming jobs in EMR in AWS with yarn. I
understand that EMR supports auto scaling but Flink doesn't. Is there a
plan for this support in 1.2.

Thanks,
Sandeep

Re: Support for Auto scaling

Posted by Till Rohrmann <tr...@apache.org>.

Hi Sandeep,

it's also not yet supported to dynamically start new TaskManagers when
needed. However, it can be done manually via the taskmanager.sh script.

We're working to enable this feature for 1.3 if everything goes according
to plan.

Cheers,
Till

On Thu, Feb 2, 2017 at 6:49 AM, Tzu-Li (Gordon) Tai <tz...@apache.org>
wrote:

> Hi Sandeep!
>
> While auto scaling jobs in Flink still isn’t possible, in Flink 1.2 you
> will be able to rescale jobs by stopping and restarting.
> This works by taking a savepoint of the job before stopping the job, and
> then redeploy the job with a higher / lower parallelism using the savepoint.
> Upon restarting the job, your states will be redistributed across the new
> operators.
>
> Changing operator / job parallelism on the fly while running is still on
> the future roadmap.
>
> Cheers,
> Gordon
>
>
> On February 2, 2017 at 8:39:39 AM, Meghashyam Sandeep V (
> vr1meghashyam@gmail.com) wrote:
>
> Hi Guys,
>
> I currently run flink 1.1.4 streaming jobs in EMR in AWS with yarn. I
> understand that EMR supports auto scaling but Flink doesn't. Is there a
> plan for this support in 1.2.
>
> Thanks,
> Sandeep
>
>

Re: Support for Auto scaling

Posted by "Tzu-Li (Gordon) Tai" <tz...@apache.org>.

Hi Sandeep!

While auto scaling jobs in Flink still isn’t possible, in Flink 1.2 you will be able to rescale jobs by stopping and restarting.
This works by taking a savepoint of the job before stopping the job, and then redeploy the job with a higher / lower parallelism using the savepoint.
Upon restarting the job, your states will be redistributed across the new operators.

Changing operator / job parallelism on the fly while running is still on the future roadmap.

Cheers,
Gordon

On February 2, 2017 at 8:39:39 AM, Meghashyam Sandeep V (vr1meghashyam@gmail.com) wrote:

Hi Guys,

I currently run flink 1.1.4 streaming jobs in EMR in AWS with yarn. I understand that EMR supports auto scaling but Flink doesn't. Is there a plan for this support in 1.2. 

Thanks,
Sandeep