Auto/Dynamic scaling in Flink

[Nauroz Khan Nausherwani <na...@etu.univ-cotedazur.fr>]

Dear Flink Contributors and users,

I am a PhD student and I was interested to know, using which matrices, and when does Flink performs scaling-in or scaling out of resources? I did search the flink's website where I could only find information about how dynamic scaling is performed in stateless or stateful operator.  It would be interesting to know which matrices Flink uses, and when actually Flink triggers auto-scaling.

Any link or reference paper with required information is appreciated.

best regards,
Nauroz

Re: Auto/Dynamic scaling in Flink

[罗齐 <lu...@bytedance.com>]

Hi Nauroz,

If you’re using Flink 1.5 on Yarn, it supports dynamic task manager allocation by default [1]. After skimming the code, it seems to me that in general if requested parallelism is larger than available task slots, new task managers will be requested via ResourceManager (please correct me if any misunderstanding).

We’re also looking into auto scaling issue in Flink, as it’s crucial in our use cases (e.g. in batch mode, we will do input with few TMs but output with much more TMs). Please kindly let me know if any further information are available.

[1] https://stackoverflow.com/questions/38054015/can-yarn-dynamically-allocate-resources-to-flink/38077626#38077626 <https://stackoverflow.com/questions/38054015/can-yarn-dynamically-allocate-resources-to-flink/38077626#38077626>

> On Nov 13, 2018, at 8:10 PM, Tzu-Li Chen <wa...@gmail.com> wrote:
> 
> Hi,
> 
> Yet Flink does not support auto-scaling. However, there is an umbrella JIRA issue[1]
> to cover the discussion about it. And I think the design doc(draft) attached[2] could help.
> 
> Best,
> tison.
> 
> [1] https://issues.apache.org/jira/browse/FLINK-10407 <https://issues.apache.org/jira/browse/FLINK-10407>
> [2] https://docs.google.com/document/d/1XKDXnrp8w45k2jIJNHxpNP2Zmr6FGru3H0dTHwYWJyE/edit <https://docs.google.com/document/d/1XKDXnrp8w45k2jIJNHxpNP2Zmr6FGru3H0dTHwYWJyE/edit>
> 
> Tzu-Li (Gordon) Tai <tzulitai@apache.org <ma...@apache.org>> 于2018年11月13日周二 下午8:05写道：
> Hi,
> 
> Flink does not support auto-scaling, yet. Rescaling operations currently are always manual, i.e take a savepoint of the Flink job, and when restoring from the savepoint, define a new parallelism for the job.
> As for the metrics to be used for auto-scaling, I can imagine that it would be possible to base this on top of metrics such as TM throughput, backpressure, etc. AFAIK, there also currently isn't any official design or discussion on going for this.
> 
> Cheers,
> Gordon
> 
> On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani <nauroz-khan.nausherwani@etu.univ-cotedazur.fr <ma...@etu.univ-cotedazur.fr>> wrote:
> Dear Flink Contributors and users,
> 
> I am a PhD student and I was interested to know, using which matrices, and when does Flink performs scaling-in or scaling out of resources? I did search the flink's website where I could only find information about how dynamic scaling is performed in stateless or stateful operator.  It would be interesting to know which matrices Flink uses, and when actually Flink triggers auto-scaling.
> 
> Any link or reference paper with required information is appreciated.
> 
> best regards,
> Nauroz

Re: Auto/Dynamic scaling in Flink

[Tzu-Li Chen <wa...@gmail.com>]

Hi,

Yet Flink does not support auto-scaling. However, there is an umbrella JIRA
issue[1]
to cover the discussion about it. And I think the design doc(draft)
attached[2] could help.

Best,
tison.

[1] https://issues.apache.org/jira/browse/FLINK-10407
[2]
https://docs.google.com/document/d/1XKDXnrp8w45k2jIJNHxpNP2Zmr6FGru3H0dTHwYWJyE/edit


Tzu-Li (Gordon) Tai <tz...@apache.org> 于2018年11月13日周二 下午8:05写道：

> Hi,
>
> Flink does not support auto-scaling, yet. Rescaling operations currently
> are always manual, i.e take a savepoint of the Flink job, and when
> restoring from the savepoint, define a new parallelism for the job.
> As for the metrics to be used for auto-scaling, I can imagine that it
> would be possible to base this on top of metrics such as TM throughput,
> backpressure, etc. AFAIK, there also currently isn't any official design or
> discussion on going for this.
>
> Cheers,
> Gordon
>
> On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani <
> nauroz-khan.nausherwani@etu.univ-cotedazur.fr> wrote:
>
>> Dear Flink Contributors and users,
>>
>> I am a PhD student and I was interested to know, using which matrices,
>> and when does Flink performs scaling-in or scaling out of resources? I did
>> search the flink's website where I could only find information about how
>> dynamic scaling is performed in stateless or stateful operator.  It would
>> be interesting to know which matrices Flink uses, and when actually Flink
>> triggers auto-scaling.
>>
>> Any link or reference paper with required information is appreciated.
>>
>> best regards,
>> Nauroz
>>
>

Re: Auto/Dynamic scaling in Flink

["Tzu-Li (Gordon) Tai" <tz...@apache.org>]

Hi,

Flink does not support auto-scaling, yet. Rescaling operations currently
are always manual, i.e take a savepoint of the Flink job, and when
restoring from the savepoint, define a new parallelism for the job.
As for the metrics to be used for auto-scaling, I can imagine that it would
be possible to base this on top of metrics such as TM throughput,
backpressure, etc. AFAIK, there also currently isn't any official design or
discussion on going for this.

Cheers,
Gordon

On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani <
nauroz-khan.nausherwani@etu.univ-cotedazur.fr> wrote:

> Dear Flink Contributors and users,
>
> I am a PhD student and I was interested to know, using which matrices, and
> when does Flink performs scaling-in or scaling out of resources? I did
> search the flink's website where I could only find information about how
> dynamic scaling is performed in stateless or stateful operator.  It would
> be interesting to know which matrices Flink uses, and when actually Flink
> triggers auto-scaling.
>
> Any link or reference paper with required information is appreciated.
>
> best regards,
> Nauroz
>