You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Olivier Nouguier <ol...@teads.com> on 2021/04/19 12:56:03 UTC
Max-parellelism limitation
Hi,
May I have the confirmation that the max-parallelism limitation only
occurs when keyed states are used ?
--
Olivier Nouguier
SSE
e | olivier.nouguier@teads.com m | 0651383971
Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
Montpellier, France
[image: image] <https://www.teads.com>
The information in this email is confidential and intended only for the
addressee(s) named above. If you are not the intended recipient any
disclosure, copying, distribution or any action taken or omitted to be
taken in reliance on it is prohibited and may be unlawful. Teads does not
warrant that any attachment(s) are free from viruses or other defects and
accept no liability for any losses resulting from infected email
transmission. Please note that any views expressed in this email may be
those of the originator and do not necessarily reflect those of the
organization.
Re: Max-parellelism limitation
Posted by Austin Cawley-Edwards <au...@gmail.com>.
Hi Olivier,
Someone will correct me if I'm wrong, but I believe the max-parallelism
limitation, where you cannot scale up past the previously defined
max-parallelism, applies to all stateful jobs no matter which type of state
you are using.
If you haven't seen it already, I think the Production Readiness checklist
[1] offers a good explanation for this behavior/ best practices for setting
it.
Best,
Austin
[1]:
https://ci.apache.org/projects/flink/flink-docs-master/docs/ops/production_ready/#set-an-explicit-max-parallelism
On Tue, Apr 20, 2021 at 9:33 AM Olivier Nouguier <ol...@teads.com>
wrote:
> Hi, thank you all for your reply, by limitation I meant the impossibility
> to resume a job when scaling up because of this max-parallelism.
> To be more precise, in our deployment, the operator (max) parallelism is
> computed from the number of available slots ( ~~ task-manager * core ),
> this approach is probably too naive, at least with keyed state.
>
>
>
> On Tue, Apr 20, 2021 at 10:46 AM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> @Olivier Could you clarify which limitation you are referring to?
>>
>> On 4/20/2021 5:23 AM, Guowei Ma wrote:
>>
>> Hi, Olivier
>> Yes. The introduction of this concept is to solve the problem of
>> rescaling the keystate.
>> Best,
>> Guowei
>>
>>
>> On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier <
>> olivier.nouguier@teads.com> wrote:
>>
>>> Hi,
>>> May I have the confirmation that the max-parallelism limitation only
>>> occurs when keyed states are used ?
>>>
>>>
>>> --
>>>
>>> Olivier Nouguier
>>>
>>> SSE
>>>
>>> e | olivier.nouguier@teads.com m | 0651383971
>>>
>>> Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
>>> Montpellier, France
>>> [image: image] <https://www.teads.com>
>>> The information in this email is confidential and intended only for the
>>> addressee(s) named above. If you are not the intended recipient any
>>> disclosure, copying, distribution or any action taken or omitted to be
>>> taken in reliance on it is prohibited and may be unlawful. Teads does
>>> not warrant that any attachment(s) are free from viruses or other defects
>>> and accept no liability for any losses resulting from infected email
>>> transmission. Please note that any views expressed in this email may be
>>> those of the originator and do not necessarily reflect those of the
>>> organization.
>>>
>>
>>
>
> --
>
> Olivier Nouguier
>
> SSE
>
> e | olivier.nouguier@teads.com m | 0651383971
>
> Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
> Montpellier, France
> [image: image] <https://www.teads.com>
>
> The information in this email is confidential and intended only for the
> addressee(s) named above. If you are not the intended recipient any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it is prohibited and may be unlawful. Teads does not
> warrant that any attachment(s) are free from viruses or other defects and
> accept no liability for any losses resulting from infected email
> transmission. Please note that any views expressed in this email may be
> those of the originator and do not necessarily reflect those of the
> organization.
>
Re: Max-parellelism limitation
Posted by Olivier Nouguier <ol...@teads.com>.
Hi, thank you all for your reply, by limitation I meant the impossibility
to resume a job when scaling up because of this max-parallelism.
To be more precise, in our deployment, the operator (max) parallelism is
computed from the number of available slots ( ~~ task-manager * core ),
this approach is probably too naive, at least with keyed state.
On Tue, Apr 20, 2021 at 10:46 AM Chesnay Schepler <ch...@apache.org>
wrote:
> @Olivier Could you clarify which limitation you are referring to?
>
> On 4/20/2021 5:23 AM, Guowei Ma wrote:
>
> Hi, Olivier
> Yes. The introduction of this concept is to solve the problem of rescaling
> the keystate.
> Best,
> Guowei
>
>
> On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier <
> olivier.nouguier@teads.com> wrote:
>
>> Hi,
>> May I have the confirmation that the max-parallelism limitation only
>> occurs when keyed states are used ?
>>
>>
>> --
>>
>> Olivier Nouguier
>>
>> SSE
>>
>> e | olivier.nouguier@teads.com m | 0651383971
>>
>> Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
>> Montpellier, France
>> [image: image] <https://www.teads.com>
>> The information in this email is confidential and intended only for the
>> addressee(s) named above. If you are not the intended recipient any
>> disclosure, copying, distribution or any action taken or omitted to be
>> taken in reliance on it is prohibited and may be unlawful. Teads does
>> not warrant that any attachment(s) are free from viruses or other defects
>> and accept no liability for any losses resulting from infected email
>> transmission. Please note that any views expressed in this email may be
>> those of the originator and do not necessarily reflect those of the
>> organization.
>>
>
>
--
Olivier Nouguier
SSE
e | olivier.nouguier@teads.com m | 0651383971
Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
Montpellier, France
[image: image] <https://www.teads.com>
The information in this email is confidential and intended only for the
addressee(s) named above. If you are not the intended recipient any
disclosure, copying, distribution or any action taken or omitted to be
taken in reliance on it is prohibited and may be unlawful. Teads does not
warrant that any attachment(s) are free from viruses or other defects and
accept no liability for any losses resulting from infected email
transmission. Please note that any views expressed in this email may be
those of the originator and do not necessarily reflect those of the
organization.
Re: Max-parellelism limitation
Posted by Chesnay Schepler <ch...@apache.org>.
@Olivier Could you clarify which limitation you are referring to?
On 4/20/2021 5:23 AM, Guowei Ma wrote:
> Hi, Olivier
> Yes. The introduction of this concept is to solve the problem of
> rescaling the keystate.
> Best,
> Guowei
>
>
> On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier
> <olivier.nouguier@teads.com <ma...@teads.com>> wrote:
>
> Hi,
> May I have the confirmation that the max-parallelism limitation
> only occurs when keyed states are used ?
>
>
> --
>
> Olivier Nouguier
>
> SSE
>
> e | olivier.nouguier@teads.com <ma...@teads.com>
> m | 0651383971
>
> Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
> Montpellier, France*
> *
>
> image <https://www.teads.com>
>
> The information in this email is confidential and intended only
> for the addressee(s) named above. If you are not the intended
> recipient any disclosure, copying, distribution or any action
> taken or omitted to be taken in reliance on it is prohibited and
> may be unlawful. Teads does not warrant that any attachment(s) are
> free from viruses or other defects and accept no liability for any
> losses resulting from infected email transmission. Please note
> that any views expressed in this email may be those of the
> originator and do not necessarily reflect those of the organization.
>
Re: Max-parellelism limitation
Posted by Guowei Ma <gu...@gmail.com>.
Hi, Olivier
Yes. The introduction of this concept is to solve the problem of rescaling
the keystate.
Best,
Guowei
On Mon, Apr 19, 2021 at 8:56 PM Olivier Nouguier <ol...@teads.com>
wrote:
> Hi,
> May I have the confirmation that the max-parallelism limitation only
> occurs when keyed states are used ?
>
>
> --
>
> Olivier Nouguier
>
> SSE
>
> e | olivier.nouguier@teads.com m | 0651383971
>
> Teads France SAS, 159 rue de Thor, Business Plaza, Bat. 4, 34000
> Montpellier, France
> [image: image] <https://www.teads.com>
>
> The information in this email is confidential and intended only for the
> addressee(s) named above. If you are not the intended recipient any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it is prohibited and may be unlawful. Teads does not
> warrant that any attachment(s) are free from viruses or other defects and
> accept no liability for any losses resulting from infected email
> transmission. Please note that any views expressed in this email may be
> those of the originator and do not necessarily reflect those of the
> organization.
>