You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Dennis-Mircea Ciupitu <de...@gmail.com> on 2026/05/18 12:35:23 UTC
Re: [DISCUSS] FLIP-XXX: Composable Parallelism Alignment Modes for Flink Autoscaler
Hi all,
Bringing this thread back up after several weeks of quiet.
The FLIP proposes composable alignment modes for the autoscaler so users
can opt into the strategy that fits their job rather than the single fixed
rule applied today (FLINK-32119 and FLINK-32120 are concrete instances of
the resulting friction). Full motivation, alternatives, and design details
are in the FLIP doc.
Design doc:
https://docs.google.com/document/d/1IbvKspt5jjjqiXl_CeHgI8EBn8LP-Pw0t7LqF85yFRw/edit?usp=sharing
Draft implementation:
https://github.com/apache/flink-kubernetes-operator/pull/1088
@Marton Balassi, @Gabor Somogyi, and @Peter Huang, based on your vote on
[1] & [2], I would really value your read here.
@Maximilian Michels, you opened FLINK-32119 and FLINK-32120, would value
your read on whether the composable-modes direction matches what you had in
mind when filing those.
Thanks,
Dennis
[1] https://lists.apache.org/thread/zffx5r6fjkjldjjxr8oqtkwsobtwc6c3
[2] https://lists.apache.org/thread/pkn8rmdfzfq5vl8smj8kw03jns9gq8ty
On Thu, Apr 23, 2026 at 2:23 PM Dennis-Mircea Ciupitu <
dennis.mircea.ciupitu@gmail.com> wrote:
> Hi David,
>
> Thanks for the feedback, I addressed them in the FLIP.
>
> Please have a look once again and let me know what you think.
>
> Best regards,
> Dennis
>
>
> On Thu, Apr 23, 2026 at 11:17 AM David Radley <da...@uk.ibm.com>
> wrote:
>
>> Hi there,
>> It would be useful in the motivation to explicitly describe the value of
>> the different options and the use cases that would drive the need for each
>> of these permutations and all best practises. Can the autoscaler work out
>> automatically which of these to use?
>> Kind regards, David.
>>
>> From: Dennis-Mircea Ciupitu <de...@gmail.com>
>> Date: Wednesday, 15 April 2026 at 08:14
>> To: dev@flink.apache.org <de...@flink.apache.org>
>> Subject: [EXTERNAL] [DISCUSS] FLIP-XXX: Composable Parallelism Alignment
>> Modes for Flink Autoscaler
>>
>> Hi all,
>>
>> [Sorry for reposting. I noticed the title in the previous message wasn’t
>> correct, so I’m sharing the updated version here.]
>>
>> I’d like to start a discussion on FLIP-XXX: Composable Parallelism
>> Alignment Modes for Flink Autoscaler [1].
>>
>> I’ve also opened a draft PR with a reference implementation to make the
>> proposal concrete and easier to review [2].
>>
>> Feedback and suggestions are very welcome.
>>
>> Best regards,
>> Dennis
>>
>> [1]
>>
>> https://docs.google.com/document/d/1IbvKspt5jjjqiXl_CeHgI8EBn8LP-Pw0t7LqF85yFRw/edit?usp=sharing
>> [2] https://github.com/apache/flink-kubernetes-operator/pull/1088
>>
>> Unless otherwise stated above:
>>
>> IBM United Kingdom Limited
>> Registered in England and Wales with number 741598
>> Registered office: Building C, IBM Hursley Office, Hursley Park Road,
>> Winchester, Hampshire SO21 2JN
>>
>
Re: [DISCUSS] FLIP-XXX: Composable Parallelism Alignment Modes for Flink Autoscaler
Posted by Dennis-Mircea Ciupitu <de...@gmail.com>.
Hi,
Thank you for the review.
I have started the voting thread for this FLIP:
https://lists.apache.org/thread/nv1103fg62dpqdzrn0g086qh57yyp96c
Best,
Dennis
On Wed, Jun 3, 2026 at 12:04 PM Gyula Fóra <gy...@gmail.com> wrote:
> Hey Dennis!
>
> Thanks for this Flip, this is pretty confusing part of the autoscaler logic
> especially with the hardcoded logic.
>
> I agree with the new behaviour and proposed new defaults. Maybe one comment
> regarding the config keys.
> I think the old key:
> job.autoscaler.scaling.key-group.partitions.adjust.mode is pretty bad, in
> terms of naming/expressivity.
>
> Maybe this is a good chance to actually change the name to something like
>
> job.autoscaler.scaling.key-group-alignment.mode
> or
> job.autoscaler.scaling.key-group.parallelism-adjustment.mode
>
> In any case we should go head with this FLIP to the vote and we can agree
> on the config naming in the meantime.
>
> Gyula
>
> On Mon, May 18, 2026 at 2:36 PM Dennis-Mircea Ciupitu <
> dennis.mircea.ciupitu@gmail.com> wrote:
>
> > Hi all,
> >
> > Bringing this thread back up after several weeks of quiet.
> >
> > The FLIP proposes composable alignment modes for the autoscaler so users
> > can opt into the strategy that fits their job rather than the single
> fixed
> > rule applied today (FLINK-32119 and FLINK-32120 are concrete instances of
> > the resulting friction). Full motivation, alternatives, and design
> details
> > are in the FLIP doc.
> >
> > Design doc:
> >
> >
> https://docs.google.com/document/d/1IbvKspt5jjjqiXl_CeHgI8EBn8LP-Pw0t7LqF85yFRw/edit?usp=sharing
> > Draft implementation:
> > https://github.com/apache/flink-kubernetes-operator/pull/1088
> >
> > @Marton Balassi, @Gabor Somogyi, and @Peter Huang, based on your vote on
> > [1] & [2], I would really value your read here.
> > @Maximilian Michels, you opened FLINK-32119 and FLINK-32120, would value
> > your read on whether the composable-modes direction matches what you had
> in
> > mind when filing those.
> >
> > Thanks,
> > Dennis
> >
> > [1] https://lists.apache.org/thread/zffx5r6fjkjldjjxr8oqtkwsobtwc6c3
> > [2] https://lists.apache.org/thread/pkn8rmdfzfq5vl8smj8kw03jns9gq8ty
> >
> > On Thu, Apr 23, 2026 at 2:23 PM Dennis-Mircea Ciupitu <
> > dennis.mircea.ciupitu@gmail.com> wrote:
> >
> > > Hi David,
> > >
> > > Thanks for the feedback, I addressed them in the FLIP.
> > >
> > > Please have a look once again and let me know what you think.
> > >
> > > Best regards,
> > > Dennis
> > >
> > >
> > > On Thu, Apr 23, 2026 at 11:17 AM David Radley <david_radley@uk.ibm.com
> >
> > > wrote:
> > >
> > >> Hi there,
> > >> It would be useful in the motivation to explicitly describe the value
> > of
> > >> the different options and the use cases that would drive the need for
> > each
> > >> of these permutations and all best practises. Can the autoscaler work
> > out
> > >> automatically which of these to use?
> > >> Kind regards, David.
> > >>
> > >> From: Dennis-Mircea Ciupitu <de...@gmail.com>
> > >> Date: Wednesday, 15 April 2026 at 08:14
> > >> To: dev@flink.apache.org <de...@flink.apache.org>
> > >> Subject: [EXTERNAL] [DISCUSS] FLIP-XXX: Composable Parallelism
> Alignment
> > >> Modes for Flink Autoscaler
> > >>
> > >> Hi all,
> > >>
> > >> [Sorry for reposting. I noticed the title in the previous message
> wasn’t
> > >> correct, so I’m sharing the updated version here.]
> > >>
> > >> I’d like to start a discussion on FLIP-XXX: Composable Parallelism
> > >> Alignment Modes for Flink Autoscaler [1].
> > >>
> > >> I’ve also opened a draft PR with a reference implementation to make
> the
> > >> proposal concrete and easier to review [2].
> > >>
> > >> Feedback and suggestions are very welcome.
> > >>
> > >> Best regards,
> > >> Dennis
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://docs.google.com/document/d/1IbvKspt5jjjqiXl_CeHgI8EBn8LP-Pw0t7LqF85yFRw/edit?usp=sharing
> > >> [2] https://github.com/apache/flink-kubernetes-operator/pull/1088
> > >>
> > >> Unless otherwise stated above:
> > >>
> > >> IBM United Kingdom Limited
> > >> Registered in England and Wales with number 741598
> > >> Registered office: Building C, IBM Hursley Office, Hursley Park Road,
> > >> Winchester, Hampshire SO21 2JN
> > >>
> > >
> >
>
Re: [DISCUSS] FLIP-XXX: Composable Parallelism Alignment Modes for Flink Autoscaler
Posted by Gyula Fóra <gy...@gmail.com>.
Hey Dennis!
Thanks for this Flip, this is pretty confusing part of the autoscaler logic
especially with the hardcoded logic.
I agree with the new behaviour and proposed new defaults. Maybe one comment
regarding the config keys.
I think the old key:
job.autoscaler.scaling.key-group.partitions.adjust.mode is pretty bad, in
terms of naming/expressivity.
Maybe this is a good chance to actually change the name to something like
job.autoscaler.scaling.key-group-alignment.mode
or
job.autoscaler.scaling.key-group.parallelism-adjustment.mode
In any case we should go head with this FLIP to the vote and we can agree
on the config naming in the meantime.
Gyula
On Mon, May 18, 2026 at 2:36 PM Dennis-Mircea Ciupitu <
dennis.mircea.ciupitu@gmail.com> wrote:
> Hi all,
>
> Bringing this thread back up after several weeks of quiet.
>
> The FLIP proposes composable alignment modes for the autoscaler so users
> can opt into the strategy that fits their job rather than the single fixed
> rule applied today (FLINK-32119 and FLINK-32120 are concrete instances of
> the resulting friction). Full motivation, alternatives, and design details
> are in the FLIP doc.
>
> Design doc:
>
> https://docs.google.com/document/d/1IbvKspt5jjjqiXl_CeHgI8EBn8LP-Pw0t7LqF85yFRw/edit?usp=sharing
> Draft implementation:
> https://github.com/apache/flink-kubernetes-operator/pull/1088
>
> @Marton Balassi, @Gabor Somogyi, and @Peter Huang, based on your vote on
> [1] & [2], I would really value your read here.
> @Maximilian Michels, you opened FLINK-32119 and FLINK-32120, would value
> your read on whether the composable-modes direction matches what you had in
> mind when filing those.
>
> Thanks,
> Dennis
>
> [1] https://lists.apache.org/thread/zffx5r6fjkjldjjxr8oqtkwsobtwc6c3
> [2] https://lists.apache.org/thread/pkn8rmdfzfq5vl8smj8kw03jns9gq8ty
>
> On Thu, Apr 23, 2026 at 2:23 PM Dennis-Mircea Ciupitu <
> dennis.mircea.ciupitu@gmail.com> wrote:
>
> > Hi David,
> >
> > Thanks for the feedback, I addressed them in the FLIP.
> >
> > Please have a look once again and let me know what you think.
> >
> > Best regards,
> > Dennis
> >
> >
> > On Thu, Apr 23, 2026 at 11:17 AM David Radley <da...@uk.ibm.com>
> > wrote:
> >
> >> Hi there,
> >> It would be useful in the motivation to explicitly describe the value
> of
> >> the different options and the use cases that would drive the need for
> each
> >> of these permutations and all best practises. Can the autoscaler work
> out
> >> automatically which of these to use?
> >> Kind regards, David.
> >>
> >> From: Dennis-Mircea Ciupitu <de...@gmail.com>
> >> Date: Wednesday, 15 April 2026 at 08:14
> >> To: dev@flink.apache.org <de...@flink.apache.org>
> >> Subject: [EXTERNAL] [DISCUSS] FLIP-XXX: Composable Parallelism Alignment
> >> Modes for Flink Autoscaler
> >>
> >> Hi all,
> >>
> >> [Sorry for reposting. I noticed the title in the previous message wasn’t
> >> correct, so I’m sharing the updated version here.]
> >>
> >> I’d like to start a discussion on FLIP-XXX: Composable Parallelism
> >> Alignment Modes for Flink Autoscaler [1].
> >>
> >> I’ve also opened a draft PR with a reference implementation to make the
> >> proposal concrete and easier to review [2].
> >>
> >> Feedback and suggestions are very welcome.
> >>
> >> Best regards,
> >> Dennis
> >>
> >> [1]
> >>
> >>
> https://docs.google.com/document/d/1IbvKspt5jjjqiXl_CeHgI8EBn8LP-Pw0t7LqF85yFRw/edit?usp=sharing
> >> [2] https://github.com/apache/flink-kubernetes-operator/pull/1088
> >>
> >> Unless otherwise stated above:
> >>
> >> IBM United Kingdom Limited
> >> Registered in England and Wales with number 741598
> >> Registered office: Building C, IBM Hursley Office, Hursley Park Road,
> >> Winchester, Hampshire SO21 2JN
> >>
> >
>