You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Dennis-Mircea Ciupitu <de...@gmail.com> on 2026/05/18 12:23:39 UTC
Re: [DISCUSS] FLIP-XXX: Expose per downstream target numRecordsOut metric
Hi all,
Bringing this thread back up after several weeks of quiet.
The FLIP proposes breaking down the task-level numRecordsOut metric per
downstream target vertex so consumers (in particular the Kubernetes
autoscaler) can see per-edge throughput instead of guessing from the
aggregate. Purely additive, the aggregate metric stays unchanged. Full
motivation, design, and alternatives are in the FLIP doc.
Design doc:
https://docs.google.com/document/d/1TlyTc6fvYGG1xlO-IFlGBC97CxilUBFdZBpLOBPdKD8/edit?usp=sharing
Draft implementation: https://github.com/apache/flink/pull/28014
@Marton Balassi, @Gabor Somogyi, and @Peter Huang, based on your vote on
[1] & [2], I would really value your read here.
@Maximilian Michels and @Rui Fan, this metric is autoscaler-adjacent, your
read would be particularly useful.
Thanks,
Dennis
[1] https://lists.apache.org/thread/zffx5r6fjkjldjjxr8oqtkwsobtwc6c3
[2] https://lists.apache.org/thread/pkn8rmdfzfq5vl8smj8kw03jns9gq8ty
On Thu, Apr 23, 2026 at 11:01 PM Dennis-Mircea Ciupitu <
dennis.mircea.ciupitu@gmail.com> wrote:
> Hi all,
>
> I’d like to start a discussion on FLIP-XXX: Expose per downstream target
> numRecordsOut metric [1].
>
> I’ve also opened a draft PR with a reference implementation to make the
> proposal concrete and easier to review [2].
>
> Feedback and suggestions are very welcome.
>
> Best regards,
> Dennis
>
> [1]
> https://docs.google.com/document/d/1TlyTc6fvYGG1xlO-IFlGBC97CxilUBFdZBpLOBPdKD8/edit?usp=sharing
> [2] https://github.com/apache/flink/pull/28014
>
Re: [DISCUSS] FLIP-XXX: Expose per downstream target numRecordsOut metric
Posted by Dennis-Mircea Ciupitu <de...@gmail.com>.
Hi,
Thank you for the review.
I have started the voting thread for this FLIP:
https://lists.apache.org/thread/m1vmt2hyhfg2yz39r2po1hdfqg17rydp
Best,
Dennis
On Wed, Jun 3, 2026 at 12:10 PM Gyula Fóra <gy...@gmail.com> wrote:
> Hi Dennis!
> This makes sense it would be a great improvement to both the autoscaler
> (simplified logic) and the Flink Web UI
>
> I think we can go ahead with this FLIP
>
> Gyula
>
> On Mon, May 18, 2026 at 2:24 PM Dennis-Mircea Ciupitu <
> dennis.mircea.ciupitu@gmail.com> wrote:
>
> > Hi all,
> >
> > Bringing this thread back up after several weeks of quiet.
> >
> > The FLIP proposes breaking down the task-level numRecordsOut metric per
> > downstream target vertex so consumers (in particular the Kubernetes
> > autoscaler) can see per-edge throughput instead of guessing from the
> > aggregate. Purely additive, the aggregate metric stays unchanged. Full
> > motivation, design, and alternatives are in the FLIP doc.
> >
> > Design doc:
> >
> >
> https://docs.google.com/document/d/1TlyTc6fvYGG1xlO-IFlGBC97CxilUBFdZBpLOBPdKD8/edit?usp=sharing
> > Draft implementation: https://github.com/apache/flink/pull/28014
> >
> > @Marton Balassi, @Gabor Somogyi, and @Peter Huang, based on your vote on
> > [1] & [2], I would really value your read here.
> > @Maximilian Michels and @Rui Fan, this metric is autoscaler-adjacent,
> your
> > read would be particularly useful.
> >
> > Thanks,
> > Dennis
> >
> > [1] https://lists.apache.org/thread/zffx5r6fjkjldjjxr8oqtkwsobtwc6c3
> > [2] https://lists.apache.org/thread/pkn8rmdfzfq5vl8smj8kw03jns9gq8ty
> >
> >
> > On Thu, Apr 23, 2026 at 11:01 PM Dennis-Mircea Ciupitu <
> > dennis.mircea.ciupitu@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I’d like to start a discussion on FLIP-XXX: Expose per downstream
> target
> > > numRecordsOut metric [1].
> > >
> > > I’ve also opened a draft PR with a reference implementation to make the
> > > proposal concrete and easier to review [2].
> > >
> > > Feedback and suggestions are very welcome.
> > >
> > > Best regards,
> > > Dennis
> > >
> > > [1]
> > >
> >
> https://docs.google.com/document/d/1TlyTc6fvYGG1xlO-IFlGBC97CxilUBFdZBpLOBPdKD8/edit?usp=sharing
> > > [2] https://github.com/apache/flink/pull/28014
> > >
> >
>
Re: [DISCUSS] FLIP-XXX: Expose per downstream target numRecordsOut metric
Posted by Gyula Fóra <gy...@gmail.com>.
Hi Dennis!
This makes sense it would be a great improvement to both the autoscaler
(simplified logic) and the Flink Web UI
I think we can go ahead with this FLIP
Gyula
On Mon, May 18, 2026 at 2:24 PM Dennis-Mircea Ciupitu <
dennis.mircea.ciupitu@gmail.com> wrote:
> Hi all,
>
> Bringing this thread back up after several weeks of quiet.
>
> The FLIP proposes breaking down the task-level numRecordsOut metric per
> downstream target vertex so consumers (in particular the Kubernetes
> autoscaler) can see per-edge throughput instead of guessing from the
> aggregate. Purely additive, the aggregate metric stays unchanged. Full
> motivation, design, and alternatives are in the FLIP doc.
>
> Design doc:
>
> https://docs.google.com/document/d/1TlyTc6fvYGG1xlO-IFlGBC97CxilUBFdZBpLOBPdKD8/edit?usp=sharing
> Draft implementation: https://github.com/apache/flink/pull/28014
>
> @Marton Balassi, @Gabor Somogyi, and @Peter Huang, based on your vote on
> [1] & [2], I would really value your read here.
> @Maximilian Michels and @Rui Fan, this metric is autoscaler-adjacent, your
> read would be particularly useful.
>
> Thanks,
> Dennis
>
> [1] https://lists.apache.org/thread/zffx5r6fjkjldjjxr8oqtkwsobtwc6c3
> [2] https://lists.apache.org/thread/pkn8rmdfzfq5vl8smj8kw03jns9gq8ty
>
>
> On Thu, Apr 23, 2026 at 11:01 PM Dennis-Mircea Ciupitu <
> dennis.mircea.ciupitu@gmail.com> wrote:
>
> > Hi all,
> >
> > I’d like to start a discussion on FLIP-XXX: Expose per downstream target
> > numRecordsOut metric [1].
> >
> > I’ve also opened a draft PR with a reference implementation to make the
> > proposal concrete and easier to review [2].
> >
> > Feedback and suggestions are very welcome.
> >
> > Best regards,
> > Dennis
> >
> > [1]
> >
> https://docs.google.com/document/d/1TlyTc6fvYGG1xlO-IFlGBC97CxilUBFdZBpLOBPdKD8/edit?usp=sharing
> > [2] https://github.com/apache/flink/pull/28014
> >
>
Re: [DISCUSS] FLIP-XXX: Expose per downstream target numRecordsOut metric
Posted by Rahul Bhattacharya <ra...@gmail.com>.
hi can you take me off this maling list
On Mon, May 18, 2026 at 7:23 AM Dennis-Mircea Ciupitu <
dennis.mircea.ciupitu@gmail.com> wrote:
> Hi all,
>
> Bringing this thread back up after several weeks of quiet.
>
> The FLIP proposes breaking down the task-level numRecordsOut metric per
> downstream target vertex so consumers (in particular the Kubernetes
> autoscaler) can see per-edge throughput instead of guessing from the
> aggregate. Purely additive, the aggregate metric stays unchanged. Full
> motivation, design, and alternatives are in the FLIP doc.
>
> Design doc:
>
> https://docs.google.com/document/d/1TlyTc6fvYGG1xlO-IFlGBC97CxilUBFdZBpLOBPdKD8/edit?usp=sharing
> Draft implementation: https://github.com/apache/flink/pull/28014
>
> @Marton Balassi, @Gabor Somogyi, and @Peter Huang, based on your vote on
> [1] & [2], I would really value your read here.
> @Maximilian Michels and @Rui Fan, this metric is autoscaler-adjacent, your
> read would be particularly useful.
>
> Thanks,
> Dennis
>
> [1] https://lists.apache.org/thread/zffx5r6fjkjldjjxr8oqtkwsobtwc6c3
> [2] https://lists.apache.org/thread/pkn8rmdfzfq5vl8smj8kw03jns9gq8ty
>
>
> On Thu, Apr 23, 2026 at 11:01 PM Dennis-Mircea Ciupitu <
> dennis.mircea.ciupitu@gmail.com> wrote:
>
> > Hi all,
> >
> > I’d like to start a discussion on FLIP-XXX: Expose per downstream target
> > numRecordsOut metric [1].
> >
> > I’ve also opened a draft PR with a reference implementation to make the
> > proposal concrete and easier to review [2].
> >
> > Feedback and suggestions are very welcome.
> >
> > Best regards,
> > Dennis
> >
> > [1]
> >
> https://docs.google.com/document/d/1TlyTc6fvYGG1xlO-IFlGBC97CxilUBFdZBpLOBPdKD8/edit?usp=sharing
> > [2] https://github.com/apache/flink/pull/28014
> >
>
--
Thanks And Regards
Rahul