You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Weiqing Yang <ya...@gmail.com> on 2026/03/03 22:58:44 UTC
Re: Re: [DISCUSS] Add UDF Metrics
Hi Zakelly,
Thanks for the feedback and sorry for the late response - I am now picking
it back up.
You raised a great point about the performance overhead, referencing
FLINK-16444 <https://issues.apache.org/jira/browse/FLINK-16444>. I've
updated the FLIP to adopt the same counter-based sampling approach used by
Flink's state latency tracking (FLINK-21736
<https://issues.apache.org/jira/browse/FLINK-21736>). Specifically:
1. New config: table.exec.udf-metric.sample-interval (default: 100 [1]) -
only every Nth invocation is measured
2. Fast path: Non-sampled invocations are a single integer increment -
negligible overhead
3. Sampled path: System.nanoTime() around the UDF call, stored in a
DescriptiveStatisticsHistogram with a bounded 128-entry circular buffer [2]
4. Metric type change: udfProcessingTime is now a Histogram (reports
p50/p75/p95/p99/mean/min/max) instead of the original Gauge
5. Exception counting: Not sampled, since exceptions are rare events and
counting each one has negligible cost
Combined with the existing feature gate (table.exec.udf-metric-enabled
defaulting to false), users have two layers of protection: the feature is
off by default, and when enabled, sampling keeps overhead minimal.
The updated FLIP is here: link
<https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1>
Would this address your concern? If so, it would be great to have your vote
on the vote thread [3].
[1] 100: state.latency-track.sample-interval default value
[2] 128: state.latency-track.history-size default value (line 55), which is
the circular buffer size for the DescriptiveStatisticsHistogram
[3] https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
Thanks,
Weiqing
On Thu, Aug 21, 2025 at 12:24 AM Zakelly Lan <za...@gmail.com> wrote:
> Hi Weiqing,
>
> Sorry for the late reply. And I have one question:
>
> I'm wondering whether the UDF processing time is measured for every
> individual UDF invocation, with the average then reported, or if sampling
> is used instead? I'm concerned about the potential overhead if we measure
> every single invocation. We've encountered similar performance issues when
> implementing state latency tracking [1].
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-16444
>
> Best,
> Zakelly
>
> On Fri, Aug 15, 2025 at 5:04 AM Weiqing Yang <ya...@gmail.com>
> wrote:
>
> > Cool - I’ll proceed to start the VOTE.
> > Thanks!
> >
> > Weiqing
> >
> > On Thu, Aug 14, 2025 at 12:53 AM Shengkai Fang <fs...@gmail.com>
> wrote:
> >
> > > I don't have any more comments.
> > >
> > > Best,
> > > Shengkai
> > >
> > > Weiqing Yang <ya...@gmail.com> 于2025年8月14日周四 14:47写道：
> > >
> > > > Thanks, Shengkai. I’ve updated the proposal doc with the recommended
> > > > configuration name. Please let me know if you have any additional
> > > feedback.
> > > >
> > > > Best,
> > > > Weiqing
> > > >
> > > > On Wed, Aug 13, 2025 at 6:58 PM Shengkai Fang <fs...@gmail.com>
> > wrote:
> > > >
> > > > > Sorry for the late response. I prefer to use
> > > > > `table.exec.udf-metric-enabled` as the option name.
> > > > >
> > > > > Best,
> > > > > Shengkai
> > > > >
> > > > > Weiqing Yang <ya...@gmail.com> 于2025年8月13日周三 23:54写道：
> > > > >
> > > > > > Hi Shengkai, Alan, Xuyang, and all,
> > > > > >
> > > > > > Since there have been no further objections, I’ll proceed to
> start
> > > the
> > > > > VOTE
> > > > > > on this proposal shortly.
> > > > > >
> > > > > > Thanks,
> > > > > > Weiqing
> > > > > >
> > > > > > On Thu, Jul 31, 2025 at 10:26 PM Weiqing Yang <
> > > > yangweiqing001@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Shengkai, Alan and Xuyang,
> > > > > > >
> > > > > > > Just checking in - do you have any concerns or feedback?
> > > > > > >
> > > > > > > If there are no further objections from anyone, I’ll mark the
> > FLIP
> > > as
> > > > > > > ready for voting.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Weiqing
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jul 14, 2025 at 9:10 PM Weiqing Yang <
> > > > yangweiqing001@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Xuyang,
> > > > > > >>
> > > > > > >> Thank you for reviewing the proposal!
> > > > > > >>
> > > > > > >> I’m planning to use: *udf.metrics.process-time* and
> > > > > > >> *udf.metrics.exception-count*. These follow the naming
> > convention
> > > > used
> > > > > > >> in Flink (e.g., RocksDB native metrics
> > > > > > >> <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics
> > > > > > >).
> > > > > > >> I’ve added these names to the proposal doc.
> > > > > > >>
> > > > > > >> Alternatively, I also considered:
> > > *metrics.udf.process-time.enabled*
> > > > > and
> > > > > > >> *metrics.udf.exception-count.enabled. *
> > > > > > >>
> > > > > > >> Happy to hear any feedback on which style might be more
> > > appropriate.
> > > > > > >>
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Weiqing
> > > > > > >>
> > > > > > >> On Mon, Jul 14, 2025 at 2:55 AM Xuyang <xy...@163.com>
> > > wrote:
> > > > > > >>
> > > > > > >>> Hi, Weiqing.
> > > > > > >>>
> > > > > > >>> Thanks for driving to improve this. I just have one
> question. I
> > > > > notice
> > > > > > a
> > > > > > >>> new configuration is introduced in this flip. I just wonder
> > what
> > > > the
> > > > > > >>> configuration name is. Could you please include the full name
> > of
> > > > this
> > > > > > >>> configuration? (just similar to the other names in
> > > MetricOptions?)
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>>
> > > > > > >>> Best！
> > > > > > >>> Xuyang
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> 在 2025-07-13 12:03:59，"Weiqing Yang" <
> yangweiqing001@gmail.com
> > >
> > > > 写道：
> > > > > > >>> >Hi Alan,
> > > > > > >>> >
> > > > > > >>> >Thanks for reviewing the proposal and for highlighting the
> > > > > ASYNC_TABLE
> > > > > > >>> work.
> > > > > > >>> >
> > > > > > >>> >Yes, I’ve updated the proposal to cover both ASYNC_SCALAR
> and
> > > > > > >>> ASYNC_TABLE.
> > > > > > >>> >For async UDFs, the plan is to instrument both the
> > invokeAsync()
> > > > > call
> > > > > > >>> and
> > > > > > >>> >the async callback handler to measure the full end-to-end
> > > latency
> > > > > > until
> > > > > > >>> the
> > > > > > >>> >result or error is returned from the future.
> > > > > > >>> >
> > > > > > >>> >Let me know if you have any further questions or
> suggestions.
> > > > > > >>> >
> > > > > > >>> >Best,
> > > > > > >>> >Weiqing
> > > > > > >>> >
> > > > > > >>> >On Thu, Jul 10, 2025 at 4:15 PM Alan Sheinberg
> > > > > > >>> ><as...@confluent.io.invalid> wrote:
> > > > > > >>> >
> > > > > > >>> >> Hi Weiqing,
> > > > > > >>> >>
> > > > > > >>> >> From your doc, the entrypoint for UDF calls in the codegen
> > is
> > > > > > >>> >> ExprCodeGenerator which should invoke
> > > > BridgingSqlFunctionCallGen,
> > > > > > >>> which
> > > > > > >>> >> could be instrumented with metrics. This works well for
> > > > > synchronous
> > > > > > >>> calls,
> > > > > > >>> >> but what about ASYNC_SCALAR and the soon to be merged
> > > > ASYNC_TABLE
> > > > > (
> > > > > > >>> >> https://github.com/apache/flink/pull/26567)? Timing
> > metrics
> > > > > would
> > > > > > >>> only
> > > > > > >>> >> account for what it takes to call invokeAsync, not for the
> > > > result
> > > > > to
> > > > > > >>> >> complete (with a result or error from the future object).
> > > > > > >>> >>
> > > > > > >>> >> There are appropriate places which can handle the async
> > > > callbacks,
> > > > > > >>> but they
> > > > > > >>> >> are in other locations. Will you be able to support those
> > as
> > > > > well?
> > > > > > >>> >>
> > > > > > >>> >> Thanks,
> > > > > > >>> >> Alan
> > > > > > >>> >>
> > > > > > >>> >> On Wed, Jul 9, 2025 at 7:52 PM Shengkai Fang <
> > > fskmine@gmail.com
> > > > >
> > > > > > >>> wrote:
> > > > > > >>> >>
> > > > > > >>> >> > I just have some questions:
> > > > > > >>> >> >
> > > > > > >>> >> > 1. The current metrics hierarchy shows that the UDF
> metric
> > > > group
> > > > > > >>> belongs
> > > > > > >>> >> to
> > > > > > >>> >> > the TaskMetricGroup. I think it would be better for the
> > UDF
> > > > > metric
> > > > > > >>> group
> > > > > > >>> >> to
> > > > > > >>> >> > belong to the OperatorMetricGroup instead, because a UDF
> > > might
> > > > > be
> > > > > > >>> used by
> > > > > > >>> >> > multiple operators.
> > > > > > >>> >> > 2. What are the naming conventions for UDF metrics?
> Could
> > > you
> > > > > > >>> provide an
> > > > > > >>> >> > example? Do the metric name contains the UDF name?
> > > > > > >>> >> > 3. Why is the UDFExceptionCount metric introduced? If a
> > UDF
> > > > > throws
> > > > > > >>> an
> > > > > > >>> >> > exception, the job fails immediately. Why do we need to
> > > track
> > > > > this
> > > > > > >>> value?
> > > > > > >>> >> >
> > > > > > >>> >> > Best
> > > > > > >>> >> > Shengkai
> > > > > > >>> >> >
> > > > > > >>> >> >
> > > > > > >>> >> > Weiqing Yang <ya...@gmail.com> 于2025年7月9日周三
> > > 12:59写道：
> > > > > > >>> >> >
> > > > > > >>> >> > > Hi all,
> > > > > > >>> >> > >
> > > > > > >>> >> > > I’d like to initiate a discussion about adding UDF
> > > metrics.
> > > > > > >>> >> > >
> > > > > > >>> >> > > *Motivation*
> > > > > > >>> >> > >
> > > > > > >>> >> > > User-defined functions (UDFs) are essential for custom
> > > logic
> > > > > in
> > > > > > >>> Flink
> > > > > > >>> >> > jobs
> > > > > > >>> >> > > but often act as black boxes, making debugging and
> > > > performance
> > > > > > >>> tuning
> > > > > > >>> >> > > difficult. When issues like high latency or frequent
> > > > > exceptions
> > > > > > >>> occur,
> > > > > > >>> >> > it's
> > > > > > >>> >> > > hard to pinpoint the root cause inside UDFs.
> > > > > > >>> >> > >
> > > > > > >>> >> > > Flink currently lacks built-in metrics for key UDF
> > aspects
> > > > > such
> > > > > > as
> > > > > > >>> >> > > per-record processing time or exception count. This
> > limits
> > > > > > >>> >> observability
> > > > > > >>> >> > > and complicates:
> > > > > > >>> >> > >
> > > > > > >>> >> > > - Debugging production issues
> > > > > > >>> >> > > - Performance tuning and resource allocation
> > > > > > >>> >> > > - Supplying reliable signals to autoscaling systems
> > > > > > >>> >> > >
> > > > > > >>> >> > > Introducing standard, opt-in UDF metrics will improve
> > > > platform
> > > > > > >>> >> > > observability and overall health.
> > > > > > >>> >> > > Here’s the proposal document: Link
> > > > > > >>> >> > > <
> > > > > > >>> >> > >
> > > > > > >>> >> >
> > > > > > >>> >>
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1
> > > > > > >>> >> > > >
> > > > > > >>> >> > >
> > > > > > >>> >> > > Your feedback and ideas are welcome to refine this
> > > feature.
> > > > > > >>> >> > >
> > > > > > >>> >> > >
> > > > > > >>> >> > > Thanks,
> > > > > > >>> >> > > Weiqing
> > > > > > >>> >> > >
> > > > > > >>> >> >
> > > > > > >>> >>
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Re: Re: [DISCUSS] Add UDF Metrics
Posted by Weiqing Yang <ya...@gmail.com>.
Thanks Xuyang and Yuepeng for the review and +1s, and everyone for the
earlier reviews.
Since there are no other concerns, I'll start a fresh [VOTE] thread now.
Thanks,
Weiqing
On Thu, Jul 23, 2026 at 7:29 PM Yuepeng Pan <pa...@apache.org> wrote:
> LGTM +1 on the design draft.
>
>
> Best Regards,
> Yuepeng Pan
>
> Xuyang <xy...@163.com> 于2026年7月24日周五 10:04写道：
>
> > Hi, Weiqing.
> > Thanks for driving this FLIP forward again, +1
> >
> >
> >
> >
> >
> > --
> >
> > Best！
> > Xuyang
> >
> >
> >
> > 在 2026-07-17 14:21:21，"Weiqing Yang" <ya...@gmail.com> 写道：
> > >Hi all,
> > >
> > >Gentle bump on this.
> > >
> > >The proposal is unchanged at a high level. It's been re-aligned with the
> > >current master, and the earlier overhead concern is addressed (details
> in
> > >my previous message).
> > >
> > >Shengkai, Alan, Xuyang, Zakelly — thank you for the thorough review last
> > >time. Yuepeng and Shengkai, thank you for your votes on the earlier
> round.
> > >Since you already know this proposal well, I'd really appreciate a quick
> > >check that the points you raised still sit right in the current version.
> > >
> > >If there are no other concerns over the next few days, I'll start a
> fresh
> > >[VOTE] thread so the votes reflect the current design.
> > >
> > >FLIP doc:
> > >
> >
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit
> > >
> > >Best,
> > >Weiqing
> > >
> > >On Thu, Jul 9, 2026 at 7:20 PM Weiqing Yang <ya...@gmail.com>
> > >wrote:
> > >
> > >> Hi all,
> > >>
> > >> I'd like to revive this discussion. This FLIP was originally proposed
> > and
> > >> went to a vote last year (it received two +1s) [1]. Since the original
> > >> proposal and vote were about a year ago, I want to revisit the
> > discussion
> > >> before starting a fresh vote to ensure the design remains fully
> aligned
> > >> with the current master branch.
> > >>
> > >> At a high level, the proposal is unchanged from the previous round.
> The
> > >> one substantive refinement since the earlier votes is the sampling
> > approach
> > >> detailed below, which directly addresses the performance overhead
> > question
> > >> Zakelly raised (the last open item on the thread).
> > >>
> > >> Overhead (Zakelly's question): Processing time is no longer measured
> on
> > >> every invocation. It now uses the same counter-based sampling as
> Flink's
> > >> state latency tracking (FLINK-21736). A new
> > >> `table.exec.udf-metric.sample-interval` option (default 100) means
> only
> > >> every Nth invocation is timed. The non-sampled fast path is a single
> > >> integer increment, and `udfProcessingTime` is now a Histogram
> > >> (p50/p75/p95/p99) backed by a bounded 128-entry circular buffer.
> > Combined
> > >> with the existing `table.exec.udf-metric-enabled` gate (off by
> default,
> > >> meaning nothing is registered when disabled), this provides two robust
> > >> layers of protection against performance overhead.
> > >>
> > >> - Proposal doc (updated): [2]
> > >>
> > >> - cwiki FLIP-485: [3]
> > >>
> > >> - Draft PR: [4] (implements the proposal; kept in draft until we
> > converge)
> > >>
> > >> I welcome your renewed feedback and any questions. If there are no
> > further
> > >> concerns, I'll start a fresh [VOTE] thread on the updated proposal so
> > the
> > >> final votes reflect the current design.
> > >>
> > >> Thanks,
> > >>
> > >> Weiqing
> > >>
> > >> [1] Previous vote thread:
> > >> https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
> > >>
> > >> [2]
> > >>
> >
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit
> > >>
> > >> [3]
> > >>
> >
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics
> > >>
> > >> [4] https://github.com/apache/flink/pull/28692
> > >>
> > >>
> > >> On Wed, Jun 3, 2026 at 4:59 PM Weiqing Yang <yangweiqing001@gmail.com
> >
> > >> wrote:
> > >>
> > >>> Hi Zakelly,
> > >>>
> > >>> Kindly pinging here to see if you had any remaining concerns
> regarding
> > >>> the FLIP.
> > >>>
> > >>> If there are no further questions or concerns from anyone, I plan to
> > >>> close this discussion thread and proceed with the vote thread.
> > >>>
> > >>> Thanks,
> > >>> Weiqing
> > >>>
> > >>> On Tue, Mar 3, 2026 at 2:58 PM Weiqing Yang <
> yangweiqing001@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Hi Zakelly,
> > >>>>
> > >>>>
> > >>>> Thanks for the feedback and sorry for the late response - I am now
> > >>>> picking it back up.
> > >>>>
> > >>>> You raised a great point about the performance overhead, referencing
> > >>>> FLINK-16444 <https://issues.apache.org/jira/browse/FLINK-16444>.
> I've
> > >>>> updated the FLIP to adopt the same counter-based sampling approach
> > used by
> > >>>> Flink's state latency tracking (FLINK-21736
> > >>>> <https://issues.apache.org/jira/browse/FLINK-21736>). Specifically:
> > >>>>
> > >>>> 1. New config: table.exec.udf-metric.sample-interval (default: 100
> > >>>> [1]) - only every Nth invocation is measured
> > >>>> 2. Fast path: Non-sampled invocations are a single integer
> > increment -
> > >>>> negligible overhead
> > >>>> 3. Sampled path: System.nanoTime() around the UDF call, stored in
> a
> > >>>> DescriptiveStatisticsHistogram with a bounded 128-entry circular
> > buffer [2]
> > >>>> 4. Metric type change: udfProcessingTime is now a Histogram
> (reports
> > >>>> p50/p75/p95/p99/mean/min/max) instead of the original Gauge
> > >>>> 5. Exception counting: Not sampled, since exceptions are rare
> events
> > >>>> and counting each one has negligible cost
> > >>>>
> > >>>> Combined with the existing feature gate
> (table.exec.udf-metric-enabled
> > >>>> defaulting to false), users have two layers of protection: the
> > feature is
> > >>>> off by default, and when enabled, sampling keeps overhead minimal.
> > >>>> The updated FLIP is here: link
> > >>>> <
> >
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1
> > >
> > >>>>
> > >>>> Would this address your concern? If so, it would be great to have
> your
> > >>>> vote on the vote thread [3].
> > >>>>
> > >>>> [1] 100: state.latency-track.sample-interval default value
> > >>>>
> > >>>> [2] 128: state.latency-track.history-size default value (line 55),
> > which
> > >>>> is the circular buffer size for the DescriptiveStatisticsHistogram
> > >>>> [3]
> https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
> > >>>>
> > >>>> Thanks,
> > >>>> Weiqing
> > >>>>
> > >>>> On Thu, Aug 21, 2025 at 12:24 AM Zakelly Lan <zakelly.lan@gmail.com
> >
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Weiqing,
> > >>>>>
> > >>>>> Sorry for the late reply. And I have one question:
> > >>>>>
> > >>>>> I'm wondering whether the UDF processing time is measured for every
> > >>>>> individual UDF invocation, with the average then reported, or if
> > >>>>> sampling
> > >>>>> is used instead? I'm concerned about the potential overhead if we
> > >>>>> measure
> > >>>>> every single invocation. We've encountered similar performance
> issues
> > >>>>> when
> > >>>>> implementing state latency tracking [1].
> > >>>>>
> > >>>>>
> > >>>>> [1] https://issues.apache.org/jira/browse/FLINK-16444
> > >>>>>
> > >>>>> Best,
> > >>>>> Zakelly
> > >>>>>
> > >>>>> On Fri, Aug 15, 2025 at 5:04 AM Weiqing Yang <
> > yangweiqing001@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>> > Cool - I’ll proceed to start the VOTE.
> > >>>>> > Thanks!
> > >>>>> >
> > >>>>> > Weiqing
> > >>>>> >
> > >>>>> > On Thu, Aug 14, 2025 at 12:53 AM Shengkai Fang <
> fskmine@gmail.com>
> > >>>>> wrote:
> > >>>>> >
> > >>>>> > > I don't have any more comments.
> > >>>>> > >
> > >>>>> > > Best,
> > >>>>> > > Shengkai
> > >>>>> > >
> > >>>>> > > Weiqing Yang <ya...@gmail.com> 于2025年8月14日周四 14:47写道：
> > >>>>> > >
> > >>>>> > > > Thanks, Shengkai. I’ve updated the proposal doc with the
> > >>>>> recommended
> > >>>>> > > > configuration name. Please let me know if you have any
> > additional
> > >>>>> > > feedback.
> > >>>>> > > >
> > >>>>> > > > Best,
> > >>>>> > > > Weiqing
> > >>>>> > > >
> > >>>>> > > > On Wed, Aug 13, 2025 at 6:58 PM Shengkai Fang <
> > fskmine@gmail.com>
> > >>>>> > wrote:
> > >>>>> > > >
> > >>>>> > > > > Sorry for the late response. I prefer to use
> > >>>>> > > > > `table.exec.udf-metric-enabled` as the option name.
> > >>>>> > > > >
> > >>>>> > > > > Best,
> > >>>>> > > > > Shengkai
> > >>>>> > > > >
> > >>>>> > > > > Weiqing Yang <ya...@gmail.com> 于2025年8月13日周三
> > 23:54写道：
> > >>>>> > > > >
> > >>>>> > > > > > Hi Shengkai, Alan, Xuyang, and all,
> > >>>>> > > > > >
> > >>>>> > > > > > Since there have been no further objections, I’ll proceed
> > to
> > >>>>> start
> > >>>>> > > the
> > >>>>> > > > > VOTE
> > >>>>> > > > > > on this proposal shortly.
> > >>>>> > > > > >
> > >>>>> > > > > > Thanks,
> > >>>>> > > > > > Weiqing
> > >>>>> > > > > >
> > >>>>> > > > > > On Thu, Jul 31, 2025 at 10:26 PM Weiqing Yang <
> > >>>>> > > > yangweiqing001@gmail.com>
> > >>>>> > > > > > wrote:
> > >>>>> > > > > >
> > >>>>> > > > > > > Hi Shengkai, Alan and Xuyang,
> > >>>>> > > > > > >
> > >>>>> > > > > > > Just checking in - do you have any concerns or
> feedback?
> > >>>>> > > > > > >
> > >>>>> > > > > > > If there are no further objections from anyone, I’ll
> mark
> > >>>>> the
> > >>>>> > FLIP
> > >>>>> > > as
> > >>>>> > > > > > > ready for voting.
> > >>>>> > > > > > >
> > >>>>> > > > > > >
> > >>>>> > > > > > > Best,
> > >>>>> > > > > > > Weiqing
> > >>>>> > > > > > >
> > >>>>> > > > > > >
> > >>>>> > > > > > > On Mon, Jul 14, 2025 at 9:10 PM Weiqing Yang <
> > >>>>> > > > yangweiqing001@gmail.com
> > >>>>> > > > > >
> > >>>>> > > > > > > wrote:
> > >>>>> > > > > > >
> > >>>>> > > > > > >> Hi Xuyang,
> > >>>>> > > > > > >>
> > >>>>> > > > > > >> Thank you for reviewing the proposal!
> > >>>>> > > > > > >>
> > >>>>> > > > > > >> I’m planning to use: *udf.metrics.process-time* and
> > >>>>> > > > > > >> *udf.metrics.exception-count*. These follow the naming
> > >>>>> > convention
> > >>>>> > > > used
> > >>>>> > > > > > >> in Flink (e.g., RocksDB native metrics
> > >>>>> > > > > > >> <
> > >>>>> > > > > >
> > >>>>> > > > >
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics
> > >>>>> > > > > > >).
> > >>>>> > > > > > >> I’ve added these names to the proposal doc.
> > >>>>> > > > > > >>
> > >>>>> > > > > > >> Alternatively, I also considered:
> > >>>>> > > *metrics.udf.process-time.enabled*
> > >>>>> > > > > and
> > >>>>> > > > > > >> *metrics.udf.exception-count.enabled. *
> > >>>>> > > > > > >>
> > >>>>> > > > > > >> Happy to hear any feedback on which style might be
> more
> > >>>>> > > appropriate.
> > >>>>> > > > > > >>
> > >>>>> > > > > > >>
> > >>>>> > > > > > >> Best,
> > >>>>> > > > > > >> Weiqing
> > >>>>> > > > > > >>
> > >>>>> > > > > > >> On Mon, Jul 14, 2025 at 2:55 AM Xuyang <
> > xyzhong131@163.com
> > >>>>> >
> > >>>>> > > wrote:
> > >>>>> > > > > > >>
> > >>>>> > > > > > >>> Hi, Weiqing.
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>> Thanks for driving to improve this. I just have one
> > >>>>> question. I
> > >>>>> > > > > notice
> > >>>>> > > > > > a
> > >>>>> > > > > > >>> new configuration is introduced in this flip. I just
> > >>>>> wonder
> > >>>>> > what
> > >>>>> > > > the
> > >>>>> > > > > > >>> configuration name is. Could you please include the
> > full
> > >>>>> name
> > >>>>> > of
> > >>>>> > > > this
> > >>>>> > > > > > >>> configuration? (just similar to the other names in
> > >>>>> > > MetricOptions?)
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>> --
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>> Best！
> > >>>>> > > > > > >>> Xuyang
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>> 在 2025-07-13 12:03:59，"Weiqing Yang" <
> > >>>>> yangweiqing001@gmail.com
> > >>>>> > >
> > >>>>> > > > 写道：
> > >>>>> > > > > > >>> >Hi Alan,
> > >>>>> > > > > > >>> >
> > >>>>> > > > > > >>> >Thanks for reviewing the proposal and for
> highlighting
> > >>>>> the
> > >>>>> > > > > ASYNC_TABLE
> > >>>>> > > > > > >>> work.
> > >>>>> > > > > > >>> >
> > >>>>> > > > > > >>> >Yes, I’ve updated the proposal to cover both
> > >>>>> ASYNC_SCALAR and
> > >>>>> > > > > > >>> ASYNC_TABLE.
> > >>>>> > > > > > >>> >For async UDFs, the plan is to instrument both the
> > >>>>> > invokeAsync()
> > >>>>> > > > > call
> > >>>>> > > > > > >>> and
> > >>>>> > > > > > >>> >the async callback handler to measure the full
> > end-to-end
> > >>>>> > > latency
> > >>>>> > > > > > until
> > >>>>> > > > > > >>> the
> > >>>>> > > > > > >>> >result or error is returned from the future.
> > >>>>> > > > > > >>> >
> > >>>>> > > > > > >>> >Let me know if you have any further questions or
> > >>>>> suggestions.
> > >>>>> > > > > > >>> >
> > >>>>> > > > > > >>> >Best,
> > >>>>> > > > > > >>> >Weiqing
> > >>>>> > > > > > >>> >
> > >>>>> > > > > > >>> >On Thu, Jul 10, 2025 at 4:15 PM Alan Sheinberg
> > >>>>> > > > > > >>> ><as...@confluent.io.invalid> wrote:
> > >>>>> > > > > > >>> >
> > >>>>> > > > > > >>> >> Hi Weiqing,
> > >>>>> > > > > > >>> >>
> > >>>>> > > > > > >>> >> From your doc, the entrypoint for UDF calls in the
> > >>>>> codegen
> > >>>>> > is
> > >>>>> > > > > > >>> >> ExprCodeGenerator which should invoke
> > >>>>> > > > BridgingSqlFunctionCallGen,
> > >>>>> > > > > > >>> which
> > >>>>> > > > > > >>> >> could be instrumented with metrics. This works
> well
> > >>>>> for
> > >>>>> > > > > synchronous
> > >>>>> > > > > > >>> calls,
> > >>>>> > > > > > >>> >> but what about ASYNC_SCALAR and the soon to be
> > merged
> > >>>>> > > > ASYNC_TABLE
> > >>>>> > > > > (
> > >>>>> > > > > > >>> >> https://github.com/apache/flink/pull/26567)?
> > Timing
> > >>>>> > metrics
> > >>>>> > > > > would
> > >>>>> > > > > > >>> only
> > >>>>> > > > > > >>> >> account for what it takes to call invokeAsync, not
> > for
> > >>>>> the
> > >>>>> > > > result
> > >>>>> > > > > to
> > >>>>> > > > > > >>> >> complete (with a result or error from the future
> > >>>>> object).
> > >>>>> > > > > > >>> >>
> > >>>>> > > > > > >>> >> There are appropriate places which can handle the
> > async
> > >>>>> > > > callbacks,
> > >>>>> > > > > > >>> but they
> > >>>>> > > > > > >>> >> are in other locations. Will you be able to
> support
> > >>>>> those
> > >>>>> > as
> > >>>>> > > > > well?
> > >>>>> > > > > > >>> >>
> > >>>>> > > > > > >>> >> Thanks,
> > >>>>> > > > > > >>> >> Alan
> > >>>>> > > > > > >>> >>
> > >>>>> > > > > > >>> >> On Wed, Jul 9, 2025 at 7:52 PM Shengkai Fang <
> > >>>>> > > fskmine@gmail.com
> > >>>>> > > > >
> > >>>>> > > > > > >>> wrote:
> > >>>>> > > > > > >>> >>
> > >>>>> > > > > > >>> >> > I just have some questions:
> > >>>>> > > > > > >>> >> >
> > >>>>> > > > > > >>> >> > 1. The current metrics hierarchy shows that the
> > UDF
> > >>>>> metric
> > >>>>> > > > group
> > >>>>> > > > > > >>> belongs
> > >>>>> > > > > > >>> >> to
> > >>>>> > > > > > >>> >> > the TaskMetricGroup. I think it would be better
> > for
> > >>>>> the
> > >>>>> > UDF
> > >>>>> > > > > metric
> > >>>>> > > > > > >>> group
> > >>>>> > > > > > >>> >> to
> > >>>>> > > > > > >>> >> > belong to the OperatorMetricGroup instead,
> > because a
> > >>>>> UDF
> > >>>>> > > might
> > >>>>> > > > > be
> > >>>>> > > > > > >>> used by
> > >>>>> > > > > > >>> >> > multiple operators.
> > >>>>> > > > > > >>> >> > 2. What are the naming conventions for UDF
> > metrics?
> > >>>>> Could
> > >>>>> > > you
> > >>>>> > > > > > >>> provide an
> > >>>>> > > > > > >>> >> > example? Do the metric name contains the UDF
> name?
> > >>>>> > > > > > >>> >> > 3. Why is the UDFExceptionCount metric
> introduced?
> > >>>>> If a
> > >>>>> > UDF
> > >>>>> > > > > throws
> > >>>>> > > > > > >>> an
> > >>>>> > > > > > >>> >> > exception, the job fails immediately. Why do we
> > need
> > >>>>> to
> > >>>>> > > track
> > >>>>> > > > > this
> > >>>>> > > > > > >>> value?
> > >>>>> > > > > > >>> >> >
> > >>>>> > > > > > >>> >> > Best
> > >>>>> > > > > > >>> >> > Shengkai
> > >>>>> > > > > > >>> >> >
> > >>>>> > > > > > >>> >> >
> > >>>>> > > > > > >>> >> > Weiqing Yang <ya...@gmail.com>
> > 于2025年7月9日周三
> > >>>>> > > 12:59写道：
> > >>>>> > > > > > >>> >> >
> > >>>>> > > > > > >>> >> > > Hi all,
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> > > I’d like to initiate a discussion about adding
> > UDF
> > >>>>> > > metrics.
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> > > *Motivation*
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> > > User-defined functions (UDFs) are essential
> for
> > >>>>> custom
> > >>>>> > > logic
> > >>>>> > > > > in
> > >>>>> > > > > > >>> Flink
> > >>>>> > > > > > >>> >> > jobs
> > >>>>> > > > > > >>> >> > > but often act as black boxes, making debugging
> > and
> > >>>>> > > > performance
> > >>>>> > > > > > >>> tuning
> > >>>>> > > > > > >>> >> > > difficult. When issues like high latency or
> > >>>>> frequent
> > >>>>> > > > > exceptions
> > >>>>> > > > > > >>> occur,
> > >>>>> > > > > > >>> >> > it's
> > >>>>> > > > > > >>> >> > > hard to pinpoint the root cause inside UDFs.
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> > > Flink currently lacks built-in metrics for key
> > UDF
> > >>>>> > aspects
> > >>>>> > > > > such
> > >>>>> > > > > > as
> > >>>>> > > > > > >>> >> > > per-record processing time or exception count.
> > This
> > >>>>> > limits
> > >>>>> > > > > > >>> >> observability
> > >>>>> > > > > > >>> >> > > and complicates:
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> > > - Debugging production issues
> > >>>>> > > > > > >>> >> > > - Performance tuning and resource
> allocation
> > >>>>> > > > > > >>> >> > > - Supplying reliable signals to autoscaling
> > >>>>> systems
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> > > Introducing standard, opt-in UDF metrics will
> > >>>>> improve
> > >>>>> > > > platform
> > >>>>> > > > > > >>> >> > > observability and overall health.
> > >>>>> > > > > > >>> >> > > Here’s the proposal document: Link
> > >>>>> > > > > > >>> >> > > <
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> >
> > >>>>> > > > > > >>> >>
> > >>>>> > > > > > >>>
> > >>>>> > > > > >
> > >>>>> > > > >
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> >
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1
> > >>>>> > > > > > >>> >> > > >
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> > > Your feedback and ideas are welcome to refine
> > this
> > >>>>> > > feature.
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> > > Thanks,
> > >>>>> > > > > > >>> >> > > Weiqing
> > >>>>> > > > > > >>> >> > >
> > >>>>> > > > > > >>> >> >
> > >>>>> > > > > > >>> >>
> > >>>>> > > > > > >>>
> > >>>>> > > > > > >>
> > >>>>> > > > > >
> > >>>>> > > > >
> > >>>>> > > >
> > >>>>> > >
> > >>>>> >
> > >>>>>
> > >>>>
> >
>
Re: Re: Re: [DISCUSS] Add UDF Metrics
Posted by Yuepeng Pan <pa...@apache.org>.
LGTM +1 on the design draft.
Best Regards,
Yuepeng Pan
Xuyang <xy...@163.com> 于2026年7月24日周五 10:04写道：
> Hi, Weiqing.
> Thanks for driving this FLIP forward again, +1
>
>
>
>
>
> --
>
> Best！
> Xuyang
>
>
>
> 在 2026-07-17 14:21:21，"Weiqing Yang" <ya...@gmail.com> 写道：
> >Hi all,
> >
> >Gentle bump on this.
> >
> >The proposal is unchanged at a high level. It's been re-aligned with the
> >current master, and the earlier overhead concern is addressed (details in
> >my previous message).
> >
> >Shengkai, Alan, Xuyang, Zakelly — thank you for the thorough review last
> >time. Yuepeng and Shengkai, thank you for your votes on the earlier round.
> >Since you already know this proposal well, I'd really appreciate a quick
> >check that the points you raised still sit right in the current version.
> >
> >If there are no other concerns over the next few days, I'll start a fresh
> >[VOTE] thread so the votes reflect the current design.
> >
> >FLIP doc:
> >
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit
> >
> >Best,
> >Weiqing
> >
> >On Thu, Jul 9, 2026 at 7:20 PM Weiqing Yang <ya...@gmail.com>
> >wrote:
> >
> >> Hi all,
> >>
> >> I'd like to revive this discussion. This FLIP was originally proposed
> and
> >> went to a vote last year (it received two +1s) [1]. Since the original
> >> proposal and vote were about a year ago, I want to revisit the
> discussion
> >> before starting a fresh vote to ensure the design remains fully aligned
> >> with the current master branch.
> >>
> >> At a high level, the proposal is unchanged from the previous round. The
> >> one substantive refinement since the earlier votes is the sampling
> approach
> >> detailed below, which directly addresses the performance overhead
> question
> >> Zakelly raised (the last open item on the thread).
> >>
> >> Overhead (Zakelly's question): Processing time is no longer measured on
> >> every invocation. It now uses the same counter-based sampling as Flink's
> >> state latency tracking (FLINK-21736). A new
> >> `table.exec.udf-metric.sample-interval` option (default 100) means only
> >> every Nth invocation is timed. The non-sampled fast path is a single
> >> integer increment, and `udfProcessingTime` is now a Histogram
> >> (p50/p75/p95/p99) backed by a bounded 128-entry circular buffer.
> Combined
> >> with the existing `table.exec.udf-metric-enabled` gate (off by default,
> >> meaning nothing is registered when disabled), this provides two robust
> >> layers of protection against performance overhead.
> >>
> >> - Proposal doc (updated): [2]
> >>
> >> - cwiki FLIP-485: [3]
> >>
> >> - Draft PR: [4] (implements the proposal; kept in draft until we
> converge)
> >>
> >> I welcome your renewed feedback and any questions. If there are no
> further
> >> concerns, I'll start a fresh [VOTE] thread on the updated proposal so
> the
> >> final votes reflect the current design.
> >>
> >> Thanks,
> >>
> >> Weiqing
> >>
> >> [1] Previous vote thread:
> >> https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
> >>
> >> [2]
> >>
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit
> >>
> >> [3]
> >>
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics
> >>
> >> [4] https://github.com/apache/flink/pull/28692
> >>
> >>
> >> On Wed, Jun 3, 2026 at 4:59 PM Weiqing Yang <ya...@gmail.com>
> >> wrote:
> >>
> >>> Hi Zakelly,
> >>>
> >>> Kindly pinging here to see if you had any remaining concerns regarding
> >>> the FLIP.
> >>>
> >>> If there are no further questions or concerns from anyone, I plan to
> >>> close this discussion thread and proceed with the vote thread.
> >>>
> >>> Thanks,
> >>> Weiqing
> >>>
> >>> On Tue, Mar 3, 2026 at 2:58 PM Weiqing Yang <ya...@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi Zakelly,
> >>>>
> >>>>
> >>>> Thanks for the feedback and sorry for the late response - I am now
> >>>> picking it back up.
> >>>>
> >>>> You raised a great point about the performance overhead, referencing
> >>>> FLINK-16444 <https://issues.apache.org/jira/browse/FLINK-16444>. I've
> >>>> updated the FLIP to adopt the same counter-based sampling approach
> used by
> >>>> Flink's state latency tracking (FLINK-21736
> >>>> <https://issues.apache.org/jira/browse/FLINK-21736>). Specifically:
> >>>>
> >>>> 1. New config: table.exec.udf-metric.sample-interval (default: 100
> >>>> [1]) - only every Nth invocation is measured
> >>>> 2. Fast path: Non-sampled invocations are a single integer
> increment -
> >>>> negligible overhead
> >>>> 3. Sampled path: System.nanoTime() around the UDF call, stored in a
> >>>> DescriptiveStatisticsHistogram with a bounded 128-entry circular
> buffer [2]
> >>>> 4. Metric type change: udfProcessingTime is now a Histogram (reports
> >>>> p50/p75/p95/p99/mean/min/max) instead of the original Gauge
> >>>> 5. Exception counting: Not sampled, since exceptions are rare events
> >>>> and counting each one has negligible cost
> >>>>
> >>>> Combined with the existing feature gate (table.exec.udf-metric-enabled
> >>>> defaulting to false), users have two layers of protection: the
> feature is
> >>>> off by default, and when enabled, sampling keeps overhead minimal.
> >>>> The updated FLIP is here: link
> >>>> <
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1
> >
> >>>>
> >>>> Would this address your concern? If so, it would be great to have your
> >>>> vote on the vote thread [3].
> >>>>
> >>>> [1] 100: state.latency-track.sample-interval default value
> >>>>
> >>>> [2] 128: state.latency-track.history-size default value (line 55),
> which
> >>>> is the circular buffer size for the DescriptiveStatisticsHistogram
> >>>> [3] https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
> >>>>
> >>>> Thanks,
> >>>> Weiqing
> >>>>
> >>>> On Thu, Aug 21, 2025 at 12:24 AM Zakelly Lan <za...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hi Weiqing,
> >>>>>
> >>>>> Sorry for the late reply. And I have one question:
> >>>>>
> >>>>> I'm wondering whether the UDF processing time is measured for every
> >>>>> individual UDF invocation, with the average then reported, or if
> >>>>> sampling
> >>>>> is used instead? I'm concerned about the potential overhead if we
> >>>>> measure
> >>>>> every single invocation. We've encountered similar performance issues
> >>>>> when
> >>>>> implementing state latency tracking [1].
> >>>>>
> >>>>>
> >>>>> [1] https://issues.apache.org/jira/browse/FLINK-16444
> >>>>>
> >>>>> Best,
> >>>>> Zakelly
> >>>>>
> >>>>> On Fri, Aug 15, 2025 at 5:04 AM Weiqing Yang <
> yangweiqing001@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>> > Cool - I’ll proceed to start the VOTE.
> >>>>> > Thanks!
> >>>>> >
> >>>>> > Weiqing
> >>>>> >
> >>>>> > On Thu, Aug 14, 2025 at 12:53 AM Shengkai Fang <fs...@gmail.com>
> >>>>> wrote:
> >>>>> >
> >>>>> > > I don't have any more comments.
> >>>>> > >
> >>>>> > > Best,
> >>>>> > > Shengkai
> >>>>> > >
> >>>>> > > Weiqing Yang <ya...@gmail.com> 于2025年8月14日周四 14:47写道：
> >>>>> > >
> >>>>> > > > Thanks, Shengkai. I’ve updated the proposal doc with the
> >>>>> recommended
> >>>>> > > > configuration name. Please let me know if you have any
> additional
> >>>>> > > feedback.
> >>>>> > > >
> >>>>> > > > Best,
> >>>>> > > > Weiqing
> >>>>> > > >
> >>>>> > > > On Wed, Aug 13, 2025 at 6:58 PM Shengkai Fang <
> fskmine@gmail.com>
> >>>>> > wrote:
> >>>>> > > >
> >>>>> > > > > Sorry for the late response. I prefer to use
> >>>>> > > > > `table.exec.udf-metric-enabled` as the option name.
> >>>>> > > > >
> >>>>> > > > > Best,
> >>>>> > > > > Shengkai
> >>>>> > > > >
> >>>>> > > > > Weiqing Yang <ya...@gmail.com> 于2025年8月13日周三
> 23:54写道：
> >>>>> > > > >
> >>>>> > > > > > Hi Shengkai, Alan, Xuyang, and all,
> >>>>> > > > > >
> >>>>> > > > > > Since there have been no further objections, I’ll proceed
> to
> >>>>> start
> >>>>> > > the
> >>>>> > > > > VOTE
> >>>>> > > > > > on this proposal shortly.
> >>>>> > > > > >
> >>>>> > > > > > Thanks,
> >>>>> > > > > > Weiqing
> >>>>> > > > > >
> >>>>> > > > > > On Thu, Jul 31, 2025 at 10:26 PM Weiqing Yang <
> >>>>> > > > yangweiqing001@gmail.com>
> >>>>> > > > > > wrote:
> >>>>> > > > > >
> >>>>> > > > > > > Hi Shengkai, Alan and Xuyang,
> >>>>> > > > > > >
> >>>>> > > > > > > Just checking in - do you have any concerns or feedback?
> >>>>> > > > > > >
> >>>>> > > > > > > If there are no further objections from anyone, I’ll mark
> >>>>> the
> >>>>> > FLIP
> >>>>> > > as
> >>>>> > > > > > > ready for voting.
> >>>>> > > > > > >
> >>>>> > > > > > >
> >>>>> > > > > > > Best,
> >>>>> > > > > > > Weiqing
> >>>>> > > > > > >
> >>>>> > > > > > >
> >>>>> > > > > > > On Mon, Jul 14, 2025 at 9:10 PM Weiqing Yang <
> >>>>> > > > yangweiqing001@gmail.com
> >>>>> > > > > >
> >>>>> > > > > > > wrote:
> >>>>> > > > > > >
> >>>>> > > > > > >> Hi Xuyang,
> >>>>> > > > > > >>
> >>>>> > > > > > >> Thank you for reviewing the proposal!
> >>>>> > > > > > >>
> >>>>> > > > > > >> I’m planning to use: *udf.metrics.process-time* and
> >>>>> > > > > > >> *udf.metrics.exception-count*. These follow the naming
> >>>>> > convention
> >>>>> > > > used
> >>>>> > > > > > >> in Flink (e.g., RocksDB native metrics
> >>>>> > > > > > >> <
> >>>>> > > > > >
> >>>>> > > > >
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics
> >>>>> > > > > > >).
> >>>>> > > > > > >> I’ve added these names to the proposal doc.
> >>>>> > > > > > >>
> >>>>> > > > > > >> Alternatively, I also considered:
> >>>>> > > *metrics.udf.process-time.enabled*
> >>>>> > > > > and
> >>>>> > > > > > >> *metrics.udf.exception-count.enabled. *
> >>>>> > > > > > >>
> >>>>> > > > > > >> Happy to hear any feedback on which style might be more
> >>>>> > > appropriate.
> >>>>> > > > > > >>
> >>>>> > > > > > >>
> >>>>> > > > > > >> Best,
> >>>>> > > > > > >> Weiqing
> >>>>> > > > > > >>
> >>>>> > > > > > >> On Mon, Jul 14, 2025 at 2:55 AM Xuyang <
> xyzhong131@163.com
> >>>>> >
> >>>>> > > wrote:
> >>>>> > > > > > >>
> >>>>> > > > > > >>> Hi, Weiqing.
> >>>>> > > > > > >>>
> >>>>> > > > > > >>> Thanks for driving to improve this. I just have one
> >>>>> question. I
> >>>>> > > > > notice
> >>>>> > > > > > a
> >>>>> > > > > > >>> new configuration is introduced in this flip. I just
> >>>>> wonder
> >>>>> > what
> >>>>> > > > the
> >>>>> > > > > > >>> configuration name is. Could you please include the
> full
> >>>>> name
> >>>>> > of
> >>>>> > > > this
> >>>>> > > > > > >>> configuration? (just similar to the other names in
> >>>>> > > MetricOptions?)
> >>>>> > > > > > >>>
> >>>>> > > > > > >>>
> >>>>> > > > > > >>>
> >>>>> > > > > > >>>
> >>>>> > > > > > >>> --
> >>>>> > > > > > >>>
> >>>>> > > > > > >>> Best！
> >>>>> > > > > > >>> Xuyang
> >>>>> > > > > > >>>
> >>>>> > > > > > >>>
> >>>>> > > > > > >>>
> >>>>> > > > > > >>>
> >>>>> > > > > > >>>
> >>>>> > > > > > >>> 在 2025-07-13 12:03:59，"Weiqing Yang" <
> >>>>> yangweiqing001@gmail.com
> >>>>> > >
> >>>>> > > > 写道：
> >>>>> > > > > > >>> >Hi Alan,
> >>>>> > > > > > >>> >
> >>>>> > > > > > >>> >Thanks for reviewing the proposal and for highlighting
> >>>>> the
> >>>>> > > > > ASYNC_TABLE
> >>>>> > > > > > >>> work.
> >>>>> > > > > > >>> >
> >>>>> > > > > > >>> >Yes, I’ve updated the proposal to cover both
> >>>>> ASYNC_SCALAR and
> >>>>> > > > > > >>> ASYNC_TABLE.
> >>>>> > > > > > >>> >For async UDFs, the plan is to instrument both the
> >>>>> > invokeAsync()
> >>>>> > > > > call
> >>>>> > > > > > >>> and
> >>>>> > > > > > >>> >the async callback handler to measure the full
> end-to-end
> >>>>> > > latency
> >>>>> > > > > > until
> >>>>> > > > > > >>> the
> >>>>> > > > > > >>> >result or error is returned from the future.
> >>>>> > > > > > >>> >
> >>>>> > > > > > >>> >Let me know if you have any further questions or
> >>>>> suggestions.
> >>>>> > > > > > >>> >
> >>>>> > > > > > >>> >Best,
> >>>>> > > > > > >>> >Weiqing
> >>>>> > > > > > >>> >
> >>>>> > > > > > >>> >On Thu, Jul 10, 2025 at 4:15 PM Alan Sheinberg
> >>>>> > > > > > >>> ><as...@confluent.io.invalid> wrote:
> >>>>> > > > > > >>> >
> >>>>> > > > > > >>> >> Hi Weiqing,
> >>>>> > > > > > >>> >>
> >>>>> > > > > > >>> >> From your doc, the entrypoint for UDF calls in the
> >>>>> codegen
> >>>>> > is
> >>>>> > > > > > >>> >> ExprCodeGenerator which should invoke
> >>>>> > > > BridgingSqlFunctionCallGen,
> >>>>> > > > > > >>> which
> >>>>> > > > > > >>> >> could be instrumented with metrics. This works well
> >>>>> for
> >>>>> > > > > synchronous
> >>>>> > > > > > >>> calls,
> >>>>> > > > > > >>> >> but what about ASYNC_SCALAR and the soon to be
> merged
> >>>>> > > > ASYNC_TABLE
> >>>>> > > > > (
> >>>>> > > > > > >>> >> https://github.com/apache/flink/pull/26567)?
> Timing
> >>>>> > metrics
> >>>>> > > > > would
> >>>>> > > > > > >>> only
> >>>>> > > > > > >>> >> account for what it takes to call invokeAsync, not
> for
> >>>>> the
> >>>>> > > > result
> >>>>> > > > > to
> >>>>> > > > > > >>> >> complete (with a result or error from the future
> >>>>> object).
> >>>>> > > > > > >>> >>
> >>>>> > > > > > >>> >> There are appropriate places which can handle the
> async
> >>>>> > > > callbacks,
> >>>>> > > > > > >>> but they
> >>>>> > > > > > >>> >> are in other locations. Will you be able to support
> >>>>> those
> >>>>> > as
> >>>>> > > > > well?
> >>>>> > > > > > >>> >>
> >>>>> > > > > > >>> >> Thanks,
> >>>>> > > > > > >>> >> Alan
> >>>>> > > > > > >>> >>
> >>>>> > > > > > >>> >> On Wed, Jul 9, 2025 at 7:52 PM Shengkai Fang <
> >>>>> > > fskmine@gmail.com
> >>>>> > > > >
> >>>>> > > > > > >>> wrote:
> >>>>> > > > > > >>> >>
> >>>>> > > > > > >>> >> > I just have some questions:
> >>>>> > > > > > >>> >> >
> >>>>> > > > > > >>> >> > 1. The current metrics hierarchy shows that the
> UDF
> >>>>> metric
> >>>>> > > > group
> >>>>> > > > > > >>> belongs
> >>>>> > > > > > >>> >> to
> >>>>> > > > > > >>> >> > the TaskMetricGroup. I think it would be better
> for
> >>>>> the
> >>>>> > UDF
> >>>>> > > > > metric
> >>>>> > > > > > >>> group
> >>>>> > > > > > >>> >> to
> >>>>> > > > > > >>> >> > belong to the OperatorMetricGroup instead,
> because a
> >>>>> UDF
> >>>>> > > might
> >>>>> > > > > be
> >>>>> > > > > > >>> used by
> >>>>> > > > > > >>> >> > multiple operators.
> >>>>> > > > > > >>> >> > 2. What are the naming conventions for UDF
> metrics?
> >>>>> Could
> >>>>> > > you
> >>>>> > > > > > >>> provide an
> >>>>> > > > > > >>> >> > example? Do the metric name contains the UDF name?
> >>>>> > > > > > >>> >> > 3. Why is the UDFExceptionCount metric introduced?
> >>>>> If a
> >>>>> > UDF
> >>>>> > > > > throws
> >>>>> > > > > > >>> an
> >>>>> > > > > > >>> >> > exception, the job fails immediately. Why do we
> need
> >>>>> to
> >>>>> > > track
> >>>>> > > > > this
> >>>>> > > > > > >>> value?
> >>>>> > > > > > >>> >> >
> >>>>> > > > > > >>> >> > Best
> >>>>> > > > > > >>> >> > Shengkai
> >>>>> > > > > > >>> >> >
> >>>>> > > > > > >>> >> >
> >>>>> > > > > > >>> >> > Weiqing Yang <ya...@gmail.com>
> 于2025年7月9日周三
> >>>>> > > 12:59写道：
> >>>>> > > > > > >>> >> >
> >>>>> > > > > > >>> >> > > Hi all,
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> > > I’d like to initiate a discussion about adding
> UDF
> >>>>> > > metrics.
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> > > *Motivation*
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> > > User-defined functions (UDFs) are essential for
> >>>>> custom
> >>>>> > > logic
> >>>>> > > > > in
> >>>>> > > > > > >>> Flink
> >>>>> > > > > > >>> >> > jobs
> >>>>> > > > > > >>> >> > > but often act as black boxes, making debugging
> and
> >>>>> > > > performance
> >>>>> > > > > > >>> tuning
> >>>>> > > > > > >>> >> > > difficult. When issues like high latency or
> >>>>> frequent
> >>>>> > > > > exceptions
> >>>>> > > > > > >>> occur,
> >>>>> > > > > > >>> >> > it's
> >>>>> > > > > > >>> >> > > hard to pinpoint the root cause inside UDFs.
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> > > Flink currently lacks built-in metrics for key
> UDF
> >>>>> > aspects
> >>>>> > > > > such
> >>>>> > > > > > as
> >>>>> > > > > > >>> >> > > per-record processing time or exception count.
> This
> >>>>> > limits
> >>>>> > > > > > >>> >> observability
> >>>>> > > > > > >>> >> > > and complicates:
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> > > - Debugging production issues
> >>>>> > > > > > >>> >> > > - Performance tuning and resource allocation
> >>>>> > > > > > >>> >> > > - Supplying reliable signals to autoscaling
> >>>>> systems
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> > > Introducing standard, opt-in UDF metrics will
> >>>>> improve
> >>>>> > > > platform
> >>>>> > > > > > >>> >> > > observability and overall health.
> >>>>> > > > > > >>> >> > > Here’s the proposal document: Link
> >>>>> > > > > > >>> >> > > <
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> >
> >>>>> > > > > > >>> >>
> >>>>> > > > > > >>>
> >>>>> > > > > >
> >>>>> > > > >
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1
> >>>>> > > > > > >>> >> > > >
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> > > Your feedback and ideas are welcome to refine
> this
> >>>>> > > feature.
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> > > Thanks,
> >>>>> > > > > > >>> >> > > Weiqing
> >>>>> > > > > > >>> >> > >
> >>>>> > > > > > >>> >> >
> >>>>> > > > > > >>> >>
> >>>>> > > > > > >>>
> >>>>> > > > > > >>
> >>>>> > > > > >
> >>>>> > > > >
> >>>>> > > >
> >>>>> > >
> >>>>> >
> >>>>>
> >>>>
>
Re:Re: Re: [DISCUSS] Add UDF Metrics
Posted by Xuyang <xy...@163.com>.
Hi, Weiqing.
Thanks for driving this FLIP forward again, +1
--
Best！
Xuyang
在 2026-07-17 14:21:21，"Weiqing Yang" <ya...@gmail.com> 写道：
>Hi all,
>
>Gentle bump on this.
>
>The proposal is unchanged at a high level. It's been re-aligned with the
>current master, and the earlier overhead concern is addressed (details in
>my previous message).
>
>Shengkai, Alan, Xuyang, Zakelly — thank you for the thorough review last
>time. Yuepeng and Shengkai, thank you for your votes on the earlier round.
>Since you already know this proposal well, I'd really appreciate a quick
>check that the points you raised still sit right in the current version.
>
>If there are no other concerns over the next few days, I'll start a fresh
>[VOTE] thread so the votes reflect the current design.
>
>FLIP doc:
>https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit
>
>Best,
>Weiqing
>
>On Thu, Jul 9, 2026 at 7:20 PM Weiqing Yang <ya...@gmail.com>
>wrote:
>
>> Hi all,
>>
>> I'd like to revive this discussion. This FLIP was originally proposed and
>> went to a vote last year (it received two +1s) [1]. Since the original
>> proposal and vote were about a year ago, I want to revisit the discussion
>> before starting a fresh vote to ensure the design remains fully aligned
>> with the current master branch.
>>
>> At a high level, the proposal is unchanged from the previous round. The
>> one substantive refinement since the earlier votes is the sampling approach
>> detailed below, which directly addresses the performance overhead question
>> Zakelly raised (the last open item on the thread).
>>
>> Overhead (Zakelly's question): Processing time is no longer measured on
>> every invocation. It now uses the same counter-based sampling as Flink's
>> state latency tracking (FLINK-21736). A new
>> `table.exec.udf-metric.sample-interval` option (default 100) means only
>> every Nth invocation is timed. The non-sampled fast path is a single
>> integer increment, and `udfProcessingTime` is now a Histogram
>> (p50/p75/p95/p99) backed by a bounded 128-entry circular buffer. Combined
>> with the existing `table.exec.udf-metric-enabled` gate (off by default,
>> meaning nothing is registered when disabled), this provides two robust
>> layers of protection against performance overhead.
>>
>> - Proposal doc (updated): [2]
>>
>> - cwiki FLIP-485: [3]
>>
>> - Draft PR: [4] (implements the proposal; kept in draft until we converge)
>>
>> I welcome your renewed feedback and any questions. If there are no further
>> concerns, I'll start a fresh [VOTE] thread on the updated proposal so the
>> final votes reflect the current design.
>>
>> Thanks,
>>
>> Weiqing
>>
>> [1] Previous vote thread:
>> https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
>>
>> [2]
>> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit
>>
>> [3]
>> https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics
>>
>> [4] https://github.com/apache/flink/pull/28692
>>
>>
>> On Wed, Jun 3, 2026 at 4:59 PM Weiqing Yang <ya...@gmail.com>
>> wrote:
>>
>>> Hi Zakelly,
>>>
>>> Kindly pinging here to see if you had any remaining concerns regarding
>>> the FLIP.
>>>
>>> If there are no further questions or concerns from anyone, I plan to
>>> close this discussion thread and proceed with the vote thread.
>>>
>>> Thanks,
>>> Weiqing
>>>
>>> On Tue, Mar 3, 2026 at 2:58 PM Weiqing Yang <ya...@gmail.com>
>>> wrote:
>>>
>>>> Hi Zakelly,
>>>>
>>>>
>>>> Thanks for the feedback and sorry for the late response - I am now
>>>> picking it back up.
>>>>
>>>> You raised a great point about the performance overhead, referencing
>>>> FLINK-16444 <https://issues.apache.org/jira/browse/FLINK-16444>. I've
>>>> updated the FLIP to adopt the same counter-based sampling approach used by
>>>> Flink's state latency tracking (FLINK-21736
>>>> <https://issues.apache.org/jira/browse/FLINK-21736>). Specifically:
>>>>
>>>> 1. New config: table.exec.udf-metric.sample-interval (default: 100
>>>> [1]) - only every Nth invocation is measured
>>>> 2. Fast path: Non-sampled invocations are a single integer increment -
>>>> negligible overhead
>>>> 3. Sampled path: System.nanoTime() around the UDF call, stored in a
>>>> DescriptiveStatisticsHistogram with a bounded 128-entry circular buffer [2]
>>>> 4. Metric type change: udfProcessingTime is now a Histogram (reports
>>>> p50/p75/p95/p99/mean/min/max) instead of the original Gauge
>>>> 5. Exception counting: Not sampled, since exceptions are rare events
>>>> and counting each one has negligible cost
>>>>
>>>> Combined with the existing feature gate (table.exec.udf-metric-enabled
>>>> defaulting to false), users have two layers of protection: the feature is
>>>> off by default, and when enabled, sampling keeps overhead minimal.
>>>> The updated FLIP is here: link
>>>> <https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1>
>>>>
>>>> Would this address your concern? If so, it would be great to have your
>>>> vote on the vote thread [3].
>>>>
>>>> [1] 100: state.latency-track.sample-interval default value
>>>>
>>>> [2] 128: state.latency-track.history-size default value (line 55), which
>>>> is the circular buffer size for the DescriptiveStatisticsHistogram
>>>> [3] https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
>>>>
>>>> Thanks,
>>>> Weiqing
>>>>
>>>> On Thu, Aug 21, 2025 at 12:24 AM Zakelly Lan <za...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Weiqing,
>>>>>
>>>>> Sorry for the late reply. And I have one question:
>>>>>
>>>>> I'm wondering whether the UDF processing time is measured for every
>>>>> individual UDF invocation, with the average then reported, or if
>>>>> sampling
>>>>> is used instead? I'm concerned about the potential overhead if we
>>>>> measure
>>>>> every single invocation. We've encountered similar performance issues
>>>>> when
>>>>> implementing state latency tracking [1].
>>>>>
>>>>>
>>>>> [1] https://issues.apache.org/jira/browse/FLINK-16444
>>>>>
>>>>> Best,
>>>>> Zakelly
>>>>>
>>>>> On Fri, Aug 15, 2025 at 5:04 AM Weiqing Yang <ya...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> > Cool - I’ll proceed to start the VOTE.
>>>>> > Thanks!
>>>>> >
>>>>> > Weiqing
>>>>> >
>>>>> > On Thu, Aug 14, 2025 at 12:53 AM Shengkai Fang <fs...@gmail.com>
>>>>> wrote:
>>>>> >
>>>>> > > I don't have any more comments.
>>>>> > >
>>>>> > > Best,
>>>>> > > Shengkai
>>>>> > >
>>>>> > > Weiqing Yang <ya...@gmail.com> 于2025年8月14日周四 14:47写道：
>>>>> > >
>>>>> > > > Thanks, Shengkai. I’ve updated the proposal doc with the
>>>>> recommended
>>>>> > > > configuration name. Please let me know if you have any additional
>>>>> > > feedback.
>>>>> > > >
>>>>> > > > Best,
>>>>> > > > Weiqing
>>>>> > > >
>>>>> > > > On Wed, Aug 13, 2025 at 6:58 PM Shengkai Fang <fs...@gmail.com>
>>>>> > wrote:
>>>>> > > >
>>>>> > > > > Sorry for the late response. I prefer to use
>>>>> > > > > `table.exec.udf-metric-enabled` as the option name.
>>>>> > > > >
>>>>> > > > > Best,
>>>>> > > > > Shengkai
>>>>> > > > >
>>>>> > > > > Weiqing Yang <ya...@gmail.com> 于2025年8月13日周三 23:54写道：
>>>>> > > > >
>>>>> > > > > > Hi Shengkai, Alan, Xuyang, and all,
>>>>> > > > > >
>>>>> > > > > > Since there have been no further objections, I’ll proceed to
>>>>> start
>>>>> > > the
>>>>> > > > > VOTE
>>>>> > > > > > on this proposal shortly.
>>>>> > > > > >
>>>>> > > > > > Thanks,
>>>>> > > > > > Weiqing
>>>>> > > > > >
>>>>> > > > > > On Thu, Jul 31, 2025 at 10:26 PM Weiqing Yang <
>>>>> > > > yangweiqing001@gmail.com>
>>>>> > > > > > wrote:
>>>>> > > > > >
>>>>> > > > > > > Hi Shengkai, Alan and Xuyang,
>>>>> > > > > > >
>>>>> > > > > > > Just checking in - do you have any concerns or feedback?
>>>>> > > > > > >
>>>>> > > > > > > If there are no further objections from anyone, I’ll mark
>>>>> the
>>>>> > FLIP
>>>>> > > as
>>>>> > > > > > > ready for voting.
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > Best,
>>>>> > > > > > > Weiqing
>>>>> > > > > > >
>>>>> > > > > > >
>>>>> > > > > > > On Mon, Jul 14, 2025 at 9:10 PM Weiqing Yang <
>>>>> > > > yangweiqing001@gmail.com
>>>>> > > > > >
>>>>> > > > > > > wrote:
>>>>> > > > > > >
>>>>> > > > > > >> Hi Xuyang,
>>>>> > > > > > >>
>>>>> > > > > > >> Thank you for reviewing the proposal!
>>>>> > > > > > >>
>>>>> > > > > > >> I’m planning to use: *udf.metrics.process-time* and
>>>>> > > > > > >> *udf.metrics.exception-count*. These follow the naming
>>>>> > convention
>>>>> > > > used
>>>>> > > > > > >> in Flink (e.g., RocksDB native metrics
>>>>> > > > > > >> <
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics
>>>>> > > > > > >).
>>>>> > > > > > >> I’ve added these names to the proposal doc.
>>>>> > > > > > >>
>>>>> > > > > > >> Alternatively, I also considered:
>>>>> > > *metrics.udf.process-time.enabled*
>>>>> > > > > and
>>>>> > > > > > >> *metrics.udf.exception-count.enabled. *
>>>>> > > > > > >>
>>>>> > > > > > >> Happy to hear any feedback on which style might be more
>>>>> > > appropriate.
>>>>> > > > > > >>
>>>>> > > > > > >>
>>>>> > > > > > >> Best,
>>>>> > > > > > >> Weiqing
>>>>> > > > > > >>
>>>>> > > > > > >> On Mon, Jul 14, 2025 at 2:55 AM Xuyang <xyzhong131@163.com
>>>>> >
>>>>> > > wrote:
>>>>> > > > > > >>
>>>>> > > > > > >>> Hi, Weiqing.
>>>>> > > > > > >>>
>>>>> > > > > > >>> Thanks for driving to improve this. I just have one
>>>>> question. I
>>>>> > > > > notice
>>>>> > > > > > a
>>>>> > > > > > >>> new configuration is introduced in this flip. I just
>>>>> wonder
>>>>> > what
>>>>> > > > the
>>>>> > > > > > >>> configuration name is. Could you please include the full
>>>>> name
>>>>> > of
>>>>> > > > this
>>>>> > > > > > >>> configuration? (just similar to the other names in
>>>>> > > MetricOptions?)
>>>>> > > > > > >>>
>>>>> > > > > > >>>
>>>>> > > > > > >>>
>>>>> > > > > > >>>
>>>>> > > > > > >>> --
>>>>> > > > > > >>>
>>>>> > > > > > >>> Best！
>>>>> > > > > > >>> Xuyang
>>>>> > > > > > >>>
>>>>> > > > > > >>>
>>>>> > > > > > >>>
>>>>> > > > > > >>>
>>>>> > > > > > >>>
>>>>> > > > > > >>> 在 2025-07-13 12:03:59，"Weiqing Yang" <
>>>>> yangweiqing001@gmail.com
>>>>> > >
>>>>> > > > 写道：
>>>>> > > > > > >>> >Hi Alan,
>>>>> > > > > > >>> >
>>>>> > > > > > >>> >Thanks for reviewing the proposal and for highlighting
>>>>> the
>>>>> > > > > ASYNC_TABLE
>>>>> > > > > > >>> work.
>>>>> > > > > > >>> >
>>>>> > > > > > >>> >Yes, I’ve updated the proposal to cover both
>>>>> ASYNC_SCALAR and
>>>>> > > > > > >>> ASYNC_TABLE.
>>>>> > > > > > >>> >For async UDFs, the plan is to instrument both the
>>>>> > invokeAsync()
>>>>> > > > > call
>>>>> > > > > > >>> and
>>>>> > > > > > >>> >the async callback handler to measure the full end-to-end
>>>>> > > latency
>>>>> > > > > > until
>>>>> > > > > > >>> the
>>>>> > > > > > >>> >result or error is returned from the future.
>>>>> > > > > > >>> >
>>>>> > > > > > >>> >Let me know if you have any further questions or
>>>>> suggestions.
>>>>> > > > > > >>> >
>>>>> > > > > > >>> >Best,
>>>>> > > > > > >>> >Weiqing
>>>>> > > > > > >>> >
>>>>> > > > > > >>> >On Thu, Jul 10, 2025 at 4:15 PM Alan Sheinberg
>>>>> > > > > > >>> ><as...@confluent.io.invalid> wrote:
>>>>> > > > > > >>> >
>>>>> > > > > > >>> >> Hi Weiqing,
>>>>> > > > > > >>> >>
>>>>> > > > > > >>> >> From your doc, the entrypoint for UDF calls in the
>>>>> codegen
>>>>> > is
>>>>> > > > > > >>> >> ExprCodeGenerator which should invoke
>>>>> > > > BridgingSqlFunctionCallGen,
>>>>> > > > > > >>> which
>>>>> > > > > > >>> >> could be instrumented with metrics. This works well
>>>>> for
>>>>> > > > > synchronous
>>>>> > > > > > >>> calls,
>>>>> > > > > > >>> >> but what about ASYNC_SCALAR and the soon to be merged
>>>>> > > > ASYNC_TABLE
>>>>> > > > > (
>>>>> > > > > > >>> >> https://github.com/apache/flink/pull/26567)? Timing
>>>>> > metrics
>>>>> > > > > would
>>>>> > > > > > >>> only
>>>>> > > > > > >>> >> account for what it takes to call invokeAsync, not for
>>>>> the
>>>>> > > > result
>>>>> > > > > to
>>>>> > > > > > >>> >> complete (with a result or error from the future
>>>>> object).
>>>>> > > > > > >>> >>
>>>>> > > > > > >>> >> There are appropriate places which can handle the async
>>>>> > > > callbacks,
>>>>> > > > > > >>> but they
>>>>> > > > > > >>> >> are in other locations. Will you be able to support
>>>>> those
>>>>> > as
>>>>> > > > > well?
>>>>> > > > > > >>> >>
>>>>> > > > > > >>> >> Thanks,
>>>>> > > > > > >>> >> Alan
>>>>> > > > > > >>> >>
>>>>> > > > > > >>> >> On Wed, Jul 9, 2025 at 7:52 PM Shengkai Fang <
>>>>> > > fskmine@gmail.com
>>>>> > > > >
>>>>> > > > > > >>> wrote:
>>>>> > > > > > >>> >>
>>>>> > > > > > >>> >> > I just have some questions:
>>>>> > > > > > >>> >> >
>>>>> > > > > > >>> >> > 1. The current metrics hierarchy shows that the UDF
>>>>> metric
>>>>> > > > group
>>>>> > > > > > >>> belongs
>>>>> > > > > > >>> >> to
>>>>> > > > > > >>> >> > the TaskMetricGroup. I think it would be better for
>>>>> the
>>>>> > UDF
>>>>> > > > > metric
>>>>> > > > > > >>> group
>>>>> > > > > > >>> >> to
>>>>> > > > > > >>> >> > belong to the OperatorMetricGroup instead, because a
>>>>> UDF
>>>>> > > might
>>>>> > > > > be
>>>>> > > > > > >>> used by
>>>>> > > > > > >>> >> > multiple operators.
>>>>> > > > > > >>> >> > 2. What are the naming conventions for UDF metrics?
>>>>> Could
>>>>> > > you
>>>>> > > > > > >>> provide an
>>>>> > > > > > >>> >> > example? Do the metric name contains the UDF name?
>>>>> > > > > > >>> >> > 3. Why is the UDFExceptionCount metric introduced?
>>>>> If a
>>>>> > UDF
>>>>> > > > > throws
>>>>> > > > > > >>> an
>>>>> > > > > > >>> >> > exception, the job fails immediately. Why do we need
>>>>> to
>>>>> > > track
>>>>> > > > > this
>>>>> > > > > > >>> value?
>>>>> > > > > > >>> >> >
>>>>> > > > > > >>> >> > Best
>>>>> > > > > > >>> >> > Shengkai
>>>>> > > > > > >>> >> >
>>>>> > > > > > >>> >> >
>>>>> > > > > > >>> >> > Weiqing Yang <ya...@gmail.com> 于2025年7月9日周三
>>>>> > > 12:59写道：
>>>>> > > > > > >>> >> >
>>>>> > > > > > >>> >> > > Hi all,
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> > > I’d like to initiate a discussion about adding UDF
>>>>> > > metrics.
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> > > *Motivation*
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> > > User-defined functions (UDFs) are essential for
>>>>> custom
>>>>> > > logic
>>>>> > > > > in
>>>>> > > > > > >>> Flink
>>>>> > > > > > >>> >> > jobs
>>>>> > > > > > >>> >> > > but often act as black boxes, making debugging and
>>>>> > > > performance
>>>>> > > > > > >>> tuning
>>>>> > > > > > >>> >> > > difficult. When issues like high latency or
>>>>> frequent
>>>>> > > > > exceptions
>>>>> > > > > > >>> occur,
>>>>> > > > > > >>> >> > it's
>>>>> > > > > > >>> >> > > hard to pinpoint the root cause inside UDFs.
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> > > Flink currently lacks built-in metrics for key UDF
>>>>> > aspects
>>>>> > > > > such
>>>>> > > > > > as
>>>>> > > > > > >>> >> > > per-record processing time or exception count. This
>>>>> > limits
>>>>> > > > > > >>> >> observability
>>>>> > > > > > >>> >> > > and complicates:
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> > > - Debugging production issues
>>>>> > > > > > >>> >> > > - Performance tuning and resource allocation
>>>>> > > > > > >>> >> > > - Supplying reliable signals to autoscaling
>>>>> systems
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> > > Introducing standard, opt-in UDF metrics will
>>>>> improve
>>>>> > > > platform
>>>>> > > > > > >>> >> > > observability and overall health.
>>>>> > > > > > >>> >> > > Here’s the proposal document: Link
>>>>> > > > > > >>> >> > > <
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> >
>>>>> > > > > > >>> >>
>>>>> > > > > > >>>
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > >
>>>>> >
>>>>> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1
>>>>> > > > > > >>> >> > > >
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> > > Your feedback and ideas are welcome to refine this
>>>>> > > feature.
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> > > Thanks,
>>>>> > > > > > >>> >> > > Weiqing
>>>>> > > > > > >>> >> > >
>>>>> > > > > > >>> >> >
>>>>> > > > > > >>> >>
>>>>> > > > > > >>>
>>>>> > > > > > >>
>>>>> > > > > >
>>>>> > > > >
>>>>> > > >
>>>>> > >
>>>>> >
>>>>>
>>>>
Re: Re: [DISCUSS] Add UDF Metrics
Posted by Weiqing Yang <ya...@gmail.com>.
Hi all,
Gentle bump on this.
The proposal is unchanged at a high level. It's been re-aligned with the
current master, and the earlier overhead concern is addressed (details in
my previous message).
Shengkai, Alan, Xuyang, Zakelly — thank you for the thorough review last
time. Yuepeng and Shengkai, thank you for your votes on the earlier round.
Since you already know this proposal well, I'd really appreciate a quick
check that the points you raised still sit right in the current version.
If there are no other concerns over the next few days, I'll start a fresh
[VOTE] thread so the votes reflect the current design.
FLIP doc:
https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit
Best,
Weiqing
On Thu, Jul 9, 2026 at 7:20 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Hi all,
>
> I'd like to revive this discussion. This FLIP was originally proposed and
> went to a vote last year (it received two +1s) [1]. Since the original
> proposal and vote were about a year ago, I want to revisit the discussion
> before starting a fresh vote to ensure the design remains fully aligned
> with the current master branch.
>
> At a high level, the proposal is unchanged from the previous round. The
> one substantive refinement since the earlier votes is the sampling approach
> detailed below, which directly addresses the performance overhead question
> Zakelly raised (the last open item on the thread).
>
> Overhead (Zakelly's question): Processing time is no longer measured on
> every invocation. It now uses the same counter-based sampling as Flink's
> state latency tracking (FLINK-21736). A new
> `table.exec.udf-metric.sample-interval` option (default 100) means only
> every Nth invocation is timed. The non-sampled fast path is a single
> integer increment, and `udfProcessingTime` is now a Histogram
> (p50/p75/p95/p99) backed by a bounded 128-entry circular buffer. Combined
> with the existing `table.exec.udf-metric-enabled` gate (off by default,
> meaning nothing is registered when disabled), this provides two robust
> layers of protection against performance overhead.
>
> - Proposal doc (updated): [2]
>
> - cwiki FLIP-485: [3]
>
> - Draft PR: [4] (implements the proposal; kept in draft until we converge)
>
> I welcome your renewed feedback and any questions. If there are no further
> concerns, I'll start a fresh [VOTE] thread on the updated proposal so the
> final votes reflect the current design.
>
> Thanks,
>
> Weiqing
>
> [1] Previous vote thread:
> https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
>
> [2]
> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit
>
> [3]
> https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics
>
> [4] https://github.com/apache/flink/pull/28692
>
>
> On Wed, Jun 3, 2026 at 4:59 PM Weiqing Yang <ya...@gmail.com>
> wrote:
>
>> Hi Zakelly,
>>
>> Kindly pinging here to see if you had any remaining concerns regarding
>> the FLIP.
>>
>> If there are no further questions or concerns from anyone, I plan to
>> close this discussion thread and proceed with the vote thread.
>>
>> Thanks,
>> Weiqing
>>
>> On Tue, Mar 3, 2026 at 2:58 PM Weiqing Yang <ya...@gmail.com>
>> wrote:
>>
>>> Hi Zakelly,
>>>
>>>
>>> Thanks for the feedback and sorry for the late response - I am now
>>> picking it back up.
>>>
>>> You raised a great point about the performance overhead, referencing
>>> FLINK-16444 <https://issues.apache.org/jira/browse/FLINK-16444>. I've
>>> updated the FLIP to adopt the same counter-based sampling approach used by
>>> Flink's state latency tracking (FLINK-21736
>>> <https://issues.apache.org/jira/browse/FLINK-21736>). Specifically:
>>>
>>> 1. New config: table.exec.udf-metric.sample-interval (default: 100
>>> [1]) - only every Nth invocation is measured
>>> 2. Fast path: Non-sampled invocations are a single integer increment -
>>> negligible overhead
>>> 3. Sampled path: System.nanoTime() around the UDF call, stored in a
>>> DescriptiveStatisticsHistogram with a bounded 128-entry circular buffer [2]
>>> 4. Metric type change: udfProcessingTime is now a Histogram (reports
>>> p50/p75/p95/p99/mean/min/max) instead of the original Gauge
>>> 5. Exception counting: Not sampled, since exceptions are rare events
>>> and counting each one has negligible cost
>>>
>>> Combined with the existing feature gate (table.exec.udf-metric-enabled
>>> defaulting to false), users have two layers of protection: the feature is
>>> off by default, and when enabled, sampling keeps overhead minimal.
>>> The updated FLIP is here: link
>>> <https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1>
>>>
>>> Would this address your concern? If so, it would be great to have your
>>> vote on the vote thread [3].
>>>
>>> [1] 100: state.latency-track.sample-interval default value
>>>
>>> [2] 128: state.latency-track.history-size default value (line 55), which
>>> is the circular buffer size for the DescriptiveStatisticsHistogram
>>> [3] https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
>>>
>>> Thanks,
>>> Weiqing
>>>
>>> On Thu, Aug 21, 2025 at 12:24 AM Zakelly Lan <za...@gmail.com>
>>> wrote:
>>>
>>>> Hi Weiqing,
>>>>
>>>> Sorry for the late reply. And I have one question:
>>>>
>>>> I'm wondering whether the UDF processing time is measured for every
>>>> individual UDF invocation, with the average then reported, or if
>>>> sampling
>>>> is used instead? I'm concerned about the potential overhead if we
>>>> measure
>>>> every single invocation. We've encountered similar performance issues
>>>> when
>>>> implementing state latency tracking [1].
>>>>
>>>>
>>>> [1] https://issues.apache.org/jira/browse/FLINK-16444
>>>>
>>>> Best,
>>>> Zakelly
>>>>
>>>> On Fri, Aug 15, 2025 at 5:04 AM Weiqing Yang <ya...@gmail.com>
>>>> wrote:
>>>>
>>>> > Cool - I’ll proceed to start the VOTE.
>>>> > Thanks!
>>>> >
>>>> > Weiqing
>>>> >
>>>> > On Thu, Aug 14, 2025 at 12:53 AM Shengkai Fang <fs...@gmail.com>
>>>> wrote:
>>>> >
>>>> > > I don't have any more comments.
>>>> > >
>>>> > > Best,
>>>> > > Shengkai
>>>> > >
>>>> > > Weiqing Yang <ya...@gmail.com> 于2025年8月14日周四 14:47写道：
>>>> > >
>>>> > > > Thanks, Shengkai. I’ve updated the proposal doc with the
>>>> recommended
>>>> > > > configuration name. Please let me know if you have any additional
>>>> > > feedback.
>>>> > > >
>>>> > > > Best,
>>>> > > > Weiqing
>>>> > > >
>>>> > > > On Wed, Aug 13, 2025 at 6:58 PM Shengkai Fang <fs...@gmail.com>
>>>> > wrote:
>>>> > > >
>>>> > > > > Sorry for the late response. I prefer to use
>>>> > > > > `table.exec.udf-metric-enabled` as the option name.
>>>> > > > >
>>>> > > > > Best,
>>>> > > > > Shengkai
>>>> > > > >
>>>> > > > > Weiqing Yang <ya...@gmail.com> 于2025年8月13日周三 23:54写道：
>>>> > > > >
>>>> > > > > > Hi Shengkai, Alan, Xuyang, and all,
>>>> > > > > >
>>>> > > > > > Since there have been no further objections, I’ll proceed to
>>>> start
>>>> > > the
>>>> > > > > VOTE
>>>> > > > > > on this proposal shortly.
>>>> > > > > >
>>>> > > > > > Thanks,
>>>> > > > > > Weiqing
>>>> > > > > >
>>>> > > > > > On Thu, Jul 31, 2025 at 10:26 PM Weiqing Yang <
>>>> > > > yangweiqing001@gmail.com>
>>>> > > > > > wrote:
>>>> > > > > >
>>>> > > > > > > Hi Shengkai, Alan and Xuyang,
>>>> > > > > > >
>>>> > > > > > > Just checking in - do you have any concerns or feedback?
>>>> > > > > > >
>>>> > > > > > > If there are no further objections from anyone, I’ll mark
>>>> the
>>>> > FLIP
>>>> > > as
>>>> > > > > > > ready for voting.
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > Best,
>>>> > > > > > > Weiqing
>>>> > > > > > >
>>>> > > > > > >
>>>> > > > > > > On Mon, Jul 14, 2025 at 9:10 PM Weiqing Yang <
>>>> > > > yangweiqing001@gmail.com
>>>> > > > > >
>>>> > > > > > > wrote:
>>>> > > > > > >
>>>> > > > > > >> Hi Xuyang,
>>>> > > > > > >>
>>>> > > > > > >> Thank you for reviewing the proposal!
>>>> > > > > > >>
>>>> > > > > > >> I’m planning to use: *udf.metrics.process-time* and
>>>> > > > > > >> *udf.metrics.exception-count*. These follow the naming
>>>> > convention
>>>> > > > used
>>>> > > > > > >> in Flink (e.g., RocksDB native metrics
>>>> > > > > > >> <
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics
>>>> > > > > > >).
>>>> > > > > > >> I’ve added these names to the proposal doc.
>>>> > > > > > >>
>>>> > > > > > >> Alternatively, I also considered:
>>>> > > *metrics.udf.process-time.enabled*
>>>> > > > > and
>>>> > > > > > >> *metrics.udf.exception-count.enabled. *
>>>> > > > > > >>
>>>> > > > > > >> Happy to hear any feedback on which style might be more
>>>> > > appropriate.
>>>> > > > > > >>
>>>> > > > > > >>
>>>> > > > > > >> Best,
>>>> > > > > > >> Weiqing
>>>> > > > > > >>
>>>> > > > > > >> On Mon, Jul 14, 2025 at 2:55 AM Xuyang <xyzhong131@163.com
>>>> >
>>>> > > wrote:
>>>> > > > > > >>
>>>> > > > > > >>> Hi, Weiqing.
>>>> > > > > > >>>
>>>> > > > > > >>> Thanks for driving to improve this. I just have one
>>>> question. I
>>>> > > > > notice
>>>> > > > > > a
>>>> > > > > > >>> new configuration is introduced in this flip. I just
>>>> wonder
>>>> > what
>>>> > > > the
>>>> > > > > > >>> configuration name is. Could you please include the full
>>>> name
>>>> > of
>>>> > > > this
>>>> > > > > > >>> configuration? (just similar to the other names in
>>>> > > MetricOptions?)
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>> --
>>>> > > > > > >>>
>>>> > > > > > >>> Best！
>>>> > > > > > >>> Xuyang
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>>
>>>> > > > > > >>> 在 2025-07-13 12:03:59，"Weiqing Yang" <
>>>> yangweiqing001@gmail.com
>>>> > >
>>>> > > > 写道：
>>>> > > > > > >>> >Hi Alan,
>>>> > > > > > >>> >
>>>> > > > > > >>> >Thanks for reviewing the proposal and for highlighting
>>>> the
>>>> > > > > ASYNC_TABLE
>>>> > > > > > >>> work.
>>>> > > > > > >>> >
>>>> > > > > > >>> >Yes, I’ve updated the proposal to cover both
>>>> ASYNC_SCALAR and
>>>> > > > > > >>> ASYNC_TABLE.
>>>> > > > > > >>> >For async UDFs, the plan is to instrument both the
>>>> > invokeAsync()
>>>> > > > > call
>>>> > > > > > >>> and
>>>> > > > > > >>> >the async callback handler to measure the full end-to-end
>>>> > > latency
>>>> > > > > > until
>>>> > > > > > >>> the
>>>> > > > > > >>> >result or error is returned from the future.
>>>> > > > > > >>> >
>>>> > > > > > >>> >Let me know if you have any further questions or
>>>> suggestions.
>>>> > > > > > >>> >
>>>> > > > > > >>> >Best,
>>>> > > > > > >>> >Weiqing
>>>> > > > > > >>> >
>>>> > > > > > >>> >On Thu, Jul 10, 2025 at 4:15 PM Alan Sheinberg
>>>> > > > > > >>> ><as...@confluent.io.invalid> wrote:
>>>> > > > > > >>> >
>>>> > > > > > >>> >> Hi Weiqing,
>>>> > > > > > >>> >>
>>>> > > > > > >>> >> From your doc, the entrypoint for UDF calls in the
>>>> codegen
>>>> > is
>>>> > > > > > >>> >> ExprCodeGenerator which should invoke
>>>> > > > BridgingSqlFunctionCallGen,
>>>> > > > > > >>> which
>>>> > > > > > >>> >> could be instrumented with metrics. This works well
>>>> for
>>>> > > > > synchronous
>>>> > > > > > >>> calls,
>>>> > > > > > >>> >> but what about ASYNC_SCALAR and the soon to be merged
>>>> > > > ASYNC_TABLE
>>>> > > > > (
>>>> > > > > > >>> >> https://github.com/apache/flink/pull/26567)? Timing
>>>> > metrics
>>>> > > > > would
>>>> > > > > > >>> only
>>>> > > > > > >>> >> account for what it takes to call invokeAsync, not for
>>>> the
>>>> > > > result
>>>> > > > > to
>>>> > > > > > >>> >> complete (with a result or error from the future
>>>> object).
>>>> > > > > > >>> >>
>>>> > > > > > >>> >> There are appropriate places which can handle the async
>>>> > > > callbacks,
>>>> > > > > > >>> but they
>>>> > > > > > >>> >> are in other locations. Will you be able to support
>>>> those
>>>> > as
>>>> > > > > well?
>>>> > > > > > >>> >>
>>>> > > > > > >>> >> Thanks,
>>>> > > > > > >>> >> Alan
>>>> > > > > > >>> >>
>>>> > > > > > >>> >> On Wed, Jul 9, 2025 at 7:52 PM Shengkai Fang <
>>>> > > fskmine@gmail.com
>>>> > > > >
>>>> > > > > > >>> wrote:
>>>> > > > > > >>> >>
>>>> > > > > > >>> >> > I just have some questions:
>>>> > > > > > >>> >> >
>>>> > > > > > >>> >> > 1. The current metrics hierarchy shows that the UDF
>>>> metric
>>>> > > > group
>>>> > > > > > >>> belongs
>>>> > > > > > >>> >> to
>>>> > > > > > >>> >> > the TaskMetricGroup. I think it would be better for
>>>> the
>>>> > UDF
>>>> > > > > metric
>>>> > > > > > >>> group
>>>> > > > > > >>> >> to
>>>> > > > > > >>> >> > belong to the OperatorMetricGroup instead, because a
>>>> UDF
>>>> > > might
>>>> > > > > be
>>>> > > > > > >>> used by
>>>> > > > > > >>> >> > multiple operators.
>>>> > > > > > >>> >> > 2. What are the naming conventions for UDF metrics?
>>>> Could
>>>> > > you
>>>> > > > > > >>> provide an
>>>> > > > > > >>> >> > example? Do the metric name contains the UDF name?
>>>> > > > > > >>> >> > 3. Why is the UDFExceptionCount metric introduced?
>>>> If a
>>>> > UDF
>>>> > > > > throws
>>>> > > > > > >>> an
>>>> > > > > > >>> >> > exception, the job fails immediately. Why do we need
>>>> to
>>>> > > track
>>>> > > > > this
>>>> > > > > > >>> value?
>>>> > > > > > >>> >> >
>>>> > > > > > >>> >> > Best
>>>> > > > > > >>> >> > Shengkai
>>>> > > > > > >>> >> >
>>>> > > > > > >>> >> >
>>>> > > > > > >>> >> > Weiqing Yang <ya...@gmail.com> 于2025年7月9日周三
>>>> > > 12:59写道：
>>>> > > > > > >>> >> >
>>>> > > > > > >>> >> > > Hi all,
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> > > I’d like to initiate a discussion about adding UDF
>>>> > > metrics.
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> > > *Motivation*
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> > > User-defined functions (UDFs) are essential for
>>>> custom
>>>> > > logic
>>>> > > > > in
>>>> > > > > > >>> Flink
>>>> > > > > > >>> >> > jobs
>>>> > > > > > >>> >> > > but often act as black boxes, making debugging and
>>>> > > > performance
>>>> > > > > > >>> tuning
>>>> > > > > > >>> >> > > difficult. When issues like high latency or
>>>> frequent
>>>> > > > > exceptions
>>>> > > > > > >>> occur,
>>>> > > > > > >>> >> > it's
>>>> > > > > > >>> >> > > hard to pinpoint the root cause inside UDFs.
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> > > Flink currently lacks built-in metrics for key UDF
>>>> > aspects
>>>> > > > > such
>>>> > > > > > as
>>>> > > > > > >>> >> > > per-record processing time or exception count. This
>>>> > limits
>>>> > > > > > >>> >> observability
>>>> > > > > > >>> >> > > and complicates:
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> > > - Debugging production issues
>>>> > > > > > >>> >> > > - Performance tuning and resource allocation
>>>> > > > > > >>> >> > > - Supplying reliable signals to autoscaling
>>>> systems
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> > > Introducing standard, opt-in UDF metrics will
>>>> improve
>>>> > > > platform
>>>> > > > > > >>> >> > > observability and overall health.
>>>> > > > > > >>> >> > > Here’s the proposal document: Link
>>>> > > > > > >>> >> > > <
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> >
>>>> > > > > > >>> >>
>>>> > > > > > >>>
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1
>>>> > > > > > >>> >> > > >
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> > > Your feedback and ideas are welcome to refine this
>>>> > > feature.
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> > > Thanks,
>>>> > > > > > >>> >> > > Weiqing
>>>> > > > > > >>> >> > >
>>>> > > > > > >>> >> >
>>>> > > > > > >>> >>
>>>> > > > > > >>>
>>>> > > > > > >>
>>>> > > > > >
>>>> > > > >
>>>> > > >
>>>> > >
>>>> >
>>>>
>>>
Re: Re: [DISCUSS] Add UDF Metrics
Posted by Weiqing Yang <ya...@gmail.com>.
Hi all,
I'd like to revive this discussion. This FLIP was originally proposed and
went to a vote last year (it received two +1s) [1]. Since the original
proposal and vote were about a year ago, I want to revisit the discussion
before starting a fresh vote to ensure the design remains fully aligned
with the current master branch.
At a high level, the proposal is unchanged from the previous round. The one
substantive refinement since the earlier votes is the sampling approach
detailed below, which directly addresses the performance overhead question
Zakelly raised (the last open item on the thread).
Overhead (Zakelly's question): Processing time is no longer measured on
every invocation. It now uses the same counter-based sampling as Flink's
state latency tracking (FLINK-21736). A new
`table.exec.udf-metric.sample-interval` option (default 100) means only
every Nth invocation is timed. The non-sampled fast path is a single
integer increment, and `udfProcessingTime` is now a Histogram
(p50/p75/p95/p99) backed by a bounded 128-entry circular buffer. Combined
with the existing `table.exec.udf-metric-enabled` gate (off by default,
meaning nothing is registered when disabled), this provides two robust
layers of protection against performance overhead.
- Proposal doc (updated): [2]
- cwiki FLIP-485: [3]
- Draft PR: [4] (implements the proposal; kept in draft until we converge)
I welcome your renewed feedback and any questions. If there are no further
concerns, I'll start a fresh [VOTE] thread on the updated proposal so the
final votes reflect the current design.
Thanks,
Weiqing
[1] Previous vote thread:
https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
[2]
https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit
[3]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/373885706/FLIP-485+Add+UDF+Metrics
[4] https://github.com/apache/flink/pull/28692
On Wed, Jun 3, 2026 at 4:59 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Hi Zakelly,
>
> Kindly pinging here to see if you had any remaining concerns regarding the
> FLIP.
>
> If there are no further questions or concerns from anyone, I plan to close
> this discussion thread and proceed with the vote thread.
>
> Thanks,
> Weiqing
>
> On Tue, Mar 3, 2026 at 2:58 PM Weiqing Yang <ya...@gmail.com>
> wrote:
>
>> Hi Zakelly,
>>
>>
>> Thanks for the feedback and sorry for the late response - I am now
>> picking it back up.
>>
>> You raised a great point about the performance overhead, referencing
>> FLINK-16444 <https://issues.apache.org/jira/browse/FLINK-16444>. I've
>> updated the FLIP to adopt the same counter-based sampling approach used by
>> Flink's state latency tracking (FLINK-21736
>> <https://issues.apache.org/jira/browse/FLINK-21736>). Specifically:
>>
>> 1. New config: table.exec.udf-metric.sample-interval (default: 100 [1])
>> - only every Nth invocation is measured
>> 2. Fast path: Non-sampled invocations are a single integer increment -
>> negligible overhead
>> 3. Sampled path: System.nanoTime() around the UDF call, stored in a
>> DescriptiveStatisticsHistogram with a bounded 128-entry circular buffer [2]
>> 4. Metric type change: udfProcessingTime is now a Histogram (reports
>> p50/p75/p95/p99/mean/min/max) instead of the original Gauge
>> 5. Exception counting: Not sampled, since exceptions are rare events
>> and counting each one has negligible cost
>>
>> Combined with the existing feature gate (table.exec.udf-metric-enabled
>> defaulting to false), users have two layers of protection: the feature is
>> off by default, and when enabled, sampling keeps overhead minimal.
>> The updated FLIP is here: link
>> <https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1>
>>
>> Would this address your concern? If so, it would be great to have your
>> vote on the vote thread [3].
>>
>> [1] 100: state.latency-track.sample-interval default value
>>
>> [2] 128: state.latency-track.history-size default value (line 55), which
>> is the circular buffer size for the DescriptiveStatisticsHistogram
>> [3] https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
>>
>> Thanks,
>> Weiqing
>>
>> On Thu, Aug 21, 2025 at 12:24 AM Zakelly Lan <za...@gmail.com>
>> wrote:
>>
>>> Hi Weiqing,
>>>
>>> Sorry for the late reply. And I have one question:
>>>
>>> I'm wondering whether the UDF processing time is measured for every
>>> individual UDF invocation, with the average then reported, or if sampling
>>> is used instead? I'm concerned about the potential overhead if we measure
>>> every single invocation. We've encountered similar performance issues
>>> when
>>> implementing state latency tracking [1].
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-16444
>>>
>>> Best,
>>> Zakelly
>>>
>>> On Fri, Aug 15, 2025 at 5:04 AM Weiqing Yang <ya...@gmail.com>
>>> wrote:
>>>
>>> > Cool - I’ll proceed to start the VOTE.
>>> > Thanks!
>>> >
>>> > Weiqing
>>> >
>>> > On Thu, Aug 14, 2025 at 12:53 AM Shengkai Fang <fs...@gmail.com>
>>> wrote:
>>> >
>>> > > I don't have any more comments.
>>> > >
>>> > > Best,
>>> > > Shengkai
>>> > >
>>> > > Weiqing Yang <ya...@gmail.com> 于2025年8月14日周四 14:47写道：
>>> > >
>>> > > > Thanks, Shengkai. I’ve updated the proposal doc with the
>>> recommended
>>> > > > configuration name. Please let me know if you have any additional
>>> > > feedback.
>>> > > >
>>> > > > Best,
>>> > > > Weiqing
>>> > > >
>>> > > > On Wed, Aug 13, 2025 at 6:58 PM Shengkai Fang <fs...@gmail.com>
>>> > wrote:
>>> > > >
>>> > > > > Sorry for the late response. I prefer to use
>>> > > > > `table.exec.udf-metric-enabled` as the option name.
>>> > > > >
>>> > > > > Best,
>>> > > > > Shengkai
>>> > > > >
>>> > > > > Weiqing Yang <ya...@gmail.com> 于2025年8月13日周三 23:54写道：
>>> > > > >
>>> > > > > > Hi Shengkai, Alan, Xuyang, and all,
>>> > > > > >
>>> > > > > > Since there have been no further objections, I’ll proceed to
>>> start
>>> > > the
>>> > > > > VOTE
>>> > > > > > on this proposal shortly.
>>> > > > > >
>>> > > > > > Thanks,
>>> > > > > > Weiqing
>>> > > > > >
>>> > > > > > On Thu, Jul 31, 2025 at 10:26 PM Weiqing Yang <
>>> > > > yangweiqing001@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Hi Shengkai, Alan and Xuyang,
>>> > > > > > >
>>> > > > > > > Just checking in - do you have any concerns or feedback?
>>> > > > > > >
>>> > > > > > > If there are no further objections from anyone, I’ll mark the
>>> > FLIP
>>> > > as
>>> > > > > > > ready for voting.
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > Best,
>>> > > > > > > Weiqing
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Mon, Jul 14, 2025 at 9:10 PM Weiqing Yang <
>>> > > > yangweiqing001@gmail.com
>>> > > > > >
>>> > > > > > > wrote:
>>> > > > > > >
>>> > > > > > >> Hi Xuyang,
>>> > > > > > >>
>>> > > > > > >> Thank you for reviewing the proposal!
>>> > > > > > >>
>>> > > > > > >> I’m planning to use: *udf.metrics.process-time* and
>>> > > > > > >> *udf.metrics.exception-count*. These follow the naming
>>> > convention
>>> > > > used
>>> > > > > > >> in Flink (e.g., RocksDB native metrics
>>> > > > > > >> <
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics
>>> > > > > > >).
>>> > > > > > >> I’ve added these names to the proposal doc.
>>> > > > > > >>
>>> > > > > > >> Alternatively, I also considered:
>>> > > *metrics.udf.process-time.enabled*
>>> > > > > and
>>> > > > > > >> *metrics.udf.exception-count.enabled. *
>>> > > > > > >>
>>> > > > > > >> Happy to hear any feedback on which style might be more
>>> > > appropriate.
>>> > > > > > >>
>>> > > > > > >>
>>> > > > > > >> Best,
>>> > > > > > >> Weiqing
>>> > > > > > >>
>>> > > > > > >> On Mon, Jul 14, 2025 at 2:55 AM Xuyang <xy...@163.com>
>>> > > wrote:
>>> > > > > > >>
>>> > > > > > >>> Hi, Weiqing.
>>> > > > > > >>>
>>> > > > > > >>> Thanks for driving to improve this. I just have one
>>> question. I
>>> > > > > notice
>>> > > > > > a
>>> > > > > > >>> new configuration is introduced in this flip. I just wonder
>>> > what
>>> > > > the
>>> > > > > > >>> configuration name is. Could you please include the full
>>> name
>>> > of
>>> > > > this
>>> > > > > > >>> configuration? (just similar to the other names in
>>> > > MetricOptions?)
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>> --
>>> > > > > > >>>
>>> > > > > > >>> Best！
>>> > > > > > >>> Xuyang
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>>
>>> > > > > > >>> 在 2025-07-13 12:03:59，"Weiqing Yang" <
>>> yangweiqing001@gmail.com
>>> > >
>>> > > > 写道：
>>> > > > > > >>> >Hi Alan,
>>> > > > > > >>> >
>>> > > > > > >>> >Thanks for reviewing the proposal and for highlighting the
>>> > > > > ASYNC_TABLE
>>> > > > > > >>> work.
>>> > > > > > >>> >
>>> > > > > > >>> >Yes, I’ve updated the proposal to cover both ASYNC_SCALAR
>>> and
>>> > > > > > >>> ASYNC_TABLE.
>>> > > > > > >>> >For async UDFs, the plan is to instrument both the
>>> > invokeAsync()
>>> > > > > call
>>> > > > > > >>> and
>>> > > > > > >>> >the async callback handler to measure the full end-to-end
>>> > > latency
>>> > > > > > until
>>> > > > > > >>> the
>>> > > > > > >>> >result or error is returned from the future.
>>> > > > > > >>> >
>>> > > > > > >>> >Let me know if you have any further questions or
>>> suggestions.
>>> > > > > > >>> >
>>> > > > > > >>> >Best,
>>> > > > > > >>> >Weiqing
>>> > > > > > >>> >
>>> > > > > > >>> >On Thu, Jul 10, 2025 at 4:15 PM Alan Sheinberg
>>> > > > > > >>> ><as...@confluent.io.invalid> wrote:
>>> > > > > > >>> >
>>> > > > > > >>> >> Hi Weiqing,
>>> > > > > > >>> >>
>>> > > > > > >>> >> From your doc, the entrypoint for UDF calls in the
>>> codegen
>>> > is
>>> > > > > > >>> >> ExprCodeGenerator which should invoke
>>> > > > BridgingSqlFunctionCallGen,
>>> > > > > > >>> which
>>> > > > > > >>> >> could be instrumented with metrics. This works well for
>>> > > > > synchronous
>>> > > > > > >>> calls,
>>> > > > > > >>> >> but what about ASYNC_SCALAR and the soon to be merged
>>> > > > ASYNC_TABLE
>>> > > > > (
>>> > > > > > >>> >> https://github.com/apache/flink/pull/26567)? Timing
>>> > metrics
>>> > > > > would
>>> > > > > > >>> only
>>> > > > > > >>> >> account for what it takes to call invokeAsync, not for
>>> the
>>> > > > result
>>> > > > > to
>>> > > > > > >>> >> complete (with a result or error from the future
>>> object).
>>> > > > > > >>> >>
>>> > > > > > >>> >> There are appropriate places which can handle the async
>>> > > > callbacks,
>>> > > > > > >>> but they
>>> > > > > > >>> >> are in other locations. Will you be able to support
>>> those
>>> > as
>>> > > > > well?
>>> > > > > > >>> >>
>>> > > > > > >>> >> Thanks,
>>> > > > > > >>> >> Alan
>>> > > > > > >>> >>
>>> > > > > > >>> >> On Wed, Jul 9, 2025 at 7:52 PM Shengkai Fang <
>>> > > fskmine@gmail.com
>>> > > > >
>>> > > > > > >>> wrote:
>>> > > > > > >>> >>
>>> > > > > > >>> >> > I just have some questions:
>>> > > > > > >>> >> >
>>> > > > > > >>> >> > 1. The current metrics hierarchy shows that the UDF
>>> metric
>>> > > > group
>>> > > > > > >>> belongs
>>> > > > > > >>> >> to
>>> > > > > > >>> >> > the TaskMetricGroup. I think it would be better for
>>> the
>>> > UDF
>>> > > > > metric
>>> > > > > > >>> group
>>> > > > > > >>> >> to
>>> > > > > > >>> >> > belong to the OperatorMetricGroup instead, because a
>>> UDF
>>> > > might
>>> > > > > be
>>> > > > > > >>> used by
>>> > > > > > >>> >> > multiple operators.
>>> > > > > > >>> >> > 2. What are the naming conventions for UDF metrics?
>>> Could
>>> > > you
>>> > > > > > >>> provide an
>>> > > > > > >>> >> > example? Do the metric name contains the UDF name?
>>> > > > > > >>> >> > 3. Why is the UDFExceptionCount metric introduced? If
>>> a
>>> > UDF
>>> > > > > throws
>>> > > > > > >>> an
>>> > > > > > >>> >> > exception, the job fails immediately. Why do we need
>>> to
>>> > > track
>>> > > > > this
>>> > > > > > >>> value?
>>> > > > > > >>> >> >
>>> > > > > > >>> >> > Best
>>> > > > > > >>> >> > Shengkai
>>> > > > > > >>> >> >
>>> > > > > > >>> >> >
>>> > > > > > >>> >> > Weiqing Yang <ya...@gmail.com> 于2025年7月9日周三
>>> > > 12:59写道：
>>> > > > > > >>> >> >
>>> > > > > > >>> >> > > Hi all,
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> > > I’d like to initiate a discussion about adding UDF
>>> > > metrics.
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> > > *Motivation*
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> > > User-defined functions (UDFs) are essential for
>>> custom
>>> > > logic
>>> > > > > in
>>> > > > > > >>> Flink
>>> > > > > > >>> >> > jobs
>>> > > > > > >>> >> > > but often act as black boxes, making debugging and
>>> > > > performance
>>> > > > > > >>> tuning
>>> > > > > > >>> >> > > difficult. When issues like high latency or frequent
>>> > > > > exceptions
>>> > > > > > >>> occur,
>>> > > > > > >>> >> > it's
>>> > > > > > >>> >> > > hard to pinpoint the root cause inside UDFs.
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> > > Flink currently lacks built-in metrics for key UDF
>>> > aspects
>>> > > > > such
>>> > > > > > as
>>> > > > > > >>> >> > > per-record processing time or exception count. This
>>> > limits
>>> > > > > > >>> >> observability
>>> > > > > > >>> >> > > and complicates:
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> > > - Debugging production issues
>>> > > > > > >>> >> > > - Performance tuning and resource allocation
>>> > > > > > >>> >> > > - Supplying reliable signals to autoscaling
>>> systems
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> > > Introducing standard, opt-in UDF metrics will
>>> improve
>>> > > > platform
>>> > > > > > >>> >> > > observability and overall health.
>>> > > > > > >>> >> > > Here’s the proposal document: Link
>>> > > > > > >>> >> > > <
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> >
>>> > > > > > >>> >>
>>> > > > > > >>>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1
>>> > > > > > >>> >> > > >
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> > > Your feedback and ideas are welcome to refine this
>>> > > feature.
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> > > Thanks,
>>> > > > > > >>> >> > > Weiqing
>>> > > > > > >>> >> > >
>>> > > > > > >>> >> >
>>> > > > > > >>> >>
>>> > > > > > >>>
>>> > > > > > >>
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>
Re: Re: [DISCUSS] Add UDF Metrics
Posted by Weiqing Yang <ya...@gmail.com>.
Hi Zakelly,
Kindly pinging here to see if you had any remaining concerns regarding the
FLIP.
If there are no further questions or concerns from anyone, I plan to close
this discussion thread and proceed with the vote thread.
Thanks,
Weiqing
On Tue, Mar 3, 2026 at 2:58 PM Weiqing Yang <ya...@gmail.com>
wrote:
> Hi Zakelly,
>
>
> Thanks for the feedback and sorry for the late response - I am now picking
> it back up.
>
> You raised a great point about the performance overhead, referencing
> FLINK-16444 <https://issues.apache.org/jira/browse/FLINK-16444>. I've
> updated the FLIP to adopt the same counter-based sampling approach used by
> Flink's state latency tracking (FLINK-21736
> <https://issues.apache.org/jira/browse/FLINK-21736>). Specifically:
>
> 1. New config: table.exec.udf-metric.sample-interval (default: 100 [1])
> - only every Nth invocation is measured
> 2. Fast path: Non-sampled invocations are a single integer increment -
> negligible overhead
> 3. Sampled path: System.nanoTime() around the UDF call, stored in a
> DescriptiveStatisticsHistogram with a bounded 128-entry circular buffer [2]
> 4. Metric type change: udfProcessingTime is now a Histogram (reports
> p50/p75/p95/p99/mean/min/max) instead of the original Gauge
> 5. Exception counting: Not sampled, since exceptions are rare events and
> counting each one has negligible cost
>
> Combined with the existing feature gate (table.exec.udf-metric-enabled
> defaulting to false), users have two layers of protection: the feature is
> off by default, and when enabled, sampling keeps overhead minimal.
> The updated FLIP is here: link
> <https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1>
>
> Would this address your concern? If so, it would be great to have your
> vote on the vote thread [3].
>
> [1] 100: state.latency-track.sample-interval default value
>
> [2] 128: state.latency-track.history-size default value (line 55), which
> is the circular buffer size for the DescriptiveStatisticsHistogram
> [3] https://lists.apache.org/thread/d0sv36839p5h03t3okv89pco2jy6vbg3
>
> Thanks,
> Weiqing
>
> On Thu, Aug 21, 2025 at 12:24 AM Zakelly Lan <za...@gmail.com>
> wrote:
>
>> Hi Weiqing,
>>
>> Sorry for the late reply. And I have one question:
>>
>> I'm wondering whether the UDF processing time is measured for every
>> individual UDF invocation, with the average then reported, or if sampling
>> is used instead? I'm concerned about the potential overhead if we measure
>> every single invocation. We've encountered similar performance issues when
>> implementing state latency tracking [1].
>>
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16444
>>
>> Best,
>> Zakelly
>>
>> On Fri, Aug 15, 2025 at 5:04 AM Weiqing Yang <ya...@gmail.com>
>> wrote:
>>
>> > Cool - I’ll proceed to start the VOTE.
>> > Thanks!
>> >
>> > Weiqing
>> >
>> > On Thu, Aug 14, 2025 at 12:53 AM Shengkai Fang <fs...@gmail.com>
>> wrote:
>> >
>> > > I don't have any more comments.
>> > >
>> > > Best,
>> > > Shengkai
>> > >
>> > > Weiqing Yang <ya...@gmail.com> 于2025年8月14日周四 14:47写道：
>> > >
>> > > > Thanks, Shengkai. I’ve updated the proposal doc with the recommended
>> > > > configuration name. Please let me know if you have any additional
>> > > feedback.
>> > > >
>> > > > Best,
>> > > > Weiqing
>> > > >
>> > > > On Wed, Aug 13, 2025 at 6:58 PM Shengkai Fang <fs...@gmail.com>
>> > wrote:
>> > > >
>> > > > > Sorry for the late response. I prefer to use
>> > > > > `table.exec.udf-metric-enabled` as the option name.
>> > > > >
>> > > > > Best,
>> > > > > Shengkai
>> > > > >
>> > > > > Weiqing Yang <ya...@gmail.com> 于2025年8月13日周三 23:54写道：
>> > > > >
>> > > > > > Hi Shengkai, Alan, Xuyang, and all,
>> > > > > >
>> > > > > > Since there have been no further objections, I’ll proceed to
>> start
>> > > the
>> > > > > VOTE
>> > > > > > on this proposal shortly.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Weiqing
>> > > > > >
>> > > > > > On Thu, Jul 31, 2025 at 10:26 PM Weiqing Yang <
>> > > > yangweiqing001@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Shengkai, Alan and Xuyang,
>> > > > > > >
>> > > > > > > Just checking in - do you have any concerns or feedback?
>> > > > > > >
>> > > > > > > If there are no further objections from anyone, I’ll mark the
>> > FLIP
>> > > as
>> > > > > > > ready for voting.
>> > > > > > >
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Weiqing
>> > > > > > >
>> > > > > > >
>> > > > > > > On Mon, Jul 14, 2025 at 9:10 PM Weiqing Yang <
>> > > > yangweiqing001@gmail.com
>> > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >> Hi Xuyang,
>> > > > > > >>
>> > > > > > >> Thank you for reviewing the proposal!
>> > > > > > >>
>> > > > > > >> I’m planning to use: *udf.metrics.process-time* and
>> > > > > > >> *udf.metrics.exception-count*. These follow the naming
>> > convention
>> > > > used
>> > > > > > >> in Flink (e.g., RocksDB native metrics
>> > > > > > >> <
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#rocksdb-native-metrics
>> > > > > > >).
>> > > > > > >> I’ve added these names to the proposal doc.
>> > > > > > >>
>> > > > > > >> Alternatively, I also considered:
>> > > *metrics.udf.process-time.enabled*
>> > > > > and
>> > > > > > >> *metrics.udf.exception-count.enabled. *
>> > > > > > >>
>> > > > > > >> Happy to hear any feedback on which style might be more
>> > > appropriate.
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> Best,
>> > > > > > >> Weiqing
>> > > > > > >>
>> > > > > > >> On Mon, Jul 14, 2025 at 2:55 AM Xuyang <xy...@163.com>
>> > > wrote:
>> > > > > > >>
>> > > > > > >>> Hi, Weiqing.
>> > > > > > >>>
>> > > > > > >>> Thanks for driving to improve this. I just have one
>> question. I
>> > > > > notice
>> > > > > > a
>> > > > > > >>> new configuration is introduced in this flip. I just wonder
>> > what
>> > > > the
>> > > > > > >>> configuration name is. Could you please include the full
>> name
>> > of
>> > > > this
>> > > > > > >>> configuration? (just similar to the other names in
>> > > MetricOptions?)
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>> --
>> > > > > > >>>
>> > > > > > >>> Best！
>> > > > > > >>> Xuyang
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>>
>> > > > > > >>> 在 2025-07-13 12:03:59，"Weiqing Yang" <
>> yangweiqing001@gmail.com
>> > >
>> > > > 写道：
>> > > > > > >>> >Hi Alan,
>> > > > > > >>> >
>> > > > > > >>> >Thanks for reviewing the proposal and for highlighting the
>> > > > > ASYNC_TABLE
>> > > > > > >>> work.
>> > > > > > >>> >
>> > > > > > >>> >Yes, I’ve updated the proposal to cover both ASYNC_SCALAR
>> and
>> > > > > > >>> ASYNC_TABLE.
>> > > > > > >>> >For async UDFs, the plan is to instrument both the
>> > invokeAsync()
>> > > > > call
>> > > > > > >>> and
>> > > > > > >>> >the async callback handler to measure the full end-to-end
>> > > latency
>> > > > > > until
>> > > > > > >>> the
>> > > > > > >>> >result or error is returned from the future.
>> > > > > > >>> >
>> > > > > > >>> >Let me know if you have any further questions or
>> suggestions.
>> > > > > > >>> >
>> > > > > > >>> >Best,
>> > > > > > >>> >Weiqing
>> > > > > > >>> >
>> > > > > > >>> >On Thu, Jul 10, 2025 at 4:15 PM Alan Sheinberg
>> > > > > > >>> ><as...@confluent.io.invalid> wrote:
>> > > > > > >>> >
>> > > > > > >>> >> Hi Weiqing,
>> > > > > > >>> >>
>> > > > > > >>> >> From your doc, the entrypoint for UDF calls in the
>> codegen
>> > is
>> > > > > > >>> >> ExprCodeGenerator which should invoke
>> > > > BridgingSqlFunctionCallGen,
>> > > > > > >>> which
>> > > > > > >>> >> could be instrumented with metrics. This works well for
>> > > > > synchronous
>> > > > > > >>> calls,
>> > > > > > >>> >> but what about ASYNC_SCALAR and the soon to be merged
>> > > > ASYNC_TABLE
>> > > > > (
>> > > > > > >>> >> https://github.com/apache/flink/pull/26567)? Timing
>> > metrics
>> > > > > would
>> > > > > > >>> only
>> > > > > > >>> >> account for what it takes to call invokeAsync, not for
>> the
>> > > > result
>> > > > > to
>> > > > > > >>> >> complete (with a result or error from the future object).
>> > > > > > >>> >>
>> > > > > > >>> >> There are appropriate places which can handle the async
>> > > > callbacks,
>> > > > > > >>> but they
>> > > > > > >>> >> are in other locations. Will you be able to support
>> those
>> > as
>> > > > > well?
>> > > > > > >>> >>
>> > > > > > >>> >> Thanks,
>> > > > > > >>> >> Alan
>> > > > > > >>> >>
>> > > > > > >>> >> On Wed, Jul 9, 2025 at 7:52 PM Shengkai Fang <
>> > > fskmine@gmail.com
>> > > > >
>> > > > > > >>> wrote:
>> > > > > > >>> >>
>> > > > > > >>> >> > I just have some questions:
>> > > > > > >>> >> >
>> > > > > > >>> >> > 1. The current metrics hierarchy shows that the UDF
>> metric
>> > > > group
>> > > > > > >>> belongs
>> > > > > > >>> >> to
>> > > > > > >>> >> > the TaskMetricGroup. I think it would be better for the
>> > UDF
>> > > > > metric
>> > > > > > >>> group
>> > > > > > >>> >> to
>> > > > > > >>> >> > belong to the OperatorMetricGroup instead, because a
>> UDF
>> > > might
>> > > > > be
>> > > > > > >>> used by
>> > > > > > >>> >> > multiple operators.
>> > > > > > >>> >> > 2. What are the naming conventions for UDF metrics?
>> Could
>> > > you
>> > > > > > >>> provide an
>> > > > > > >>> >> > example? Do the metric name contains the UDF name?
>> > > > > > >>> >> > 3. Why is the UDFExceptionCount metric introduced? If a
>> > UDF
>> > > > > throws
>> > > > > > >>> an
>> > > > > > >>> >> > exception, the job fails immediately. Why do we need to
>> > > track
>> > > > > this
>> > > > > > >>> value?
>> > > > > > >>> >> >
>> > > > > > >>> >> > Best
>> > > > > > >>> >> > Shengkai
>> > > > > > >>> >> >
>> > > > > > >>> >> >
>> > > > > > >>> >> > Weiqing Yang <ya...@gmail.com> 于2025年7月9日周三
>> > > 12:59写道：
>> > > > > > >>> >> >
>> > > > > > >>> >> > > Hi all,
>> > > > > > >>> >> > >
>> > > > > > >>> >> > > I’d like to initiate a discussion about adding UDF
>> > > metrics.
>> > > > > > >>> >> > >
>> > > > > > >>> >> > > *Motivation*
>> > > > > > >>> >> > >
>> > > > > > >>> >> > > User-defined functions (UDFs) are essential for
>> custom
>> > > logic
>> > > > > in
>> > > > > > >>> Flink
>> > > > > > >>> >> > jobs
>> > > > > > >>> >> > > but often act as black boxes, making debugging and
>> > > > performance
>> > > > > > >>> tuning
>> > > > > > >>> >> > > difficult. When issues like high latency or frequent
>> > > > > exceptions
>> > > > > > >>> occur,
>> > > > > > >>> >> > it's
>> > > > > > >>> >> > > hard to pinpoint the root cause inside UDFs.
>> > > > > > >>> >> > >
>> > > > > > >>> >> > > Flink currently lacks built-in metrics for key UDF
>> > aspects
>> > > > > such
>> > > > > > as
>> > > > > > >>> >> > > per-record processing time or exception count. This
>> > limits
>> > > > > > >>> >> observability
>> > > > > > >>> >> > > and complicates:
>> > > > > > >>> >> > >
>> > > > > > >>> >> > > - Debugging production issues
>> > > > > > >>> >> > > - Performance tuning and resource allocation
>> > > > > > >>> >> > > - Supplying reliable signals to autoscaling
>> systems
>> > > > > > >>> >> > >
>> > > > > > >>> >> > > Introducing standard, opt-in UDF metrics will improve
>> > > > platform
>> > > > > > >>> >> > > observability and overall health.
>> > > > > > >>> >> > > Here’s the proposal document: Link
>> > > > > > >>> >> > > <
>> > > > > > >>> >> > >
>> > > > > > >>> >> >
>> > > > > > >>> >>
>> > > > > > >>>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/document/d/1ZTN_kSxTMXKyJcrtmP6I9wlZmfPkK8748_nA6EVuVA0/edit?tab=t.0#heading=h.ljww281maxj1
>> > > > > > >>> >> > > >
>> > > > > > >>> >> > >
>> > > > > > >>> >> > > Your feedback and ideas are welcome to refine this
>> > > feature.
>> > > > > > >>> >> > >
>> > > > > > >>> >> > >
>> > > > > > >>> >> > > Thanks,
>> > > > > > >>> >> > > Weiqing
>> > > > > > >>> >> > >
>> > > > > > >>> >> >
>> > > > > > >>> >>
>> > > > > > >>>
>> > > > > > >>
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>