[jira] [Updated] (FLINK-39059) Add unified metrics support to AsyncPredictFunction and PredictFunction

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-39059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated FLINK-39059:
-----------------------------------
    Labels: pull-request-available  (was: )

> Add unified metrics support to AsyncPredictFunction and PredictFunction
> -----------------------------------------------------------------------
>
>                 Key: FLINK-39059
>                 URL: https://issues.apache.org/jira/browse/FLINK-39059
>             Project: Flink
>          Issue Type: Sub-task
>            Reporter: featzhang
>            Priority: Major
>              Labels: pull-request-available
>
> h3. Subtask: Add Built-in Metrics for Model Inference Functions
> *Description*
> Introduce unified, built-in metrics support for model inference in Flink by enhancing both {{PredictFunction}} and {{{}AsyncPredictFunction{}}}. The goal is to provide consistent observability for inference workloads without requiring changes in individual model implementations.
> *Scope*
>  * Add common metrics instrumentation to the base inference function classes.
>  * Ensure both synchronous and asynchronous inference paths are covered.
>  * Automatically enable metrics for all existing and future model connectors (e.g., OpenAI, Triton).
> *Metrics Included*
>  * {{{}inference_requests{}}}: Total number of inference requests.
>  * {{{}inference_requests_success{}}}: Number of successful inference requests.
>  * {{{}inference_requests_failure{}}}: Number of failed inference requests.
>  * {{{}inference_latency{}}}: Histogram of inference latency in milliseconds.
>  * {{{}inference_rows_output{}}}: Total number of output rows produced by inference.
> *Extensibility*
>  * Provide a {{createLatencyHistogram()}} hook method.
>  * Allow subclasses to customize latency histogram behavior (e.g., bucket configuration).
> *Acceptance Criteria*
>  * Metrics are registered automatically without modifying existing model implementations.
>  * Metrics are exposed consistently for both {{PredictFunction}} and {{{}AsyncPredictFunction{}}}.
>  * No regression in existing inference functionality.
>  * Metrics names and semantics are aligned with Flink metrics conventions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)