[jira] [Commented] (FLINK-39617) Add batch REST endpoints for aggregated subtask metrics across multiple job vertices

["Herbert Wang (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-39617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18078728#comment-18078728 ] 

Herbert Wang commented on FLINK-39617:
--------------------------------------

I am opening this ticket to discuss the API shape before submitting any implementation.

>   Add batch REST endpoints for aggregated subtask metrics across multiple job vertices
> --------------------------------------------------------------------------------------
>
>                 Key: FLINK-39617
>                 URL: https://issues.apache.org/jira/browse/FLINK-39617
>             Project: Flink
>          Issue Type: Improvement
>          Components: Runtime / Metrics, Runtime / REST
>            Reporter: Herbert Wang
>            Priority: Major
>              Labels: Metrics, metrics, rest_api
>
> The JobManager REST API currently exposes aggregated subtask metrics per job vertex via:
> {code}
> GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics
> {code}
> Clients that need the same metric set for many vertices, such as autoscalers or monitoring integrations, must issue one request per vertex for metric-name discovery and another request per vertex for metric values. For jobs with many vertices this creates avoidable REST fan-out, repeated MetricFetcher updates, and large repeated payloads.
> h2. Proposal
> Add two batch JobManager REST endpoints for aggregated subtask metrics across multiple vertices:
> {code}
> POST /jobs/:jobid/vertices/subtasks/metrics/names
> POST /jobs/:jobid/vertices/subtasks/metrics/values
> {code}
> The existing single-vertex endpoint should remain unchanged for compatibility.
> The endpoints are intentionally split rather than using one POST endpoint with mode-switching behavior, so OpenAPI schemas, code generation, and capability detection remain straightforward.
> h3. Name discovery endpoint
> Request:
> {code:json}
> {
>   "vertexIds": ["<jobVertexId>", "<jobVertexId>"],
>   "regex": [".*busyTime.*", ".*numRecords.*"]
> }
> {code}
> Response:
> {code:json}
> [
>   {
>     "vertexId": "<jobVertexId>",
>     "metrics": [{ "id": "busyTimeMsPerSecond" }]
>   }
> ]
> {code}
> h3. Value aggregation endpoint
> Request:
> {code:json}
> {
>   "vertices": [
>     { "vertexId": "<jobVertexId>", "metrics": ["busyTimeMsPerSecond"] },
>     { "vertexId": "<jobVertexId>", "metrics": ["numRecordsInPerSecond"] }
>   ],
>   "agg": ["min", "max", "avg"]
> }
> {code}
> Response:
> {code:json}
> [
>   {
>     "vertexId": "<jobVertexId>",
>     "metrics": [{ "id": "busyTimeMsPerSecond", "min": 0.0, "max": 1.0, "avg": 0.5 }]
>   }
> ]
> {code}
> h2. Compatibility
> This is additive. The existing endpoint remains unchanged:
> {code}
> GET /jobs/:jobid/vertices/:vertexid/subtasks/metrics
> {code}
> Clients can feature-detect the new endpoints and fall back to the existing per-vertex endpoint when unavailable, or we can cherry-pick to earlier 2.x versions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)