You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Admaing (via GitHub)" <gi...@apache.org> on 2026/10/20 02:01:41 UTC
[PR] [FLINK-37751][autoscaler] Add vertex name to the autoscaler scaling metrics [flink-kubernetes-operator]
Admaing opened a new pull request, #1226:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1226
## What is the purpose of the change
The autoscaler exposes a `Current` / `Average` gauge per job vertex for every scaling metric, keyed by the hex `jobVertexID`. The id is not human readable, and metric backends that refuse high cardinality identifiers drop it, which makes the autoscaler metrics hard to read and to filter.
This change carries the job vertex name reported by the JobManager through `VertexInfo` and adds it as a `jobVertexName` scope to the registered scaling metrics:
```
AutoScaler.jobVertexID.<jobVertexID>.jobVertexName.<vertexName>.<ScalingMetric>.Current
```
The names are taken from the `JobDetailsInfo` that `ScalingMetricCollector` already fetches on every cycle, so no additional REST call is introduced. Vertices for which no name is reported keep exactly the previous metric identifiers.
## Brief change log
- `VertexInfo` gained an optional `name`, next to the existing vertex information
- `JobTopology.fromJsonPlan` takes the vertex names and populates them on the parsed vertices
- `ScalingMetricCollector#getJobTopology` builds the name map from the already fetched `JobDetailsInfo`
- `AutoscalerFlinkMetrics#registerScalingMetrics` takes the `JobTopology` instead of the vertex id list and adds the `jobVertexName` scope when a name is available
- `docs/content/docs/operations/metrics.md` documents the additional scope (the `content.zh` mirror is kept in sync, as it is identical to the English page)
## Verifying this change
This change added tests and can be verified as follows:
- `AutoScalerFlinkMetricsTest#testVertexNameAddedToMetricScope` asserts the registered metric identifiers carry the vertex name; it fails without the `jobVertexName` group
- `JobTopologyTest#testTopologyFromJson` asserts the names parsed from a real JSON plan
- `ScalingMetricCollectorTest#testJobTopologyParsingFromJobDetails` asserts the names flow from `JobDetailsInfo` into the topology
- `mvn clean test -pl flink-autoscaler,flink-autoscaler-standalone,flink-autoscaler-plugin-jdbc,flink-kubernetes-operator -am` passes
- `mvn spotless:check checkstyle:check` passes
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changes to the `CustomResourceDescriptors`: no
- Core observer or reconciler logic that is regularly executed: no (only the metric registration path is touched, no scaling decision changes)
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? docs and JavaDocs
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: DeepSeek Harness (deepseek-v4.1-flash-1017)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37751][autoscaler] Add vertex name to the autoscaler scaling metrics [flink-kubernetes-operator]
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora closed pull request #1226: [FLINK-37751][autoscaler] Add vertex name to the autoscaler scaling metrics
URL: https://github.com/apache/flink-kubernetes-operator/pull/1226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37751][autoscaler] Add vertex name to the autoscaler scaling metrics [flink-kubernetes-operator]
Posted by "Admaing (via GitHub)" <gi...@apache.org>.
Admaing commented on PR #1226:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1226#issuecomment-6399198867
Thanks @Dennis-Mircea for the review. Both points are fair — let me answer them separately.
Breaking change: agreed, and it's worse than my PR description says. `JobVertexDetailsInfo#getName()`
is always populated for a running job (it comes from `ExecutionJobVertex#getName()`), so the
"name not reported -> old identifiers" fallback I documented basically never triggers in practice:
every existing `AutoScaler.jobVertexID.<id>.<ScalingMetric>.{Current,Average}` series would be
renamed and users' dashboards/alerts would break. That's too much for a minor improvement.
Label: I looked into it, and Flink's metrics API has no label support for regular metrics.
`MetricGroup#addGroup(key, value)` creates two *named* child groups, so both land in the logical
scope; labeling reporters (Prometheus/Datadog/Influx) fold the logical scope into the metric *name*,
and only scope *variables* become labels -- those are contributed by the operator's own metric-group
classes (`KubernetesOperatorMetricGroup#putVariables`, etc.). The autoscaler registers on a generic
`MetricGroup`, so `jobVertexID.<id>` is already part of the name rather than a label, and making
`jobVertexName` a real label would mean a vertex-scoped metric group in the operator plus a way for
the autoscaler context to hand it out -- considerably bigger than this ticket.
So I see two ways forward:
1. keep the identifiers unchanged by default and gate the extra scope behind
`kubernetes.operator.autoscaler.metrics.vertex-name.enabled` (default `false`, documented,
resolved value logged at INFO) -- no breakage, readable metrics for whoever opts in;
2. drop the metric rename and just log the vertex name at registration
(`Registering scaling metrics for vertex <name> (<id>)`), and follow up separately with the
label-based exposure.
I lean towards (1).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-37751][autoscaler] Add vertex name to the autoscaler scaling metrics [flink-kubernetes-operator]
Posted by "gyfora (via GitHub)" <gi...@apache.org>.
gyfora commented on PR #1226:
URL: https://github.com/apache/flink-kubernetes-operator/pull/1226#issuecomment-6400907328
Closing this PR as apparently no human thought, care and review went into generating a good solution. This is a waste of the reviewers time. Please do not reopen in the same way.
If the reviewers wanted to talk the AI they could have done this themselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org