[jira] [Commented] (FLINK-40372) [Kubernetes Operator] JOSDK per-resource metrics are not unregistered after resource cleanup

["liting liu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-40372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18103972#comment-18103972 ] 

liting liu commented on FLINK-40372:
------------------------------------

I'm glad to fix this

> [Kubernetes Operator] JOSDK per-resource metrics are not unregistered after resource cleanup
> --------------------------------------------------------------------------------------------
>
>                 Key: FLINK-40372
>                 URL: https://issues.apache.org/jira/browse/FLINK-40372
>             Project: Flink
>          Issue Type: Bug
>          Components: Kubernetes Operator
>    Affects Versions: 1.13
>            Reporter: liting liu
>            Priority: Critical
>
> ## Description
> `OperatorJosdkMetrics` creates Flink metric groups and counters whose scopes contain
> the namespace and name of every reconciled Kubernetes custom resource.
> After a resource is successfully deleted, `cleanupDone()` only increments a cleanup
> counter. It does not close or remove the metric group, and it does not remove the
> associated entries from the internal maps.
> As resource names change over time, especially for automatically generated
> `FlinkStateSnapshot` resources such as periodic savepoints, the number of registered
> metrics can grow monotonically.
> This increases metric cardinality and retains metric groups, counters and their scope
> strings for resources that no longer need to be monitored. When the SLF4J reporter is
> enabled, every reporting cycle also iterates over and formats the accumulated metrics.
> ## Affected versions
> Observed with an Apache Flink Kubernetes Operator 1.13 based build.
> The same lifecycle issue is still present on Apache `main` at:
> - commit: `fdfcc695daba16e4f4a8c9a6c08b7f470787de00`
> - date: 2026-08-11
> JOSDK version in the 1.13 based build: 5.1.2.
> ## Relevant configuration
> The SLF4J reporter was configured with a five-minute interval:
>     kubernetes.operator.metrics.reporter.slf4j.factory.class: org.apache.flink.metrics.slf4j.Slf4jReporterFactory
>     kubernetes.operator.metrics.reporter.slf4j.interval: 5 MINUTE
> The Operator container had a 1 GiB memory limit.
> ## Source analysis
> `OperatorJosdkMetrics` keeps the following maps for the lifetime of the Operator:
> ```java
> private final Map<ResourceID, KubernetesResourceNamespaceMetricGroup>
>         resourceNsMetricGroups = new ConcurrentHashMap<>();
> private final Map<ResourceID, KubernetesResourceMetricGroup>
>         resourceMetricGroups = new ConcurrentHashMap<>();
> private final Map<List<String>, Histogram> histograms = new ConcurrentHashMap<>();
> private final Map<List<String>, Counter> counters = new ConcurrentHashMap<>();



--
This message was sent by Atlassian Jira
(v8.20.10#820010)