You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Alexander Bij (Jira)" <ji...@apache.org> on 2026/09/04 11:27:00 UTC
[jira] [Updated] (NIFI-16296) Stale Prometheus cluster_connected_node_count metrics accumulate indefinitely due to unbounded connected_nodes label values
[ https://issues.apache.org/jira/browse/NIFI-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Bij updated NIFI-16296:
---------------------------------
Description:
The Prometheus flow metrics endpoint (/nifi-api/flow/metrics/prometheus) exposes cluster_connected_node_count with a connected_nodes label containing a human-readable ratio string, e.g. connected_nodes="3 / 4". Because this label's value changes over the lifetime of the process (as nodes join/leave the cluster) and the underlying Gauge is never cleared, every distinct ratio the cluster has ever reported remains permanently exposed as a separate, frozen time series – alongside the current, correct one.
For a 4-node cluster this means a scrape can return multiple lines like:
{noformat}
cluster_connected_node_count{instance="<uuid>",connected_nodes="3 / 4",} 3.0
cluster_connected_node_count{instance="<uuid>",connected_nodes="4 / 4",} 4.0
{noformat}
...even though the cluster is currently fully connected (4/4). The "3 / 4" series is stale – a snapshot from an earlier, transient state – but is never removed, so it is scraped forever (until the node's JVM restarts), making it impossible to build a reliable Prometheus/Grafana panel for cluster health without manually filtering out stale label combinations you can't distinguish from current ones.
h2. Root cause
ClusterMetricsRegistry (org.apache.nifi.prometheusutil.ClusterMetricsRegistry) is a permanent singleton field on StandardNiFiServiceFacade:
{code:java}
private final ClusterMetricsRegistry clusterMetricsRegistry = new ClusterMetricsRegistry();
{code}
Its cluster_connected_node_count gauge is declared with connected_nodes as a {*}label{*}, not just a numeric value:
{code:java}
nameToGaugeMap.put("CONNECTED_NODE_COUNT", Gauge.build()
.name("cluster_connected_node_count")
.help("The number of connected nodes in this cluster")
.labelNames("instance", "connected_nodes") # <<-- ISSUE HERE
.register(registry));
{code}
was:
The Prometheus flow metrics endpoint (/nifi-api/flow/metrics/prometheus) exposes cluster_connected_node_count with a connected_nodes label containing a human-readable ratio string, e.g. connected_nodes="3 / 4". Because this label's value changes over the lifetime of the process (as nodes join/leave the cluster) and the underlying Gauge is never cleared, every distinct ratio the cluster has ever reported remains permanently exposed as a separate, frozen time series -- alongside the current, correct one.
For a 4-node cluster this means a scrape can return multiple lines like:
{noformat}
cluster_connected_node_count\{instance="<uuid>",connected_nodes="3 / 4"} 3.0
cluster_connected_node_count\{instance="<uuid>",connected_nodes="4 / 4"} 4.0
{noformat}
...even though the cluster is currently fully connected (4/4). The "3 / 4" series is stale -- a snapshot from an earlier, transient state -- but is never removed, so it is scraped forever (until the node's JVM restarts), making it impossible to build a reliable Prometheus/Grafana panel for cluster health without manually filtering out stale label combinations you can't distinguish from current ones.
h2. Root cause
ClusterMetricsRegistry (org.apache.nifi.prometheusutil.ClusterMetricsRegistry) is a permanent singleton field on StandardNiFiServiceFacade:
{code:java}
private final ClusterMetricsRegistry clusterMetricsRegistry = new ClusterMetricsRegistry();
{code}
Its cluster_connected_node_count gauge is declared with connected_nodes as a *label*, not just a numeric value:
{code:java}
nameToGaugeMap.put("CONNECTED_NODE_COUNT", Gauge.build()
.name("cluster_connected_node_count")
.help("The number of connected nodes in this cluster")
.labelNames("instance", "connected_nodes") # <<-- ISSUE HERE
.register(registry));
{code}
> Stale Prometheus cluster_connected_node_count metrics accumulate indefinitely due to unbounded connected_nodes label values
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16296
> URL: https://issues.apache.org/jira/browse/NIFI-16296
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.10.0, 2.11.0
> Reporter: Alexander Bij
> Priority: Major
> Labels: Matrices, monitoring, prometheus
>
> The Prometheus flow metrics endpoint (/nifi-api/flow/metrics/prometheus) exposes cluster_connected_node_count with a connected_nodes label containing a human-readable ratio string, e.g. connected_nodes="3 / 4". Because this label's value changes over the lifetime of the process (as nodes join/leave the cluster) and the underlying Gauge is never cleared, every distinct ratio the cluster has ever reported remains permanently exposed as a separate, frozen time series – alongside the current, correct one.
>
> For a 4-node cluster this means a scrape can return multiple lines like:
> {noformat}
> cluster_connected_node_count{instance="<uuid>",connected_nodes="3 / 4",} 3.0
> cluster_connected_node_count{instance="<uuid>",connected_nodes="4 / 4",} 4.0
> {noformat}
> ...even though the cluster is currently fully connected (4/4). The "3 / 4" series is stale – a snapshot from an earlier, transient state – but is never removed, so it is scraped forever (until the node's JVM restarts), making it impossible to build a reliable Prometheus/Grafana panel for cluster health without manually filtering out stale label combinations you can't distinguish from current ones.
>
> h2. Root cause
> ClusterMetricsRegistry (org.apache.nifi.prometheusutil.ClusterMetricsRegistry) is a permanent singleton field on StandardNiFiServiceFacade:
> {code:java}
> private final ClusterMetricsRegistry clusterMetricsRegistry = new ClusterMetricsRegistry();
> {code}
> Its cluster_connected_node_count gauge is declared with connected_nodes as a {*}label{*}, not just a numeric value:
> {code:java}
> nameToGaugeMap.put("CONNECTED_NODE_COUNT", Gauge.build()
> .name("cluster_connected_node_count")
> .help("The number of connected nodes in this cluster")
> .labelNames("instance", "connected_nodes") # <<-- ISSUE HERE
> .register(registry));
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)