You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@trafficserver.apache.org by "cmcfarlen (via GitHub)" <gi...@apache.org> on 2026/08/06 00:07:30 UTC

[PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

cmcfarlen opened a new pull request, #13506:
URL: https://github.com/apache/trafficserver/pull/13506

   Publishes per-upstream-server connection metrics aggregated per hostname, using hidden metrics as the inputs and a derived metric for the aggregation, so the aggregation work happens on the stat sync task rather than on every connection.
   
   > **Depends on #13505**, which adds the hidden-metric store and `Derived::add_source` / `Op::MAX` that this builds on. The first 8 commits here are that PR; only the last two are new. Please merge #13505 first, after which this diff reduces to those two commits.
   
   ### `metric_enabled` becomes a level
   
   `proxy.config.http.per_server.connection.metric_enabled` changes from a flag to a level:
   
   | Value | Behavior |
   |---|---|
   | `0` | No per server metrics. |
   | `1` | Per group metrics stay hidden; the per hostname aggregates are published. |
   | `2` | As `1`, plus the per group metrics are mirrored into the published store. |
   
   The per group metrics are **always** created in the hidden store whenever metrics are enabled at all. Only what is registered for publication varies by level, so changing the level at runtime never has to migrate a metric between the two stores.
   
   ### Metrics
   
   Three hidden per group metrics (`current_connection`, `total_connection`, `blocked_connection`) feed four published per hostname aggregates: those three summed across the hostname's groups, plus `current_connection_max`, the `MAX` of the groups' current counts.
   
   `current_connection_max` is deliberately the maximum of the groups' *instantaneous* counts, not a monotone high-water mark. It answers "how close is the busiest group of this hostname to `per_server.connection.max` right now", which matters because that limit is enforced per group rather than per hostname. Because it rises and falls, a monitoring system can compute max-over-time over any window from it; a monotone value would collapse the time dimension and only report that a peak happened at some point, not when. There is no per group `current_connection_max` — it exists only as an aggregate.
   
   Aggregates are registered only for the `both` match type, the only one with more than one group per hostname. For `host`, `Group::metric_name` already returns the bare FQDN, so a host aggregate would collide with the single group's own name on one metric.
   
   ### No new hot-path work
   
   `TxnState::reserve`, `release` and `blocked` are unchanged. They already branch on `_count_metric != nullptr` and use the typed mutators, and `createHiddenPtr` returns the same pointer types, so those branches keep working with the metrics simply being hidden now. `git diff master -- include/iocore/net/ConnectionTracker.h` shows no logic added inside them.
   
   ### Also fixes the per group peak count
   
   `Group::_count_max`, reported as the `max` field of the connection tracker group dump, had two defects, both pre-existing:
   
   - `update_max_count()` made a single `compare_exchange_weak` attempt with no retry, so a racing update — or a spurious failure of the weak form, which is permitted by the standard and does occur on LL/SC architectures — silently discarded the sample.
   - It was only called when a maximum was configured. With metrics enabled and no configured maximum the count was reserved and then discarded, so the peak stayed at zero.
   
   ### Testing
   
   `tests/gold_tests/origin_connection/per_server_connection_max.test.py` is extended to cover:
   
   - Level 1: host aggregates published, and all three per group names absent from a normal query.
   - Level 2: host aggregates plus the per group metrics published.
   - Both levels: per group metrics visible via `traffic_ctl metric match per_server --include-hidden`.
   - **An aggregate genuinely spanning two groups** — one hostname mapped to two origin ports under `match: both`, with different concurrency per group (2 and 3) so the `SUM` (5) and the `MAX` (3) are distinguishable rather than coincidentally equal. Nothing previously covered a multi-group aggregate.
   - **Both gauges draining back to 0** after traffic stops and a further sync interval passes. This is the assertion that distinguishes an instantaneous gauge from a monotone peak, which could never satisfy it.
   
   Aggregates are recomputed on `ET_TASK` every `REC_RAW_STAT_SYNC_INTERVAL_MS` (5000 ms) and no record drives that interval, so the test sleeps past a tick rather than trying to configure a faster one; reading too early would silently compare against zeros.
   
   The autest passes. Documentation is added for `metric_enabled` and `metric_prefix`, neither of which was documented before, plus the metrics themselves under the monitoring guide.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#issuecomment-5402548493

   @serrislew @bryancall — on the `add_source` linear scan, since you both landed on it independently.
   
   I measured rather than guessed. `DerivedMetric` is 40 bytes (`IdType` is `int32_t`, plus a 24-byte source vector and the op), so `std::find_if` is a contiguous sweep comparing one `int32_t` at stride 40:
   
   | distinct group identities | derived entries (default `metric_aggregate: 0`, 3/group) | per group creation (3 scans) |
   |---|---|---|
   | ~33 | ~100 | ~4 KB swept, ~0.5 µs |
   | 500 | ~1500 | ~180 KB swept, ~10–20 µs |
   
   For context on what shares that call path: the `Group` constructor already does 3–7 `std::string` concatenations (a heap allocation each) plus 3–7 `_create` calls, each taking the metrics-store mutex and hashing a ~50 character name — on the order of 1–3 µs. So at the cardinality this actually runs at, the scan is **cheaper than the string work sitting next to it**, and an id-indexed map would not be measurable. It starts to matter in the high hundreds of distinct upstreams, and even then it is ~10 µs on a path that also does a DNS lookup and a TCP connect.
   
   The contention concern is the sharper version of this, and the map does not address it either: the serialization is `_outbound_table._mutex` held across the whole constructor, plus `metrics_lock`, which `update()` also takes every sync tick while walking every entry. Making the lookup O(1) shortens the hold slightly without changing which locks are held or for how long in aggregate.
   
   What actually makes the linear scan the right structure is bounding N, which is the registry lifetime issue Bryan raised separately: `~Group` never unregisters, so entries accumulate per distinct `(fqdn, addr, port)` identity ever seen, and `update()` walks dead groups forever. Worth being precise about the growth, though — `add_source` dedups by source pointer and the hidden slots are name-keyed, so a reaped group that comes back reuses its existing entry. Churn is free; growth is bounded by distinct upstream identity cardinality, which plateaus for a fixed origin set. Not a per-connection leak.
   
   We are deferring that cleanup to a follow-up rather than growing this PR, because the design needs a discussion first:
   
   1. Whether real deployments churn the identity space enough for the growth to matter at all.
   2. If so, `remove_source` needs a final refresh before erasing an entry (otherwise a published gauge freezes at its last synced value instead of dropping to 0, since the hot path only writes the hidden copy) and refcounted registrations (otherwise an overlapping reap and re-create can strip a live group's publication).
   3. Retiring metrics from `Storage` is the harder half. Since pointers handed out are raw and never invalidated, and a slot is only 8 bytes plus a name, reclamation buys almost nothing — the costs that bite are enumeration and `update()`. A tombstone that hides a metric from both, without ever invalidating a pointer, looks like the right first step.
   
   Filing that as a separate issue so it does not get lost.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#discussion_r3724980510


##########
doc/admin-guide/monitoring/statistics/core/http-connection.en.rst:
##########
@@ -201,6 +201,61 @@ HTTP Connection
    Current number of TCP connections for tunnels where the far end is the server,
    except for those counted by ``proxy.process.tunnel.current_server_connections_tls``
 
+.. _per-server-connection-metrics:
+
+Per Server Connection Metrics
+-----------------------------
+
+Unlike the metrics above these do not have fixed names. They are created dynamically, one set per
+upstream server group as defined by :ts:cv:`proxy.config.http.per_server.connection.match`, and,
+when the match type is ``both``, one aggregate set per hostname. Whether any of them are published,
+and at what granularity, is controlled by
+:ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. An optional
+:ts:cv:`proxy.config.http.per_server.connection.metric_prefix` can be inserted into the names.
+
+Per group names are ``proxy.process.http.per_server.<counter>.<group>``, where ``<group>`` depends on
+the match type: an IP address, an ``address:port`` pair, a hostname, or, for ``both``,
+``<hostname>.<address:port>``. Per hostname names are
+``proxy.process.http.per_server.<counter>.<hostname>``. Aggregates exist only for match type
+``both``, because that is the only match type with more than one group per hostname; for match type
+``host`` the group name is already the bare hostname, so an aggregate would carry the same name as
+the single group it summarises.
+
+For a group, ``<counter>`` is one of:
+
+current_connection
+   Gauge. The number of connections currently open to the group.
+
+total_connection
+   Counter. The total number of connections ever opened to the group. Never decreases.
+
+blocked_connection
+   Counter. The total number of connection attempts to the group blocked by
+   :ts:cv:`proxy.config.http.per_server.connection.max`. Never decreases.
+
+For a hostname aggregate, ``<counter>`` is one of those three, each summed across the groups of that
+hostname, plus:
+
+current_connection_max
+   Gauge. The largest ``current_connection`` value among the groups of that hostname at the moment
+   of sampling, so the maximum rather than the sum of the groups' current counts. This is useful
+   because :ts:cv:`proxy.config.http.per_server.connection.max` is enforced per group rather than
+   per hostname, so the busiest group is what determines whether connections are about to be
+   blocked. Like ``current_connection`` it rises and falls with traffic and is not a high-water
+   mark. There is no per group ``current_connection_max``; it exists only as a hostname aggregate.
+
+The aggregates are recomputed periodically, currently every 5 seconds, rather than on every
+connection event, so a reader sees a value up to that interval old. This applies to
+``current_connection_max`` as well: it reports the maximum across groups as of the last sample, not
+a running peak. To obtain the peak over a longer window, compute a maximum over time from this gauge
+in the monitoring system.

Review Comment:
   This section says only the hostname aggregates are recomputed periodically, but at `metric_enabled` level 2 the per-group metrics are also published via `Derived` mirroring, so they will lag by up to the same sync interval as well. The docs should call out that any derived-published per-server metric (including level-2 group mirrors) is sampled periodically, not updated on each connection event.



##########
src/records/RecordsConfig.cc:
##########
@@ -395,7 +395,7 @@ static constexpr RecordElement RecordsConfig[] =
         ,
   {RECT_CONFIG, "proxy.config.http.per_server.connection.min", RECD_INT, "0", RECU_DYNAMIC, RR_NULL, RECC_STR, "^[0-9]+$", RECA_NULL}
   ,
-  {RECT_CONFIG, "proxy.config.http.per_server.connection.metric_enabled", RECD_INT, "0", RECU_DYNAMIC, RR_NULL, RECC_STR, "[0-1]", RECA_NULL}
+  {RECT_CONFIG, "proxy.config.http.per_server.connection.metric_enabled", RECD_INT, "0", RECU_DYNAMIC, RR_NULL, RECC_STR, "[0-2]", RECA_NULL}

Review Comment:
   The validation regex `[0-2]` will accept multi-digit values like `10` because the check uses a non-anchored regex match. This means invalid `metric_enabled` values can pass records.yaml validation even though the implementation later clamps them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#issuecomment-5413723690

   Thanks Bryan — this is a genuinely useful review, and the two corrections to our own claims are the valuable part. Pushed `4e57a3e183` and `8038a05aa9`.
   
   ## Applied
   
   **1. Reload directionality.** Took your wording on `metric_enabled` and spelled the caveat out on `metric_aggregate` rather than leaving it as a back-reference, since as you say `2` is the value someone reaches for specifically to cut cardinality and it's the one direction a reload can't deliver. Kept `:reloadable:`.
   
   **2. The wrong premise.** You're right, and I verified it: `Group::equal` keys `MATCH_IP` on the address alone, so one hostname with several A records already yields several groups. Worth flagging that the claim had spread to **four** places, not one — `http-connection.en.rst:220`, `records.yaml.en.rst:2051`, and both `ConnectionTracker.h:211` and `:491` — so I fixed all of them with your reasoning: `ip`/`port` keys carry no hostname and are shared by every hostname resolving to that address. Good catch on "someone will eventually relax the restriction on the strength of a premise that is false"; that's exactly the failure mode.
   
   **3. Subset aggregates and the cross-match-type collision.** Both documented, on the monitoring page and on `host_metric_name`. Confirmed `server_match` is overridable at `OverridableConfigDefs.h:229`, so the `MATCH_HOST`/`MATCH_BOTH` collision is reachable by configuration, and `add_source` does keep the first caller's op. Agreed that including the match type in the aggregate name, and deciding membership per hostname, are the real fixes and belong with the follow-up.
   
   **4. `per_server_metric_enabled.test.py`.** Fixed — it now sets `raw_stat_sync_interval_ms` the same way the other file does. Your analysis was right: a 6 second wait against a 5000ms default is at least one second of margin and never more.
   
   **7 (partial).** Corrected the "single source SUM is an identity" comment. You're right that it contradicts the design it sits in: a single-source SUM combines nothing, but the published value is still a sample, and it reads 0 from creation until the first tick.
   
   ## One correction back
   
   The unanchored-pattern note isn't right as stated — though I suspect it's a holdover from reviewing the first iteration of this work, which did define `metric_enabled` as a three-valued `0/1/2` level before we split it into two settings for backward compatibility. Against the pattern that's actually on master, a `records.yaml` carrying `metric_enabled: 2` did **not** load: the pre-PR pattern was unanchored `[0-1]`, and `recordRegexCheck` searching `"2"` for `[0-1]` finds no match, so validation failed then too. What actually regresses is multi-character values containing a `0` or `1` — `10`, `01`, `12` — which used to match on a substring and now don't. Real incompatibility, different trigger, and vanishingly unlikely in a real config, so I'd rather not put a note in the PR body implying `2` used to be accepted. Happy to add one about the substring case if you think it earns the space.
   
   ## Deferred, with reasons
   
   - **5, upgrading notes.** Skipping this one here. `upgrading.en.rst` is still headed "Upgrading to ATS v10.x" and this change is on 11.x-dev, so starting that section is a bigger editorial call than this PR should make. The sampling-latency note lands with the follow-up, which also has to document the retirement behaviour, and the monitoring page already covers the mechanism in the meantime.
   
   - **6, `metric_prefix` global + unsynchronised write.** Agreed on both halves. The race is pre-existing and the fix wants to be the same change that makes registration lifetime-aware, so it goes with the follow-up.
   - **7, remaining items.** The diagnostics on the converters and reload callbacks and the more informative `Registered per_server_connection` debug line are worth doing — your point that "why can I not see my metric" will be the common question is well taken, given how many ways this feature can decline to publish. Dropping the `Group` constructor defaults and taking `bool metric_enabled` is a good call too; the adjacent-`int` transposition hazard is real. All code changes, none of them blocking, and I'd rather not reopen the platform builds on this PR for them.
   - **`MultiGroupAggregateTest` hold time.** Fair, and widening it costs nothing at a 500ms interval. Also right that the comment's reasoning is wrong — go-httpbin returns 400 above its cap and `--fail` makes that a hard failure, not a clamp. Folding both into the follow-up unless you'd rather see it here.
   
   Following up separately on the `MatchType` `static_cast` UB you spotted — you're right that it's a real find independent of this change, and it deserves its own issue rather than being buried in this thread.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#issuecomment-5398110197

   [approve ci autest 2]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#issuecomment-5432363468

   Cherry-picked to the 10.2.x branch as 1a8102d12f37a1004d11275deae9a2162c808936 for the 10.2.1 release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen merged PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on code in PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#discussion_r3854592636


##########
doc/admin-guide/files/records.yaml.en.rst:
##########
@@ -2007,6 +2007,80 @@ Origin Server Connect Attempts
    the connection. Useful when the origin supports keep-alive, removing the time needed to set up a
    new connection from the next request at the expense of added (inactive) connections.
 
+.. ts:cv:: CONFIG proxy.config.http.per_server.connection.metric_enabled INT 0
+   :reloadable:
+   :overridable:
+
+   Enable per upstream server connection metrics. These metrics are dynamically named, one set per
+   upstream server group, so the number of them scales with the number of distinct upstream servers
+   seen. See :ref:`per-server-connection-metrics`.
+
+   ===== ======================================================================================
+   Value Effect
+   ===== ======================================================================================
+   ``0`` No per server connection metrics.
+   ``1`` Per server connection metrics are collected for each upstream server group.
+   ===== ======================================================================================
+
+   What is published from them is controlled separately by
+   :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate`, which by default publishes the
+   per group metrics themselves.
+
+   Because this is overridable, metrics can be enabled for the upstreams of interest and left off
+   for the rest, for example with :ref:`admin-plugins-conf-remap` on a specific mapping.
+
+   The value is applied when a connection group is created. Where two mappings that disagree about
+   this setting resolve to the same group -- that is, the same key under
+   :ts:cv:`proxy.config.http.per_server.connection.match` -- the transaction that creates the group
+   determines its metrics, and later transactions do not change them. A group is discarded once its
+   connection count reaches zero, so the choice is made again the next time that upstream is
+   reopened. This affects only which metrics exist; enforcement of

Review Comment:
   This is the sentence I would change. "the choice is made again the next time that upstream is reopened" reads as full reversibility, and it only holds for increasing publication. Turning metrics off, or switching `metric_aggregate` to `2`, leaves the already-published names in place for the life of the process.
   
   ```suggestion
      determines its metrics, and later transactions do not change them. A group is discarded once its
      connection count reaches zero, so *raising* the level of publication is picked up the next time
      that upstream is reopened: enabling metrics, or enabling the aggregates, takes effect as upstreams
      reconnect. Lowering it does not. Metrics are never retired once published, so disabling this
      setting, or switching
      :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate` to ``2``, leaves the names that
      are already published in place, frozen at their last sampled value, until Traffic Server is
      restarted. This affects only which metrics exist; enforcement of
   ```
   
   The same caveat is needed on `metric_aggregate` below, which currently just points back here with "the same consequence for mappings that disagree and resolve to the same group". It is worth spelling out there too, since `2` is the value an operator reaches for specifically to reduce a metric count, and that is the one direction a reload cannot deliver.



##########
doc/admin-guide/monitoring/statistics/core/http-connection.en.rst:
##########
@@ -201,6 +201,64 @@ HTTP Connection
    Current number of TCP connections for tunnels where the far end is the server,
    except for those counted by ``proxy.process.tunnel.current_server_connections_tls``
 
+.. _per-server-connection-metrics:
+
+Per Server Connection Metrics
+-----------------------------
+
+Unlike the metrics above these do not have fixed names. They are created dynamically, one set per
+upstream server group as defined by :ts:cv:`proxy.config.http.per_server.connection.match`, and,
+when the match type is ``both``, one aggregate set per hostname. Whether they are collected at all is
+controlled by :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`, and which of them are
+published by :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate`. An optional
+:ts:cv:`proxy.config.http.per_server.connection.metric_prefix` can be inserted into the names.
+
+Per group names are ``proxy.process.http.per_server.<counter>.<group>``, where ``<group>`` depends on
+the match type: an IP address, an ``address:port`` pair, a hostname, or, for ``both``,
+``<hostname>.<address:port>``. Per hostname names are
+``proxy.process.http.per_server.<counter>.<hostname>``. Aggregates exist only for match type
+``both``, because that is the only match type with more than one group per hostname; for match type
+``host`` the group name is already the bare hostname, so an aggregate would carry the same name as
+the single group it summarises.
+
+For a group, ``<counter>`` is one of:
+
+current_connection
+   Gauge. The number of connections currently open to the group.
+
+total_connection
+   Counter. The total number of connections ever opened to the group. Never decreases.
+
+blocked_connection
+   Counter. The total number of connection attempts to the group blocked by
+   :ts:cv:`proxy.config.http.per_server.connection.max`. Never decreases.
+
+For a hostname aggregate, ``<counter>`` is one of those three, each summed across the groups of that
+hostname, plus:

Review Comment:
   "each summed across the groups of that hostname" is what an operator will rely on, and it can be false. Aggregate membership is registered per group, gated on the `metric_aggregate` value of the transaction that created that group, so under `match: both` a mapping with `metric_aggregate: 1` joins the aggregate and a mapping with `0` for the same hostname does not. The sum then covers a strict subset, and `current_connection_max` takes its maximum over that subset, with nothing indicating it.
   
   ```suggestion
   For a hostname aggregate, ``<counter>`` is one of those three, each summed across the groups of that
   hostname which have aggregation enabled, plus:
   ```
   
   I would add a short paragraph after the `current_connection_max` entry as well, along the lines of: because `metric_aggregate` is overridable, a group joins its hostname's aggregate only if the mapping that first opened that upstream had aggregation enabled, so mappings that disagree for one hostname produce an aggregate over part of it. Making membership a per-hostname decision is the better fix, but that can go with the `remove_source` follow-up.



##########
doc/admin-guide/monitoring/statistics/core/http-connection.en.rst:
##########
@@ -201,6 +201,64 @@ HTTP Connection
    Current number of TCP connections for tunnels where the far end is the server,
    except for those counted by ``proxy.process.tunnel.current_server_connections_tls``
 
+.. _per-server-connection-metrics:
+
+Per Server Connection Metrics
+-----------------------------
+
+Unlike the metrics above these do not have fixed names. They are created dynamically, one set per
+upstream server group as defined by :ts:cv:`proxy.config.http.per_server.connection.match`, and,
+when the match type is ``both``, one aggregate set per hostname. Whether they are collected at all is
+controlled by :ts:cv:`proxy.config.http.per_server.connection.metric_enabled`, and which of them are
+published by :ts:cv:`proxy.config.http.per_server.connection.metric_aggregate`. An optional
+:ts:cv:`proxy.config.http.per_server.connection.metric_prefix` can be inserted into the names.
+
+Per group names are ``proxy.process.http.per_server.<counter>.<group>``, where ``<group>`` depends on
+the match type: an IP address, an ``address:port`` pair, a hostname, or, for ``both``,
+``<hostname>.<address:port>``. Per hostname names are
+``proxy.process.http.per_server.<counter>.<hostname>``. Aggregates exist only for match type
+``both``, because that is the only match type with more than one group per hostname; for match type
+``host`` the group name is already the bare hostname, so an aggregate would carry the same name as
+the single group it summarises.

Review Comment:
   "the only match type with more than one group per hostname" is not accurate, and the same claim appears in `records.yaml.en.rst`, in the `MetricAggregate` doc comment, and on `host_metric_name`. `Group::hash` and `Group::equal` key `MATCH_IP` on the address alone, so one hostname with several A records already yields several groups.
   
   The real reason is narrower and, I think, more useful to a future reader: `ip` and `port` keys carry no hostname at all, and one such group is shared by every hostname that resolves to that address, so there is nothing to aggregate it under. As written, someone will eventually relax the restriction on the strength of a premise that is false.
   
   ```suggestion
   ``proxy.process.http.per_server.<counter>.<hostname>``. Aggregates exist only for match type
   ``both``, because that is the only match type whose group key carries the hostname. An ``ip`` or
   ``port`` group is keyed on the address alone and is shared by every hostname that resolves to it, so
   there is no single hostname to aggregate it under. For match type ``host`` the group name is already
   the bare hostname, so an aggregate would carry the same name as the single group it summarises.
   ```



##########
src/iocore/net/ConnectionTracker.cc:
##########
@@ -440,11 +475,45 @@ ConnectionTracker::Group::Group(DirectionType direction, Key const &key, std::st
 {
   Metrics::Gauge::increment(net_rsb.connection_tracker_table_size);
   // only add metrics for server connections
-  if (_global_config->metric_enabled && direction == DirectionType::OUTBOUND) {
+  if (metric_enabled && direction == DirectionType::OUTBOUND) {
     std::string _metric_name = metric_name(key, fqdn, _global_config->metric_prefix);
-    _count_metric            = Metrics::Gauge::createPtr("proxy.process.http.per_server.current_connection.", _metric_name);
-    _count_total_metric      = Metrics::Counter::createPtr("proxy.process.http.per_server.total_connection.", _metric_name);
-    _blocked_metric          = Metrics::Counter::createPtr("proxy.process.http.per_server.blocked_connection.", _metric_name);
+    // Per group metrics always live in the hidden store. metric_aggregate controls what is
+    // published from them (see MetricAggregate), not whether they exist.
+    _count_metric       = Metrics::Gauge::createHiddenPtr("proxy.process.http.per_server.current_connection.", _metric_name);
+    _count_total_metric = Metrics::Counter::createHiddenPtr("proxy.process.http.per_server.total_connection.", _metric_name);
+    _blocked_metric     = Metrics::Counter::createHiddenPtr("proxy.process.http.per_server.blocked_connection.", _metric_name);
+
+    // Only MATCH_BOTH groups have siblings sharing a hostname to aggregate across.
+    std::string _host_metric_name = host_metric_name(key, fqdn, _global_config->metric_prefix);
+    bool const  has_aggregate     = !_host_metric_name.empty();
+
+    if (has_aggregate && metric_aggregate != AGGREGATE_NONE) {
+      Metrics::Derived::add_source("proxy.process.http.per_server.current_connection." + _host_metric_name,
+                                   Metrics::MetricType::GAUGE, _count_metric, Metrics::Derived::Op::SUM);
+      Metrics::Derived::add_source("proxy.process.http.per_server.total_connection." + _host_metric_name,
+                                   Metrics::MetricType::COUNTER, _count_total_metric, Metrics::Derived::Op::SUM);
+      Metrics::Derived::add_source("proxy.process.http.per_server.blocked_connection." + _host_metric_name,
+                                   Metrics::MetricType::COUNTER, _blocked_metric, Metrics::Derived::Op::SUM);
+      // The largest current count among this hostname's groups, sampled. Deliberately taken over
+      // the instantaneous gauge rather than each group's all time peak, so the value falls again
+      // and a maximum over time can be computed by whatever scrapes it.
+      Metrics::Derived::add_source("proxy.process.http.per_server.current_connection_max." + _host_metric_name,
+                                   Metrics::MetricType::GAUGE, _count_metric, Metrics::Derived::Op::MAX);
+    }
+
+    // AGGREGATE_ONLY suppresses the per group metrics to keep the published count proportional to
+    // hostnames. Without an aggregate to stand in for them there would be nothing at all reported
+    // for this group, so in that case publish them regardless.
+    if (metric_aggregate != AGGREGATE_ONLY || !has_aggregate) {
+      // Mirror the per group metrics into the published store under their own name. A single
+      // source SUM is an identity: the published value always equals the hidden source.

Review Comment:
   "the published value always equals the hidden source" is the one claim in this change that contradicts its own design. The published copy is stale between derived ticks and reads 0 from creation until the first one, which is precisely the property `http-connection.en.rst` takes care to explain. A single-source SUM does not combine anything, but it is still a sample rather than an identity.
   
   ```suggestion
         // Mirror the per group metrics into the published store under their own name. A single
         // source SUM combines nothing, but the published value is still a sample: it is whatever
         // the last derived tick read, and it reads 0 from creation until that first tick.
   ```



##########
src/iocore/net/ConnectionTracker.cc:
##########
@@ -70,10 +72,28 @@ const MgmtConverter ConnectionTracker::SERVER_MATCH_CONV{
     }
   }};
 
+// Neither of these clamps, for the same reason as SERVER_MATCH_CONV above: the InkAPITest
+// regression test requires an arbitrary integer to round trip through the setter and getter. The
+// records paths do the range checking instead -- records.yaml validates the value and the reload
+// callbacks below clamp -- so an out of range value is only reachable by a plugin that sets one
+// deliberately. Both settings degrade safely if that happens: any non-zero metric_enabled enables
+// metrics, and any metric_aggregate outside 0..2 publishes both the aggregate and the per group
+// metrics, the same as AGGREGATE_GROUP.
+const MgmtConverter ConnectionTracker::METRIC_ENABLED_CONV{

Review Comment:
   Not blocking, and I accept the reasoning in the comment above for why neither converter clamps. What I would still like is a diagnostic somewhere on this path, because right now there is none anywhere in the new code.
   
   `SERVER_MATCH_CONV`, three converters up, has a formatter and calls `Warning_Bad_Match_Type`. Both new reload callbacks clamp silently, and their `return false` on a wrong `dtype` is swallowed, since the `RecCore` lambda returns `REC_ERR_OKAY` unconditionally and `traffic_ctl` reports success either way. The sibling `Config_Update_Conntrack_Match` does at least emit `Warning("Invalid type for '%s'")`. The net effect is that a plugin setting `metric_aggregate = 7` reads back 7, behaves as `AGGREGATE_GROUP`, and leaves no trace of either fact.
   
   Related, and the same size of fix: the debug print at line 520 says `Registered per_server_connection.{}` without saying what was actually registered, hidden or published, aggregate or per group. That line is the first place an operator will look to answer "why can I not see my metric", which is going to be a common question given how many ways this feature can decline to publish something.



##########
include/iocore/net/ConnectionTracker.h:
##########
@@ -171,6 +206,20 @@ class ConnectionTracker
     std::time_t        get_last_alert_epoch_time() const;
     static std::string metric_name(const Key &key, std::string_view fqdn, std::string metric_prefix);
 
+    /** Name of the metric which aggregates a value across all groups of a hostname.
+     *
+     * Only @c MATCH_BOTH groups have more than one group per hostname. For @c MATCH_HOST there is
+     * exactly one group per hostname, so an aggregate would be over a set of one, and
+     * @c Group::metric_name already returns the FQDN alone for that match type - identical to what
+     * this would return, so publishing both would collide on one name.

Review Comment:
   Two problems in this paragraph. The first sentence repeats the inaccurate reason described on the monitoring page. The second is that the collision argument is right within one match type and does not hold across two, which matters because `server_match` is itself overridable: foo.com can be `MATCH_HOST` on one mapping and `MATCH_BOTH` on another, at which point `metric_name` for the host group and `host_metric_name` for the aggregate produce the identical string. `add_source` then finds the existing entry by id, appends the source, and ignores the later caller's op, so one published name silently carries the union of a per-group mirror and a cross-group aggregate.
   
   ```suggestion
        * Only @c MATCH_BOTH keys carry both a hostname and an address, so it is the only match type
        * whose groups can be gathered by hostname at all. @c MATCH_IP and @c MATCH_PORT key on the
        * address alone and one such group is shared by every hostname resolving to it, so there is no
        * single hostname to aggregate it under. For @c MATCH_HOST there is exactly one group per
        * hostname, so an aggregate would be over a set of one, and @c Group::metric_name already
        * returns the FQDN alone for that match type - identical to what this would return, so
        * publishing both would collide on one name.
        *
        * Note that reasoning holds within a single match type. @c TxnConfig::server_match is
        * overridable, so one hostname can be @c MATCH_HOST on one mapping and @c MATCH_BOTH on
        * another, and then that group's own published name and this aggregate name are the same
        * string and are merged into one derived metric.
   ```
   
   Documenting it is enough for this PR. Including the match type in the aggregate name, or warning on the collision, is the actual fix and can go with the follow-up.



##########
include/iocore/net/ConnectionTracker.h:
##########
@@ -155,8 +187,11 @@ class ConnectionTracker
      * @param key A populated @c Key structure - values are copied to the @c Group.
      * @param fqdn The full FQDN.
      * @param min_keep_alive The minimum number of origin keep alive connections to maintain.
+     * @param metric_enabled Whether the transaction creating this group wants per server metrics.
+     * @param metric_aggregate What that transaction wants published, see @c MetricAggregate.
      */
-    Group(DirectionType direction, Key const &key, std::string_view fqdn, int min_keep_alive);
+    Group(DirectionType direction, Key const &key, std::string_view fqdn, int min_keep_alive, int metric_enabled = 0,
+          MetricAggregate metric_aggregate = AGGREGATE_NONE);

Review Comment:
   Suggestion only. I would drop both default arguments. There are two construction sites, and only the INBOUND one 21 lines away in `obtain_inbound` uses the defaults, so the defaults buy one short call and cost a silent failure mode: a future outbound site that forgets these arguments compiles cleanly and gets no metrics.
   
   The sharper version is that `min_keep_alive` and `metric_enabled` are adjacent, both `int`, so transposing them also compiles and produces a group with metrics keyed off a keep-alive count. Taking `bool metric_enabled` would remove that particular hazard, and it matches how the value is actually used here, which is a plain truth test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on code in PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#discussion_r3730335911


##########
src/records/RecordsConfig.cc:
##########
@@ -395,7 +395,7 @@ static constexpr RecordElement RecordsConfig[] =
         ,
   {RECT_CONFIG, "proxy.config.http.per_server.connection.min", RECD_INT, "0", RECU_DYNAMIC, RR_NULL, RECC_STR, "^[0-9]+$", RECA_NULL}
   ,
-  {RECT_CONFIG, "proxy.config.http.per_server.connection.metric_enabled", RECD_INT, "0", RECU_DYNAMIC, RR_NULL, RECC_STR, "[0-1]", RECA_NULL}
+  {RECT_CONFIG, "proxy.config.http.per_server.connection.metric_enabled", RECD_INT, "0", RECU_DYNAMIC, RR_NULL, RECC_STR, "[0-2]", RECA_NULL}

Review Comment:
   Good catch, fixed to `^[0-2]$`.
   
   Confirmed the mechanism: `recordRegexCheck` in `RecUtils.cc` does `regex.compile(pattern) && regex.exec(value)`, which is an unanchored search, so `10` matched on the `0`. The anchored form is also the dominant convention in this file (124 entries use `^...$` versus 6 that do not, including a `^[0-1]$`).
   
   For completeness: the looseness was pre-existing rather than introduced here, since the previous value was `[0-1]`, equally unanchored, and it was never a functional bug because the config update handler clamps the value. Two other unanchored `[0-2]` entries remain elsewhere in the file; happy to anchor those too, though it seemed out of scope for this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "serrislew (via GitHub)" <gi...@apache.org>.
serrislew commented on PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#issuecomment-5246815358

   Similar to `proxy.config.http.per_server.connection.max`, could we make this `metric_enabled` feature overridable


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "serrislew (via GitHub)" <gi...@apache.org>.
serrislew commented on code in PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#discussion_r3799622712


##########
src/tsutil/Metrics.cc:
##########
@@ -257,6 +295,26 @@ namespace details
       metrics.push_back(std::move(m));
     }
 
+    void
+    add_source(Metrics::IdType id, Metrics::AtomicType *source, Metrics::Derived::Op op)
+    {
+      if (!source) {
+        return;
+      }
+
+      std::lock_guard l(metrics_lock);
+      auto            it = std::find_if(metrics.begin(), metrics.end(), [id](DerivedMetric const &m) { return m.metric == id; });

Review Comment:
    since `add_source` is O(# of derived metrics) and is called from the Group ctor under the outbound table lock, maybe an id-indexed map would help here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on code in PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#discussion_r3853504531


##########
src/tsutil/Metrics.cc:
##########
@@ -257,6 +295,26 @@ namespace details
       metrics.push_back(std::move(m));
     }
 
+    void
+    add_source(Metrics::IdType id, Metrics::AtomicType *source, Metrics::Derived::Op op)
+    {
+      if (!source) {
+        return;
+      }
+
+      std::lock_guard l(metrics_lock);
+      auto            it = std::find_if(metrics.begin(), metrics.end(), [id](DerivedMetric const &m) { return m.metric == id; });

Review Comment:
   Did some analysis with Claude it with small numbers of sources (dozens) the overhead is tiny.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#issuecomment-5246809302

   [approve ci autest 1]


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "cmcfarlen (via GitHub)" <gi...@apache.org>.
cmcfarlen commented on code in PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#discussion_r3730337223


##########
doc/admin-guide/monitoring/statistics/core/http-connection.en.rst:
##########
@@ -201,6 +201,61 @@ HTTP Connection
    Current number of TCP connections for tunnels where the far end is the server,
    except for those counted by ``proxy.process.tunnel.current_server_connections_tls``
 
+.. _per-server-connection-metrics:
+
+Per Server Connection Metrics
+-----------------------------
+
+Unlike the metrics above these do not have fixed names. They are created dynamically, one set per
+upstream server group as defined by :ts:cv:`proxy.config.http.per_server.connection.match`, and,
+when the match type is ``both``, one aggregate set per hostname. Whether any of them are published,
+and at what granularity, is controlled by
+:ts:cv:`proxy.config.http.per_server.connection.metric_enabled`. An optional
+:ts:cv:`proxy.config.http.per_server.connection.metric_prefix` can be inserted into the names.
+
+Per group names are ``proxy.process.http.per_server.<counter>.<group>``, where ``<group>`` depends on
+the match type: an IP address, an ``address:port`` pair, a hostname, or, for ``both``,
+``<hostname>.<address:port>``. Per hostname names are
+``proxy.process.http.per_server.<counter>.<hostname>``. Aggregates exist only for match type
+``both``, because that is the only match type with more than one group per hostname; for match type
+``host`` the group name is already the bare hostname, so an aggregate would carry the same name as
+the single group it summarises.
+
+For a group, ``<counter>`` is one of:
+
+current_connection
+   Gauge. The number of connections currently open to the group.
+
+total_connection
+   Counter. The total number of connections ever opened to the group. Never decreases.
+
+blocked_connection
+   Counter. The total number of connection attempts to the group blocked by
+   :ts:cv:`proxy.config.http.per_server.connection.max`. Never decreases.
+
+For a hostname aggregate, ``<counter>`` is one of those three, each summed across the groups of that
+hostname, plus:
+
+current_connection_max
+   Gauge. The largest ``current_connection`` value among the groups of that hostname at the moment
+   of sampling, so the maximum rather than the sum of the groups' current counts. This is useful
+   because :ts:cv:`proxy.config.http.per_server.connection.max` is enforced per group rather than
+   per hostname, so the busiest group is what determines whether connections are about to be
+   blocked. Like ``current_connection`` it rises and falls with traffic and is not a high-water
+   mark. There is no per group ``current_connection_max``; it exists only as a hostname aggregate.
+
+The aggregates are recomputed periodically, currently every 5 seconds, rather than on every
+connection event, so a reader sees a value up to that interval old. This applies to
+``current_connection_max`` as well: it reports the maximum across groups as of the last sample, not
+a running peak. To obtain the peak over a longer window, compute a maximum over time from this gauge
+in the monitoring system.

Review Comment:
   Correct, and fixed. The paragraph now says that *every* published per server metric is sampled on that interval, and calls out explicitly that the level 2 per group metrics are mirrored by the same periodic mechanism rather than written as connections open and close.
   
   I also added that `--include-hidden` reads the internal per group metrics directly and so is *not* subject to the sampling delay, which seemed worth stating since it is the difference between the two ways of observing the same underlying value.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Per upstream server connection metrics via hidden and derived metrics [trafficserver]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #13506:
URL: https://github.com/apache/trafficserver/pull/13506#discussion_r3800438642


##########
include/shared/rpc/RPCRequests.h:
##########
@@ -117,6 +117,8 @@ struct ClientRequestNotification : JSONRPCRequest {
 // handy definitions.
 static const std::vector<int> CONFIG_REC_TYPES = {1, 16};
 static const std::vector<int> METRIC_REC_TYPES = {2, 4, 32};
+// Same as METRIC_REC_TYPES, plus 64 (RECT_HIDDEN_METRIC, see RecDefs.h) to also match hidden metrics.
+static const std::vector<int> METRIC_REC_TYPES_INCLUDE_HIDDEN = {2, 4, 32, 64};

Review Comment:
   METRIC_REC_TYPES(_INCLUDE_HIDDEN) does not include RECT_PROCESS (1), but RecLookupMatchingRecords synthesizes ts::Metrics-backed results with rec_type=RECT_PROCESS. That combination makes JSONRPC filtering treat those metrics as a REQUESTED_TYPE_MISMATCH and drop them. Include RECT_PROCESS in the metric request type set so ts::Metrics-backed metrics (including derived/per_server metrics) are returned consistently.



##########
src/records/RecCore.cc:
##########
@@ -612,6 +612,22 @@ RecLookupMatchingRecords(unsigned rec_type, const char *match, void (*callback)(
     });
   }
 
+  if (rec_type & RECT_HIDDEN_METRIC) {
+    // Opt-in only: hidden metrics are never reachable through RECT_ALL, see RecDefs.h.
+    for (auto &&[name, type, val] : ts::Metrics::hidden_instance()) {
+      if (regex.exec(name.data())) {
+        RecRecord tmp;
+
+        tmp.rec_type = RECT_PROCESS;
+
+        tmp.name         = name.data();

Review Comment:
   Hidden metrics are added only when RECT_HIDDEN_METRIC is requested, but the pseudo-record is currently tagged as RECT_PROCESS. The JSONRPC handler filters results by (requested_mask & record->rec_type), so this will be dropped as a type mismatch even when hidden metrics were explicitly requested.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org