You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@trafficserver.apache.org by "masaori335 (via GitHub)" <gi...@apache.org> on 2026/05/18 05:53:21 UTC

[I] Connect Attempts Bugs and Fixes [trafficserver]

masaori335 opened a new issue, #13169:
URL: https://github.com/apache/trafficserver/issues/13169

   # Report of connect attempts bugs
   
   - affected versions: 10.1.x and smaller
   
   Relevant configs:
   
   - `proxy.config.http.connect_attempts_max_retries`
   - `proxy.config.http.connect_attempts_max_retries_suspect_server`
     (replaces the deprecated `proxy.config.http.connect_attempts_max_retries_down_server` —
     see PR #13102)
   - `proxy.config.http.connect_attempts_rr_retries`
   - `proxy.config.http.down_server.cache_time` (the `fail_window`)
   
   PRs:
   
   - https://github.com/apache/trafficserver/pull/12846
   - https://github.com/apache/trafficserver/pull/12880
   - https://github.com/apache/trafficserver/pull/13092
   - https://github.com/apache/trafficserver/pull/13102
   
   ## Net effect between 10.1.x and later
   
   On 10.1.x and earlier, the three `connect_attempts_*` configs behaved erratically:
   
   - `connect_attempts_max_retries_down_server` was effectively unreachable because the `is_down()`
     check was inverted (#12846).
   - Single-address origins ignored down-state entirely — `select_best_http` unconditionally returned
     the first (and only) entry (#12880).
   - The "mark host DOWN" threshold was gated by `connect_attempts_rr_retries` instead of
     `connect_attempts_max_retries` (#13102).
   - The fail-window clock used `client_request_time` instead of the actual failure time, so the window
     could start in the past or be pre-expired (#13102).
   - After `select_next_rr()`, `dst_addr` was not updated, so RR "switches" silently kept hitting the
     same target (#13102).
   - `mark_host_failure()` ran at end-of-txn against the post-switch `dns_info.active`, so failures on
     A could be charged to B (#13102).
   - RR exhaustion terminated retries early even when the current target still had per-host budget
     (#13102).
   
   The combined effect of PRs #12846, #12880, #13092, and #13102 is a coherent model in which each
   config has a well-defined role tied to an explicit `UP / DOWN / SUSPECT` state for the active
   target.
   
   ### Worked examples
   
   All examples use the ATS defaults:
   
   | Config                                          | Default |
   |-------------------------------------------------|---------|
   | `connect_attempts_max_retries`                  | 3       |
   | `connect_attempts_max_retries_suspect_server`   | 1       |
   | `connect_attempts_rr_retries`                   | 3       |
   | `down_server.cache_time` (`fail_window`)        | 60s     |
   
   #### Example 1 — Single-address origin goes offline
   
   Client sends repeated requests to an origin whose only A record points to a down IP. How many
   connect attempts does ATS make before the IP is marked DOWN in HostDB?
   
   **10.1.x behavior — IP marked DOWN after 12 connect attempts (3 failed transactions).**
   
   Why: `mark_host_failure()` is only called from `do_hostdb_update_if_necessary()`, which is only
   reached at the end of a transaction (via `handle_server_connection_not_open`). So `fail_count`
   increments **once per failed transaction**, not once per attempt. The threshold is
   `connect_attempts_rr_retries` (default `3`) — the wrong config, but that's what 10.1.x uses.
   Per-transaction there are `1 + connect_attempts_max_retries = 4` attempts.
   
   | Attempt | Txn | `fail_count` after | marked DOWN? |
   |---------|-----|--------------------|--------------|
   | 1       | 1   | 0                  | no           |
   | 2       | 1   | 0                  | no           |
   | 3       | 1   | 0                  | no           |
   | 4       | 1   | 1 (end of txn)     | no           |
   | 5       | 2   | 1                  | no           |
   | 6       | 2   | 1                  | no           |
   | 7       | 2   | 1                  | no           |
   | 8       | 2   | 2 (end of txn)     | no           |
   | 9       | 3   | 2                  | no           |
   | 10      | 3   | 2                  | no           |
   | 11      | 3   | 2                  | no           |
   | **12**  | 3   | 3 ≥ 3              | **yes**      |
   
   And because `select_best_http()` returns `info[0]` unconditionally for single-address records
   (#12880), every one of those 12 attempts actually hits the network. Even after the IP is marked
   DOWN, subsequent requests keep trying it because the single-address code path ignores down-state.
   
   **Later behavior — IP marked DOWN after 4 connect attempts (1 failed transaction).**
   
   Why: `do_hostdb_update_if_necessary()` is now called after **each** failure in
   `handle_response_from_server()`, so `fail_count` increments **once per attempt**. The threshold is
   `max_retries + 1 = 4` (for an UP target). Per-transaction there are still `1 + max_retries = 4`
   attempts, so the threshold is reached on the last attempt of the first failed transaction.
   
   | Attempt | Txn | `fail_count` after | marked DOWN? |
   |---------|-----|--------------------|--------------|
   | 1       | 1   | 1                  | no           |
   | 2       | 1   | 2                  | no           |
   | 3       | 1   | 3                  | no           |
   | **4**   | 1   | 4 ≥ 4              | **yes**      |
   
   Subsequent requests within `fail_window` (60s): `select_best_http()` now honors down-state for
   single-address records, so HostDB lookup returns `nullptr` and the SM reports `OriginDown`
   immediately — **zero connect attempts**.
   
   After `fail_window` elapses, the entry is SUSPECT. The next request gets
   `connect_attempts_max_retries_suspect_server + 1 = 2` attempts (1 probe + 1 retry). Success → UP;
   failure → back to DOWN for another 60s.
   
   #### Example 2 — Round-robin with one bad backend (A fails, B/C healthy)
   
   Defaults apply: `rr_retries = 3`, `max_retries = 3`. A is unresponsive, B and C are fine. Initial
   active = A.
   
   **10.1.x behavior — client gets 502, and the wrong host is blamed.**
   
   Two compounding bugs: (a) after `select_next_rr()` is called, neither `dns_info.addr` nor
   `server_info.dst_addr` is updated, so the next connect still goes to A's address. (b)
   `mark_host_failure()` is called once at end-of-txn against `dns_info.active`, which by that point is
   the *post-switch* host (B), so B's `fail_count` is incremented for A's failures.
   
   | Attempt | dst_addr (target) | dns_info.active | retry_attempts after | `(retry+1) % 3 == 0`? | Result |
   |---------|-------------------|-----------------|----------------------|----------------------|--------|
   | 1       | A                 | A               | 1                    | no                   | fail   |
   | 2       | A                 | A               | 2                    | no                   | fail   |
   | 3       | A                 | A → switched to B (`select_next_rr`); dst_addr untouched | 3 | **yes** | fail |
   | 4       | A (still!)        | B               | 3 ≥ 3, give up       | n/a                  | fail → 502 to client |
   
   End of txn: `mark_host_failure()` increments `B.fail_count` even though B was never tried. Repeated
   requests do this 3 times → **B gets marked DOWN** while A (the actually-broken host) keeps getting
   hit on every transaction.
   
   **Later behavior — client gets 200; A's failures are correctly recorded against A.**
   
   `do_hostdb_update_if_necessary()` runs after each failure (so the correct `dns_info.active` is in
   scope), and after `select_next_rr()` the SM explicitly assigns `dns_info.addr` and
   `server_info.dst_addr` to the new target.
   
   | Attempt | dst_addr (target) | dns_info.active | A.fail_count after | retry_attempts after | RR switch? | Result |
   |---------|-------------------|-----------------|--------------------|----------------------|------------|--------|
   | 1       | A                 | A               | 1                  | 1                    | no         | fail   |
   | 2       | A                 | A               | 2                  | 2                    | no         | fail   |
   | 3       | A                 | A               | 3                  | 3                    | **yes** → active=B, dst_addr=B | fail |
   | **4**   | **B**             | B               | 3 (unchanged)      | n/a                  | n/a        | **success → 200** |
   
   A finishes this transaction with `fail_count = 3` (not yet at the threshold of
   `max_retries + 1 = 4`). The next request that hashes to A bumps it to 4 and marks A DOWN; subsequent
   requests within `fail_window` skip A entirely until the window expires.
   
   If B and C had also been DOWN (no selectable alternate), `select_next_rr()` returns false and the SM
   stays on A using its remaining per-host retry budget instead of giving up — new in #13102.
   
   **Later behavior**
   
   - Failure timestamp is `ts_clock::now()` at the moment `mark_down` is called, so the full
     `fail_window` is honored regardless of how long the transaction took to decide the host was bad.
   
   ## Summary of bug-fix PRs
   
   ### PR #12846 — Fix `HostDBInfo::is_down` condition
   
   **Bug.** `HostDBInfo::is_down()` had an inverted comparison:
   
   ```cpp
   // Before
   return (last_fail != TS_TIME_ZERO) && (last_fail + fail_window < now);
   // After
   return (last_fail != TS_TIME_ZERO) && (now <= last_fail + fail_window);
   ```
   
   The pre-fix code reported a host as DOWN only *after* `fail_window` had elapsed — the opposite of
   intent. During the fail window (when the host should be considered blocked) it returned `false`.
   
   **Behavior change.** A failed host is now correctly treated as DOWN for the entire `fail_window`
   following `last_failure`, and returns to UP only after the window has elapsed. Every downstream path
   that calls `is_down()` (retry budget, RR selection, negative-cached check) is impacted.
   
   ### PR #12880 — Check state of HostDBInfo in `select_best_http`
   
   **Bug.** For single-address (non round-robin) records, `HostDBRecord::select_best_http()` bypassed
   the state check:
   
   ```cpp
   // Before
   best_alive = &info[0];                       // unconditional
   // After
   if (info[0].select(now, fail_window)) {
     best_alive = &info[0];
   }
   ```
   
   A single-address host that was marked down would still be returned from HostDB and reused for the
   next request.
   
   **Behavior change.** Single-address origins now honor `down_server` state the same way RR origins
   do: if the only target is DOWN and still within the fail window, HostDB lookup fails (`nullptr`),
   the SM reports `OriginDown`, and no connection attempt is made until the window expires. The gold
   autest `dns_host_down` was updated accordingly (second request returns 500 instead of 502 once the
   first failure marked the IP down).
   
   ### PR #13092 — Clarify `HostDBInfo` state
   
   **Not a bug fix — a model/refactor.** Introduces an explicit tri-state for upstream health and
   reshapes the API around it:
   
   ```cpp
   enum class HostDBInfo::State { UP, DOWN, SUSPECT };
   ```
   
   - **UP** — no known failure; normal selection.
   - **DOWN** — `_last_failure` set, `now` is within `fail_window`; not eligible.
   - **SUSPECT** — `fail_window` has elapsed; a probe is permitted. A successful response transitions
     it to UP via `mark_up()`; another failure returns it to DOWN.
   
   API renames: `is_alive` → `is_up`, `mark_active_server_alive` → `mark_active_server_up`.
   `mark_down` now takes `fail_window`. `select()` is replaced by callers using `is_down()` directly.
   `last_failure` / `fail_count` become private atomics (`_last_failure`, `_fail_count`).
   
   **Behavior change.** The previous two-state model collapsed SUSPECT into UP once `fail_window`
   expired, which meant a recovering host was treated identically to a never-failed host for retry
   sizing. With SUSPECT explicit, callers (notably the retry machinery in #13102) can apply
   `connect_attempts_max_retries_suspect_server` to probing traffic and `connect_attempts_max_retries`
   to UP traffic.
   
   ### PR #13102 — Fix connect attempt retries
   
   **Bugs.** The retry machinery in `HttpTransact::handle_response_from_server()` and
   `HttpSM::mark_host_failure()` had several interlocking issues:
   
   1. **Wrong threshold used to mark host down.** `mark_host_failure()` called
      `increment_fail_count(..., connect_attempts_rr_retries, ...)`. The RR switch-over config was
      reused as the "mark down" threshold, so a host was marked DOWN after
      `connect_attempts_rr_retries` failures instead of `connect_attempts_max_retries + 1` total
      attempts.
   2. **Wrong clock for failure timestamp.** `do_hostdb_update_if_necessary()` recorded the failure at
      `t_state.client_request_time` (request-receipt time), not `ts_clock::now()`. For long requests
      the fail window started in the past, sometimes expiring before the failure was even recorded.
   3. **RR exhaustion short-circuited all retries.** When the `connect_attempts_rr_retries` boundary
      was hit and no other RR member was selectable, the old code gave up, even though the current
      target still had per-host retry budget.
   4. **`connect_attempts_max_retries_down_server` had no effect in practice.** It was only consulted
      through `is_server_negative_cached()`, which depended on the inverted `is_down()` from #12846. In
      many cases it was either never applied or applied when it shouldn't have been.
   5. **`connect_attempts_max_retries_down_server == 0` blocked the probe.** A separate code path in
      `do_http_server_open()` refused to connect at all when the target was negative-cached and the
      config was zero — preventing the SUSPECT-state probe that the fail window was designed to
      allow.
   6. **Retry config range allowed overflow.** `[0-255]` with `uint8_t` threshold = `max_retries + 1`
      wraps to 0.
   7. **Config name didn't match its actual effect.** The "down_server" config never applied to
      DOWN-state origins (DOWN-state retries are now hardcoded to 0); it controls the probe budget for
      SUSPECT-state origins. Renamed to `proxy.config.http.connect_attempts_max_retries_suspect_server`
      to align with `HostDBInfo::State::SUSPECT`. The previous
      `proxy.config.http.connect_attempts_max_retries_down_server` record is kept and marked
      deprecated; if only it is set, its value is mirrored forward into the new record with a warning,
      and if both are set the new record wins. The legacy
      `TS_CONFIG_HTTP_CONNECT_ATTEMPTS_MAX_RETRIES_DOWN_SERVER` plugin enum continues to work — it
      aliases to the same internal field as the new `_SUSPECT_SERVER` enum.
   
   **Behavior change.**
   
   - Retry budget is now driven by HostDB state (via the new helper
     `HttpTransact::origin_server_connect_attempts_max_retries`):
     - `UP`      → `connect_attempts_max_retries`
     - `DOWN`    → 0 (bail out immediately)
     - `SUSPECT` → `connect_attempts_max_retries_suspect_server`
   - `mark_host_failure` uses `max_retries + 1` as the attempt budget and `ts_clock::now()` as the
     failure time.
   - RR switch and per-host retry are separated: if RR has no selectable alternate, the SM stays on
     the current target and keeps retrying up to the per-host budget rather than giving up.
   - The `do_http_server_open()` early-bail on "negative cached + down_server=0" is removed — DOWN vs
     SUSPECT is now the single source of truth.
   - `proxy.config.http.connect_attempts_max_retries`, `…_max_retries_suspect_server`, and
     `…_rr_retries` clamped to `[0-254]` to avoid the `uint8_t` overflow when adding 1 for the
     total-attempt count.
   - A new warning is logged at config reconfigure when
     `connect_attempts_max_retries_suspect_server == 0` and `connect_attempts_rr_retries > 0`, because
     that combination prevents any probe of a recovering (SUSPECT) origin.
   - `proxy.config.http.connect_attempts_max_retries_down_server` is now a deprecated alias of
     `…_suspect_server`. The struct field
     `OverridableHttpConfigParams::connect_attempts_max_retries_down_server` was removed; both
     records.yaml entries and both `TSOverridableConfigKey` enum values write into a single field, so
     existing operator configs and plugins keep working while the codebase has one source of truth.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ☂️ Connect Attempts Bugs and Fixes [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall closed issue #13169: ☂️ Connect Attempts Bugs and Fixes
URL: https://github.com/apache/trafficserver/issues/13169


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ☂️ Connect Attempts Bugs and Fixes [trafficserver]

Posted by "masaori335 (via GitHub)" <gi...@apache.org>.
masaori335 commented on issue #13169:
URL: https://github.com/apache/trafficserver/issues/13169#issuecomment-4530469672

   This is an umbrella issue to track PRs, just waiting open PRs are merged to go next step. Nothing to do now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] ☂️ Connect Attempts Bugs and Fixes [trafficserver]

Posted by "bryancall (via GitHub)" <gi...@apache.org>.
bryancall commented on issue #13169:
URL: https://github.com/apache/trafficserver/issues/13169#issuecomment-4828331417

   All four PRs tracked by this umbrella are now merged and milestoned for the 10.2.0 release: #12846 (fix HostDBInfo::is_down condition), #12880 (check HostDBInfo state in select_best_http), #13092 (clarify HostDBInfo UP/DOWN/SUSPECT state), and #13102 (fix connect attempt retries). #13102, the last open one, merged on 2026-06-16, which satisfies the only remaining condition noted here ("just waiting open PRs are merged"). I confirmed on master that #13102 landed and that the new proxy.config.http.connect_attempts_max_retries_suspect_server configuration is in place. Since the tracking purpose is complete, I am closing this. First release: 10.2.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Connect Attempts Bugs and Fixes [trafficserver]

Posted by "Arihant-Golchha (via GitHub)" <gi...@apache.org>.
Arihant-Golchha commented on issue #13169:
URL: https://github.com/apache/trafficserver/issues/13169#issuecomment-4528374484

   hey @masaori335 i was looking forward to contribut in this repo i would love to know if this issue is still open
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@trafficserver.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org