You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by kr...@apache.org on 2026/10/25 15:35:12 UTC

(iggy) 01/01: fix(gateways): stop Kafka Produce sends at the request deadline

This is an automated email from the ASF dual-hosted git repository.

krishvishal pushed a commit to branch kafka-produce-deadline
in repository https://gitbox.apache.org/repos/asf/iggy.git

commit 8eb61d127303dd7547a7c95b504c78dcdeaa5284
Author: Krishna Vishal <kr...@gmail.com>
AuthorDate: Sun Oct 25 18:08:44 2026 +0530

    fix(gateways): stop Kafka Produce sends at the request deadline
    
    A send that outlived its request landed later, so a client retry stored the batch twice; failed sends that wrote nothing answer 6, not -1.
---
 Cargo.lock                                        |   1 +
 gateways/kafka/Cargo.toml                         |   3 +
 gateways/kafka/README.md                          |  16 ++-
 gateways/kafka/docs/TEST_SUITE.md                 |   2 +-
 gateways/kafka/src/bridge/error.rs                |  57 +++++++++
 gateways/kafka/src/bridge/iggy_bridge/produce.rs  |  23 +++-
 gateways/kafka/src/protocol/handlers/produce.rs   |  20 ++-
 gateways/kafka/tests/produce_real_bridge_tests.rs | 148 +++++++++++++++++++++-
 8 files changed, 254 insertions(+), 16 deletions(-)

diff --git a/Cargo.lock b/Cargo.lock
index ad0b60756..a056a50b6 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -7290,6 +7290,7 @@ dependencies = [
  "flate2",
  "futures",
  "iggy",
+ "iggy_common",
  "kafka-protocol",
  "libc",
  "lz4",
diff --git a/gateways/kafka/Cargo.toml b/gateways/kafka/Cargo.toml
index bd4afe645..45eaa4962 100644
--- a/gateways/kafka/Cargo.toml
+++ b/gateways/kafka/Cargo.toml
@@ -44,6 +44,9 @@ bytes = { workspace = true }
 flate2 = { workspace = true }
 futures = { workspace = true }
 iggy = { workspace = true }
+# `with_binary_deadline`, which hands a Produce request's deadline to the SDK. The prelude does not
+# export it.
+iggy_common = { workspace = true }
 # The codec features stay on for the encoder, which the batch tests use to build fixtures.
 kafka-protocol = { workspace = true, features = ["gzip", "lz4", "snappy", "zstd"] }
 libc = { workspace = true }
diff --git a/gateways/kafka/README.md b/gateways/kafka/README.md
index 692aa6a2c..fafa983c4 100644
--- a/gateways/kafka/README.md
+++ b/gateways/kafka/README.md
@@ -474,7 +474,7 @@ One partition, one Iggy send. Each partition answers for itself.
 | `acks` | `0`, `1`, `-1` write the same. Other values: 21, before any per-partition check. |
 | `acks=0` | Writes, answers nothing. Any failed partition closes the connection. |
 | Topics | Creates none. Missing: 3. Bad name: 17. |
-| `timeout_ms` | Honored, max 20 s. Past it: 7. |
+| `timeout_ms` | Honored, max 20 s. The Iggy send stops by then too. Past it: 7. A send that never went out: 6. |
 | Compression | gzip, snappy, lz4. zstd from v7, else 76. |
 | Producer id | Picks the producer's Iggy session. See [Concurrency ceiling](#concurrency-ceiling). |
 | Producer epoch, sequence | Ignored, so a retry writes twice. |
@@ -492,8 +492,8 @@ One partition, one Iggy send. Each partition answers for itself.
 | 10 | Record, send or partition too large, even alone | Java splits multi-record batches. Else fails. |
 | 87 | Record the gateway cannot map. Reason in `error_message` from v8. | Fails. |
 | 35 | Transactional or control batch | Fails. |
-| 6 | Request budget ran out. Nothing written. | Retries. |
-| 7 | Deadline passed: in the send, waiting for decode memory, or behind the producer's own earlier send. Or connection lost mid-send. May be written. | Retries. Can duplicate. |
+| 6 | Request budget ran out. Or the send did not reach Iggy by the deadline, or Iggy refused it: no primary, partition busy or recreated, client shut down. Nothing written. | Retries. |
+| 7 | Deadline passed: after the send went out, waiting for decode memory, or behind the producer's own earlier send. Or connection lost mid-send. Possibly written. | Retries. Can duplicate. |
 
 Partition cap: `max_frame_size` decompressed bytes, `max_frame_size / 64` record slots, 3 headers
 per slot. Past it: 10.
@@ -605,8 +605,10 @@ Iggy becomes unreachable later.
 `send_records` uses the Produce deadline instead:
 
 - One request of a producer runs at a time, on that producer's client.
-- A send past its deadline keeps the producer's turn until it ends. That producer's requests answer
-  7 meanwhile. Other producers send on.
+- The SDK stops retrying the send 50 ms before the deadline. So it sends nothing after the request
+  answered, and the client's retry is the only copy of a send that never went out.
+- A send still running past its deadline keeps the producer's turn until it ends. That producer's
+  requests answer 7 meanwhile. Other producers send on.
 - After 45 s the gateway gives up on the send. It then closes that connection before the turn
   frees, so the send cannot land after a later one. The SDK ends its write within 30 s more,
   unless Iggy stopped reading: the producer then waits until that connection breaks.
@@ -816,6 +818,10 @@ instead of -1, because the Java `AdminClient` fails a partition on any non-retri
 incarnation of the partition, as after a delete and recreate; the retry probes again and answers 3
 if the topic is gone. ListOffsets reads do not get `RequestTooOld` today, so that arm is defensive.
 
+Produce also answers `HistoryUnavailable`, `LifecycleBusy` and `ClientShutdown` with 6 instead of
+-1 (`to_produce_error_code`). The SDK returns them only for a send that wrote nothing. A send that
+possibly landed ends as `OperationUncertain`, which answers 7.
+
 ### Server limits the gateway inherits
 
 These are Iggy server limits, not gateway settings. A Kafka client cannot act on any of them, so
diff --git a/gateways/kafka/docs/TEST_SUITE.md b/gateways/kafka/docs/TEST_SUITE.md
index b32f374b0..3229839ec 100644
--- a/gateways/kafka/docs/TEST_SUITE.md
+++ b/gateways/kafka/docs/TEST_SUITE.md
@@ -83,7 +83,7 @@ file under `tests/` anymore.
 | [`bridge_iggy_integration_tests.rs`](../tests/bridge_iggy_integration_tests.rs) | `IggyBridge` against a real, spawned `iggy-server` — provisioning idempotency, high watermark, credential/connection edge cases, and the bridge dialing a TLS-only server over TLS | No (needs the `iggy-server` binary - see Prerequisites) |
 | [`list_offsets_real_bridge_tests.rs`](../tests/list_offsets_real_bridge_tests.rs) | ListOffsets (key 2) through the whole handler against a real, spawned `iggy-server`: LATEST, EARLIEST, lookups by time and `MAX_TIMESTAMP`, the `-1` answers, code 3, several partitions looked up by time in one request, and a stalled lookup leaving the later partitions their answers | No (needs the `iggy-server` binary - see Prerequisites) |
 | [`fetch_real_bridge_tests.rs`](../tests/fetch_real_bridge_tests.rs) | Fetch (key 1) through the whole handler against a real, spawned `iggy-server` — records go in through the Iggy SDK or the Produce handler and come back through `kafka_protocol`'s client decoder, at every version, with codes 0, 1, 3, 6, -1 and 70, paged polls, and the wait | No (needs the `iggy-server` binary - see Prerequisites) |
-| [`produce_real_bridge_tests.rs`](../tests/produce_real_bridge_tests.rs) | Produce (key 0) through the whole handler against a real, spawned `iggy-server` — records go in as Kafka wire bytes and come back through the Iggy SDK, plus one error code per partition | No (needs the `iggy-server` binary - see Prerequisites) |
+| [`produce_real_bridge_tests.rs`](../tests/produce_real_bridge_tests.rs) | Produce (key 0) through the whole handler against a real, spawned `iggy-server` — records go in as Kafka wire bytes and come back through the Iggy SDK, plus one error code per partition. Through a relay that cuts Iggy off, a send that misses its deadline answers 6, and the client's retry stores the batch once | No (needs the `iggy-server` binary - see Prerequisites) |
 | [`create_partitions_real_bridge_tests.rs`](../tests/create_partitions_real_bridge_tests.rs) | CreatePartitions (key 37) through the whole handler against a real, spawned `iggy-server`: a grow appends indices and shows in Metadata's listing, the grow-only rule (37), a missing topic (3), `validate_only`, duplicate names (42), the manual assignment checks (39), the per-request and topic caps, the `manage_topics` gate (29), a duplicate answered before authorization (42, then 29 for the res [...]
 | [`offsets_real_bridge_tests.rs`](../tests/offsets_real_bridge_tests.rs) | OffsetCommit (key 8) and OffsetFetch (key 9) through the whole handler against a real, spawned `iggy-server`: offsets stored as sent, the Iggy external group key, a negative commit as a delete, -1 for nothing committed, a null topic list, and a gateway restart ([#3542](https://github.com/apache/iggy/issues/3542) acceptance) | No (needs the `iggy-server` binary - see Prerequisites) |
 | [`offset_groups_real_bridge_tests.rs`](../tests/offset_groups_real_bridge_tests.rs) | ListGroups (key 16), DescribeGroups (key 15), DeleteGroups (key 42) and OffsetDelete (key 47) through the whole handler against a real, spawned `iggy-server`. A group with offsets and no members is `Empty`, and a live group is listed once. Also the filters, a commit after the last scan, a new gateway, a native Iggy group, and only the live groups once a scan failed with Iggy down. DeleteGroups refuses [...]
diff --git a/gateways/kafka/src/bridge/error.rs b/gateways/kafka/src/bridge/error.rs
index f35bd6884..08700cb23 100644
--- a/gateways/kafka/src/bridge/error.rs
+++ b/gateways/kafka/src/bridge/error.rs
@@ -196,6 +196,22 @@ impl BridgeError {
         }
     }
 
+    /// [`Self::to_kafka_error_code`] for Produce. The SDK returns `HistoryUnavailable`,
+    /// `LifecycleBusy` and `ClientShutdown` only for a send that wrote nothing, because a send
+    /// that may have landed ends as `OperationUncertain`. So these answer 6, which clients retry,
+    /// not -1, which loses the records.
+    #[must_use]
+    pub const fn to_produce_error_code(&self) -> i16 {
+        match self {
+            Self::Iggy(
+                IggyError::HistoryUnavailable
+                | IggyError::LifecycleBusy
+                | IggyError::ClientShutdown,
+            ) => ERROR_NOT_LEADER_OR_FOLLOWER,
+            _ => self.to_kafka_error_code(),
+        }
+    }
+
     /// True when Iggy rejected the bridge's own login. The Kafka client only sees -1.
     #[must_use]
     pub const fn is_bridge_login_rejected(&self) -> bool {
@@ -408,6 +424,47 @@ mod tests {
         }
     }
 
+    #[test]
+    fn given_a_send_that_wrote_nothing_when_mapped_for_produce_should_answer_retriable() {
+        let wrote_nothing = [
+            IggyError::HistoryUnavailable,
+            IggyError::LifecycleBusy,
+            IggyError::ClientShutdown,
+        ];
+        for error in wrote_nothing {
+            let err = BridgeError::Iggy(error);
+            assert_eq!(
+                err.to_kafka_error_code(),
+                ERROR_UNKNOWN_SERVER_ERROR,
+                "{err}"
+            );
+            assert_eq!(
+                err.to_produce_error_code(),
+                ERROR_NOT_LEADER_OR_FOLLOWER,
+                "{err}"
+            );
+        }
+    }
+
+    #[test]
+    fn given_a_send_that_may_have_landed_when_mapped_for_produce_should_keep_the_shared_code() {
+        let may_have_landed = [
+            (
+                BridgeError::Iggy(IggyError::OperationUncertain(1)),
+                ERROR_REQUEST_TIMED_OUT,
+            ),
+            (BridgeError::Timeout, ERROR_REQUEST_TIMED_OUT),
+            // A retry after the client was evicted could write it twice.
+            (
+                BridgeError::Iggy(IggyError::RequestTooOld),
+                ERROR_UNKNOWN_SERVER_ERROR,
+            ),
+        ];
+        for (err, code) in may_have_landed {
+            assert_eq!(err.to_produce_error_code(), code, "{err}");
+        }
+    }
+
     #[test]
     fn given_any_other_failure_when_mapped_for_list_offsets_should_keep_the_shared_code() {
         let others = [
diff --git a/gateways/kafka/src/bridge/iggy_bridge/produce.rs b/gateways/kafka/src/bridge/iggy_bridge/produce.rs
index d5dc03266..196692465 100644
--- a/gateways/kafka/src/bridge/iggy_bridge/produce.rs
+++ b/gateways/kafka/src/bridge/iggy_bridge/produce.rs
@@ -27,6 +27,7 @@ use std::time::Duration;
 
 use futures::future::join_all;
 use iggy::prelude::{IggyClient, IggyError, IggyMessage, MessageClient, Partitioning};
+use iggy_common::{binary_operation_deadline, with_binary_deadline};
 use tokio::runtime::Handle;
 use tokio::sync::{OwnedSemaphorePermit, Semaphore, oneshot};
 use tokio::time::{Instant, MissedTickBehavior, interval, sleep_until, timeout, timeout_at};
@@ -39,6 +40,10 @@ use crate::bridge::error::BridgeError;
 /// Frame bytes besides the messages: request header, batch header and ids.
 const SEND_FRAME_ROOM: u64 = 4 * 1024;
 
+/// How much sooner than its request the SDK stops a send. The SDK's own answer then tells whether
+/// the send went out, before the request gives up on it with 7.
+const SDK_STOP_ROOM: Duration = Duration::from_millis(50);
+
 /// Sessions that connect and sign in at once. A sign-in costs Iggy a password hash on a shard
 /// thread, so a burst of new producers must not start them all together.
 const SESSION_OPENS: usize = 4;
@@ -374,6 +379,9 @@ impl IggyBridge {
     /// `Ok(None)`: committed, but the server named no offset (a duplicate request, or a journal
     /// entry that is gone). Do not retry it.
     ///
+    /// The SDK stops the send by `deadline`, so no attempt of it starts after the request
+    /// answered, and a client retry is the only copy of a send that never went out.
+    ///
     /// A send still running at `deadline` keeps `turn` and `kept`, and hands back `None` for
     /// them. So the producer's later requests answer 7 meanwhile and cannot overtake it, and what
     /// `kept` reserves for `messages` stays reserved. Other producers send on.
@@ -386,7 +394,8 @@ impl IggyBridge {
     /// [`BridgeError::Timeout`] if `deadline` passes. The send may have landed.
     /// [`BridgeError::SendLost`] if the connection broke during the send. It may have landed.
     /// [`BridgeError::Iggy`] if the session cannot connect, if the stream, topic or partition is
-    /// missing, or if the send is too large.
+    /// missing, or if the send is too large. Also when the SDK stopped the send at `deadline`:
+    /// `OperationUncertain` if it went out, so it may have landed, and its refusal if it did not.
     pub async fn send_records<K: Send + 'static>(
         &self,
         turn: ProduceTurn,
@@ -408,7 +417,13 @@ impl IggyBridge {
             return (Err(BridgeError::Timeout), Some((turn, kept)));
         }
         let (stream_id, topic_id) = (target.stream_id.clone(), target.topic_id.clone());
-        let recovery_deadline = (Instant::now() + SLOT_LIMIT).into_std();
+        // With less than the room left, the SDK keeps the whole deadline, and the send answers 7.
+        // Never past the SDK's own limit, which `SLOT_LIMIT` counts on.
+        let sdk_deadline = deadline
+            .checked_sub(SDK_STOP_ROOM)
+            .filter(|stop| *stop > Instant::now())
+            .unwrap_or(deadline)
+            .min(binary_operation_deadline());
         let send = async move {
             // Owns the messages, so a send cut off at the limit drops them.
             let send = {
@@ -421,13 +436,15 @@ impl IggyBridge {
                             .await
                         {
                             Err(IggyError::LaneBlocked(operation)) => {
-                                resolve_blocker(&client, operation, recovery_deadline).await?;
+                                resolve_blocker(&client, operation, sdk_deadline.into_std())
+                                    .await?;
                             }
                             result => return result,
                         }
                     }
                 }
             };
+            let send = with_binary_deadline(sdk_deadline, send);
             within_slot_limit(&session, &client, kept, send).await
         };
         // The SDK cannot cancel a send. One that outlives this call keeps the turn in its task,
diff --git a/gateways/kafka/src/protocol/handlers/produce.rs b/gateways/kafka/src/protocol/handlers/produce.rs
index d006c9b4a..4b09aa8ff 100644
--- a/gateways/kafka/src/protocol/handlers/produce.rs
+++ b/gateways/kafka/src/protocol/handlers/produce.rs
@@ -517,10 +517,11 @@ fn request_timeout(timeout_ms: i32) -> Duration {
         .min(REQUEST_DEADLINE)
 }
 
-/// Sends one partition's messages, or answers 7 once the deadline has passed. Hands `held` back,
+/// Sends one partition's messages, and gives up on them at the deadline. Hands `held` back,
 /// unless the send still runs and keeps it.
 ///
-/// 7 means nothing was written, or the send may still land and a retry duplicate it.
+/// 7 means nothing was written, or the send may still land and a retry duplicate it. 6 means that
+/// it never went out.
 async fn send_by(
     bridge: &IggyBridge,
     (turn, admission): Held,
@@ -535,22 +536,31 @@ async fn send_by(
         .await;
     let sent = match sent {
         Ok(base_offset) => Ok(base_offset),
-        Err(BridgeError::Timeout) => {
+        // The SDK retries a send until the deadline, and then ends it with one of these two: not
+        // sent, or in doubt.
+        Err(
+            error @ (BridgeError::Timeout
+            | BridgeError::Iggy(
+                IggyError::TransientNotAccepted | IggyError::OperationUncertain(_),
+            )),
+        ) => {
             // The client picks `timeout_ms`, so its expiry is no signal to the operator.
             if deadline.ceiling {
                 tracing::warn!(
+                    %error,
                     kafka_topic,
                     partition_id,
                     "Produce ran past the gateway deadline"
                 );
             } else {
                 tracing::debug!(
+                    %error,
                     kafka_topic,
                     partition_id,
                     "Produce ran past the client deadline"
                 );
             }
-            Err(ERROR_REQUEST_TIMED_OUT)
+            Err(error.to_produce_error_code())
         }
         Err(error) => Err(refusal_code(kafka_topic, &error)),
     };
@@ -559,7 +569,7 @@ async fn send_by(
 
 /// The Kafka code for a bridge refusal, logged at a level that matches who can fix it.
 fn refusal_code(kafka_topic: &str, error: &BridgeError) -> i16 {
-    let code = error.to_kafka_error_code();
+    let code = error.to_produce_error_code();
     if error.is_bridge_login_rejected() {
         tracing::error!(%error, kafka_topic, "Iggy rejected the bridge's credentials");
     } else if matches!(code, ERROR_UNKNOWN_SERVER_ERROR) {
diff --git a/gateways/kafka/tests/produce_real_bridge_tests.rs b/gateways/kafka/tests/produce_real_bridge_tests.rs
index 3a70c1825..8b09ae2f6 100644
--- a/gateways/kafka/tests/produce_real_bridge_tests.rs
+++ b/gateways/kafka/tests/produce_real_bridge_tests.rs
@@ -21,6 +21,7 @@
 //! the same way in both directions cannot pass.
 
 use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
 use std::time::Duration;
 
 use bytes::{Bytes, BytesMut};
@@ -35,9 +36,11 @@ use kafka_protocol::records::{
     RecordBatchEncoder, RecordEncodeOptions, TimestampType,
 };
 use serial_test::serial;
+use tokio::net::{TcpListener, TcpStream};
+use tokio::sync::watch;
 use tokio_util::sync::CancellationToken;
 
-use iggy_gateway_kafka::bridge::{IggyBridge, Producer};
+use iggy_gateway_kafka::bridge::{IggyBridge, IggyBridgeConfig, Producer};
 use iggy_gateway_kafka::group::{GroupCoordinator, GroupCoordinatorConfig};
 use iggy_gateway_kafka::protocol::api::{
     API_KEY_INIT_PRODUCER_ID, API_KEY_PRODUCE, ERROR_NONE, ERROR_NOT_LEADER_OR_FOLLOWER,
@@ -68,7 +71,12 @@ const ERROR_INVALID_RECORD: i16 = 87;
 
 /// A live gateway with the topic already provisioned, since Produce creates nothing.
 async fn gateway_with_topic(server: &TestServer, partitions: u32) -> GatewayState {
-    let bridge = IggyBridge::connect(server.test_config())
+    gateway_on(server.test_config(), partitions).await
+}
+
+/// [`gateway_with_topic`], with a bridge that dials as `config` says.
+async fn gateway_on(config: IggyBridgeConfig, partitions: u32) -> GatewayState {
+    let bridge = IggyBridge::connect(config)
         .await
         .expect("bridge should connect to a ready server");
     bridge
@@ -664,6 +672,142 @@ async fn given_a_stopped_server_when_handled_should_answer_retriable() {
     );
 }
 
+/// The SDK stops a send that cannot reach Iggy by its request's deadline. Otherwise the send lands
+/// once Iggy is back, after the client got its answer, and the client's retry writes the batch a
+/// second time.
+#[tokio::test]
+#[serial]
+async fn given_iggy_out_of_reach_past_the_deadline_when_the_client_retries_should_store_the_batch_once()
+ {
+    let data_dir = tempfile::tempdir().expect("tempdir");
+    let server = TestServer::spawn(data_dir.path()).await;
+    let relay = OutageRelay::start(&server).await;
+    let mut config = server.test_config();
+    config.address.clone_from(&relay.address);
+    let state = gateway_on(config, 1).await;
+    // Each request here comes on a new connection, so only a producer id keeps one Iggy session.
+    let id = issued_producer_id(&state).await;
+    let first = [(0, idempotent_batch(id, b"first"))];
+    assert_eq!(
+        produce(&state, 3, 1, TOPIC, &first).await,
+        vec![(0, ERROR_NONE, 0)],
+        "the producer's Iggy session is open"
+    );
+
+    relay.go_down().await;
+    let retried = [(0, idempotent_batch(id, b"retried"))];
+    let body = encode_timed_request(3, 1, 1_000, TOPIC, &retried);
+    let response = handle_request_bounded(&state, API_KEY_PRODUCE, 3, body)
+        .await
+        .expect_response("acks=1 is answered");
+    assert_eq!(
+        partition_results(3, response),
+        vec![(0, ERROR_NOT_LEADER_OR_FOLLOWER, -1)],
+        "the send never went out, so the answer tells the client that a retry is safe"
+    );
+
+    relay.come_up();
+    tokio::time::sleep(SDK_RECONNECT).await;
+    assert_eq!(
+        produce(&state, 3, 1, TOPIC, &retried).await,
+        vec![(0, ERROR_NONE, 1)],
+        "the client's retry lands right after the first batch"
+    );
+    let payloads: Vec<Bytes> = stored(&server, 0, 10)
+        .await
+        .into_iter()
+        .map(|message| message.payload)
+        .collect();
+    assert_eq!(
+        payloads,
+        [&b"first"[..], b"retried"],
+        "one copy of each batch"
+    );
+}
+
+/// Longer than the SDK takes to connect again and send what it still holds.
+const SDK_RECONNECT: Duration = Duration::from_secs(3);
+
+/// A TCP relay to a server that can go down. Down, it cuts its connections, and holds the ones that
+/// open until it is up again. So a client behind it cannot send, and does not learn why.
+struct OutageRelay {
+    address: String,
+    up: watch::Sender<bool>,
+    connections: Arc<Connections>,
+}
+
+#[derive(Default)]
+struct Connections {
+    accepted: AtomicUsize,
+    open: AtomicUsize,
+}
+
+impl OutageRelay {
+    async fn start(server: &TestServer) -> Self {
+        let listener = TcpListener::bind("127.0.0.1:0")
+            .await
+            .expect("bind the relay");
+        let address = listener.local_addr().expect("relay address").to_string();
+        let upstream = server.test_config().address;
+        let (up, _) = watch::channel(true);
+        let connections = Arc::new(Connections::default());
+        let (switch, counts) = (up.clone(), Arc::clone(&connections));
+        tokio::spawn(async move {
+            while let Ok((inbound, _)) = listener.accept().await {
+                counts.accepted.fetch_add(1, Ordering::SeqCst);
+                counts.open.fetch_add(1, Ordering::SeqCst);
+                let (upstream, up, counts) =
+                    (upstream.clone(), switch.subscribe(), Arc::clone(&counts));
+                tokio::spawn(async move {
+                    relay_while_up(inbound, &upstream, up).await;
+                    counts.open.fetch_sub(1, Ordering::SeqCst);
+                });
+            }
+        });
+        Self {
+            address,
+            up,
+            connections,
+        }
+    }
+
+    /// Goes down, and returns once each client it cut off dialed again. A client that has not
+    /// seen the cut yet can still write a request into its old connection, and then cannot know
+    /// whether the request arrived.
+    async fn go_down(&self) {
+        let cut = self.connections.open.load(Ordering::SeqCst);
+        let redialed = self.connections.accepted.load(Ordering::SeqCst) + cut;
+        self.up.send_replace(false);
+        let reached = async {
+            while self.connections.accepted.load(Ordering::SeqCst) < redialed {
+                tokio::time::sleep(Duration::from_millis(20)).await;
+            }
+        };
+        tokio::time::timeout(STEP, reached)
+            .await
+            .expect("every client that the relay cut off dials again");
+    }
+
+    fn come_up(&self) {
+        self.up.send_replace(true);
+    }
+}
+
+/// Copies both ways between a client and the server while the relay is up. A connection that
+/// opens while it is down waits, and one that is open ends when it goes down.
+async fn relay_while_up(mut inbound: TcpStream, upstream: &str, mut up: watch::Receiver<bool>) {
+    if up.wait_for(|up| *up).await.is_err() {
+        return;
+    }
+    let Ok(mut outbound) = TcpStream::connect(upstream).await else {
+        return;
+    };
+    tokio::select! {
+        _ = tokio::io::copy_bidirectional(&mut inbound, &mut outbound) => {}
+        _ = up.wait_for(|up| !*up) => {}
+    }
+}
+
 /// Long enough for any one step against a local server.
 const STEP: Duration = Duration::from_secs(10);