You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by hg...@apache.org on 2026/06/30 08:53:55 UTC
(iggy) branch metadata-caught-up created (now 281d858f2)
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a change to branch metadata-caught-up
in repository https://gitbox.apache.org/repos/asf/iggy.git
at 281d858f2 fix(metadata): wait out transient catch-up lag on in-process submits
This branch includes the following new commits:
new 281d858f2 fix(metadata): wait out transient catch-up lag on in-process submits
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(iggy) 01/01: fix(metadata): wait out transient catch-up lag on in-process submits
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch metadata-caught-up
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 281d858f2f1ce74143482af0a66d3bdf9b580fe3
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jun 24 00:53:09 2026 +0200
fix(metadata): wait out transient catch-up lag on in-process submits
In-process metadata submits (register, logout, all replicated ops)
gated on commit_min == commit_max and silently dropped the request
when the primary was not yet caught up. During steady-state
pipelining commit_max runs microseconds ahead of commit_min while a
concurrent on_ack applies an in-flight op, so a request landing in
that window was rejected with no reply. The only recovery was the
client's 30s read-timeout and reconnect, so a few concurrent cold
connects stalled the whole run ~30s (a pp/pc bench finishing in
0.4s otherwise took 30s).
The catch-up gate stays: after a view change a fresh Register for a
client whose inherited Register is still unapplied would panic the
session-equality assert on replay. So instead of removing it, wait
for the transient to clear. advance_commit_min now fires a
CommitSignal, and submits await it (arm-then-recheck to avoid lost
wakeups, bounded by a 1s timeout that falls back to the original
silent drop for a genuinely stuck primary). The wait is event-driven
rather than a poll, and lives in one shared helper so the whole
replicated-op class benefits, not just register.
CommitSignal is a single-thread waker matching the existing oneshot
primitive; Notified deregisters on drop so the waiter roster stays
bounded to live waiters.
---
core/consensus/src/commit_signal.rs | 247 ++++++++++++++++++++++++++++++++++++
core/consensus/src/impls.rs | 26 ++++
core/consensus/src/lib.rs | 3 +
core/metadata/Cargo.toml | 2 +-
core/metadata/src/impls/metadata.rs | 182 +++++++++++++++++++++-----
5 files changed, 430 insertions(+), 30 deletions(-)
diff --git a/core/consensus/src/commit_signal.rs b/core/consensus/src/commit_signal.rs
new file mode 100644
index 000000000..95715e042
--- /dev/null
+++ b/core/consensus/src/commit_signal.rs
@@ -0,0 +1,247 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Single-threaded edge-triggered notify.
+//!
+//! Lets a parked caller wake when `commit_min` advances. Multiple callers may
+//! park concurrently (one per in-flight in-process submit); a notify wakes
+//! every parked waiter, each re-checks its own gate and re-parks if still not
+//! satisfied.
+//!
+//! Edge-triggered: a notify with no parked waiter is lost. Once a caller has
+//! decided it must wait, it MUST arm-then-check: build the [`Notified`] future
+//! (registering intent) BEFORE re-reading the gated state, so a notify landing
+//! between that read and the await is not missed. Check the gate FIRST and only
+//! arm when a wait is unavoidable - arming registers a roster slot, and the hot
+//! already-satisfied path should leave the roster untouched.
+//!
+//! Each [`Notified`] deregisters its slot on drop (and `notify_all` drains the
+//! roster), so the roster stays bounded to currently-live waiters.
+
+use std::cell::RefCell;
+use std::future::Future;
+use std::pin::Pin;
+use std::rc::Rc;
+use std::task::{Context, Poll, Waker};
+
+/// Slot for one parked waiter. `Rc`-shared between the [`Notified`] future and
+/// the [`CommitSignal`]'s roster so a wake reaches the future's task.
+#[derive(Default)]
+struct Slot {
+ waker: RefCell<Option<Waker>>,
+ notified: RefCell<bool>,
+}
+
+/// Edge-triggered single-threaded notify. Not `Send`/`Sync`: state is
+/// `Rc<RefCell<_>>`, thread-sharing would race at borrow.
+///
+/// The roster is an `Rc<RefCell<_>>` so each [`Notified`] keeps a handle to
+/// deregister its own slot on drop, bounding the roster to live waiters.
+#[derive(Debug, Default)]
+pub struct CommitSignal {
+ signal: Rc<RefCell<Vec<Rc<Slot>>>>,
+}
+
+impl std::fmt::Debug for Slot {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("Slot").finish_non_exhaustive()
+ }
+}
+
+impl CommitSignal {
+ /// Wake every parked waiter. Drains the roster: each waiter re-arms on its
+ /// next `commit_advanced()` if its gate is still unsatisfied. A drained
+ /// slot's [`Notified::drop`] becomes a no-op (the entry is already gone).
+ pub fn notify_all(&self) {
+ let slots = std::mem::take(&mut *self.signal.borrow_mut());
+ for slot in slots {
+ *slot.notified.borrow_mut() = true;
+ if let Some(waker) = slot.waker.borrow_mut().take() {
+ waker.wake();
+ }
+ }
+ }
+
+ /// Arm a single-use wait. Build this only once a wait is unavoidable and
+ /// re-check the gate AFTER (arm-then-check) so a notify landing in between
+ /// is not lost. The returned future resolves on the next `notify_all`.
+ #[must_use]
+ pub fn notified(&self) -> Notified {
+ let slot = Rc::new(Slot::default());
+ self.signal.borrow_mut().push(Rc::clone(&slot));
+ Notified {
+ slot,
+ signal: Rc::clone(&self.signal),
+ }
+ }
+
+ #[cfg(test)]
+ fn parked_count(&self) -> usize {
+ self.signal.borrow().len()
+ }
+}
+
+/// One-shot future that resolves on the next [`CommitSignal::notify_all`].
+/// Dropping before resolution deregisters its slot, so the roster stays
+/// bounded to currently-live waiters.
+pub struct Notified {
+ slot: Rc<Slot>,
+ signal: Rc<RefCell<Vec<Rc<Slot>>>>,
+}
+
+impl Future for Notified {
+ type Output = ();
+
+ fn poll(self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<()> {
+ if *self.slot.notified.borrow() {
+ return Poll::Ready(());
+ }
+ *self.slot.waker.borrow_mut() = Some(cx.waker().clone());
+ Poll::Pending
+ }
+}
+
+impl Drop for Notified {
+ fn drop(&mut self) {
+ // Idempotent: if `notify_all` already drained the slot it is absent.
+ self.signal
+ .borrow_mut()
+ .retain(|parked| !Rc::ptr_eq(parked, &self.slot));
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use std::task::Context;
+
+ #[test]
+ fn notify_wakes_armed_waiter() {
+ let signal = CommitSignal::default();
+ let mut fut = signal.notified();
+
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ assert!(Pin::new(&mut fut).poll(&mut cx).is_pending());
+
+ signal.notify_all();
+ assert!(Pin::new(&mut fut).poll(&mut cx).is_ready());
+ }
+
+ #[test]
+ fn notify_before_poll_is_not_lost() {
+ // Arm-then-check: future created (armed) before the notify; the notify
+ // must still resolve it even though it was never polled Pending first.
+ let signal = CommitSignal::default();
+ let mut fut = signal.notified();
+ signal.notify_all();
+
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ assert!(Pin::new(&mut fut).poll(&mut cx).is_ready());
+ }
+
+ #[test]
+ fn notify_wakes_all_parked_waiters() {
+ let signal = CommitSignal::default();
+ let mut a = signal.notified();
+ let mut b = signal.notified();
+
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ assert!(Pin::new(&mut a).poll(&mut cx).is_pending());
+ assert!(Pin::new(&mut b).poll(&mut cx).is_pending());
+
+ signal.notify_all();
+ assert!(Pin::new(&mut a).poll(&mut cx).is_ready());
+ assert!(Pin::new(&mut b).poll(&mut cx).is_ready());
+ }
+
+ #[test]
+ fn notify_with_no_waiter_is_dropped() {
+ // Edge-triggered: a notify with nothing parked is lost. A waiter armed
+ // afterwards stays pending (it must re-arm to catch the next edge).
+ let signal = CommitSignal::default();
+ signal.notify_all();
+
+ let mut fut = signal.notified();
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ assert!(Pin::new(&mut fut).poll(&mut cx).is_pending());
+ }
+
+ #[test]
+ fn dropping_unpolled_notified_deregisters_slot() {
+ let signal = CommitSignal::default();
+ assert_eq!(signal.parked_count(), 0);
+
+ let armed = signal.notified();
+ assert_eq!(signal.parked_count(), 1, "arming registers a slot");
+ drop(armed);
+ assert_eq!(signal.parked_count(), 0, "drop deregisters the slot");
+
+ // A surviving waiter still wakes exactly once: the abandoned slot left
+ // no stale entry to double-wake.
+ let mut live = signal.notified();
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ assert!(Pin::new(&mut live).poll(&mut cx).is_pending());
+ signal.notify_all();
+ assert!(Pin::new(&mut live).poll(&mut cx).is_ready());
+ }
+
+ #[test]
+ fn repeated_arm_drop_keeps_roster_bounded() {
+ // Models the hot path's transient parks: many arm-then-abandon cycles
+ // must not grow the roster (the leak this Drop impl prevents).
+ let signal = CommitSignal::default();
+ for _ in 0..1000 {
+ let armed = signal.notified();
+ assert_eq!(signal.parked_count(), 1);
+ drop(armed);
+ assert_eq!(signal.parked_count(), 0);
+ }
+ assert_eq!(signal.parked_count(), 0, "roster bounded to live waiters");
+ }
+
+ #[test]
+ fn notify_all_then_drop_is_idempotent_no_op() {
+ // A slot drained by notify_all is already gone; the eventual drop must
+ // not panic or remove an unrelated waiter.
+ let signal = CommitSignal::default();
+ let mut resolved = signal.notified();
+ let other = signal.notified();
+ assert_eq!(signal.parked_count(), 2);
+
+ signal.notify_all();
+ assert_eq!(signal.parked_count(), 0, "notify_all drains the roster");
+
+ let waker = futures::task::noop_waker();
+ let mut cx = Context::from_waker(&waker);
+ assert!(Pin::new(&mut resolved).poll(&mut cx).is_ready());
+
+ // Re-arm a fresh waiter, then drop the already-drained ones: their drop
+ // must not touch the fresh slot.
+ let fresh = signal.notified();
+ assert_eq!(signal.parked_count(), 1);
+ drop(resolved);
+ drop(other);
+ assert_eq!(signal.parked_count(), 1, "drained slots' drop is a no-op");
+ drop(fresh);
+ assert_eq!(signal.parked_count(), 0);
+ }
+}
diff --git a/core/consensus/src/impls.rs b/core/consensus/src/impls.rs
index 0edcb2fee..38846e62a 100644
--- a/core/consensus/src/impls.rs
+++ b/core/consensus/src/impls.rs
@@ -15,6 +15,7 @@
// specific language governing permissions and limitations
// under the License.
+use crate::commit_signal::{CommitSignal, Notified};
use crate::oneshot::{self, Receiver, Sender};
use crate::vsr_timeout::{TimeoutKind, TimeoutManager};
use crate::{
@@ -683,6 +684,13 @@ where
/// Monotonic timestamp from the most recent accepted commit heartbeat.
/// Old/replayed commit messages with a lower timestamp are ignored.
heartbeat_timestamp: Cell<u64>,
+
+ /// Wakes in-process submit callers parked on the catch-up gate when
+ /// `commit_min` advances (fired in `advance_commit_min`). Lets a submit
+ /// that arrives during the steady-state `commit_min < commit_max`
+ /// pipelining transient wait out the microsecond lag instead of being
+ /// silently dropped.
+ commit_signal: CommitSignal,
}
impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
@@ -739,6 +747,7 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
sent_own_do_view_change: Cell::new(false),
timeouts: RefCell::new(TimeoutManager::new(timeout_seed)),
heartbeat_timestamp: Cell::new(0),
+ commit_signal: CommitSignal::default(),
}
}
@@ -798,6 +807,18 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
);
self.commit_min.set(op);
assert!(self.commit_max.get() >= self.commit_min.get());
+ // Single chokepoint for local-apply progress: wake any submit caller
+ // parked on the catch-up gate so it re-checks `commit_min`.
+ self.commit_signal.notify_all();
+ }
+
+ /// Arm a wait that resolves on the next `commit_min` advance. Callers MUST
+ /// build this BEFORE re-reading the catch-up gate (arm-then-check) to avoid
+ /// a lost wakeup: a commit landing between the gate read and the await is
+ /// otherwise missed. See [`CommitSignal`].
+ #[must_use]
+ pub fn commit_advanced(&self) -> Notified {
+ self.commit_signal.notified()
}
/// Restore local commit progress from already-applied state during bootstrap.
@@ -825,6 +846,11 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
);
self.commit_max.set(commit_max);
self.commit_min.set(commit_min);
+ // No `commit_signal.notify_all()`: the fresh-instance asserts above
+ // guarantee this runs at bootstrap, before any `commit_advanced`
+ // waiter can park, so there is no edge to deliver. If this ever runs
+ // after a waiter parks, that waiter would miss the edge (masked only by
+ // the submit-side 1s catch-up timeout).
}
/// Maximum number of faulty replicas that can be tolerated.
diff --git a/core/consensus/src/lib.rs b/core/consensus/src/lib.rs
index 4ae31c5df..29e7b73e5 100644
--- a/core/consensus/src/lib.rs
+++ b/core/consensus/src/lib.rs
@@ -147,6 +147,9 @@ pub use client_table::{CachedReply, ClientTable};
// One-shot per `PipelineEntry` for in-process commit awaiters.
pub(crate) mod oneshot;
pub use oneshot::{Canceled, Receiver};
+// Edge-triggered notify for `commit_min`-advanced waiters.
+pub(crate) mod commit_signal;
+pub use commit_signal::{CommitSignal, Notified};
mod impls;
pub use impls::*;
diff --git a/core/metadata/Cargo.toml b/core/metadata/Cargo.toml
index ac68c96d4..2802b67b3 100644
--- a/core/metadata/Cargo.toml
+++ b/core/metadata/Cargo.toml
@@ -32,6 +32,7 @@ publish = false
ahash = { workspace = true }
bytemuck = { workspace = true }
bytes = { workspace = true }
+compio = { workspace = true }
consensus = { workspace = true }
iggy_binary_protocol = { workspace = true }
iggy_common = { workspace = true }
@@ -47,7 +48,6 @@ tracing = { workspace = true }
[dev-dependencies]
bytes = { workspace = true }
-compio = { workspace = true }
tempfile = { workspace = true }
[lints.clippy]
diff --git a/core/metadata/src/impls/metadata.rs b/core/metadata/src/impls/metadata.rs
index e0a7fa3c5..c13517a25 100644
--- a/core/metadata/src/impls/metadata.rs
+++ b/core/metadata/src/impls/metadata.rs
@@ -903,6 +903,61 @@ where
}
}
+/// Safety-net cap on how long an in-process submit waits for the catch-up
+/// gate to open. Far under the SDK 30s read-timeout, so a genuinely stuck or
+/// lost-primary node still falls back to the silent-drop path well before the
+/// client gives up. Never fires in steady state: the gate reopens on the next
+/// `commit_min` advance, which is microsecond-scale.
+const CAUGHT_UP_WAIT_TIMEOUT: std::time::Duration = std::time::Duration::from_secs(1);
+
+/// Park until this node is a caught-up primary, then return `Ok`.
+///
+/// Absorbs the steady-state `commit_min < commit_max` pipelining transient: a
+/// submit arriving while a sibling `on_ack` is mid-flight waits the microsecond
+/// lag out instead of being silently dropped (which forced the SDK into a full
+/// 30s read-timeout + reconnect).
+///
+/// Returns immediately with the caller's silent-drop variant when the node is
+/// genuinely not a primary (`NotPrimary`) or has not caught up within
+/// [`CAUGHT_UP_WAIT_TIMEOUT`] (`NotCaughtUp`); both preserve the by-design
+/// silent drop so the client retry lands on a peer or here post-catch-up.
+///
+/// # Safety
+/// The catch-up gate is correctness-critical (see
+/// [`is_caught_up_primary`]); this only waits for it to open, it never weakens
+/// it. No `RefCell` borrow of `consensus` is held across the await.
+#[allow(clippy::future_not_send)]
+async fn await_caught_up<B: MessageBus>(
+ consensus: &VsrConsensus<B>,
+) -> Result<(), MetadataSubmitError> {
+ let wait = async {
+ loop {
+ // Check FIRST so the hot already-caught-up path never arms a roster
+ // slot (arming-then-abandoning every common submit would grow the
+ // roster unboundedly).
+ if is_caught_up_primary(consensus) {
+ return Ok(());
+ }
+ if !(consensus.is_primary() && consensus.is_normal() && !consensus.is_syncing()) {
+ // Genuinely not primary: do not wait, the client must try a peer.
+ return Err(MetadataSubmitError::NotPrimary);
+ }
+ // Must wait: arm, then re-check AFTER arming so a commit landing
+ // between the first check and the await is not lost (it fires
+ // `notify_all` into the now-armed waiter). See `CommitSignal`.
+ let notified = consensus.commit_advanced();
+ if is_caught_up_primary(consensus) {
+ return Ok(());
+ }
+ notified.await;
+ }
+ };
+ match compio::time::timeout(CAUGHT_UP_WAIT_TIMEOUT, wait).await {
+ Ok(result) => result,
+ Err(_elapsed) => Err(MetadataSubmitError::NotCaughtUp),
+ }
+}
+
impl<B, J, S, M> IggyMetadata<VsrConsensus<B>, J, S, M>
where
B: MessageBus,
@@ -948,17 +1003,10 @@ where
return Ok(session);
}
- // Status + catch-up gate (see doc). Split variants for telemetry:
- // NotPrimary (try peer) vs NotCaughtUp (retry). Caller policy same.
- if !is_caught_up_primary(consensus) {
- return Err(
- if consensus.is_primary() && consensus.is_normal() && !consensus.is_syncing() {
- MetadataSubmitError::NotCaughtUp
- } else {
- MetadataSubmitError::NotPrimary
- },
- );
- }
+ // Status + catch-up gate (see doc). Waits out the steady-state
+ // `commit_min < commit_max` pipelining transient; only a genuine
+ // not-primary / lost-primary falls through to the silent-drop error.
+ await_caught_up(consensus).await?;
// Mirror wire-path register_preflight: a racing second prepare fails
// check_register on commit. Surface pre-synthesis.
@@ -1048,15 +1096,7 @@ where
return Ok(consensus.commit_min());
}
- if !is_caught_up_primary(consensus) {
- return Err(
- if consensus.is_primary() && consensus.is_normal() && !consensus.is_syncing() {
- MetadataSubmitError::NotCaughtUp
- } else {
- MetadataSubmitError::NotPrimary
- },
- );
- }
+ await_caught_up(consensus).await?;
if consensus
.pipeline()
@@ -1303,15 +1343,7 @@ where
.as_ref()
.expect("submit_request_in_process: consensus only exists on shard 0");
- if !is_caught_up_primary(consensus) {
- return Err(
- if consensus.is_primary() && consensus.is_normal() && !consensus.is_syncing() {
- MetadataSubmitError::NotCaughtUp
- } else {
- MetadataSubmitError::NotPrimary
- },
- );
- }
+ await_caught_up(consensus).await?;
// Dedup / session / eviction. shard 0 cannot route by the VSR
// consensus `client_id` (its top bits are random, not home-shard
@@ -1947,7 +1979,9 @@ mod tests {
use super::*;
use crate::stm::stream::Streams;
use crate::stm::user::Users;
+ use consensus::LocalPipeline;
use iggy_common::variadic;
+ use message_bus::IggyMessageBus;
use std::cell::RefCell;
use std::rc::Rc;
@@ -2043,4 +2077,94 @@ mod tests {
"cleared notifier must stay quiet"
);
}
+
+ /// Single-replica primary (`replica_count = 1` -> always primary) parked on a
+ /// closed catch-up gate (`commit_min < commit_max`), with the gate opened
+ /// one op at a time. Proves the steady-state transient is absorbed instead
+ /// of silently dropped, and that every parked waiter is woken (no
+ /// lost-wakeup) when `commit_min` finally reaches `commit_max`.
+ fn caught_up_consensus(commit_max: u64) -> Rc<VsrConsensus<IggyMessageBus>> {
+ let consensus = VsrConsensus::new(
+ 0,
+ 0,
+ 1,
+ server_common::sharding::METADATA_CONSENSUS_NAMESPACE,
+ IggyMessageBus::new(0),
+ LocalPipeline::new(),
+ );
+ consensus.init();
+ consensus.advance_commit_max(commit_max);
+ Rc::new(consensus)
+ }
+
+ #[compio::test]
+ async fn given_many_parked_registers_when_commit_min_catches_up_should_wake_all() {
+ const WAITERS: u64 = 64;
+ const TARGET: u64 = 8;
+
+ let consensus = caught_up_consensus(TARGET);
+ // Gate is closed: commit_min (0) < commit_max (TARGET).
+ assert!(!is_caught_up_primary(&consensus));
+
+ let waiters: Vec<_> = (0..WAITERS)
+ .map(|_| {
+ let consensus = Rc::clone(&consensus);
+ compio::runtime::spawn(async move { await_caught_up(&consensus).await })
+ })
+ .collect();
+
+ // Let every waiter poll once and park on the closed gate before the
+ // gate is opened, so the wake path (not the fast path) is exercised.
+ compio::time::sleep(std::time::Duration::from_millis(20)).await;
+
+ // Open the gate one op at a time; each advance fires the commit signal.
+ for op in 1..=TARGET {
+ consensus.advance_commit_min(op);
+ }
+ assert!(is_caught_up_primary(&consensus));
+
+ for waiter in waiters {
+ match waiter.await {
+ Ok(Ok(())) => {}
+ other => panic!("waiter must resolve Ok(()), got {other:?}"),
+ }
+ }
+ }
+
+ #[compio::test]
+ async fn given_non_primary_when_await_caught_up_should_return_not_primary() {
+ // replica 1 of 2 in view 0 -> primary is replica 0, so this node is a
+ // backup. The gate can never open here, so the helper must fail fast
+ // with NotPrimary instead of waiting out the timeout.
+ let consensus = VsrConsensus::<IggyMessageBus>::new(
+ 0,
+ 1,
+ 2,
+ server_common::sharding::METADATA_CONSENSUS_NAMESPACE,
+ IggyMessageBus::new(0),
+ LocalPipeline::new(),
+ );
+ consensus.init();
+
+ assert_eq!(
+ await_caught_up(&consensus).await,
+ Err(MetadataSubmitError::NotPrimary)
+ );
+ }
+
+ #[compio::test]
+ async fn given_primary_that_never_advances_when_await_caught_up_should_timeout_not_caught_up() {
+ // Primary with a permanently closed gate (commit_min < commit_max, no
+ // advance ever fires): the helper parks, then the safety-net timeout
+ // surfaces NotCaughtUp instead of hanging. The cancelled `Notified` is
+ // dropped, which deregisters its slot (roster bounding is asserted in
+ // the consensus-crate `CommitSignal` tests).
+ let consensus = caught_up_consensus(1);
+ assert!(!is_caught_up_primary(&consensus));
+
+ assert_eq!(
+ await_caught_up(&consensus).await,
+ Err(MetadataSubmitError::NotCaughtUp)
+ );
+ }
}