You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by hg...@apache.org on 2026/10/07 09:46:56 UTC
(iggy) 01/10: perf(partitions): keep shard traffic moving during file I/O
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch perf/task-per-partition
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit fb817ea596a6781b7e49d5b9ddddb0b0b79088a3
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Tue Oct 6 19:26:43 2026 +0200
perf(partitions): keep shard traffic moving during file I/O
Partition file writes and durability barriers held the shard pump.
Unrelated partitions waited until storage completed, and Replicated
commits also waited for the WAL writer and its checkpoints. On a
3-node cluster, Replicated p999 rose from 2-6 ms to 21-140 ms.
Run bounded, owned file jobs on the existing runtime and return
results to the pump for identity checks and ordered publication.
Commit, NoAck, purge, checkpoint and lifecycle continuations stay
with their owners, so durability and reply ordering hold.
A Replicated commit may pass a body the WAL writer has not written
yet, up to a 16 MiB backlog, and its flush takes only the written
prefix. Persisted commits wait while a WAL checkpoint is pending.
So do offset writes, because the checkpoint syncs the offset files
it lists. A session retirement follows partition durability.
Shutdown stops originating prepares, answers queued requests and
flushes committed ops. Each partition, a fatal one too, publishes
its WAL frontier before the drain. The tick now runs between job
phases, so it no longer extends the offset reservation during
shutdown or certifies the WAL during an install.
A state transfer pins its plan, so commits between rounds do not
restart the build. A receiver refuses only snapshots behind an op
it committed. Offset writes admitted before a purge get the
terminal errors of #4325.
Replicated p999 is back near master: 5.3 ms for 20 pinned
producers, 2.7 ms for one, 6.6 ms for producing consumers and
4.2 ms for balanced producers.
Add controlled I/O regressions for progress, capacity, stale
completions, lifecycle transitions, purge, shutdown, transfers
and Replicated run-ahead.
---
core/configs/src/server_config/sharding.rs | 61 +
core/consensus/src/impls.rs | 139 +-
core/consensus/src/lib.rs | 5 +
core/consensus/src/metadata_helpers.rs | 23 +-
core/consensus/src/plane_helpers.rs | 33 +
core/journal/src/local_gate.rs | 50 +-
core/partitions/src/consumer_offset_capacity.rs | 5 +
core/partitions/src/iggy_index_writer.rs | 31 +-
core/partitions/src/iggy_partition.rs | 12564 ++++++++++++++-------
core/partitions/src/iggy_partitions.rs | 177 +-
core/partitions/src/io.rs | 1425 +++
core/partitions/src/journal.rs | 171 +-
core/partitions/src/lib.rs | 10 +-
core/partitions/src/messages_writer.rs | 21 +-
core/partitions/src/offset_storage.rs | 266 +-
core/partitions/src/persistence.rs | 240 +-
core/partitions/src/state_transfer.rs | 2657 ++---
core/partitions/tests/consumer_offset_history.rs | 219 +-
core/server/config.toml | 5 +
core/server/src/boot/recovery.rs | 4 +
core/server/src/boot/threads.rs | 4 +
core/server/src/partition_reconciler.rs | 10 +
core/server/src/server_error.rs | 2 +
core/server_common/src/iobuf.rs | 35 +
core/server_common/src/poll.rs | 2 +-
core/shard/src/builder.rs | 14 +-
core/shard/src/lib.rs | 573 +-
core/shard/src/metrics.rs | 58 +
core/shard/src/partition_io.rs | 1952 ++++
core/shard/src/poll.rs | 4 +-
core/shard/src/poll/completion_tests.rs | 15 +-
core/shard/src/poll/timeout_tests.rs | 17 +-
core/shard/src/router.rs | 217 +-
core/simulator/src/storage/purge.rs | 127 +-
core/simulator/src/workload/state_checker.rs | 1 +
35 files changed, 15233 insertions(+), 5904 deletions(-)
diff --git a/core/configs/src/server_config/sharding.rs b/core/configs/src/server_config/sharding.rs
index 03fb95f94..a4ea87b08 100644
--- a/core/configs/src/server_config/sharding.rs
+++ b/core/configs/src/server_config/sharding.rs
@@ -101,6 +101,11 @@ pub struct ShardingConfig {
/// rejects new disk polls before I/O. Main and reply inbox traffic uses
/// separate capacity. This counts operations, not retained message bytes.
pub poll_completion_capacity: usize,
+ /// Active partition file jobs and results awaiting owner acceptance.
+ pub partition_io_capacity: usize,
+ /// Retained job allocations, resolved at boot against the largest legal record.
+ /// Omission selects max(256 MiB, the single-job minimum).
+ pub partition_io_bytes_max: Option<usize>,
/// Wall-clock budget for a single shard's bus drain on shutdown.
/// Drives `IggyMessageBus::shutdown(..)` from the per-shard watchdog
/// and the parallel-join survivor path. Sized larger than typical
@@ -149,6 +154,8 @@ impl Default for ShardingConfig {
inbox_capacity: SERVER_CONFIG.sharding.inbox_capacity as usize,
reply_inbox_capacity: SERVER_CONFIG.sharding.reply_inbox_capacity as usize,
poll_completion_capacity: SERVER_CONFIG.sharding.poll_completion_capacity as usize,
+ partition_io_capacity: SERVER_CONFIG.sharding.partition_io_capacity as usize,
+ partition_io_bytes_max: None,
shutdown_drain_timeout: SERVER_CONFIG
.sharding
.shutdown_drain_timeout
@@ -175,6 +182,17 @@ impl Default for ShardingConfig {
impl Validatable<ConfigurationError> for ShardingConfig {
fn validate(&self) -> Result<(), ConfigurationError> {
+ if self.partition_io_capacity == 0
+ || self.partition_io_capacity > INBOX_CAPACITY_MAX
+ || self
+ .partition_io_bytes_max
+ .is_some_and(|bytes| bytes == 0 || bytes > isize::MAX as usize)
+ {
+ eprintln!(
+ "Invalid sharding configuration: partition I/O limits must be positive and fit addressable capacity"
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
if self.inbox_capacity == 0 {
eprintln!(
"Invalid sharding configuration: inbox_capacity must be > 0 (crossfire silently \
@@ -320,6 +338,49 @@ mod tests {
assert!(ShardingConfig::default().validate().is_ok());
}
+ #[test]
+ fn partition_io_limits_preserve_omission_and_accept_explicit_env_mappings() {
+ let omitted: ShardingConfig = Figment::new()
+ .merge(Toml::string("inbox_capacity = 7"))
+ .extract()
+ .unwrap();
+ assert_eq!(omitted.partition_io_capacity, 16);
+ assert_eq!(omitted.partition_io_bytes_max, None);
+
+ let configured: ShardingConfig = Figment::new()
+ .merge(Toml::string(
+ "partition_io_capacity = 3\npartition_io_bytes_max = 8589934592",
+ ))
+ .extract()
+ .unwrap();
+ assert_eq!(configured.partition_io_capacity, 3);
+ assert_eq!(
+ configured.partition_io_bytes_max,
+ Some(8 * 1024 * 1024 * 1024)
+ );
+ assert!(configured.validate().is_ok());
+ let mappings = <ShardingConfig as configs::ConfigEnvMappings>::env_mappings();
+ for name in ["PARTITION_IO_CAPACITY", "PARTITION_IO_BYTES_MAX"] {
+ assert!(mappings.iter().any(|mapping| mapping.env_name == name));
+ }
+ for (capacity, bytes) in [
+ (0, None),
+ (INBOX_CAPACITY_MAX + 1, None),
+ (1, Some(0)),
+ (1, Some(usize::MAX)),
+ ] {
+ let invalid = ShardingConfig {
+ partition_io_capacity: capacity,
+ partition_io_bytes_max: bytes,
+ ..ShardingConfig::default()
+ };
+ assert!(
+ invalid.validate().is_err(),
+ "capacity={capacity}, bytes={bytes:?}"
+ );
+ }
+ }
+
#[test]
fn given_invalid_poll_completion_capacity_when_validated_should_reject() {
for capacity in [0, INBOX_CAPACITY_MAX + 1] {
diff --git a/core/consensus/src/impls.rs b/core/consensus/src/impls.rs
index cd17643cc..f5a90aedc 100644
--- a/core/consensus/src/impls.rs
+++ b/core/consensus/src/impls.rs
@@ -40,9 +40,29 @@ use server_common::poll::{AutoCommitReservation, PollHistoryId};
use server_common::sharding::{IggyNamespace, METADATA_GROUP};
use std::cell::{Cell, RefCell};
use std::collections::{HashSet, VecDeque};
+use std::num::NonZeroU64;
use std::rc::Rc;
use std::time::Duration;
+#[derive(Clone)]
+pub struct LoopbackNotifier(Rc<dyn Fn()>);
+
+impl LoopbackNotifier {
+ pub fn new(notify: impl Fn() + 'static) -> Self {
+ Self(Rc::new(notify))
+ }
+
+ fn notify(&self) {
+ self.0();
+ }
+}
+
+impl std::fmt::Debug for LoopbackNotifier {
+ fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ formatter.write_str("LoopbackNotifier")
+ }
+}
+
/// Injected time source for primary-stamped prepare timestamps.
///
/// The consensus core never reads the wall clock directly,
@@ -197,9 +217,14 @@ pub const CLIENTS_TABLE_MAX: usize = 8192;
/// `[partition] dedup_clients_max` default by a bootstrap assert.
pub const PARTITION_DEDUP_CLIENTS_MAX: usize = 4096;
+/// Admission order on one partition owner. It is never replicated or recovered.
+#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)]
+pub struct LocalRequestOrder(pub NonZeroU64);
+
#[derive(Debug)]
pub struct PipelineEntry {
pub header: PrepareHeader,
+ pub local_order: Option<LocalRequestOrder>,
/// Bitmap of replicas that have acknowledged this prepare.
pub ok_from_replicas: BitSet<u32>,
/// Whether we've received a quorum of `prepare_ok` messages.
@@ -217,6 +242,7 @@ impl PipelineEntry {
pub fn new(header: PrepareHeader) -> Self {
Self {
header,
+ local_order: None,
ok_from_replicas: BitSet::with_capacity(REPLICAS_MAX),
ok_quorum_received: false,
reply_sender: None,
@@ -241,6 +267,7 @@ impl PipelineEntry {
pub fn with_sender(header: PrepareHeader, sender: Sender<Message<ReplyHeader>>) -> Self {
Self {
header,
+ local_order: None,
ok_from_replicas: BitSet::with_capacity(REPLICAS_MAX),
ok_quorum_received: false,
reply_sender: Some(sender),
@@ -293,6 +320,7 @@ pub struct AutoCommitRequestContext {
/// Accepted request waiting in `request_queue` for a prepare slot.
#[derive(Debug)]
pub struct RequestEntry {
+ pub local_order: Option<LocalRequestOrder>,
/// Automatic commit context owned by this entry until promotion or removal.
auto_commit: Option<AutoCommitRequestContext>,
/// Offset writes must not cross a reset of the owner's message history.
@@ -391,6 +419,7 @@ impl RequestEntry {
Self {
auto_commit: None,
consumer_offset_history: None,
+ local_order: None,
message,
received_at: 0,
reply_sender,
@@ -594,6 +623,23 @@ impl LocalPipeline {
self.request_queue.len()
}
+ #[must_use]
+ pub const fn request_queue_capacity(&self) -> usize {
+ self.request_queue_max
+ }
+
+ /// Rebuilt entries have no local marker and precede new local requests.
+ #[must_use]
+ pub fn has_request_before(&self, order: LocalRequestOrder) -> bool {
+ self.prepare_queue
+ .iter()
+ .any(|entry| entry.local_order.is_none_or(|entry| entry < order))
+ || self
+ .request_queue
+ .iter()
+ .any(|entry| entry.local_order.is_none_or(|entry| entry < order))
+ }
+
#[must_use]
pub fn request_queue_full(&self) -> bool {
self.request_queue.len() >= self.request_queue_max
@@ -625,6 +671,11 @@ impl LocalPipeline {
self.request_queue.iter()
}
+ #[must_use]
+ pub fn request_head(&self) -> Option<&RequestEntry> {
+ self.request_queue.front()
+ }
+
/// True iff `prepare_queue` is full (NOT including `request_queue`).
/// Callers branch on this between direct push and [`Self::push_request`].
#[must_use]
@@ -960,6 +1011,10 @@ impl Pipeline for LocalPipeline {
Self::pop_request(self)
}
+ fn request_head(&self) -> Option<&Self::Request> {
+ Self::request_head(self)
+ }
+
fn request_queue_len(&self) -> usize {
Self::request_queue_len(self)
}
@@ -1232,6 +1287,7 @@ where
message_bus: B,
loopback_queue: RefCell<VecDeque<Message<GenericHeader>>>,
+ loopback_notifier: RefCell<Option<LoopbackNotifier>>,
/// Tracks start view change messages received from all replicas (including self)
start_view_change_from_all_replicas: RefCell<BitSet<u32>>,
/// Consecutive unanswered `RequestStartView` probes while Recovering;
@@ -1630,6 +1686,7 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
prepare_queue_max,
message_bus,
loopback_queue: RefCell::new(VecDeque::with_capacity(prepare_queue_max)),
+ loopback_notifier: RefCell::new(None),
start_view_change_from_all_replicas: RefCell::new(BitSet::with_capacity(REPLICAS_MAX)),
probe_attempts: Cell::new(0),
probe_attempts_max: Cell::new(PROBE_ATTEMPTS_MAX),
@@ -2285,6 +2342,25 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
self.push_prepare_entry(plane, message, entry);
}
+ /// Preserve local admission order when a queued request becomes a prepare.
+ ///
+ /// # Panics
+ /// If not primary, as with other pipeline insertion methods.
+ pub fn pipeline_message_with_order(
+ &self,
+ plane: PlaneKind,
+ message: &Message<PrepareHeader>,
+ sender: Option<Sender<Message<ReplyHeader>>>,
+ order: Option<LocalRequestOrder>,
+ ) {
+ let mut entry = sender.map_or_else(
+ || PipelineEntry::new(*message.header()),
+ |sender| PipelineEntry::with_sender(*message.header(), sender),
+ );
+ entry.local_order = order;
+ self.push_prepare_entry(plane, message, entry);
+ }
+
#[must_use]
pub const fn cluster(&self) -> u128 {
self.cluster
@@ -2572,6 +2648,15 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
/// Returns a list of actions to take based on fired timeouts.
/// Empty vec means no actions needed.
pub fn tick(&self, plane: PlaneKind) -> Vec<VsrAction> {
+ self.tick_with_history_fence(plane, false)
+ }
+
+ /// Timers and retransmissions continue while storage leases defer history changes.
+ pub fn tick_with_history_fence(
+ &self,
+ plane: PlaneKind,
+ history_fenced: bool,
+ ) -> Vec<VsrAction> {
let mut actions = Vec::new();
let mut timeouts = self.timeouts.borrow_mut();
@@ -2579,7 +2664,7 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
timeouts.tick();
// Phase 2: Handle fired timeouts
- if timeouts.fired(TimeoutKind::NormalHeartbeat) {
+ if !history_fenced && timeouts.fired(TimeoutKind::NormalHeartbeat) {
drop(timeouts);
actions.extend(self.handle_normal_heartbeat_timeout(plane));
timeouts = self.timeouts.borrow_mut();
@@ -2609,7 +2694,7 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
timeouts = self.timeouts.borrow_mut();
}
- if timeouts.fired(TimeoutKind::RequestStartViewMessage) {
+ if !history_fenced && timeouts.fired(TimeoutKind::RequestStartViewMessage) {
drop(timeouts);
// Two probers share this timeout, both asking "resend me the
// current StartView":
@@ -2684,7 +2769,7 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
timeouts = self.timeouts.borrow_mut();
}
- if timeouts.fired(TimeoutKind::ViewChangeStatus) {
+ if !history_fenced && timeouts.fired(TimeoutKind::ViewChangeStatus) {
drop(timeouts);
actions.extend(self.handle_view_change_status_timeout(plane));
// timeouts = self.timeouts.borrow_mut(); // Not needed if last
@@ -4176,22 +4261,25 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
}
}
- /// Enqueue a self-addressed message for processing in the next loopback drain.
- ///
- /// Only `PrepareOk` reaches here (via `send_or_loopback`), and deliberately:
- /// it is a message to a peer that happens to be this replica, so a one-drain
- /// delay costs nothing. A replica's own SVC/DVC is not that shape -- it is the
- /// local decision to change view, recorded synchronously with the view and
- /// status writes in [`Self::enter_view_change`]. Routing it here instead would
- /// leave a window where the replica has entered a view change without counting
- /// itself, which on a solo group is the entire quorum.
+ /// Enqueue a self-ack for the next owner service round.
pub(crate) fn push_loopback(&self, message: Message<GenericHeader>) {
- assert!(
- self.loopback_queue.borrow().len() < self.prepare_queue_max,
- "loopback queue overflow: {} items",
- self.loopback_queue.borrow().len()
- );
- self.loopback_queue.borrow_mut().push_back(message);
+ let notify = {
+ let mut queue = self.loopback_queue.borrow_mut();
+ assert!(
+ queue.len() < self.prepare_queue_max,
+ "loopback queue overflow: {} items",
+ queue.len()
+ );
+ let was_empty = queue.is_empty();
+ queue.push_back(message);
+ was_empty
+ };
+ if notify {
+ let notifier = self.loopback_notifier.borrow().clone();
+ if let Some(notifier) = notifier {
+ notifier.notify();
+ }
+ }
}
/// Drain all pending loopback messages into `buf`, leaving the queue empty.
@@ -4201,6 +4289,21 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
buf.extend(self.loopback_queue.borrow_mut().drain(..));
}
+ /// Registers existing work as well as future empty-to-nonempty edges.
+ pub fn set_loopback_notifier(&self, notifier: Option<LoopbackNotifier>) {
+ self.loopback_notifier.borrow_mut().clone_from(¬ifier);
+ if self.has_loopback()
+ && let Some(notifier) = notifier
+ {
+ notifier.notify();
+ }
+ }
+
+ #[must_use]
+ pub fn has_loopback(&self) -> bool {
+ !self.loopback_queue.borrow().is_empty()
+ }
+
/// Send a message to `target`, routing self-addressed messages through the loopback queue.
// VsrConsensus uses Cell/RefCell for single-threaded compio shards; futures are intentionally !Send.
#[allow(clippy::future_not_send)]
diff --git a/core/consensus/src/lib.rs b/core/consensus/src/lib.rs
index 1de069145..c952d399d 100644
--- a/core/consensus/src/lib.rs
+++ b/core/consensus/src/lib.rs
@@ -107,6 +107,11 @@ pub trait Pipeline {
fn pop_request(&mut self) -> Option<Self::Request> {
None
}
+
+ /// Inspect the original queue head without transferring reply ownership.
+ fn request_head(&self) -> Option<&Self::Request> {
+ None
+ }
}
pub type RequestMessage<C> = <C as Consensus>::Message<<C as Consensus>::RoutedRequestHeader>;
diff --git a/core/consensus/src/metadata_helpers.rs b/core/consensus/src/metadata_helpers.rs
index 8b1e2e34b..2885781f8 100644
--- a/core/consensus/src/metadata_helpers.rs
+++ b/core/consensus/src/metadata_helpers.rs
@@ -401,6 +401,20 @@ fn build_eviction_from_header(header: EvictionHeader) -> Message<EvictionHeader>
/// `commit_min == commit_max` -- a transferring replica also fails that. Do
/// not weaken commit-equality.
pub fn is_caught_up_primary<B, P>(consensus: &VsrConsensus<B, P>) -> bool
+where
+ B: MessageBus,
+ P: Pipeline<Entry = PipelineEntry>,
+{
+ is_settled_primary(consensus) && consensus.commit_min() == consensus.commit_max()
+}
+
+/// [`is_caught_up_primary`] without commit-equality.
+///
+/// The primary of a Normal view that it has not ceded, not syncing, with its
+/// recovered suffix re-committed, however far it still is from applying
+/// `commit_max`. Enough only for a caller whose reads stop at `commit_min` by
+/// construction.
+pub fn is_settled_primary<B, P>(consensus: &VsrConsensus<B, P>) -> bool
where
B: MessageBus,
P: Pipeline<Entry = PipelineEntry>,
@@ -409,7 +423,6 @@ where
&& !consensus.has_ceded_primaryship()
&& consensus.is_normal()
&& !consensus.is_transferring()
- && consensus.commit_min() == consensus.commit_max()
// Recovery re-pipelines the WAL's prepared-but-uncommitted suffix;
// those ops were acked to clients before the restart, so admitting
// new requests (a login is a Register write) before the suffix
@@ -1026,16 +1039,22 @@ mod tests {
primary.set_state_transfer_stage(crate::StateTransferStage::Idle);
assert!(is_caught_up_primary(&primary));
- // commit_min < commit_max -> false.
+ // commit_min < commit_max -> false, yet still the settled primary.
primary.advance_commit_max(5);
assert_ne!(primary.commit_min(), primary.commit_max());
assert!(!is_caught_up_primary(&primary));
+ assert!(is_settled_primary(&primary));
+
+ // Mid-transfer -> not settled either.
+ primary.begin_state_transfer_await();
+ assert!(!is_settled_primary(&primary));
// Backup -> false.
let backup = VsrConsensus::new(1, 1, 3, 0, ClientSpyBus::new(), LocalPipeline::new());
backup.init();
assert!(!backup.is_primary());
assert!(!is_caught_up_primary(&backup));
+ assert!(!is_settled_primary(&backup));
}
#[test]
diff --git a/core/consensus/src/plane_helpers.rs b/core/consensus/src/plane_helpers.rs
index d88fc4f84..ff42cfb4e 100644
--- a/core/consensus/src/plane_helpers.rs
+++ b/core/consensus/src/plane_helpers.rs
@@ -985,7 +985,9 @@ mod tests {
use iggy_common::calculate_checksum;
use message_bus::{BusMessage, SendError};
use server_common::{MESSAGE_ALIGN, iobuf::Frozen};
+ use std::cell::Cell;
use std::collections::BTreeMap;
+ use std::rc::Rc;
/// `PrepareHeader`'s alignment, which every suffix body has to satisfy.
const BODY_ALIGN: usize = align_of::<PrepareHeader>();
@@ -1449,6 +1451,37 @@ mod tests {
assert!(buf.is_empty());
}
+ #[test]
+ fn loopback_notifier_registers_existing_work_and_only_notifies_new_edges() {
+ let consensus = Rc::new(VsrConsensus::new(1, 0, 3, 0, NoopBus, LocalPipeline::new()));
+ consensus.init();
+ let message =
+ || Message::<PrepareOkHeader>::new(size_of::<PrepareOkHeader>()).into_generic();
+ consensus.push_loopback(message());
+ let notifications = Rc::new(Cell::new(0));
+ let observed = Rc::clone(¬ifications);
+ let owner = Rc::downgrade(&consensus);
+ consensus.set_loopback_notifier(Some(crate::LoopbackNotifier::new(move || {
+ assert!(
+ owner.upgrade().unwrap().has_loopback(),
+ "notification releases the queue borrow"
+ );
+ observed.set(observed.get() + 1);
+ })));
+ assert_eq!(notifications.get(), 1);
+ consensus.push_loopback(message());
+ assert_eq!(notifications.get(), 1);
+ let mut messages = Vec::new();
+ consensus.drain_loopback_into(&mut messages);
+ assert_eq!(messages.len(), 2);
+ consensus.push_loopback(message());
+ assert_eq!(notifications.get(), 2);
+ consensus.set_loopback_notifier(None);
+ consensus.drain_loopback_into(&mut messages);
+ consensus.push_loopback(message());
+ assert_eq!(notifications.get(), 2, "retired owners do not notify");
+ }
+
#[test]
fn loopback_cleared_on_view_change_reset() {
let consensus = VsrConsensus::new(1, 0, 3, 0, NoopBus, LocalPipeline::new());
diff --git a/core/journal/src/local_gate.rs b/core/journal/src/local_gate.rs
index ca9998f69..e5a716df1 100644
--- a/core/journal/src/local_gate.rs
+++ b/core/journal/src/local_gate.rs
@@ -29,6 +29,7 @@
//! holder that re-acquires deadlocks itself.
use std::cell::{Cell, RefCell};
+use std::rc::Rc;
/// See the module docs. Callers hold the returned guard across the awaited
/// critical section; dropping it releases the gate and wakes every waiter.
@@ -59,6 +60,37 @@ impl LocalGate {
pub const fn acquire(&self) -> LocalGateAcquire<'_> {
LocalGateAcquire { gate: self }
}
+
+ /// A dropped owned lease leaves the resource fenced. Only an owner that
+ /// observed physical completion may release it.
+ #[must_use]
+ pub fn try_acquire_owned(self: &Rc<Self>) -> Option<OwnedLocalGateGuard> {
+ if self.busy.replace(true) {
+ return None;
+ }
+ Some(OwnedLocalGateGuard {
+ gate: Rc::clone(self),
+ })
+ }
+
+ fn release(&self) {
+ self.busy.set(false);
+ let waiters = std::mem::take(&mut *self.waiters.borrow_mut());
+ for waker in waiters {
+ waker.wake();
+ }
+ }
+}
+
+#[must_use = "release only after physical work has settled"]
+pub struct OwnedLocalGateGuard {
+ gate: Rc<LocalGate>,
+}
+
+impl OwnedLocalGateGuard {
+ pub fn release(self) {
+ self.gate.release();
+ }
}
impl Default for LocalGate {
@@ -103,14 +135,10 @@ pub struct LocalGateGuard<'a> {
impl Drop for LocalGateGuard<'_> {
fn drop(&mut self) {
- self.gate.busy.set(false);
// Move the waiters out before waking: `wake()` only schedules under
// compio today, but a waker that ever polled a waiter inline would
// re-enter `acquire`'s `waiters.borrow_mut()` and panic the RefCell.
- let waiters = std::mem::take(&mut *self.gate.waiters.borrow_mut());
- for waker in waiters {
- waker.wake();
- }
+ self.gate.release();
}
}
@@ -119,7 +147,17 @@ mod tests {
use super::*;
use futures::FutureExt;
use futures::future::join;
- use std::rc::Rc;
+
+ #[test]
+ fn owned_lease_requires_explicit_settlement_before_reuse() {
+ let gate = Rc::new(LocalGate::new());
+ let held = gate.try_acquire_owned().unwrap();
+ assert!(gate.try_acquire_owned().is_none());
+ held.release();
+ let interrupted = gate.try_acquire_owned().unwrap();
+ drop(interrupted);
+ assert!(gate.try_acquire_owned().is_none());
+ }
/// Two writers queued on the same gate run one after the other, never
/// interleaved -- the property a torn superblock slot depends on.
diff --git a/core/partitions/src/consumer_offset_capacity.rs b/core/partitions/src/consumer_offset_capacity.rs
index f4554c642..d1cf35e34 100644
--- a/core/partitions/src/consumer_offset_capacity.rs
+++ b/core/partitions/src/consumer_offset_capacity.rs
@@ -324,6 +324,7 @@ impl ConsumerOffsetCapacity {
}
/// See [`Self::set_pending_count`] for the serialization contract.
+ #[cfg(test)]
pub(crate) fn release_reservation(&self, id: u32) {
let mut pending = self.pending.borrow_mut();
let Some(count) = pending.get_mut(&id) else {
@@ -383,6 +384,10 @@ impl ConsumerOffsetCapacity {
self.stranded.borrow().len()
}
+ pub(crate) fn extend_stranded_ids(&self, ids: &mut HashSet<u32>) {
+ ids.extend(self.stranded.borrow().iter().copied());
+ }
+
pub(crate) fn rearm_if_below_limit(&self, durable: &DurableConsumerOffsets) {
if !self.durable_warned.get()
|| self.uncertain.get()
diff --git a/core/partitions/src/iggy_index_writer.rs b/core/partitions/src/iggy_index_writer.rs
index 400cf81bd..7a86dc064 100644
--- a/core/partitions/src/iggy_index_writer.rs
+++ b/core/partitions/src/iggy_index_writer.rs
@@ -99,7 +99,7 @@ impl IggyIndexWriter {
})
}
- /// Appends encoded sparse index bytes at the current write cursor and
+ /// Writes encoded sparse index bytes at a captured file position and
/// returns how many bytes landed. The cursor is left where it was: the
/// caller advances it with `advance` once the companion segment save has
/// also succeeded.
@@ -107,21 +107,28 @@ impl IggyIndexWriter {
/// # Errors
///
/// Returns an error if the index bytes cannot be written or synced to disk.
- pub(crate) async fn save_indexes(&self, indexes: Vec<u8>) -> Result<u64, IggyError> {
- let saved = self.save_indexes_buffered(indexes).await?;
+ pub(crate) async fn save_indexes_at(
+ &self,
+ indexes: Vec<u8>,
+ position: u64,
+ ) -> Result<u64, IggyError> {
+ let saved = self.save_indexes_buffered_at(indexes, position).await?;
if saved > 0 && self.fsync {
self.fsync().await?;
}
Ok(saved)
}
- pub(crate) async fn save_indexes_buffered(&self, indexes: Vec<u8>) -> Result<u64, IggyError> {
+ pub(crate) async fn save_indexes_buffered_at(
+ &self,
+ indexes: Vec<u8>,
+ position: u64,
+ ) -> Result<u64, IggyError> {
if indexes.is_empty() {
return Ok(0);
}
let len = indexes.len();
- let position = self.index_size_bytes.load(Ordering::Relaxed);
let file = &self.file;
(&*file)
.write_all_at(indexes, position)
@@ -150,6 +157,10 @@ impl IggyIndexWriter {
&self.file_path
}
+ pub(crate) fn position(&self) -> u64 {
+ self.index_size_bytes.load(Ordering::Relaxed)
+ }
+
/// Flushes buffered index file contents to disk.
///
/// Uses `fdatasync` (data only): index files are append-only and the
@@ -178,9 +189,15 @@ mod tests {
let writer = IggyIndexWriter::new("/dev/null", Rc::new(AtomicU64::new(0)), true, false)
.await
.unwrap();
- assert_eq!(writer.save_indexes_buffered(vec![1; 32]).await.unwrap(), 32);
+ assert_eq!(
+ writer
+ .save_indexes_buffered_at(vec![1; 32], 0)
+ .await
+ .unwrap(),
+ 32
+ );
assert!(writer.fsync().await.is_err());
- assert!(writer.save_indexes(vec![1; 32]).await.is_err());
+ assert!(writer.save_indexes_at(vec![1; 32], 0).await.is_err());
}
#[compio::test]
diff --git a/core/partitions/src/iggy_partition.rs b/core/partitions/src/iggy_partition.rs
index 9260bb383..bae775ebf 100644
--- a/core/partitions/src/iggy_partition.rs
+++ b/core/partitions/src/iggy_partition.rs
@@ -16,18 +16,14 @@
// under the License.
use crate::consumer_offset_capacity::{
- ConsumerOffsetCapacity, ConsumerOffsetCapacityError, DurableConsumerOffsets,
+ ConsumerOffsetCapacity, ConsumerOffsetCapacityError, DurableConsumerOffsets, DurableOffsetState,
+};
+use crate::journal::{
+ MessageLookup, PartitionJournal, PartitionJournalMemStorage, RetainedBatchMeta,
};
-use crate::iggy_index_writer::IggyIndexWriter;
-use crate::journal::{MessageLookup, PartitionJournal, PartitionJournalMemStorage};
use crate::log::JournalInfo;
use crate::log::SegmentedLog;
-use crate::messages_writer::MessagesWriter;
-use crate::offset_storage::{
- PURGE_GENERATION_FILE, delete_persisted_offset, delete_persisted_offset_with_storage,
- persist_offset, persist_offset_max, persist_purge_generation_with_storage,
- read_purge_generation,
-};
+use crate::offset_storage::{PURGE_GENERATION_FILE, read_purge_generation};
use crate::persistence::{
CheckpointBarrier, PartitionPersistence, PersistenceCompletion, PersistenceNotifier,
};
@@ -36,25 +32,26 @@ use crate::poll_plan::{
PollTier, ResidentTailSnapshot,
};
use crate::segment::Segment;
-use crate::state_transfer::{PartitionTransferSession, PendingTransferRearm};
+use crate::state_transfer::{
+ PartitionTransferSession, PartitionTransferUnavailable, PendingTransferRearm, TransferPlanSlot,
+};
use crate::types::{COMMIT_WALK_OPS_MAX, FatalCommit, RepairConclusion, RepairSession};
use crate::{
AppendResult, Partition, PartitionOffsets, PartitionsConfig, PollFragments, PollQueryResult,
PollingArgs, PollingConsumer,
};
use consensus::Pipeline;
-use consensus::client_table::CachedReply;
+use consensus::client_table::{CachedReply, ClientTableWireError};
use consensus::{
AutoCommitRequestContext, ClientTable, ClientTableMode, CommitLogEvent, Consensus,
- PartitionDiagEvent, PipelineEntry, PlaneKind, Project, ReplicaLogContext, RequestLogEvent,
- Sequencer, SimEventKind, VsrConsensus, ack_preflight, ack_quorum_reached,
+ LocalRequestOrder, PartitionDiagEvent, PipelineEntry, PlaneKind, Project, ReplicaLogContext,
+ RequestLogEvent, Sequencer, SimEventKind, VsrConsensus, ack_preflight, ack_quorum_reached,
build_deny_reply_from_request, build_reply_from_request, build_reply_message,
- drain_committable_prefix, emit_namespace_progress_event, emit_partition_diag, emit_sim_event,
+ emit_namespace_progress_event, emit_partition_diag, emit_sim_event,
fence_old_prepare_by_commit, repair_session_live, repaired_frontier_update,
- replicate_frozen_to_next_in_chain, replicate_preflight, report_uncommittable_head,
- restamp_prepare_view, send_prepare_ok as send_prepare_ok_common, verify_prepare_integrity,
+ replicate_frozen_to_next_in_chain, replicate_preflight, restamp_prepare_view,
+ send_prepare_ok as send_prepare_ok_common, verify_prepare_integrity,
};
-use futures::{StreamExt, TryStreamExt};
use iggy_binary_protocol::primitives::consumer::WireConsumer;
use iggy_binary_protocol::requests::consumer_offsets::{
DeleteConsumerOffsetRequest, StoreConsumerOffsetRequest,
@@ -91,11 +88,10 @@ use server_common::{
sharding::IggyNamespace,
};
use std::cell::{Cell, RefCell};
-use std::collections::{BTreeMap, HashMap, HashSet};
+use std::collections::{BTreeMap, HashMap, HashSet, VecDeque};
use std::fmt;
use std::hash::Hash;
-use std::num::NonZeroU32;
-use std::path::Path;
+use std::num::{NonZeroU32, NonZeroU64};
use std::rc::Rc;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
@@ -230,9 +226,23 @@ where
/// into a no-op, so the shard's tick can observe the fault and shut the
/// server down without the partition moving again in the meantime.
fatal: Option<FatalCommit>,
+ pending_commit: Option<PendingCommit>,
+ local_request_order: Cell<u64>,
+ no_ack: VecDeque<PendingNoAck>,
+ materialization: Option<PendingMaterialization>,
+ maintenance: PendingPartitionMaintenance,
+ io_notifier: RefCell<Option<crate::PartitionIoNotifier>>,
+ io_bytes_max: Cell<usize>,
+ io_sequence: Cell<u64>,
+ io_active: Rc<crate::io::PartitionIoQuiescence>,
+ pub(crate) transition: Option<PendingPartitionTransition>,
+ io_replies: VecDeque<(PipelineEntry, Option<CachedReply>)>,
+ io_promotions: usize,
pub(crate) pending_consumer_offset_commits: HashMap<u64, PendingConsumerOffsetCommit>,
/// Identity shared with pending polls and replaced when their history retires.
poll_history: PollHistoryId,
+ incarnation: crate::PartitionIncarnation,
+ workerless_offset_files: crate::offset_storage::RetainedOffsetFiles<compio::fs::File>,
/// Committed consumer-offset membership and values. This is deliberately
/// separate from the eager poll maps because follower-local and uncommitted
/// auto-commit progress must never consume a durable slot or enter a state
@@ -291,7 +301,11 @@ where
/// it while both report success. Per partition, not per shard -- every
/// group owns its own two-file store, so writes to different partitions
/// never contend.
- superblock_lock: LocalGate,
+ superblock_lock: Rc<LocalGate>,
+ superblock_intent: Cell<Option<SuperblockIntent>>,
+ superblock_running: Cell<Option<(SuperblockIntent, bool)>>,
+ superblock_completed: Cell<Option<(SuperblockIntent, bool)>>,
+ pending_wire_actions: RefCell<Option<PendingWireActions>>,
/// Consecutive failed superblock writes, and the clock reading after which
/// the next attempt may run. A persistent `ENOSPC` / `EIO` would otherwise
/// re-run a full `atomic_replace` on every 10 ms consensus tick. Reset on
@@ -392,6 +406,10 @@ where
/// different bytes) and released by the shard's offer-expiry sweep.
pub(crate) transfer_offer_cache:
RefCell<Option<Rc<crate::state_transfer::PartitionStateTransferOffer>>>,
+ /// Serving-side plan the rounds of one offer build share. Released when
+ /// the build ends, with the offer cache, and with the history it
+ /// describes.
+ pub(crate) transfer_plan: RefCell<Option<TransferPlanSlot>>,
}
impl<B, SB> fmt::Debug for IggyPartition<B, SB>
@@ -463,6 +481,7 @@ enum Disposition {
/// stale counter as durable truth.
#[derive(Debug)]
pub enum PurgeError {
+ Pending,
/// The frontier reset could not be recorded. NOTHING was mutated: the
/// segments, the counters and `applied_purge_generation` are all untouched,
/// so the reconciler's `committed > applied` gate re-issues this purge on
@@ -512,6 +531,7 @@ pub enum PurgeError {
impl fmt::Display for PurgeError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
+ Self::Pending => f.write_str("partition purge is pending"),
Self::FrontierNotRecorded => write!(
f,
"could not record the purge's offset-frontier reset; nothing was mutated"
@@ -554,6 +574,265 @@ enum PendingConsumerOffsetMutation {
Delete,
}
+#[derive(Clone, Copy)]
+struct ConsumerOffsetEffect {
+ pending: PendingConsumerOffsetCommit,
+ state: Option<DurableOffsetState>,
+}
+
+struct PendingNoAck {
+ request_header: Box<RoutedRequestHeader>,
+ pending: PendingConsumerOffsetCommit,
+ reply: Option<consensus::Sender<Message<ReplyHeader>>>,
+ poll_history: PollHistoryId,
+ local_order: LocalRequestOrder,
+ reservation: AutoCommitReservation,
+ phase: NoAckPhase,
+}
+
+enum NoAckPhase {
+ Validate,
+ OffsetReady(ConsumerOffsetPlan),
+ OffsetSubmitted(ConsumerOffsetPlan),
+ Directory,
+ DirectorySubmitted,
+ Reply(Result<(), IggyError>),
+}
+
+enum NoAckStep {
+ Progress,
+ Pending,
+ Ready,
+ Reply,
+}
+
+pub enum PendingPartitionTransition {
+ View {
+ message: server_common::MessageBag,
+ drain: Option<crate::PersistenceDrain>,
+ },
+ Truncate {
+ from_op: u64,
+ drain: Option<crate::PersistenceDrain>,
+ actions: Vec<consensus::VsrAction>,
+ peer: Option<u8>,
+ },
+ Purge(PendingPurge),
+ Install(Box<crate::state_transfer::PendingInstall>),
+ Quarantine {
+ frontier: Option<u64>,
+ drain: Option<crate::PersistenceDrain>,
+ phase: QuarantinePhase,
+ },
+}
+
+pub enum QuarantinePhase {
+ Drain,
+ Frontier,
+ Files,
+ Submitted,
+ Done(std::io::Result<Option<String>>),
+}
+
+pub struct PendingPurge {
+ generation: u64,
+ drain: Option<crate::PersistenceDrain>,
+ phase: PurgePhase,
+ /// Only the `purge.gen` write of an earlier attempt failed.
+ record_only: bool,
+}
+
+enum PurgePhase {
+ Drain,
+ Frontier,
+ Segments,
+ Cleanup,
+ Plant,
+ PartitionDirectory,
+ Offsets(usize),
+ OffsetDirectory(usize),
+ Generation,
+ FinalFrontier,
+ Submitted(PurgeFilePhase),
+ Done(Result<(), PurgeError>),
+}
+
+enum PurgeFilePhase {
+ Segment,
+ Cleanup,
+ Plant,
+ PartitionDirectory,
+ Offsets(usize),
+ OffsetDirectory(usize),
+ Generation,
+}
+
+#[derive(Clone, Copy)]
+struct ConsumerOffsetPlan {
+ pending: PendingConsumerOffsetCommit,
+ tracked: Option<DurableOffsetState>,
+ persisted: bool,
+ action: ConsumerOffsetAction,
+}
+
+#[derive(Clone, Copy)]
+enum ConsumerOffsetAction {
+ Ready(ConsumerOffsetEffect),
+ Write { offset: u64, fold_max: bool },
+ Delete,
+}
+
+/// Selected headers and replies remain in their original owners until all
+/// physical phases settle. Journal eviction is deferred for the same lifetime.
+struct PendingCommit {
+ headers: Vec<PrepareHeader>,
+ journal_headers: Vec<Option<PrepareHeader>>,
+ batch_stats: Vec<Option<CommittedBatchStats>>,
+ retirements: Vec<(u64, RetireSessionsRequest)>,
+ offset_effects: Vec<ConsumerOffsetEffect>,
+ incarnation: crate::PartitionIncarnation,
+ poll_history: PollHistoryId,
+ applied_floor: u64,
+ from_pipeline: bool,
+ send_client_replies: bool,
+ materialize: bool,
+ phase: CommitPhase,
+ next_entry: usize,
+ evictable: usize,
+}
+
+impl PendingCommit {
+ fn has_sends(&self) -> bool {
+ self.headers
+ .iter()
+ .any(|header| header.operation == Operation::SendMessages)
+ }
+}
+
+#[derive(Clone, Copy)]
+enum CommitPhase {
+ Start,
+ Materialize,
+ Offsets,
+ OffsetReady(ConsumerOffsetPlan),
+ OffsetSubmitted(ConsumerOffsetPlan),
+ SegmentDirectory,
+ SegmentDirectorySubmitted,
+ OffsetDirectories,
+ OffsetDirectoriesSubmitted,
+ Publish,
+}
+
+enum CommitStep {
+ Progress,
+ Pending,
+ Ready,
+ Publish,
+}
+
+struct PendingMaterialization {
+ incarnation: crate::PartitionIncarnation,
+ poll_history: PollHistoryId,
+ through_op: u64,
+ prefix_len: usize,
+ next_entry: usize,
+ durable_offset: Option<u64>,
+ phase: MaterializationPhase,
+ save: Option<PendingMaterializationSave>,
+}
+
+#[derive(Clone, Copy)]
+struct PendingMaterializationSave {
+ chunk: MaterializationChunk,
+ written: u64,
+}
+
+#[derive(Default)]
+struct PendingPartitionMaintenance {
+ flush_through: Option<u64>,
+ flush_drain: Option<crate::PersistenceDrain>,
+ flush_completed_through: Option<u64>,
+ transfer_flush: bool,
+ shutdown: Option<PartitionShutdownPhase>,
+ shutdown_drain: Option<crate::PersistenceDrain>,
+ checkpoint: Option<PendingCheckpoint>,
+ retention: Option<u64>,
+ removing: Option<Segment>,
+}
+
+struct PendingWireActions {
+ view: u32,
+ log_view: u32,
+ actions: Vec<consensus::VsrAction>,
+}
+
+enum PendingCheckpoint {
+ /// The receipt table at the checkpoint op. Commits keep advancing while the
+ /// checkpoint runs, so it is captured before any other phase.
+ Receipts(u64, Vec<u8>),
+ ReceiptsWriting(u64),
+ Flush(u64),
+ Index(u64),
+ Syncing(u64),
+ /// Carries the index file already synced through its original writer.
+ Enqueue(u64, Option<String>),
+ Wal(u64),
+ /// Receipt files older than the completed WAL checkpoint.
+ Reclaim(u64),
+ Reclaiming,
+}
+
+#[derive(Clone, Copy, PartialEq, Eq)]
+enum PartitionShutdownPhase {
+ DrainAccepted,
+ Flush,
+ DrainWriters,
+ Collapse,
+ Complete,
+}
+
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum SuperblockIntent {
+ Advance { frontier: u64, reserved: u64 },
+ Reset(u64),
+ Install(u64),
+ Quarantine(Option<u64>),
+ Collapse { frontier: u64, reserved: u64 },
+}
+
+impl SuperblockIntent {
+ const fn final_write(self) -> bool {
+ matches!(self, Self::Quarantine(_) | Self::Collapse { .. })
+ }
+}
+
+#[derive(Clone, Copy)]
+enum MaterializationPhase {
+ Inspect,
+ Save(MaterializationChunk),
+ Rotate(u64),
+ Saving(MaterializationChunk),
+ Rotating,
+}
+
+#[derive(Clone, Copy)]
+struct MaterializationChunk {
+ first_entry: usize,
+ end_entry: usize,
+ persisted_end: Option<u64>,
+ buffers: usize,
+ allocation_charge: usize,
+ index: Option<crate::iggy_index::IggyIndex>,
+ info: JournalInfo,
+}
+
+enum MaterializationStep {
+ Progress,
+ Pending,
+ Ready,
+ Complete,
+}
+
impl PendingConsumerOffsetCommit {
const fn upsert(kind: ConsumerKind, consumer_id: u32, offset: u64) -> Self {
Self {
@@ -583,6 +862,27 @@ impl PendingConsumerOffsetCommit {
}
}
+ /// An explicit client mutation: a store for `Some(offset)`, a delete for
+ /// `None`.
+ const fn explicit(kind: ConsumerKind, consumer_id: u32, offset: Option<u64>) -> Self {
+ match offset {
+ Some(offset) => Self::upsert(kind, consumer_id, offset),
+ None => Self::delete(kind, consumer_id),
+ }
+ }
+
+ /// Terminal refusal for a mutation admitted before a reset of the poll
+ /// history. A retry would store the old offset, or delete a checkpoint, in
+ /// the replacement history.
+ const fn history_reset_refusal(&self) -> IggyError {
+ match self.mutation {
+ PendingConsumerOffsetMutation::Upsert(offset) => IggyError::InvalidOffset(offset),
+ PendingConsumerOffsetMutation::Delete => {
+ IggyError::ConsumerOffsetNotFound(self.consumer_id as usize)
+ }
+ }
+ }
+
fn try_from_polling_consumer(
consumer: PollingConsumer,
offset: u64,
@@ -614,6 +914,7 @@ where
B: MessageBus,
SB: SuperblockStore,
{
+ #[allow(clippy::too_many_lines)]
pub fn new(stats: Arc<PartitionStats>, consensus: VsrConsensus<B>) -> Self {
let observed_view = consensus.view();
let single_replica = consensus.replica_count() == 1;
@@ -653,7 +954,21 @@ where
installed_frontier: None,
fatal: None,
pending_consumer_offset_commits: HashMap::new(),
+ pending_commit: None,
+ local_request_order: Cell::new(0),
+ no_ack: VecDeque::new(),
+ materialization: None,
+ maintenance: PendingPartitionMaintenance::default(),
+ io_notifier: RefCell::new(None),
+ io_bytes_max: Cell::new(isize::MAX as usize),
+ io_sequence: Cell::new(0),
+ io_active: Rc::default(),
+ transition: None,
+ io_replies: VecDeque::new(),
+ io_promotions: 0,
poll_history: PollHistoryId::default(),
+ incarnation: crate::PartitionIncarnation::default(),
+ workerless_offset_files: crate::offset_storage::RetainedOffsetFiles::default(),
durable_consumer_offsets: DurableConsumerOffsets::default(),
consumer_offset_capacities: ConsumerKind::ALL
.map(|kind| ConsumerOffsetCapacity::new(kind, crate::DEFAULT_CONSUMER_OFFSETS_MAX)),
@@ -674,7 +989,11 @@ where
created_revision: 0,
purge_floor_op: 0,
superblock: None,
- superblock_lock: LocalGate::new(),
+ superblock_lock: Rc::new(LocalGate::new()),
+ superblock_intent: Cell::new(None),
+ superblock_running: Cell::new(None),
+ superblock_completed: Cell::new(None),
+ pending_wire_actions: RefCell::new(None),
superblock_write_failures: Cell::new(0),
superblock_retry_after_micros: Cell::new(0),
purge_deferred: false,
@@ -691,6 +1010,7 @@ where
segment_checksum_cache: RefCell::new(std::collections::HashMap::new()),
reuse_scan_memo: RefCell::new(None),
transfer_offer_cache: RefCell::new(None),
+ transfer_plan: RefCell::new(None),
};
if single_replica {
partition.log.journal().inner.set_repair_retention(false);
@@ -725,7 +1045,9 @@ where
}
/// A report covers only an applied barrier on a serving replica. WAL replay
- /// or the paired checkpoint must restore this floor before admission.
+ /// or the paired checkpoint must restore this floor before admission. A
+ /// crash can lose the barrier of a Replicated partition, see
+ /// `requires_persistence`.
#[must_use]
pub fn session_retired(&self, identity: SessionIdentity) -> bool {
self.fatal.is_none()
@@ -877,6 +1199,114 @@ where
}
}
+ pub fn capture_teardown(&self) -> crate::PartitionTeardown {
+ self.io_active.delete();
+ self.workerless_offset_files.clear();
+ crate::PartitionTeardown {
+ io: Rc::clone(&self.io_active),
+ persistence: self.persistence.clone(),
+ }
+ }
+
+ pub fn stop_io_admission(&self) {
+ self.io_active.retire();
+ }
+
+ pub fn begin_quarantine(&mut self, frontier: Option<u64>) {
+ if self
+ .superblock_intent
+ .get()
+ .is_some_and(|intent| !intent.final_write())
+ {
+ self.superblock_intent.set(None);
+ }
+ if !matches!(
+ self.transition,
+ Some(PendingPartitionTransition::Quarantine { .. })
+ ) {
+ self.transition = Some(PendingPartitionTransition::Quarantine {
+ frontier,
+ drain: None,
+ phase: QuarantinePhase::Drain,
+ });
+ }
+ self.io_active.retire();
+ self.notify_io();
+ }
+
+ async fn drive_quarantine_io(&mut self) -> crate::PartitionIoStep {
+ let Some(PendingPartitionTransition::Quarantine {
+ frontier,
+ mut drain,
+ mut phase,
+ }) = self.transition.take()
+ else {
+ return crate::PartitionIoStep::Pending;
+ };
+ let mut step = crate::PartitionIoStep::Progress;
+ match phase {
+ QuarantinePhase::Drain => {
+ self.pending_commit = None;
+ self.materialization = None;
+ self.no_ack.clear();
+ self.workerless_offset_files.clear();
+ if let Some(persistence) = &self.persistence {
+ if drain.is_none() {
+ persistence.retire();
+ drain = Some(persistence.begin_drain());
+ }
+ match persistence
+ .observe_drain(drain.as_ref().expect("quarantine drain exists"))
+ {
+ Ok(true) => phase = QuarantinePhase::Frontier,
+ Ok(false) => step = crate::PartitionIoStep::Pending,
+ Err(error) => phase = QuarantinePhase::Done(Err(error)),
+ }
+ } else {
+ phase = QuarantinePhase::Frontier;
+ }
+ }
+ QuarantinePhase::Frontier => match self
+ .transition_superblock(SuperblockIntent::Quarantine(frontier))
+ .await
+ {
+ crate::PartitionIoVerdict::Pending => step = crate::PartitionIoStep::Pending,
+ verdict => {
+ if verdict == crate::PartitionIoVerdict::Failed {
+ error!(
+ namespace_raw = self.namespace().inner(),
+ "quarantine frontier write failed; retaining the last durable record"
+ );
+ }
+ phase = QuarantinePhase::Files;
+ }
+ },
+ QuarantinePhase::Files => {
+ if self.partition_dir.is_none() {
+ phase = QuarantinePhase::Done(Ok(None));
+ } else {
+ match self.io_plan(crate::PartitionIoContinuation::Quarantine) {
+ Ok(ready) => step = ready,
+ Err(error) => {
+ phase = QuarantinePhase::Done(Err(std::io::Error::other(error)));
+ }
+ }
+ }
+ }
+ QuarantinePhase::Submitted => step = crate::PartitionIoStep::Pending,
+ QuarantinePhase::Done(outcome) => {
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::Complete);
+ return crate::PartitionIoStep::QuarantineFinished(outcome);
+ }
+ }
+ self.transition = Some(PendingPartitionTransition::Quarantine {
+ frontier,
+ drain,
+ phase,
+ });
+ step
+ }
+
/// # Errors
/// Returns an error if durable prepare history cannot be opened or replayed.
pub async fn open_persistence(&mut self) -> Result<(), IggyError> {
@@ -1002,6 +1432,7 @@ where
if !self.materialization_missing && persistence.recovered_frontier() {
self.consensus.set_recovery_election_allowed(true);
}
+ self.workerless_offset_files.clear();
if self.materialization_missing {
self.persistence = Some(persistence);
return Ok(());
@@ -1139,6 +1570,7 @@ where
{
self.commit_journal(config).await;
}
+ self.notify_io();
}
pub fn needs_persistence_checkpoint(&self) -> bool {
@@ -1165,6 +1597,26 @@ where
if through_op <= previous_checkpoint || through_op > persistence.head() {
return;
}
+ if self.has_io_dispatcher() {
+ if self.maintenance.checkpoint.is_none() {
+ match self.encode_retry_checkpoint(through_op) {
+ Ok(receipts) => {
+ self.maintenance.checkpoint =
+ Some(PendingCheckpoint::Receipts(through_op, receipts));
+ self.notify_io();
+ }
+ Err(error) => {
+ error!(%error, namespace_raw = self.namespace().inner(), "partition receipt checkpoint failed");
+ self.fatal = Some(FatalCommit {
+ namespace_raw: self.namespace().inner(),
+ op: through_op,
+ operation: None,
+ });
+ }
+ }
+ }
+ return;
+ }
match self.commit_messages_inner(config, true, through_op).await {
Ok(true) => {}
Ok(false) => return,
@@ -1274,6 +1726,162 @@ where
)
}
+ fn drive_checkpoint_io(
+ &mut self,
+ config: &PartitionsConfig,
+ ) -> Result<Option<crate::PartitionIoStep>, IggyError> {
+ let Some(checkpoint) = &self.maintenance.checkpoint else {
+ return Ok(None);
+ };
+ match *checkpoint {
+ PendingCheckpoint::Receipts(..) | PendingCheckpoint::Reclaim(_) => self
+ .io_plan(crate::PartitionIoContinuation::Checkpoint)
+ .map(Some),
+ PendingCheckpoint::ReceiptsWriting(_)
+ | PendingCheckpoint::Syncing(_)
+ | PendingCheckpoint::Reclaiming => Ok(Some(crate::PartitionIoStep::Pending)),
+ PendingCheckpoint::Flush(through) => {
+ if !self.begin_materialization(through)? {
+ return Ok(Some(crate::PartitionIoStep::Pending));
+ }
+ if self.materialization.is_none() {
+ self.maintenance.checkpoint = Some(PendingCheckpoint::Index(through));
+ }
+ Ok(Some(crate::PartitionIoStep::Progress))
+ }
+ PendingCheckpoint::Index(through) => {
+ if self
+ .log
+ .index_writers()
+ .last()
+ .and_then(Option::as_ref)
+ .is_some()
+ {
+ self.io_plan(crate::PartitionIoContinuation::Checkpoint)
+ .map(Some)
+ } else {
+ self.maintenance.checkpoint = Some(PendingCheckpoint::Enqueue(through, None));
+ Ok(Some(crate::PartitionIoStep::Progress))
+ }
+ }
+ PendingCheckpoint::Enqueue(through, ref synced_index) => {
+ let barriers = synced_index
+ .iter()
+ .map(CheckpointBarrier::already_synced)
+ .collect();
+ let (files, directories) = self.persistence_checkpoint_files(config);
+ if let Some(persistence) = &self.persistence {
+ persistence.checkpoint_files(through, files, directories, barriers);
+ self.start_persistence();
+ }
+ self.maintenance.checkpoint = Some(PendingCheckpoint::Wal(through));
+ Ok(Some(crate::PartitionIoStep::Progress))
+ }
+ PendingCheckpoint::Wal(through) => {
+ let Some(persistence) = &self.persistence else {
+ self.maintenance.checkpoint = None;
+ return Ok(None);
+ };
+ let checkpoint_op = persistence.checkpoint_op();
+ if checkpoint_op < through {
+ return Ok(None);
+ }
+ self.maintenance.checkpoint = Some(PendingCheckpoint::Reclaim(checkpoint_op));
+ Ok(Some(crate::PartitionIoStep::Progress))
+ }
+ }
+ }
+
+ fn capture_checkpoint_phase(
+ &mut self,
+ ) -> Result<Option<crate::io::PartitionIoJob<SB>>, IggyError> {
+ let job = match &mut self.maintenance.checkpoint {
+ Some(PendingCheckpoint::Receipts(through, receipts)) => {
+ let through = *through;
+ let bytes = std::mem::take(receipts);
+ let path = self
+ .retry_checkpoint_path(through)
+ .map_err(|_| IggyError::CannotWriteToFile)?;
+ self.maintenance.checkpoint = Some(PendingCheckpoint::ReceiptsWriting(through));
+ crate::PartitionIoJob::RetryCheckpoint { path, bytes }
+ }
+ Some(PendingCheckpoint::Index(through)) => {
+ let through = *through;
+ let writer = self
+ .log
+ .index_writers()
+ .last()
+ .and_then(Option::as_ref)
+ .ok_or(IggyError::CannotSyncFile)?;
+ let job = crate::PartitionIoJob::IndexSync(Rc::clone(writer));
+ self.maintenance.checkpoint = Some(PendingCheckpoint::Syncing(through));
+ job
+ }
+ Some(PendingCheckpoint::Reclaim(through)) => {
+ let through = *through;
+ let current = self
+ .retry_checkpoint_path(through)
+ .map_err(|_| IggyError::CannotDeleteFile)?;
+ self.maintenance.checkpoint = Some(PendingCheckpoint::Reclaiming);
+ crate::PartitionIoJob::ReclaimRetryCheckpoints {
+ current,
+ keep_current: through > 0,
+ }
+ }
+ _ => return Ok(None),
+ };
+ Ok(Some(job))
+ }
+
+ fn accept_checkpoint_io(&mut self, result: crate::PartitionIoResult) -> Result<(), IggyError> {
+ match (self.maintenance.checkpoint.take(), result) {
+ (
+ Some(PendingCheckpoint::ReceiptsWriting(through)),
+ crate::PartitionIoResult::RetryCheckpoint(outcome),
+ ) => {
+ if let Err(error) = outcome {
+ error!(%error, namespace_raw = self.namespace().inner(), "partition receipt checkpoint failed");
+ return Err(IggyError::CannotSyncFile);
+ }
+ self.maintenance.checkpoint = Some(PendingCheckpoint::Flush(through));
+ Ok(())
+ }
+ (
+ Some(PendingCheckpoint::Syncing(through)),
+ crate::PartitionIoResult::IndexSync { writer, outcome },
+ ) => {
+ outcome?;
+ if !self
+ .log
+ .index_writers()
+ .last()
+ .and_then(Option::as_ref)
+ .is_some_and(|installed| Rc::ptr_eq(installed, &writer))
+ {
+ return Err(IggyError::CannotSyncFile);
+ }
+ self.maintenance.checkpoint = Some(PendingCheckpoint::Enqueue(
+ through,
+ Some(writer.path().to_owned()),
+ ));
+ Ok(())
+ }
+ (
+ Some(PendingCheckpoint::Reclaiming),
+ crate::PartitionIoResult::ReclaimRetryCheckpoints(outcome),
+ ) => {
+ if let Err(error) = outcome {
+ warn!(%error, namespace_raw = self.namespace().inner(), "cannot reclaim obsolete receipt checkpoints");
+ }
+ Ok(())
+ }
+ (checkpoint, _) => {
+ self.maintenance.checkpoint = checkpoint;
+ Err(IggyError::CannotSyncFile)
+ }
+ }
+ }
+
pub fn take_persistence_metrics(&self) -> Option<crate::persistence::PersistenceMetrics> {
self.persistence
.as_ref()
@@ -1291,6 +1899,21 @@ where
.is_some_and(|persistence| persistence.checkpoint_pending())
}
+ /// Persisted commits keep their earned quorum and reply slots in the
+ /// pipeline while the checkpoint worker synchronizes this partition's
+ /// materialized files. Replicated commits continue.
+ fn commit_waits_for_checkpoint(&self) -> bool {
+ self.durability().is_persisted() && self.persistence_checkpoint_pending()
+ }
+
+ /// The checkpoint worker syncs the retained offset handles and opens the
+ /// dirty offset paths. A write in its window reopens a file with a
+ /// truncation and a delete removes a listed path, so the checkpoint could
+ /// sync an empty file or fail.
+ fn offset_writes_wait_for_checkpoint(&self) -> bool {
+ self.persistence_checkpoint_pending()
+ }
+
pub async fn drive_persistence(&mut self) {
if self.fatal.is_some() {
return;
@@ -1352,7 +1975,12 @@ where
const fn requires_persistence(&self, operation: Operation) -> bool {
match operation {
- Operation::RetireSession => true,
+ // Follows the partition policy, so that a retirement sweep does
+ // not sync a Replicated WAL. A crash can then lose the fence and
+ // bring back the receipts that the last checkpoint holds.
+ Operation::RetireSession => {
+ self.durability().is_persisted() || self.consumer_offset_durability().is_persisted()
+ }
Operation::SendMessages => self.durability().is_persisted(),
Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset => {
self.consumer_offset_durability().is_persisted()
@@ -1368,7 +1996,15 @@ where
if persistence.certified_log_view() == Some(self.consensus.log_view()) {
return true;
}
- if self.materialization_missing || !self.consensus.is_normal() {
+ // An install certifies the WAL it reset. Until it publishes its consensus,
+ // the sequencer still names the history that the install replaces.
+ if self.materialization_missing
+ || !self.consensus.is_normal()
+ || matches!(
+ self.transition,
+ Some(PendingPartitionTransition::Install(_))
+ )
+ {
return false;
}
self.persist_repaired_prefix();
@@ -1512,81 +2148,75 @@ where
/// `true` when the send may proceed, either because the state is now
/// durable or because there was nothing to persist (no store attached --
/// in-memory / simulated partitions -- or an unchanged view). `false`
- /// only when a write was attempted and failed, and the caller must
- /// withhold the send. The in-memory view stays ahead of the durable one,
- /// which a crash safely rolls back, and the next tick retries.
+ /// while a write is queued, in progress, or failed. Callers retain wire
+ /// actions with [`Self::defer_wire_actions`] until persistence completes.
#[allow(clippy::future_not_send)]
#[must_use = "the bool is the durability verdict; dropping it silently ignores a failed write"]
pub async fn persist_superblock_if_needed(&self) -> bool {
if !self.materialization_missing && !self.ensure_wal_view() {
return false;
}
- let Some(superblock) = self.superblock.as_ref() else {
- // No store (in-memory / simulated partitions): nothing can be
- // recorded, so keep the durable cells current instead. The
- // dispatch tripwire asserts `needs_superblock_persist()` is clear
- // on every view-scoped send, and for a storeless group "current"
- // is trivially true -- leaving the cells behind would trip it on
- // the first view change.
+ if self.superblock.is_none() {
self.consensus
.mark_superblock_durable(self.consensus.view(), self.consensus.log_view());
return true;
- };
- // Lock-free fast path: the steady state is an unchanged view with
- // nothing to write, and skipping the lock keeps every gated send off
- // it, notably `send_prepare_ok`, which runs this per prepare. Safe
- // because `view`/`log_view` advance only on this single-threaded
- // executor and no `.await` sits between the `Cell` read and the
- // return; a concurrent advance is caught by the re-check below.
- if !self.consensus.needs_superblock_persist() {
- return true;
- }
- // A write that keeps failing must not re-run a full `atomic_replace`
- // on every 10 ms tick. Back off first, while still reporting `false`
- // so the send stays withheld: fail-closed is the point of this gate,
- // and the backoff only bounds what the retry costs.
- if self.superblock_write_is_backed_off() {
- return false;
}
- // Re-check needs-persist AFTER acquiring the lock so check and write
- // are atomic and a redundant caller coalesces, finding the state
- // already made durable by the writer it queued behind.
- let _superblock_guard = self.superblock_lock.acquire().await;
if !self.consensus.needs_superblock_persist() {
return true;
}
- self.write_superblock(superblock.as_ref(), self.offset_frontier())
- .await
+ self.complete_superblock_intent(SuperblockIntent::Advance {
+ frontier: self.offset_frontier(),
+ reserved: 0,
+ })
+ .await
}
- /// Write the current VSR state under [`Self::superblock_lock`].
- ///
- /// The caller must hold that lock. The state is captured HERE rather than
- /// passed in: with writes serialized and no await between the capture and
- /// the write, the last writer carries the freshest view, so the durable
- /// view cannot regress. `mark_superblock_durable` takes the WRITTEN
- /// values, never a re-read, because the in-memory view can advance across
- /// the write's `.await`.
- ///
- /// # Terminal policy
- /// There is none beyond staying fenced: a replica that cannot record the
- /// view it is in must not act in it, so it withholds every view-scoped
- /// send for this group, goes quiet, and its peers elect around it. Only
- /// THIS partition's group is fenced; the rest of the node keeps serving.
- /// A RUN of failures is terminal for the process, not the group: the shard
- /// tick fail-stops on `superblock_wedged`.
- #[allow(clippy::future_not_send)]
- async fn write_superblock(&self, superblock: &SB, offset_frontier: u64) -> bool {
- self.write_superblock_advancing(superblock, offset_frontier, 0)
- .await
+ pub fn defer_wire_actions(&self, actions: Vec<consensus::VsrAction>) {
+ if actions.is_empty() || !self.has_io_dispatcher() {
+ return;
+ }
+ let mut pending = self.pending_wire_actions.borrow_mut();
+ let view = self.consensus.view();
+ let log_view = self.consensus.log_view();
+ if pending
+ .as_ref()
+ .is_some_and(|pending| pending.view != view || pending.log_view != log_view)
+ {
+ *pending = None;
+ }
+ let pending = pending.get_or_insert_with(|| PendingWireActions {
+ view,
+ log_view,
+ actions: Vec::new(),
+ });
+ for action in actions {
+ // Timers keep firing during a held write. Retain only their latest snapshot.
+ let previous = match action {
+ consensus::VsrAction::SendStartViewChange { .. }
+ | consensus::VsrAction::SendDoViewChange { .. }
+ | consensus::VsrAction::SendRequestStartView { .. }
+ | consensus::VsrAction::SendCommit { .. }
+ | consensus::VsrAction::RetransmitPrepares { .. } => {
+ pending.actions.iter_mut().find(|pending| {
+ std::mem::discriminant(*pending) == std::mem::discriminant(&action)
+ })
+ }
+ _ => None,
+ };
+ if let Some(previous) = previous {
+ *previous = action;
+ } else {
+ pending.actions.push(action);
+ }
+ }
+ self.notify_io();
}
- /// [`Self::write_superblock`] for a caller that also has a reservation to
- /// claim. Both fields advance, neither can regress.
+ /// Initial claims on an isolated partition preserve both durable bounds.
#[allow(clippy::future_not_send)]
async fn write_superblock_advancing(
&self,
- superblock: &SB,
+ superblock: &Rc<SB>,
offset_frontier: u64,
offset_reserved: u64,
) -> bool {
@@ -1633,55 +2263,137 @@ where
#[allow(clippy::future_not_send)]
#[must_use = "the bool is the durability verdict; dropping it silently ignores a failed write"]
pub async fn install_offset_frontier_at(&self, frontier: u64) -> bool {
- let Some(superblock) = self.superblock.as_ref().map(Rc::clone) else {
+ self.complete_superblock_intent(SuperblockIntent::Install(frontier))
+ .await
+ }
+
+ const fn offset_range_is_reserved(&self, end_offset: u64) -> bool {
+ self.consensus.replica_count() > 1
+ || self.superblock.is_none()
+ || self.durable_offset_reserved.get() > end_offset
+ }
+
+ fn superblock_state(&self, intent: SuperblockIntent) -> consensus::VsrState {
+ let mut state = self.consensus.vsr_state(0, 0);
+ let (frontier, reserved) = match intent {
+ SuperblockIntent::Advance { frontier, reserved } => (
+ self.advanced_frontier(frontier),
+ reserved.max(self.durable_offset_reserved.get()),
+ ),
+ SuperblockIntent::Reset(frontier) => (frontier, frontier),
+ SuperblockIntent::Install(frontier) => (self.advanced_frontier(frontier), frontier),
+ SuperblockIntent::Quarantine(intended) => (
+ intended.unwrap_or_else(|| self.advanced_frontier(self.offset_frontier())),
+ self.durable_offset_reserved.get(),
+ ),
+ SuperblockIntent::Collapse { frontier, reserved } => (frontier, reserved),
+ };
+ state.offset_frontier = frontier;
+ state.offset_reserved = reserved.max(frontier);
+ state
+ }
+
+ fn request_superblock_intent(&self, intent: SuperblockIntent) -> crate::PartitionIoVerdict {
+ if self.superblock.is_none() {
+ return crate::PartitionIoVerdict::Ready;
+ }
+ if let SuperblockIntent::Advance { frontier, reserved } = intent {
+ if !self.consensus.needs_superblock_persist()
+ && self.durable_offset_frontier.get() >= frontier
+ && self.durable_offset_reserved.get() >= reserved
+ {
+ return crate::PartitionIoVerdict::Ready;
+ }
+ } else if let Some((completed, succeeded)) = self.superblock_completed.get()
+ && completed == intent
+ {
+ self.superblock_completed.set(None);
+ return if succeeded {
+ crate::PartitionIoVerdict::Ready
+ } else {
+ crate::PartitionIoVerdict::Failed
+ };
+ }
+ if self.superblock_write_is_backed_off() && !intent.final_write() {
+ return crate::PartitionIoVerdict::Failed;
+ }
+ if self
+ .superblock_running
+ .get()
+ .is_some_and(|(running, _)| running == intent)
+ {
+ return crate::PartitionIoVerdict::Pending;
+ }
+ let pending = match (self.superblock_intent.get(), intent) {
+ (
+ Some(SuperblockIntent::Advance {
+ frontier: old_frontier,
+ reserved: old_reserved,
+ }),
+ SuperblockIntent::Advance { frontier, reserved },
+ ) => SuperblockIntent::Advance {
+ frontier: old_frontier.max(frontier),
+ reserved: old_reserved.max(reserved),
+ },
+ (Some(pending), _) if pending != intent => return crate::PartitionIoVerdict::Pending,
+ _ => intent,
+ };
+ self.superblock_intent.set(Some(pending));
+ self.notify_io();
+ crate::PartitionIoVerdict::Pending
+ }
+
+ async fn complete_superblock_intent(&self, intent: SuperblockIntent) -> bool {
+ if self.io_notifier.borrow().is_some() {
+ return self.request_superblock_intent(intent) == crate::PartitionIoVerdict::Ready;
+ }
+ let Some(superblock) = self.superblock.as_ref() else {
return true;
};
- if self.superblock_write_is_backed_off() {
+ if self.superblock_write_is_backed_off() && !intent.final_write() {
return false;
}
- let _superblock_guard = self.superblock_lock.acquire().await;
- let advanced = self.advanced_frontier(frontier);
- self.write_superblock_inner(superblock.as_ref(), advanced, frontier)
- .await
+ let _guard = self.superblock_lock.acquire().await;
+ let state = self.superblock_state(intent);
+ let result = crate::io::SuperblockIoJob {
+ superblock: Rc::clone(superblock),
+ state,
+ }
+ .execute()
+ .await;
+ self.accept_superblock(result, self.superblock_write_is_backed_off())
}
- /// The write itself; the advance and reset directions differ only in the
- /// values they hand in.
#[allow(clippy::future_not_send)]
async fn write_superblock_inner(
&self,
- superblock: &SB,
+ superblock: &Rc<SB>,
offset_frontier: u64,
offset_reserved: u64,
) -> bool {
- // The pairing fields stay `(0, 0)` and `commit_max` is a dead write
- // on this plane: nothing reads either back (`restore_partition_view`
- // restores view/log_view only), because recovery re-derives the
- // install floor from the installed segments at boot -- a crash after
- // an install does not re-run the transfer. Written anyway so the
- // record shape matches the metadata plane's.
- //
- // `offset_frontier` is NOT dead: it is the only durable carrier of the
- // group's offset space once the segments that named it are gone. Every
- // write stamps the current counter, so whichever write lands last (a
- // view change, or the explicit persist an install issues) leaves a
- // lower bound boot can re-seed from.
- // Sampled BEFORE the write: the writers that bypass the gate (the append
- // fence, the quarantine record, the shutdown collapse) attempt inside an
- // open window, and counting those makes the wedge threshold a function of
- // producer retry rate rather than of elapsed time.
let inside_backoff_window = self.superblock_write_is_backed_off();
let mut state = self.consensus.vsr_state(0, 0);
state.offset_frontier = offset_frontier;
- // A frontier of N says offsets below N exist, so a reservation under it
- // is not a reservation. Clamped here rather than per caller so the
- // reset direction gets it too, where a stale higher reservation would
- // seed the next boot into the offset space the reset just erased.
state.offset_reserved = offset_reserved.max(offset_frontier);
- match superblock.write(&state.to_bytes()).await {
- Ok(()) => {
- self.consensus
- .mark_superblock_durable(state.view, state.log_view);
+ let result = crate::io::SuperblockIoJob {
+ superblock: Rc::clone(superblock),
+ state,
+ }
+ .execute()
+ .await;
+ self.accept_superblock(result, inside_backoff_window)
+ }
+
+ fn accept_superblock(
+ &self,
+ result: crate::io::SuperblockIoResult,
+ inside_backoff_window: bool,
+ ) -> bool {
+ let state = result.state;
+ match result.outcome {
+ Ok(()) => {
+ self.consensus
+ .mark_superblock_durable(state.view, state.log_view);
self.durable_offset_frontier.set(state.offset_frontier);
self.durable_offset_reserved.set(state.offset_reserved);
self.superblock_write_failures.set(0);
@@ -2191,16 +2903,7 @@ where
#[allow(clippy::future_not_send)]
#[must_use = "the bool is the durability verdict; dropping it silently ignores a failed write"]
pub async fn reset_offset_frontier_at(&self, frontier: u64) -> bool {
- let Some(superblock) = self.superblock.as_ref().map(Rc::clone) else {
- return true;
- };
- if self.superblock_write_is_backed_off() {
- return false;
- }
- let _superblock_guard = self.superblock_lock.acquire().await;
- // Reservation reset with it: left above, it would seed the next boot
- // back into the offset space this reset just left behind.
- self.write_superblock_inner(superblock.as_ref(), frontier, frontier)
+ self.complete_superblock_intent(SuperblockIntent::Reset(frontier))
.await
}
@@ -2226,21 +2929,8 @@ where
#[allow(clippy::future_not_send)]
#[must_use = "the bool is the durability verdict; dropping it silently ignores a failed write"]
pub async fn record_frontier_before_quarantine(&self, intended: Option<u64>) -> bool {
- let Some(superblock) = self.superblock.as_ref().map(Rc::clone) else {
- return true;
- };
- let _superblock_guard = self.superblock_lock.acquire().await;
- match intended {
- Some(frontier) => {
- let reserved = frontier.max(self.durable_offset_reserved.get());
- self.write_superblock_inner(superblock.as_ref(), frontier, reserved)
- .await
- }
- None => {
- self.write_superblock(superblock.as_ref(), self.offset_frontier())
- .await
- }
- }
+ self.complete_superblock_intent(SuperblockIntent::Quarantine(intended))
+ .await
}
/// Whether a recent write failure's backoff window is still open.
@@ -2283,17 +2973,15 @@ where
#[allow(clippy::future_not_send)]
#[must_use = "the bool is the durability verdict; a failed collapse leaves a gap"]
pub async fn collapse_offset_reservation(&self) -> bool {
- let append_point = self.mint_frontier();
- if self.durable_offset_reserved.get() <= append_point {
+ let reserved = self.mint_frontier();
+ if self.durable_offset_reserved.get() <= reserved {
return true;
}
- let Some(superblock) = self.superblock.as_ref().map(Rc::clone) else {
- return true;
- };
- let _superblock_guard = self.superblock_lock.acquire().await;
- let held = self.advanced_frontier(0);
- self.write_superblock_inner(superblock.as_ref(), held, append_point)
- .await
+ self.complete_superblock_intent(SuperblockIntent::Collapse {
+ frontier: self.advanced_frontier(0),
+ reserved,
+ })
+ .await
}
/// [`Self::persist_offset_frontier`] for a frontier this replica has not
@@ -2309,14 +2997,11 @@ where
#[allow(clippy::future_not_send)]
#[must_use = "the bool is the durability verdict; dropping it silently ignores a failed write"]
pub async fn persist_offset_frontier_at(&self, frontier: u64) -> bool {
- let Some(superblock) = self.superblock.as_ref().map(Rc::clone) else {
- return true;
- };
- if self.superblock_write_is_backed_off() {
- return false;
- }
- let _superblock_guard = self.superblock_lock.acquire().await;
- self.write_superblock(superblock.as_ref(), frontier).await
+ self.complete_superblock_intent(SuperblockIntent::Advance {
+ frontier,
+ reserved: 0,
+ })
+ .await
}
/// Whether the offset reservation is close enough to being consumed that it
@@ -2339,7 +3024,11 @@ where
/// the append path.
#[must_use]
pub fn needs_offset_reservation_extension(&self) -> bool {
- if self.consensus.replica_count() > 1 || self.superblock.is_none() {
+ // The shutdown collapse must stay the last superblock write.
+ if self.consensus.replica_count() > 1
+ || self.superblock.is_none()
+ || self.maintenance.shutdown.is_some()
+ {
return false;
}
if !self.offset_space.append_live {
@@ -2368,15 +3057,12 @@ where
/// so a ceiling that somehow fell behind still comes forward.
#[allow(clippy::future_not_send)]
pub async fn extend_offset_reservation(&self) -> bool {
- let Some(superblock) = self.superblock.as_ref().map(Rc::clone) else {
- return true;
- };
- if self.superblock_write_is_backed_off() {
- return false;
- }
- let _superblock_guard = self.superblock_lock.acquire().await;
let ceiling = self.durable_offset_reserved.get().max(self.mint_frontier());
- self.write_claim_from(superblock.as_ref(), ceiling).await
+ self.complete_superblock_intent(SuperblockIntent::Advance {
+ frontier: 0,
+ reserved: ceiling.saturating_add(self.offset_reservation_lease),
+ })
+ .await
}
/// Upper bound on the offsets a pending `SendMessages` request will mint, for
@@ -2475,25 +3161,24 @@ where
#[allow(clippy::future_not_send)]
#[must_use = "the bool is the fence verdict; dropping it lets the append escape unreserved"]
pub async fn reserve_offsets_through(&self, end_offset: u64) -> bool {
- if self.consensus.replica_count() > 1 {
- return true;
- }
- // A frontier: offsets strictly below it are permitted, so covering
- // `end_offset` needs a record strictly above it.
- if self.durable_offset_reserved.get() > end_offset {
+ if self.offset_range_is_reserved(end_offset) {
return true;
}
- let Some(superblock) = self.superblock.as_ref().map(Rc::clone) else {
- return true;
+ let Some(exclusive) = end_offset.checked_add(1) else {
+ return false;
};
- let _superblock_guard = self.superblock_lock.acquire().await;
- // A batch queued behind another append's write finds the block already
- // extended.
- if self.durable_offset_reserved.get() > end_offset {
- return true;
+ if self.io_notifier.borrow().is_none() {
+ let Some(superblock) = self.superblock.as_ref() else {
+ return true;
+ };
+ let _guard = self.superblock_lock.acquire().await;
+ return self.write_offset_claim(superblock, end_offset).await;
}
- self.write_offset_claim(superblock.as_ref(), end_offset)
- .await
+ self.complete_superblock_intent(SuperblockIntent::Advance {
+ frontier: 0,
+ reserved: exclusive.saturating_add(self.offset_reservation_lease),
+ })
+ .await
}
/// [`Self::reserve_offsets_through`] for the ADMITTED path, where a refusal
@@ -2512,47 +3197,41 @@ where
/// buys nothing and starves the shard pump for as long as the fault lasts.
#[allow(clippy::future_not_send)]
#[must_use = "the bool is the fence verdict; dropping it lets the append escape unreserved"]
- async fn reserve_offsets_through_retryable(&self, end_offset: u64) -> bool {
- if self.consensus.replica_count() > 1 || self.durable_offset_reserved.get() > end_offset {
- return true;
+ async fn reserve_offsets_through_retryable(
+ &self,
+ end_offset: u64,
+ ) -> crate::PartitionIoVerdict {
+ if self.offset_range_is_reserved(end_offset) {
+ return crate::PartitionIoVerdict::Ready;
}
- if self.superblock_write_is_backed_off() {
- return false;
+ let Some(exclusive) = end_offset.checked_add(1) else {
+ return crate::PartitionIoVerdict::Failed;
+ };
+ let intent = SuperblockIntent::Advance {
+ frontier: 0,
+ reserved: exclusive.saturating_add(self.offset_reservation_lease),
+ };
+ if self.io_notifier.borrow().is_some() {
+ return self.request_superblock_intent(intent);
+ }
+ if self.complete_superblock_intent(intent).await {
+ crate::PartitionIoVerdict::Ready
+ } else {
+ crate::PartitionIoVerdict::Failed
}
- self.reserve_offsets_through(end_offset).await
}
- /// The reservation preflight every admitted send passes, whether it arrives
- /// at [`Self::on_request`] or is promoted out of the request queue.
- ///
- /// `true` when the send may be projected. `false` when it was ANSWERED here
- /// and must go no further: the client holds a `TransientNotAccepted`, which
- /// admitted nothing, so it may re-issue anywhere without double-apply risk.
- ///
- /// Two ways to come back `false`, neither reaching the mint: an open
- /// superblock backoff window, and a claim that was attempted and failed.
- ///
- /// `waiter` is the submit's in-process reply channel, taken only on a
- /// refusal: the deny goes there because `header.client` is then the VSR
- /// consensus id, which the bus cannot route.
+ /// Check reservation coverage before direct sequencing or queue promotion.
+ /// Pending retains the original request and reply in the bounded request queue.
#[allow(clippy::future_not_send)]
async fn admit_reserved_send(
&self,
message: &Message<RoutedRequestHeader>,
- waiter: &mut Option<consensus::Sender<Message<ReplyHeader>>>,
- ) -> bool {
- if message.header().operation != Operation::SendMessages {
- return true;
- }
- let Some(ceiling) = self.request_mint_ceiling(message) else {
- return true;
- };
- if !self.reserve_offsets_through_retryable(ceiling).await {
- self.deny_unreserved_send(message.header(), waiter.take())
- .await;
- return false;
+ ) -> crate::PartitionIoVerdict {
+ match self.request_mint_ceiling(message) {
+ Some(ceiling) => self.reserve_offsets_through_retryable(ceiling).await,
+ None => crate::PartitionIoVerdict::Ready,
}
- true
}
/// Answer a send the reservation would not cover with a retryable transient,
@@ -2604,7 +3283,7 @@ where
/// an offset a client already holds -- the one defect this whole path exists
/// to prevent, at the one offset where it would be silent.
#[allow(clippy::future_not_send)]
- async fn write_offset_claim(&self, superblock: &SB, end_offset: u64) -> bool {
+ async fn write_offset_claim(&self, superblock: &Rc<SB>, end_offset: u64) -> bool {
let Some(exclusive) = end_offset.checked_add(1) else {
tracing::error!(
namespace_raw = self.consensus.group(),
@@ -2624,7 +3303,7 @@ where
/// every `end_offset` below it, so the coverage test still holds. Only the
/// successor itself can push the frontier off the end of the space.
#[allow(clippy::future_not_send)]
- async fn write_claim_from(&self, superblock: &SB, exclusive: u64) -> bool {
+ async fn write_claim_from(&self, superblock: &Rc<SB>, exclusive: u64) -> bool {
let claim = exclusive.saturating_add(self.offset_reservation_lease);
self.write_superblock_advancing(superblock, 0, claim).await
}
@@ -2868,13 +3547,9 @@ where
}
}
- /// Stage a consumer offset upsert for the replicated op. The prepare
- /// must already have been appended to `self.log.journal` by the caller
- /// so `VsrAction::RetransmitPrepares` can recover it during a view
- /// change. The on-disk offset table is NOT touched here: persist runs
- /// from [`Self::apply_staged_consumer_offset_commit`] at commit time so a
- /// view-change rollback of the in-memory pending entry also rolls
- /// back the disk write (by never having performed it).
+ /// Stage an already journaled mutation without touching its offset file.
+ /// The commit driver persists it before publication, so discarding an
+ /// uncommitted prepare does not require rolling back a file write.
pub(crate) fn stage_consumer_offset_upsert(
&mut self,
op: u64,
@@ -2916,6 +3591,7 @@ where
}
}
+ #[cfg(test)]
pub(crate) async fn apply_staged_consumer_offset_commit(
&mut self,
op: u64,
@@ -2956,166 +3632,298 @@ where
Ok(())
}
+ #[cfg(test)]
async fn write_consumer_offset(
&self,
path: &str,
offset: u64,
persisted: bool,
) -> Result<(), IggyError> {
- if let Some(persistence) = &self.persistence {
- let (result, file) = crate::offset_storage::persist_offset_retained(
- path,
- offset,
- persistence.take_offset_file(path),
- )
- .await?;
- let result = result.and(persistence.retain_offset_file(path.to_owned(), file).await);
- result.map_err(|error| {
- // Unlike WAL admission, a failed write can leave a partial record.
- persistence.fail_operation(error, Operation::StoreConsumerOffset);
- IggyError::CannotWriteToFile
- })
- } else {
- persist_offset(path, offset, persisted).await
- }
+ let job = self.prepare_consumer_offset_write(path, offset, persisted, false);
+ let result = job.execute().await;
+ self.accept_consumer_offset_write(result).map(|_| ())
}
+ #[cfg(test)]
async fn write_cold_consumer_offset(
&self,
path: &str,
offset: u64,
persisted: bool,
) -> Result<(u64, bool), IggyError> {
- if self.persistence.is_some() {
- let result = crate::offset_storage::read_offset_max(path, offset).await?;
- if result.written {
- self.write_consumer_offset(path, result.offset, false)
- .await?;
+ let job = self.prepare_consumer_offset_write(path, offset, persisted, true);
+ let result = job.execute().await;
+ self.accept_consumer_offset_write(result)
+ .map(|value| (value.offset, value.written))
+ }
+
+ fn prepare_consumer_offset_write(
+ &self,
+ path: &str,
+ offset: u64,
+ persisted: bool,
+ fold_max: bool,
+ ) -> crate::io::OffsetIoJob {
+ let (owner, checked_out) = match (&self.persistence, persisted) {
+ (Some(persistence), _) => (
+ crate::io::OffsetFileOwner::Wal,
+ persistence.checkout_offset_file(path),
+ ),
+ (None, false) => self.workerless_offset_files.checkout(path).map_or(
+ (crate::io::OffsetFileOwner::Uncached, None),
+ |checked_out| (crate::io::OffsetFileOwner::Workerless, Some(checked_out)),
+ ),
+ (None, true) => {
+ self.workerless_offset_files.retire(path);
+ self.workerless_offset_files.discard_retired();
+ (crate::io::OffsetFileOwner::Uncached, None)
+ }
+ };
+ let (file, permit) =
+ checked_out.map_or((None, None), |(file, permit)| (file, Some(permit)));
+ crate::io::OffsetIoJob {
+ path: path.to_owned(),
+ offset,
+ fold_max,
+ persisted,
+ owner,
+ file,
+ permit,
+ }
+ }
+
+ fn accept_consumer_offset_write(
+ &self,
+ result: crate::io::OffsetIoResult,
+ ) -> Result<crate::offset_storage::PersistedOffset, IggyError> {
+ let crate::io::OffsetIoResult {
+ path,
+ owner,
+ file,
+ permit,
+ outcome,
+ write_error,
+ } = result;
+ match owner {
+ crate::io::OffsetFileOwner::Wal => {
+ let persistence = self
+ .persistence
+ .as_ref()
+ .ok_or(IggyError::CannotWriteToFile)?;
+ if let (Some(file), Some(permit)) = (file, permit) {
+ persistence.return_offset_file(&path, file, permit);
+ } else {
+ persistence.retire_offset_file(&path);
+ }
+ if let Some(error) = write_error {
+ persistence.fail_operation(error, Operation::StoreConsumerOffset);
+ }
+ }
+ crate::io::OffsetFileOwner::Workerless => {
+ if write_error.is_some() || file.is_none() {
+ self.workerless_offset_files.retire(&path);
+ }
+ if let (Some(file), Some(permit)) = (file, permit) {
+ self.workerless_offset_files.put(&path, file, permit);
+ }
+ self.workerless_offset_files.discard_retired();
}
- Ok((result.offset, result.written))
+ crate::io::OffsetFileOwner::Uncached => {}
+ }
+ outcome
+ }
+
+ fn retire_consumer_offset_file(&self, path: &str) {
+ if let Some(persistence) = &self.persistence {
+ persistence.retire_offset_file(path);
} else {
- let result = persist_offset_max(path, offset, persisted).await?;
- Ok((result.offset, result.written))
+ self.workerless_offset_files.retire(path);
+ self.workerless_offset_files.discard_retired();
+ }
+ }
+
+ fn prepare_consumer_offset_delete(&self, path: &str) -> crate::io::OffsetDeleteIoJob {
+ self.retire_consumer_offset_file(path);
+ crate::io::OffsetDeleteIoJob {
+ path: path.to_owned(),
}
}
+ #[cfg(test)]
async fn persist_consumer_offset_commit(
&self,
pending: PendingConsumerOffsetCommit,
) -> Result<(), IggyError> {
- // For either offset policy, the WAL protects these updates until checkpoint
- // syncs their retained writers and directories before reclaiming history.
- let persisted =
- self.consumer_offset_durability().is_persisted() && self.persistence.is_none();
- let path = self.persisted_offset_path(pending.kind, pending.consumer_id);
- let capacity = self.consumer_offset_capacity_for(pending.kind);
- match pending.mutation {
- // A server auto-commit persists monotonically: its op offset can
- // trail the durably-recorded value (disk-tier polls replicate in
- // IO-completion order), so a plain overwrite would rewind the file
- // and re-deliver on restart. The durable offset tracker keeps
- // the fold off the file: a covered offset skips the write, an
- // advancing one blind-writes, and only a cold key (first commit
- // after boot) reads the file once. Explicit client stores
- // overwrite, so a deliberate offset reset still holds. Mirrors the
- // in-memory `upsert_offset_max` vs `upsert_offset` split in the
- // commit-apply.
- PendingConsumerOffsetMutation::Upsert(offset) if pending.auto_commit => {
- let tracked = self
- .durable_consumer_offsets
- .get(pending.kind, pending.consumer_id);
- let (persisted_high_water, written) = match (path.as_deref(), tracked) {
- (None, _) => (offset, false),
- (Some(_), Some(state)) if offset <= state.persisted_high_water => {
- (state.persisted_high_water, false)
+ let tracked = self
+ .durable_consumer_offsets
+ .get(pending.kind, pending.consumer_id);
+ let effect = self
+ .persist_consumer_offset_effect(pending, tracked)
+ .await?;
+ self.publish_consumer_offset_effect(effect);
+ Ok(())
+ }
+
+ #[cfg(test)]
+ async fn persist_consumer_offset_effect(
+ &self,
+ pending: PendingConsumerOffsetCommit,
+ tracked: Option<DurableOffsetState>,
+ ) -> Result<ConsumerOffsetEffect, IggyError> {
+ let plan = self.plan_consumer_offset_effect(pending, tracked);
+ if let ConsumerOffsetAction::Ready(effect) = plan.action {
+ return Ok(effect);
+ }
+ let job = self.capture_consumer_offset_effect(&plan)?;
+ self.accept_consumer_offset_effect(plan, job.execute().await)
+ }
+
+ fn plan_consumer_offset_effect(
+ &self,
+ pending: PendingConsumerOffsetCommit,
+ tracked: Option<DurableOffsetState>,
+ ) -> ConsumerOffsetPlan {
+ let has_path = self.consumer_offsets_dir(pending.kind).is_some();
+ let action = match pending.mutation {
+ PendingConsumerOffsetMutation::Upsert(offset) => {
+ if !has_path {
+ ConsumerOffsetAction::Ready(consumer_offset_effect(pending, tracked, offset))
+ } else if pending.auto_commit
+ && let Some(state) =
+ tracked.filter(|state| offset <= state.persisted_high_water)
+ {
+ ConsumerOffsetAction::Ready(consumer_offset_effect(
+ pending,
+ tracked,
+ state.persisted_high_water,
+ ))
+ } else {
+ ConsumerOffsetAction::Write {
+ offset: if pending.auto_commit {
+ tracked.map_or(offset, |state| state.committed_offset.max(offset))
+ } else {
+ offset
+ },
+ fold_max: pending.auto_commit && tracked.is_none(),
}
- (Some(path), Some(state)) => {
- let value = state.committed_offset.max(offset);
- self.write_consumer_offset(path, value, persisted).await?;
- (value, true)
+ }
+ }
+ PendingConsumerOffsetMutation::Delete => {
+ if has_path {
+ ConsumerOffsetAction::Delete
+ } else {
+ ConsumerOffsetAction::Ready(ConsumerOffsetEffect {
+ pending,
+ state: None,
+ })
+ }
+ }
+ };
+ ConsumerOffsetPlan {
+ pending,
+ tracked,
+ persisted: self.consumer_offset_durability().is_persisted()
+ && self.persistence.is_none(),
+ action,
+ }
+ }
+
+ fn capture_consumer_offset_effect(
+ &self,
+ plan: &ConsumerOffsetPlan,
+ ) -> Result<crate::io::PartitionIoJob<SB>, IggyError> {
+ let path = self
+ .persisted_offset_path(plan.pending.kind, plan.pending.consumer_id)
+ .ok_or(IggyError::CannotWriteToFile)?;
+ match plan.action {
+ ConsumerOffsetAction::Write { offset, fold_max } => {
+ Ok(crate::io::PartitionIoJob::OffsetWrite(
+ self.prepare_consumer_offset_write(&path, offset, plan.persisted, fold_max),
+ ))
+ }
+ ConsumerOffsetAction::Delete => Ok(crate::io::PartitionIoJob::OffsetDelete(
+ self.prepare_consumer_offset_delete(&path),
+ )),
+ ConsumerOffsetAction::Ready(_) => Err(IggyError::CannotWriteToFile),
+ }
+ }
+
+ fn accept_consumer_offset_effect(
+ &self,
+ plan: ConsumerOffsetPlan,
+ result: crate::io::PartitionIoResult,
+ ) -> Result<ConsumerOffsetEffect, IggyError> {
+ let pending = plan.pending;
+ match (plan.action, result) {
+ (
+ ConsumerOffsetAction::Write { .. },
+ crate::io::PartitionIoResult::OffsetWrite(result),
+ ) => {
+ let value = self.accept_consumer_offset_write(result)?;
+ if value.written && plan.persisted {
+ self.mark_consumer_offset_dir_dirty(pending.kind);
+ }
+ Ok(consumer_offset_effect(pending, plan.tracked, value.offset))
+ }
+ (ConsumerOffsetAction::Delete, crate::io::PartitionIoResult::OffsetDelete(outcome)) => {
+ match outcome {
+ Ok(removed) => {
+ if removed && plan.persisted {
+ self.mark_consumer_offset_dir_dirty(pending.kind);
+ }
+ Ok(ConsumerOffsetEffect {
+ pending,
+ state: None,
+ })
}
- (Some(path), None) => {
- self.write_cold_consumer_offset(path, offset, persisted)
- .await?
+ Err(error) => {
+ self.consumer_offset_capacity_for(pending.kind)
+ .record_stranded(pending.consumer_id);
+ warn!(
+ namespace_raw = self.namespace().inner(),
+ kind = ?pending.kind, consumer_id = pending.consumer_id, %error,
+ "committed consumer offset delete could not remove its file"
+ );
+ Err(error)
}
- };
+ }
+ }
+ _ => Err(IggyError::CannotWriteToFile),
+ }
+ }
+
+ fn publish_consumer_offset_effect(&self, effect: ConsumerOffsetEffect) {
+ let ConsumerOffsetEffect { pending, state } = effect;
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
+ if let Some(state) = state {
+ let created = !self
+ .durable_consumer_offsets
+ .contains(pending.kind, pending.consumer_id);
+ if pending.auto_commit {
self.durable_consumer_offsets.record_auto_commit(
pending.kind,
pending.consumer_id,
- if tracked.is_none() {
- persisted_high_water
- } else {
- offset
- },
- persisted_high_water,
+ state.committed_offset,
+ state.persisted_high_water,
);
- if written && persisted {
- self.mark_consumer_offset_dir_dirty(pending.kind);
- }
- capacity.clear_stranded(pending.consumer_id);
- if keyed_by_group(pending.kind) && tracked.is_none() {
- self.mark_consumer_group_offsets_need_reconcile();
- }
- Ok(())
- }
- PendingConsumerOffsetMutation::Upsert(offset) => {
- if let Some(path) = path.as_deref() {
- self.write_consumer_offset(path, offset, persisted).await?;
- }
- let created = self.durable_consumer_offsets.record_explicit(
+ } else {
+ self.durable_consumer_offsets.record_explicit(
pending.kind,
pending.consumer_id,
- offset,
- offset,
+ state.committed_offset,
+ state.persisted_high_water,
);
- if path.is_some() && persisted {
- self.mark_consumer_offset_dir_dirty(pending.kind);
- }
- capacity.clear_stranded(pending.consumer_id);
- if keyed_by_group(pending.kind) && created {
- self.mark_consumer_group_offsets_need_reconcile();
- }
- Ok(())
}
- PendingConsumerOffsetMutation::Delete => {
- if let Some(path) = path.as_deref() {
- // Keep the logical state until the file is removed. The
- // partition journal is memory-only, so acknowledging an
- // unsuccessful unlink would let boot resurrect the key.
- match delete_persisted_offset(path).await {
- Ok(removed) => {
- if let Some(persistence) = &self.persistence {
- persistence.retire_offset_file(path);
- }
- if removed && persisted {
- self.mark_consumer_offset_dir_dirty(pending.kind);
- }
- capacity.clear_stranded(pending.consumer_id);
- }
- Err(error) => {
- capacity.record_stranded(pending.consumer_id);
- warn!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- replica_id = self.consensus.replica(),
- namespace_raw = self.namespace().inner(),
- kind = ?pending.kind,
- consumer_id = pending.consumer_id,
- path,
- %error,
- "committed consumer offset delete could not remove its file"
- );
- return Err(error);
- }
- }
- }
- self.durable_consumer_offsets
- .remove(pending.kind, pending.consumer_id);
- capacity.forget_inactive_provisional(pending.consumer_id);
- capacity.rearm_if_below_limit(&self.durable_consumer_offsets);
- Ok(())
+ if keyed_by_group(pending.kind) && created {
+ self.mark_consumer_group_offsets_need_reconcile();
}
+ } else {
+ self.durable_consumer_offsets
+ .remove(pending.kind, pending.consumer_id);
+ capacity.forget_inactive_provisional(pending.consumer_id);
+ capacity.rearm_if_below_limit(&self.durable_consumer_offsets);
}
+ capacity.clear_stranded(pending.consumer_id);
}
/// Accept a read only while its message history still belongs to this owner.
@@ -3190,10 +3998,12 @@ where
/// Invalidate after preflight, before replacing data or progress.
/// Preserve the identity when preflight leaves served state unchanged.
/// Queued automatic commits also belong to the old history, even when
- /// their reads have already completed.
+ /// their reads have already completed. So does a pinned transfer plan.
pub(crate) fn invalidate_poll_history(&mut self) {
+ self.workerless_offset_files.clear();
self.poll_history = PollHistoryId::default();
self.discard_queued_auto_commits();
+ self.transfer_plan.borrow_mut().take();
}
fn discard_queued_auto_commits(&self) {
@@ -3217,15 +4027,16 @@ where
return Err(IggyError::TransientNotAccepted);
}
self.check_local_poll_key(kind, consumer_id)
- .map_err(|error| self.poll_capacity_error(error))?;
+ .map_err(|error| self.consumer_offset_capacity_error(error))?;
// Already durable: the commit this poll would make has happened and
// replicated, so applying it locally syncs this replica to something
// the group agreed. Safe wherever the read was served, and checked
// before the role so a caught-up backup is not refused for a
// commit nobody needs.
- if self
- .durable_consumer_offsets
- .covers(kind, consumer_id, offset)
+ if !self.has_unsettled_local_requests()
+ && self
+ .durable_consumer_offsets
+ .covers(kind, consumer_id, offset)
{
return Ok(None);
}
@@ -3236,14 +4047,18 @@ where
// poll would report progress the group never agreed, and a later
// read on the primary would hand the same messages out again.
// Refusing keeps the outcome retriable on a replica that can commit.
- if !consensus.is_primary() || !consensus.is_normal() || consensus.is_transferring() {
+ if !consensus.is_primary()
+ || !consensus.is_normal()
+ || consensus.is_transferring()
+ || self.refuses_prepares()
+ {
return Err(IggyError::TransientNotAccepted);
}
let reservation = self
.consumer_offset_capacity_for(kind)
.reserve_provisional(consumer_id, &self.durable_consumer_offsets)
- .map_err(|error| self.poll_capacity_error(error))?;
+ .map_err(|error| self.consumer_offset_capacity_error(error))?;
let request = self.build_poll_auto_commit_request(kind, consumer_id, offset)?;
// Automatic commits bypass the request handler, so check journal
// capacity here before queueing or assigning an operation.
@@ -3254,24 +4069,22 @@ where
{
return Err(IggyError::TransientNotAccepted);
}
- if self.consensus.pipeline_is_full() {
+ let order = self.next_local_request_order()?;
+ if self.consensus.pipeline_is_full() || self.has_unsettled_local_requests() {
let context = AutoCommitRequestContext {
history: self.poll_history,
reservation,
};
- self.consensus
- .push_queued_request(consensus::RequestEntry::with_auto_commit(request, context))
- .map_err(|_| IggyError::TransientNotAccepted)?;
+ self.queue_local_request(
+ consensus::RequestEntry::with_auto_commit(request, context),
+ order,
+ )
+ .map_err(|_| IggyError::TransientNotAccepted)?;
Ok(None)
} else {
self.reserve_consumer_offset(kind, consumer_id)
- .map_err(|error| self.poll_capacity_error(error))?;
- let prepare = consensus::seal_prepare_capacity(
- request.project(self.consensus()),
- self.consensus().retry_capacity(&self.dedup),
- );
- self.consensus
- .pipeline_message(PlaneKind::Partitions, &prepare);
+ .map_err(|error| self.consumer_offset_capacity_error(error))?;
+ let prepare = self.pipeline_local_request(request, None, Some(order));
Ok(Some(PollReplication {
prepare,
reservation,
@@ -3279,6 +4092,67 @@ where
}
}
+ fn next_local_request_order(&self) -> Result<LocalRequestOrder, IggyError> {
+ self.local_request_order
+ .get()
+ .checked_add(1)
+ .and_then(NonZeroU64::new)
+ .map(LocalRequestOrder)
+ .ok_or(IggyError::TransientNotAccepted)
+ }
+
+ fn has_unsettled_local_requests(&self) -> bool {
+ !self.no_ack.is_empty()
+ || self.pending_commit.is_some()
+ || self
+ .consensus
+ .with_pipeline(|pipeline| !pipeline.is_empty())
+ }
+
+ /// `on_replicate` journals nothing during a transition, retirement or
+ /// shutdown, so a primary must not originate a prepare while this holds.
+ fn refuses_prepares(&self) -> bool {
+ self.transition.is_some()
+ || self.io_active.is_retiring()
+ || self.maintenance.shutdown.is_some()
+ }
+
+ fn pipeline_local_request(
+ &self,
+ message: Message<RoutedRequestHeader>,
+ sender: Option<consensus::Sender<Message<ReplyHeader>>>,
+ order: Option<LocalRequestOrder>,
+ ) -> Message<PrepareHeader> {
+ debug_assert!(
+ !self.refuses_prepares(),
+ "on_replicate would drop a prepare that the pipeline already holds"
+ );
+ let prepare = consensus::seal_prepare_capacity(
+ message.project(&self.consensus),
+ self.consensus.retry_capacity(&self.dedup),
+ );
+ self.consensus.verify_pipeline();
+ self.consensus
+ .pipeline_message_with_order(PlaneKind::Partitions, &prepare, sender, order);
+ if let Some(order) = order {
+ self.local_request_order
+ .set(self.local_request_order.get().max(order.0.get()));
+ }
+ prepare
+ }
+
+ fn queue_local_request(
+ &self,
+ mut request: consensus::RequestEntry,
+ order: LocalRequestOrder,
+ ) -> Result<(), consensus::RequestEntry> {
+ request.local_order = Some(order);
+ self.consensus.push_queued_request(request)?;
+ self.local_request_order.set(order.0.get());
+ self.notify_io();
+ Ok(())
+ }
+
fn auto_commit_admission_ready(&self, kind: ConsumerKind, consumer_id: u32) -> bool {
self.observed_view == self.consensus.view()
&& !self.offset_reservations_need_resync.get()
@@ -3337,11 +4211,11 @@ where
}
}
- fn poll_capacity_error(&self, error: ConsumerOffsetCapacityError) -> IggyError {
+ fn consumer_offset_capacity_error(&self, error: ConsumerOffsetCapacityError) -> IggyError {
if error.first_in_episode {
warn!(namespace_raw = self.namespace().inner(), kind = ?error.kind,
occupied = error.occupied, limit = error.limit, uncertain = error.uncertain,
- "consumer offset admission refused during poll completion");
+ "consumer offset admission refused");
}
error.into()
}
@@ -3554,109 +4428,269 @@ where
.remove(&ConsumerGroupId(group_id as usize));
}
- /// `AckLevel::NoAck` fast path: persist, apply, send reply, no
- /// replication. Single-replica durability. Never recorded in the dedup
- /// slice: it does not replicate, so folding it in would fork the slice
- /// across replicas. Session lifecycle lives on metadata.
+ /// The FIFO owns capacity and replies until the local mutation's barrier settles.
#[allow(clippy::future_not_send)]
async fn apply_consumer_offset_no_ack(
- &self,
+ &mut self,
request_header: Box<RoutedRequestHeader>,
kind: ConsumerKind,
consumer_id: u32,
offset: Option<u64>,
waiter: Option<consensus::Sender<Message<ReplyHeader>>>,
) {
- let pending = offset.map_or_else(
- || PendingConsumerOffsetCommit::delete(kind, consumer_id),
- |value| PendingConsumerOffsetCommit::upsert(kind, consumer_id, value),
- );
-
- if let Err(error) = self.persist_consumer_offset_commit(pending).await {
- if offset.is_some() {
- self.release_consumer_offset_reservation(kind, consumer_id);
+ let admission = self.reserve_no_ack(kind, consumer_id);
+ let (order, reservation) = match admission {
+ Ok(admission) => admission,
+ Err(error) => {
+ Self::send_partition_deny_or_log(
+ &self.consensus,
+ &request_header,
+ error.as_code(),
+ "no_ack admission reply failed",
+ waiter,
+ )
+ .await;
+ return;
}
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(self.diag_ctx(), "no_ack offset persist failed")
- .with_operation(request_header.operation)
- .with_error(error.to_string()),
- );
- Self::send_partition_deny_or_log(
- &self.consensus,
- &request_header,
- error.as_code(),
- "no_ack offset failure reply send failed",
- waiter,
- )
- .await;
- return;
- }
- self.apply_consumer_offset_commit(pending);
- // Only this request's kind: dirt on the other directory belongs to
- // whichever path left it, and its failure is not this client's answer.
- // Visibility does not prove crash durability. A failed required barrier
- // must remain an error even after the mutation becomes visible.
- let mut kinds = [false; ConsumerKind::COUNT];
- kinds[kind.index()] = true;
- let failed = self.flush_consumer_offset_directories_for(kinds).await;
- if failed.iter().any(|failed| *failed) {
- if offset.is_some() {
- self.release_consumer_offset_reservation(kind, consumer_id);
- } else {
- // Retain retry admission after the visible deletion so a new
- // delete can retry its directory barrier instead of returning
- // ConsumerOffsetNotFound.
- self.consumer_offset_capacity_for(kind)
- .record_stranded(consumer_id);
- }
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "no_ack offset directory sync failed after a visible local mutation",
- )
- .with_operation(request_header.operation),
- );
- Self::send_partition_deny_or_log(
- &self.consensus,
- &request_header,
- IggyError::CannotSyncFile.as_code(),
- "no_ack offset directory sync failure reply send failed",
- waiter,
- )
- .await;
+ };
+ let pending = PendingConsumerOffsetCommit::explicit(kind, consumer_id, offset);
+ self.no_ack.push_back(PendingNoAck {
+ request_header,
+ pending,
+ reply: waiter,
+ poll_history: self.poll_history,
+ local_order: order,
+ reservation,
+ phase: NoAckPhase::Validate,
+ });
+ self.local_request_order.set(order.0.get());
+ self.drain_no_ack().await;
+ }
+
+ fn reserve_no_ack(
+ &self,
+ kind: ConsumerKind,
+ consumer_id: u32,
+ ) -> Result<(LocalRequestOrder, AutoCommitReservation), IggyError> {
+ if self.no_ack.len()
+ >= self
+ .consensus
+ .with_pipeline(consensus::LocalPipeline::request_queue_capacity)
+ || self
+ .no_ack
+ .front()
+ .is_some_and(|head| head.poll_history != self.poll_history)
+ {
+ return Err(IggyError::TransientNotAccepted);
+ }
+ let order = self.next_local_request_order()?;
+ let reservation = self
+ .consumer_offset_capacity_for(kind)
+ .reserve_provisional(consumer_id, &self.durable_consumer_offsets)
+ .map_err(|error| self.consumer_offset_capacity_error(error))?;
+ Ok((order, reservation))
+ }
+
+ async fn drain_no_ack(&mut self) {
+ if self.has_io_dispatcher() {
+ self.notify_io();
return;
}
+ loop {
+ match self.drive_no_ack() {
+ NoAckStep::Progress => {}
+ NoAckStep::Pending => return,
+ NoAckStep::Ready => match self.capture_no_ack_phase() {
+ Ok(job) => {
+ let result = job.execute().await;
+ self.accept_no_ack_phase(result);
+ }
+ Err(error) => self.fail_no_ack(error, false),
+ },
+ NoAckStep::Reply => self.deliver_no_ack_reply().await,
+ }
+ }
+ }
- let reply = build_reply_from_request(
- &self.consensus,
- &request_header,
- committed_reply_body(request_header.operation),
- );
- if offset.is_some() {
- self.release_consumer_offset_reservation(kind, consumer_id);
+ fn drive_no_ack(&mut self) -> NoAckStep {
+ let Some(head) = self.no_ack.front() else {
+ return NoAckStep::Pending;
+ };
+ match &head.phase {
+ NoAckPhase::Validate => {
+ let pending = head.pending;
+ if head.poll_history != self.poll_history {
+ self.fail_no_ack(pending.history_reset_refusal(), false);
+ return NoAckStep::Reply;
+ }
+ if self.fatal.is_some()
+ || !self.consensus.is_primary()
+ || !self.consensus.is_normal()
+ || self.consensus.is_transferring()
+ || !self
+ .consumer_offset_capacity_for(pending.kind)
+ .owns(&head.reservation)
+ {
+ self.fail_no_ack(IggyError::TransientNotAccepted, false);
+ return NoAckStep::Reply;
+ }
+ if self.pending_commit.is_some() {
+ return NoAckStep::Pending;
+ }
+ if self
+ .consensus
+ .with_pipeline(|pipeline| pipeline.has_request_before(head.local_order))
+ {
+ if self.maintenance.shutdown.is_some() {
+ self.fail_no_ack(IggyError::TransientNotAccepted, false);
+ return NoAckStep::Reply;
+ }
+ return NoAckStep::Pending;
+ }
+ let validation = match pending.mutation {
+ PendingConsumerOffsetMutation::Upsert(offset) => self
+ .store_offset_range_error(pending.kind, offset)
+ .map_or(Ok(()), Err),
+ PendingConsumerOffsetMutation::Delete => {
+ self.ensure_consumer_offset_exists(pending.kind, pending.consumer_id)
+ }
+ };
+ if let Err(error) = validation {
+ self.fail_no_ack(error, false);
+ return NoAckStep::Reply;
+ }
+ let tracked = self
+ .durable_consumer_offsets
+ .get(pending.kind, pending.consumer_id);
+ let plan = self.plan_consumer_offset_effect(pending, tracked);
+ if let ConsumerOffsetAction::Ready(effect) = plan.action {
+ self.publish_no_ack_effect(effect);
+ } else {
+ self.no_ack.front_mut().expect("no_ack head exists").phase =
+ NoAckPhase::OffsetReady(plan);
+ }
+ NoAckStep::Progress
+ }
+ NoAckPhase::OffsetReady(_) if self.offset_writes_wait_for_checkpoint() => {
+ NoAckStep::Pending
+ }
+ NoAckPhase::OffsetReady(_) => NoAckStep::Ready,
+ NoAckPhase::Directory => {
+ if self.consumer_offset_dirs_dirty[head.pending.kind.index()].get() {
+ NoAckStep::Ready
+ } else {
+ self.no_ack.front_mut().expect("no_ack head exists").phase =
+ NoAckPhase::Reply(Ok(()));
+ NoAckStep::Reply
+ }
+ }
+ NoAckPhase::OffsetSubmitted(_) | NoAckPhase::DirectorySubmitted => NoAckStep::Pending,
+ NoAckPhase::Reply(_) => NoAckStep::Reply,
}
- // Same rule as the committed path: a submit's waiter takes the reply,
- // because `header.client` is then the VSR consensus id.
- if let Some(waiter) = waiter {
- let _ = waiter.send(reply);
+ }
+
+ fn capture_no_ack_phase(&mut self) -> Result<crate::io::PartitionIoJob<SB>, IggyError> {
+ let head = self.no_ack.front().ok_or(IggyError::CannotWriteToFile)?;
+ let (job, submitted) = match &head.phase {
+ NoAckPhase::OffsetReady(plan) => (
+ self.capture_consumer_offset_effect(plan)?,
+ NoAckPhase::OffsetSubmitted(*plan),
+ ),
+ NoAckPhase::Directory => {
+ let mut kinds = [false; ConsumerKind::COUNT];
+ kinds[head.pending.kind.index()] = true;
+ (
+ crate::io::PartitionIoJob::OffsetDirectories(
+ self.prepare_consumer_offset_directories(kinds),
+ ),
+ NoAckPhase::DirectorySubmitted,
+ )
+ }
+ _ => return Err(IggyError::CannotWriteToFile),
+ };
+ self.no_ack.front_mut().expect("no_ack head exists").phase = submitted;
+ Ok(job)
+ }
+
+ fn accept_no_ack_phase(&mut self, result: crate::io::PartitionIoResult) {
+ let Some(head) = self.no_ack.front() else {
return;
+ };
+ match (&head.phase, result) {
+ (NoAckPhase::OffsetSubmitted(plan), result) => {
+ match self.accept_consumer_offset_effect(*plan, result) {
+ Ok(effect) => self.publish_no_ack_effect(effect),
+ Err(error) => self.fail_no_ack(error, true),
+ }
+ }
+ (
+ NoAckPhase::DirectorySubmitted,
+ crate::io::PartitionIoResult::OffsetDirectories(result),
+ ) => {
+ self.accept_consumer_offset_directories(&result);
+ if result.failed.iter().any(|failed| *failed) {
+ self.fail_no_ack(IggyError::CannotSyncFile, true);
+ } else {
+ self.no_ack.front_mut().expect("no_ack head exists").phase =
+ NoAckPhase::Reply(Ok(()));
+ }
+ }
+ _ => self.fail_no_ack(IggyError::CannotWriteToFile, true),
}
- let reply_buffers = reply.into_generic().into_frozen();
- if let Err(error) = self
- .consensus
- .message_bus()
- .send_to_client(request_header.client, reply_buffers)
- .await
- {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(self.diag_ctx(), "no_ack reply send failed")
- .with_operation(request_header.operation)
- .with_error(error.to_string()),
- );
+ }
+
+ fn publish_no_ack_effect(&mut self, effect: ConsumerOffsetEffect) {
+ self.publish_consumer_offset_effect(effect);
+ self.apply_consumer_offset_commit(effect.pending);
+ self.no_ack.front_mut().expect("no_ack head exists").phase = NoAckPhase::Directory;
+ }
+
+ fn fail_no_ack(&mut self, error: IggyError, file_attempted: bool) {
+ let Some(head) = self.no_ack.front() else {
+ return;
+ };
+ if file_attempted {
+ self.consumer_offset_capacity_for(head.pending.kind)
+ .record_stranded(head.pending.consumer_id);
}
+ warn!(
+ namespace_raw = self.namespace().inner(),
+ operation = ?head.request_header.operation, %error,
+ "local consumer offset mutation failed"
+ );
+ self.no_ack.front_mut().expect("no_ack head exists").phase = NoAckPhase::Reply(Err(error));
+ }
+
+ async fn deliver_no_ack_reply(&mut self) {
+ let Some(head) = self.no_ack.pop_front() else {
+ return;
+ };
+ let NoAckPhase::Reply(outcome) = head.phase else {
+ self.no_ack.push_front(head);
+ return;
+ };
+ let reply = match outcome {
+ Ok(()) => build_reply_from_request(
+ &self.consensus,
+ &head.request_header,
+ committed_reply_body(head.request_header.operation),
+ ),
+ Err(error) => build_deny_reply_from_request(
+ &self.consensus,
+ &head.request_header,
+ error.as_code(),
+ ),
+ };
+ Self::deliver_reply_or_log(
+ &self.consensus,
+ &head.request_header,
+ reply.into(),
+ head.reply,
+ "no_ack reply send failed",
+ )
+ .await;
+ drop(head.reservation);
+ self.refresh_consumer_offset_reservation(head.pending.kind, head.pending.consumer_id);
}
pub(crate) fn persisted_offset_path(
@@ -3705,11 +4739,6 @@ where
.try_reserve(consumer_id, &self.durable_consumer_offsets)
}
- fn release_consumer_offset_reservation(&self, kind: ConsumerKind, consumer_id: u32) {
- self.consumer_offset_capacity_for(kind)
- .release_reservation(consumer_id);
- }
-
fn refresh_consumer_offset_reservation(&self, kind: ConsumerKind, consumer_id: u32) {
let count = self
.pending_consumer_offset_commits
@@ -4247,2286 +5276,4187 @@ where
IggyNamespace::from_raw(self.consensus.group())
}
- /// The commit fault that fenced this partition, if one has.
#[must_use]
- pub const fn fatal(&self) -> Option<&FatalCommit> {
- self.fatal.as_ref()
+ pub const fn incarnation(&self) -> crate::PartitionIncarnation {
+ self.incarnation
}
- /// Consecutive superblock write failures for this group, for the shard's
- /// wedge fail-stop. A partition that cannot record its state withholds every
- /// view-scoped send and refuses every append, so past some window it is
- /// serving nothing and a supervisor should be handling it instead.
- #[must_use]
- pub const fn superblock_write_failures(&self) -> u64 {
- self.superblock_write_failures.get()
+ pub fn set_io_notifier(&self, notifier: crate::PartitionIoNotifier, bytes_max: usize) {
+ self.io_bytes_max.set(bytes_max);
+ *self.io_notifier.borrow_mut() = Some(notifier);
+ self.notify_io();
}
- /// Fence this partition after the shutdown flush failed to persist its
- /// committed journal prefix: that data is cluster-committed and now lives
- /// only in this process's memory, so the shard must not report a clean
- /// exit over it. A fault the commit path already recorded is kept, since
- /// it names the op that first diverged; `commit_min` here only bounds
- /// where the unpersisted prefix ends.
- pub fn fence_flush_failure(&mut self) {
- if self.fatal.is_none() {
- self.fatal = Some(FatalCommit {
- namespace_raw: self.namespace().inner(),
- op: self.consensus.commit_min(),
- operation: Some(Operation::SendMessages),
- });
+ pub fn notify_io(&self) {
+ let notifier = self.io_notifier.borrow().clone();
+ if let Some(notifier) = notifier {
+ notifier(self.namespace(), self.incarnation);
}
}
- pub(crate) const fn fence_install_failure(&mut self, op: u64) {
- if self.fatal.is_none() {
- self.fatal = Some(FatalCommit {
- namespace_raw: self.consensus.group(),
- op,
- operation: Some(Operation::SendMessages),
- });
- }
+ #[must_use]
+ pub fn io_active(&self) -> Option<crate::PartitionIoIdentity> {
+ self.io_active.get()
}
- fn partition_dir(&self) -> Option<String> {
- if self.partition_dir.is_some() {
- return self.partition_dir.clone();
- }
- // Writer-derived fallback for partitions built without
- // `set_partition_dir`. Unreliable mid-rotation: sealed segments
- // drop their writer, so prefer the stored path above.
- self.log
- .messages_writers()
- .iter()
- .rev()
- .flatten()
- .next()
- .and_then(|writer| {
- std::path::Path::new(&writer.path())
- .parent()
- .map(|dir| dir.to_string_lossy().into_owned())
+ #[must_use]
+ pub const fn has_pending_commit(&self) -> bool {
+ self.pending_commit.is_some()
+ }
+
+ #[must_use]
+ pub fn read_history_is_changing(&self) -> bool {
+ self.io_active.is_retiring()
+ || matches!(
+ self.transition,
+ Some(
+ PendingPartitionTransition::Purge(_)
+ | PendingPartitionTransition::Install(_)
+ | PendingPartitionTransition::Quarantine { .. }
+ )
+ )
+ }
+
+ pub fn history_is_busy(&self) -> bool {
+ self.history_writes_pending() || !self.no_ack.is_empty()
+ }
+
+ pub fn history_writes_pending(&self) -> bool {
+ self.io_active.get().is_some()
+ || self.io_active.is_retiring()
+ || self.pending_commit.is_some()
+ || self.materialization.is_some()
+ || self.maintenance.flush_through.is_some()
+ || !self.io_replies.is_empty()
+ || self.no_ack.front().is_some_and(|head| {
+ !matches!(
+ head.phase,
+ NoAckPhase::Validate | NoAckPhase::OffsetReady(_)
+ )
})
+ || self.transition.is_some()
}
- /// [`Self::partition_dir`] upgraded with the reason a dir is absent, so a
- /// disk poll can tell file-less (simulated) storage from a live partition
- /// whose dir is transiently unresolvable mid-rotation. Storage readers,
- /// unlike writers, survive segment sealing, so any present reader or
- /// writer proves file-backed data exists behind the missing dir.
- fn partition_dir_resolution(&self) -> PartitionDirResolution {
- if let Some(dir) = self.partition_dir() {
- return PartitionDirResolution::Resolved(dir);
- }
- let file_backed =
- self.log.storages().iter().any(|storage| {
- storage.messages_reader.is_some() || storage.messages_size.is_some()
+ pub fn defer_view_transition(&mut self, message: server_common::MessageBag) {
+ if self.transition.is_none() {
+ self.transition = Some(PendingPartitionTransition::View {
+ message,
+ drain: None,
});
- if file_backed {
- PartitionDirResolution::Unresolvable
- } else {
- PartitionDirResolution::NoFiles
}
+ self.notify_io();
}
- fn has_persisted_segment_bytes(&self) -> bool {
- self.log
- .segments()
- .iter()
- .any(|segment| segment.size.as_bytes_u64() > 0)
+ pub fn has_io_dispatcher(&self) -> bool {
+ self.io_notifier.borrow().is_some()
}
- /// Mean encoded bytes per committed message, including its share of the
- /// batch headers, or `None` while the partition has committed nothing.
- ///
- /// Both counters are relaxed loads that retention also decrements, so this
- /// is a hint and nothing reads it as a bound. Its one consumer sizes the
- /// first read of a disk poll, where being wrong costs an extra read.
- fn mean_encoded_message_size(&self) -> Option<u32> {
- let messages = self.stats.messages_count_inconsistent();
- let bytes = self.stats.size_bytes_inconsistent();
- (messages > 0).then(|| u32::try_from(bytes / messages).unwrap_or(u32::MAX))
+ pub fn defer_view_truncation(
+ &mut self,
+ from_op: u64,
+ actions: Vec<consensus::VsrAction>,
+ peer: Option<u8>,
+ ) {
+ self.transition = Some(PendingPartitionTransition::Truncate {
+ from_op,
+ drain: None,
+ actions,
+ peer,
+ });
+ self.notify_io();
}
- /// Widest observed committed batch, for the disk walk's chunk floor.
- /// Recovered history starts unknown; each walk learns its own batch floor
- /// when an incomplete batch requires an exact reread.
- ///
- /// A high-water, never lowered: retention cannot make an older batch
- /// narrower, and the read path clamps it to the chunk ceiling anyway, so
- /// the worst a stale value costs is the fixed-size read polls did before
- /// they were sized at all.
- const fn widest_committed_batch(&self) -> u64 {
- self.widest_batch_bytes.get()
+ #[must_use]
+ pub fn retains_io_reservation(&self, identity: crate::PartitionIoIdentity) -> bool {
+ identity.incarnation == self.incarnation
+ && !self.io_active.is_retiring()
+ && identity.continuation == crate::PartitionIoContinuation::NoAck
+ && self
+ .no_ack
+ .front()
+ .is_some_and(|head| Some(head.local_order) == identity.local_order)
}
- /// Starting `(segment index, byte position)` for a disk poll, resolved
- /// via each segment's sparse index cache. An index miss starts at the
- /// segment's first byte (the walk filters precisely).
- fn disk_poll_start(&self, query: &MessageLookup) -> (usize, u64, Option<u64>) {
- let segments = self.log.segments();
- match query {
- MessageLookup::Offset { offset, .. } => {
- let segment_index = segments
- .iter()
- .rposition(|segment| segment.start_offset <= *offset)
- .unwrap_or(0);
- let entry = self
- .log
- .segment_indexes(segment_index)
- .and_then(|cache| cache.offset_lower_bound(*offset));
- let position = entry.map_or(0, |index| index.position);
- let entry_offset = entry.map(|index| index.offset).or_else(|| {
- segments
- .get(segment_index)
- .map(|segment| segment.start_offset)
- });
- (segment_index, position, entry_offset)
- }
- MessageLookup::Timestamp { timestamp, .. } => {
- // Resolve the starting SEGMENT from segment metadata, not from
- // the per-segment index caches: sealed segments drop their
- // cache at rotation, and a cache miss must not read as "the
- // timestamp is not in this segment" (skipping a sealed segment
- // loses its messages). Timestamps are monotone across
- // segments, so the first segment whose max timestamp reaches
- // the query is the correct start; the walk filters precisely,
- // so an early start is safe.
- let segment_index = segments
- .iter()
- .position(|segment| segment.max_timestamp >= *timestamp)
- .unwrap_or_else(|| segments.len().saturating_sub(1));
- let position = self
- .log
- .segment_indexes(segment_index)
- .and_then(|cache| cache.timestamp_lower_bound(*timestamp))
- .map_or(0, |index| index.position);
- (segment_index, position, None)
+ /// One complete owner mutation per call. File execution is always separate.
+ pub async fn resume_io(&mut self, config: &PartitionsConfig) -> crate::PartitionIoStep {
+ match self.resume_io_inner(config).await {
+ Ok(step) => step,
+ Err(error) => {
+ self.fail_io_work(&error);
+ if self.maintenance.shutdown.is_some() {
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::Complete);
+ }
+ crate::PartitionIoStep::Pending
}
}
}
- /// Project a client request into a prepare.
- ///
- /// A replay of a committed `(client, request)` is absorbed by this group's
- /// dedup slice; anything above the watermark projects into a prepare.
- /// Session lifecycle + eviction live on the metadata plane.
- ///
- /// `reply` is the in-process channel a `PartitionSubmit` carried in. When
- /// present the committed reply fires on it instead of going to the bus:
- /// the connection-owning shard writes it to the socket it holds, because
- /// `header.client` is the VSR consensus id and carries no home-shard
- /// routing. `None` keeps the bus path (auto-commit ops, tests).
- ///
- /// # Panics
- /// Panics if called when this partition's consensus instance is not the
- /// primary, is not in normal status, or is currently syncing.
- #[allow(clippy::future_not_send, clippy::too_many_lines)]
#[allow(clippy::too_many_lines)]
- pub async fn on_request(
+ async fn resume_io_inner(
&mut self,
- message: Message<RoutedRequestHeader>,
- reply: Option<consensus::Sender<Message<ReplyHeader>>>,
- ) {
- // Taken by whichever arm answers: the deny paths, the NoAck fast path,
- // or the pipeline entry that fires it at commit. Exactly one runs.
- let mut reply = reply;
- self.resynchronize_consumer_offset_reservations();
- let namespace = IggyNamespace::from_raw(message.header().group);
- let client_id = message.header().client;
- let request = message.header().request;
-
- let disposition = {
- let consensus = self.consensus();
- emit_sim_event(
- SimEventKind::ClientRequestReceived,
- &RequestLogEvent {
- replica: ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- client_id,
- request_id: request,
- operation: message.header().operation,
- },
- );
-
- let message = if message.header().operation == Operation::SendMessages {
- // Skip the batch-checksum pass: on the partition ingest path
- // nothing reads it before `stamp_prepare_for_persistence`
- // recomputes it over the stamped header. An already-canonical
- // batch (the plane's pre-encrypt convert output) returns early
- // inside the convert, so Skip only affects the wire-form
- // admission, whose output goes straight to project/stamp.
- match convert_request_message(namespace, message, ChecksumMode::Skip) {
- Ok(message) => message,
- Err(error) => {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- "failed to convert send_messages request",
- )
- .with_operation(Operation::SendMessages)
- .with_error(error.to_string()),
- );
- return;
- }
+ config: &PartitionsConfig,
+ ) -> Result<crate::PartitionIoStep, IggyError> {
+ if self.io_active.get().is_some() {
+ if self.io_active.is_interrupted() && self.maintenance.shutdown.is_some() {
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::Complete);
+ }
+ return Ok(crate::PartitionIoStep::Pending);
+ }
+ if self.io_active.is_retiring() {
+ if !self.io_active.is_deleting()
+ && matches!(
+ self.transition,
+ Some(PendingPartitionTransition::Quarantine { .. })
+ )
+ {
+ if self.superblock_intent.get().is_some() {
+ return self.io_plan(crate::PartitionIoContinuation::Superblock);
}
+ return Ok(self.drive_quarantine_io().await);
+ }
+ return Ok(crate::PartitionIoStep::Pending);
+ }
+ if let Some(reply) = self.io_replies.pop_front() {
+ self.deliver_commit_replies(std::iter::once(reply)).await;
+ return Ok(crate::PartitionIoStep::Progress);
+ }
+ if let Some(transition) = &self.transition
+ && let Some(head) = self.no_ack.front()
+ && matches!(
+ head.phase,
+ NoAckPhase::Validate | NoAckPhase::OffsetReady(_)
+ )
+ {
+ // Admission stops while a transition is pending, so the head
+ // predates the purge that replaces its history.
+ let error = if matches!(transition, PendingPartitionTransition::Purge(_)) {
+ head.pending.history_reset_refusal()
} else {
- message
+ IggyError::TransientNotAccepted
};
-
- // Recovered commits need receipt restoration; live pipeline entries
- // already carry the pending identity and immutable capacity.
- let committed_history_pending = consensus.commit_min() < consensus.commit_max()
- && consensus.with_pipeline(|pipeline| {
- pipeline.entry_by_op(consensus.commit_min() + 1).is_none()
- || pipeline.entry_by_op(consensus.commit_max()).is_none()
- });
- // State-dependent admission belongs to the primary. A backup can
- // lag the committed offset table or message frontier and would
- // otherwise turn a routing artifact into a terminal 404 or 400.
- // Reject it first with the only response that proves the request
- // was never admitted, so the caller may safely retry elsewhere.
- if self.materialization_missing
- || consensus.is_follower()
- || !consensus.is_normal()
- || consensus.is_transferring()
- || committed_history_pending
- {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- if committed_history_pending {
- "rejecting client request while committed history is pending"
- } else {
- "rejecting client request on unavailable partition replica"
- },
- )
- .with_operation(message.header().operation),
- );
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- IggyError::TransientNotAccepted.as_code(),
- "non-primary transient reply send failed",
- reply.take(),
- )
- .await;
- return;
+ self.fail_no_ack(error, false);
+ }
+ if self.no_ack_ready() {
+ match self.drive_no_ack() {
+ NoAckStep::Progress => return Ok(crate::PartitionIoStep::Progress),
+ NoAckStep::Ready => {
+ return match self.io_plan(crate::PartitionIoContinuation::NoAck) {
+ Ok(step) => Ok(step),
+ Err(error) => {
+ self.fail_no_ack(error, false);
+ Ok(crate::PartitionIoStep::Progress)
+ }
+ };
+ }
+ NoAckStep::Reply => {
+ self.deliver_no_ack_reply().await;
+ return Ok(crate::PartitionIoStep::Progress);
+ }
+ NoAckStep::Pending => {}
}
-
- let frame_bytes = message.as_slice().len();
- if !is_auto_commit_client(client_id)
- && message.header().metadata_watermark < self.required_metadata_frontier
+ }
+ if self.superblock_intent.get().is_some()
+ && (!self.superblock_write_is_backed_off()
+ || self
+ .superblock_intent
+ .get()
+ .is_some_and(SuperblockIntent::final_write))
+ {
+ return self.io_plan(crate::PartitionIoContinuation::Superblock);
+ }
+ if self.fatal.is_some()
+ && !matches!(
+ self.transition,
+ Some(PendingPartitionTransition::Install(_))
+ )
+ {
+ // A fatal partition skips the flush, yet still publishes its WAL.
+ if self
+ .maintenance
+ .shutdown
+ .is_some_and(|phase| phase != PartitionShutdownPhase::Complete)
{
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- IggyError::TransientNotAccepted.as_code(),
- "partition requires newer metadata",
- reply.take(),
- )
- .await;
- return;
+ match self.drain_wal_for_shutdown() {
+ None => return Ok(crate::PartitionIoStep::Pending),
+ Some(Err(error)) => {
+ warn!(%error, "fatal partition could not drain its WAL on shutdown");
+ }
+ Some(Ok(())) => {}
+ }
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::Complete);
}
- if self.persistence.is_some()
- && frame_bytes > journal::partition_journal::PREPARE_BYTES_MAX
+ return Ok(crate::PartitionIoStep::Pending);
+ }
+ let has_pending_wire_actions = self.pending_wire_actions.borrow().is_some();
+ if has_pending_wire_actions && self.persist_superblock_if_needed().await {
+ let pending = self.pending_wire_actions.borrow_mut().take();
+ if let Some(pending) = pending
+ && pending.view == self.consensus.view()
+ && pending.log_view == self.consensus.log_view()
{
- let error = IggyError::InvalidMessagesSize(
- u32::try_from(frame_bytes).unwrap_or(u32::MAX),
- u32::try_from(journal::partition_journal::PREPARE_BYTES_MAX)
- .expect("prepare limit fits wire size"),
- );
- warn!(%error, namespace_raw = self.namespace().inner(), "persisted topic rejected oversized prepare");
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- error.as_code(),
- "oversized prepare rejection failed",
- reply.take(),
- )
- .await;
- return;
+ return Ok(crate::PartitionIoStep::WireActions(pending.actions));
}
-
- // Retirement and the retiring identity must never occupy the same
- // pipeline. Otherwise a later commit could recreate its receipt.
- let retirement_conflict = if message.header().operation == Operation::RetireSession {
- let Ok(retirement) = RetireSessionsRequest::decode_from(message.body()) else {
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- IggyError::InvalidCommand.as_code(),
- "invalid retirement reply failed",
- reply.take(),
- )
- .await;
- return;
- };
- retirement.identities.iter().any(|identity| {
- consensus.pending_request(identity.client_id).is_some()
- || self.pending_session_retirement(identity.client_id, identity.session)
- })
- } else {
- !is_auto_commit_client(client_id)
- && self.pending_session_retirement(client_id, message.header().session)
+ return Ok(crate::PartitionIoStep::Progress);
+ }
+ if self.resume_pending_prepare_ack().await {
+ return Ok(crate::PartitionIoStep::Progress);
+ }
+ if self.pending_commit.is_some() {
+ return match self.drive_commit()? {
+ CommitStep::Progress => Ok(crate::PartitionIoStep::Progress),
+ CommitStep::Pending => Ok(crate::PartitionIoStep::Pending),
+ CommitStep::Ready => self.io_plan(crate::PartitionIoContinuation::Commit),
+ CommitStep::Publish => {
+ self.queue_committed_replies();
+ Ok(crate::PartitionIoStep::Progress)
+ }
};
- if retirement_conflict {
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- IggyError::TransientNotAccepted.as_code(),
- "session retirement admission reply failed",
- reply.take(),
- )
- .await;
- return;
+ }
+ if self.materialization.is_some() {
+ return match self.drive_materialization()? {
+ MaterializationStep::Progress | MaterializationStep::Complete => {
+ Ok(crate::PartitionIoStep::Progress)
+ }
+ MaterializationStep::Pending => Ok(crate::PartitionIoStep::Pending),
+ MaterializationStep::Ready => {
+ self.io_plan(crate::PartitionIoContinuation::Materialization)
+ }
+ };
+ }
+ if let Some(through) = self.maintenance.flush_through {
+ if let Some(drain) = self.maintenance.flush_drain.as_ref()
+ && let Some(persistence) = self.persistence.as_ref()
+ && !persistence
+ .observe_drain(drain)
+ .map_err(|_| IggyError::CannotSyncFile)?
+ {
+ return Ok(crate::PartitionIoStep::Pending);
}
-
- // Parse once for both the delete-existence check and AckLevel dispatch.
- let consumer_offset = match message.header().operation {
- Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset => {
- match Self::parse_consumer_offset_request(message.header().operation, &message)
- {
- Ok(parsed) => Some(parsed),
- Err(error) => {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(
- consensus,
- PlaneKind::Partitions,
- ),
- "failed to parse consumer offset request",
- )
- .with_operation(message.header().operation)
- .with_error(error.to_string()),
- );
- return;
- }
+ let ready_through = self.flushable_through(through);
+ if self.begin_materialization(ready_through)?
+ && (self.materialization.is_some() || ready_through == through)
+ {
+ if self.materialization.is_none() {
+ self.finish_flush();
+ }
+ return Ok(crate::PartitionIoStep::Progress);
+ }
+ }
+ if let Some(PendingPartitionTransition::View { drain, .. }) = &mut self.transition {
+ if let Some(persistence) = &self.persistence {
+ let drain = drain.get_or_insert_with(|| persistence.begin_drain());
+ if !persistence
+ .observe_drain(drain)
+ .map_err(|_| IggyError::CannotSyncFile)?
+ {
+ return Ok(crate::PartitionIoStep::Pending);
+ }
+ persistence.finish_drain(drain);
+ }
+ let Some(PendingPartitionTransition::View { message, .. }) = self.transition.take()
+ else {
+ unreachable!("ready view transition exists");
+ };
+ return Ok(crate::PartitionIoStep::Transition(message));
+ }
+ if let Some(PendingPartitionTransition::Truncate { from_op, drain, .. }) =
+ &mut self.transition
+ {
+ if let Some(persistence) = &self.persistence {
+ if drain.is_none() {
+ persistence.truncate_from(*from_op);
+ if persistence.start() {
+ self.consensus
+ .message_bus()
+ .spawn(Rc::clone(persistence).run());
}
+ *drain = Some(persistence.begin_drain());
}
- _ => None,
+ let drain = drain.as_ref().expect("truncate drain exists");
+ if !persistence
+ .observe_drain(drain)
+ .map_err(|_| IggyError::CannotSyncFile)?
+ {
+ return Ok(crate::PartitionIoStep::Pending);
+ }
+ persistence.finish_drain(drain);
+ }
+ let Some(PendingPartitionTransition::Truncate {
+ from_op,
+ actions,
+ peer,
+ ..
+ }) = self.transition.take()
+ else {
+ unreachable!("truncate transition exists");
};
-
- // Dedup BEFORE the admission checks below: a replay of an
- // already-committed delete must answer the success its original
- // earned, not the typed 404 the existence check would raise now
- // that the offset is gone.
- //
- // One unresolved mutation per session/group protects the original
- // receipt until the caller resolves it. An exact pending retry is
- // still uncertain; a different request waits for that original
- // instead of displacing its receipt.
- //
- // HTTP handlers and pipelining SDKs can deliver a client's ids out of order:
- // a write refused transiently here is replayed after its
- // successors committed. The slice therefore keeps a committed-id
- // window under the watermark (`consensus::COMMITTED_WINDOW_BITS`)
- // and admits an unmarked id inside it instead of absorbing it.
- if !is_auto_commit_client(client_id) {
- if let Some((pending_session, pending_request, pending_operation)) =
- consensus.pending_request(client_id)
- {
- let error = if pending_request != request {
- IggyError::TransientNotAccepted
- } else if pending_session != message.header().session
- || pending_operation != message.header().operation
- {
- IggyError::InvalidCommand
- } else {
- IggyError::TransientNotCommitted
- };
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- error.as_code(),
- "in-flight dedup transient reply send failed",
- reply.take(),
- )
- .await;
- return;
- }
- match self.dedup.check_partition_request(
- client_id,
- message.header().user_id,
- message.header().session,
- request,
- message.header().operation,
- ) {
- consensus::client_table::RequestStatus::New => {}
- consensus::client_table::RequestStatus::Duplicate(cached) => {
- Self::deliver_reply_or_log(
- &self.consensus,
- message.header(),
- cached,
- reply.take(),
- "duplicate reply send failed",
- )
- .await;
- return;
- }
- status => {
- let error = match status {
- consensus::client_table::RequestStatus::AlreadyApplied { .. } => {
- emit_partition_diag(
- tracing::Level::DEBUG,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(
- consensus,
- PlaneKind::Partitions,
- ),
- "rejecting client request aged out of the dedup window",
- )
- .with_operation(message.header().operation)
- .with_client(client_id)
- .with_request(request),
- );
- IggyError::RequestTooOld
- }
- consensus::client_table::RequestStatus::OperationMismatch {
- ..
- } => IggyError::InvalidCommand,
- _ => IggyError::Unauthenticated,
- };
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- error.as_code(),
- "dedup rejection reply failed",
- reply.take(),
- )
- .await;
- return;
- }
- }
- if !consensus.has_retry_capacity(&self.dedup, client_id) {
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- IggyError::TransientNotAccepted.as_code(),
- "retry protection capacity reached",
- reply.take(),
- )
- .await;
- return;
- }
+ self.truncate_resident_from(from_op).await?;
+ return Ok(crate::PartitionIoStep::ViewApplied { actions, peer });
+ }
+ if matches!(self.transition, Some(PendingPartitionTransition::Purge(_))) {
+ return Ok(self.drive_purge_io().await);
+ }
+ if matches!(
+ self.transition,
+ Some(PendingPartitionTransition::Install(_))
+ ) {
+ return Ok(self.drive_install_io().await);
+ }
+ if self.transfer.as_ref().is_some_and(|session| {
+ session.target_accepted
+ && session
+ .artifacts
+ .iter()
+ .all(consensus::ChunkProgress::complete)
+ }) && self.consensus.state_transfer_stage() == consensus::StateTransferStage::Fetching
+ {
+ return Ok(crate::PartitionIoStep::TransferReady);
+ }
+ if self.materialization_missing {
+ if self.maintenance.shutdown == Some(PartitionShutdownPhase::DrainAccepted) {
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::DrainWriters);
}
-
+ if self.maintenance.shutdown.is_some() {
+ return Ok(self.drive_shutdown_io());
+ }
+ return Ok(crate::PartitionIoStep::Pending);
+ }
+ if self.maintenance.shutdown.is_none() && self.drive_transfer_plan() {
+ return Ok(crate::PartitionIoStep::Progress);
+ }
+ if let Some(step) = self.drive_checkpoint_io(config)? {
+ return Ok(step);
+ }
+ if let Some(through) = self.maintenance.retention {
if self
.persistence
.as_ref()
- .is_some_and(|persistence| !persistence.has_capacity(message.as_slice().len()))
+ .is_some_and(|persistence| !persistence.is_quiescent())
{
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- IggyError::TransientNotAccepted.as_code(),
- "partition WAL backpressure reply failed",
- reply.take(),
- )
- .await;
- return;
+ return Ok(crate::PartitionIoStep::Pending);
}
-
- if matches!(message.header().operation, Operation::DeleteConsumerOffset)
- && let Some((kind, consumer_id, _, _)) = consumer_offset
- && let Err(error) = self.ensure_consumer_offset_exists(kind, consumer_id)
- {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- "rejecting delete_consumer_offset for missing offset",
- )
- .with_operation(message.header().operation)
- .with_error(error.to_string()),
- );
- // Deny on the primary before the op enters the pipeline: nothing
- // replicates, so backups never see the rejected delete, and the
- // client gets a typed failure instead of waiting out its reply
- // timeout. The code rides `ReplyHeader.status` (not the result
- // body): the HTTP listener's `classify_partition_reply` reads the
- // status field to render the typed 404.
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- error.as_code(),
- "delete_consumer_offset deny reply send failed",
- reply.take(),
- )
- .await;
- return;
+ if self.retention_is_ready(through) {
+ return self.io_plan(crate::PartitionIoContinuation::Retention);
}
-
- // Reject an out-of-range consumer-offset store at admission,
- // mirroring the legacy `validate_partition_offset`: an empty
- // partition accepts no offset, and a stored offset may not run ahead
- // of the committed offset. Done here so the doomed op is never
- // replicated. Like the delete-offset deny above, the typed
- // `InvalidOffset` rides `ReplyHeader.status` (op=0, empty body): the
- // status-only `classify_partition_reply` would misread a result-body
- // code on this committed-shaped frame (op=commit_max) as success.
- if matches!(message.header().operation, Operation::StoreConsumerOffset)
- && let Some((kind, _, Some(requested_offset), _)) = consumer_offset
- && let Some(error) = self.store_offset_range_error(kind, requested_offset)
- {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- "rejecting store_consumer_offset for out-of-range offset",
- )
- .with_operation(message.header().operation)
- .with_error(error.to_string()),
- );
- Self::send_partition_deny_or_log(
- consensus,
- message.header(),
- error.as_code(),
- "store_consumer_offset deny reply send failed",
- reply.take(),
- )
- .await;
- return;
+ self.maintenance.retention = None;
+ }
+ if self.maintenance.shutdown.is_some() && self.consensus.request_queue_len() > 0 {
+ self.deny_queued_requests_at_shutdown().await;
+ return Ok(crate::PartitionIoStep::Progress);
+ }
+ if self.io_promotions > 0 || self.queued_requests_ready() {
+ self.io_promotions = self.io_promotions.saturating_sub(1);
+ let queued = self.consensus.request_queue_len();
+ self.drain_request_queue_into_prepares(1).await;
+ if self.consensus.request_queue_len() < queued {
+ return Ok(crate::PartitionIoStep::Progress);
}
+ }
+ let mut headers = self.select_persistable_commits(config);
+ let reply = !headers.is_empty() && self.consensus.is_primary();
+ if headers.is_empty() {
+ headers = self.collect_committable_from_journal(COMMIT_WALK_OPS_MAX, config);
+ }
+ if self.select_commit(headers, config, reply) {
+ return Ok(crate::PartitionIoStep::Progress);
+ }
+ Ok(self.drive_shutdown_io())
+ }
- // The node-local fast path is safe only for a single-replica
- // partition. Every mutation in a replicated partition enters VSR
- // regardless of the acknowledgement byte.
- if let Some((kind, consumer_id, offset, AckLevel::NoAck)) = consumer_offset
- && consensus.replica_count() == 1
- && matches!(
- message.header().operation,
- Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset,
- )
- {
- if offset.is_some()
- && !self
- .admit_consumer_offset_key(message.header(), kind, consumer_id, &mut reply)
- .await
- {
- return;
- }
- Disposition::NoAck {
- request_header: Box::new(*message.header()),
- kind,
- consumer_id,
- offset,
- }
- } else {
- // Fence AHEAD of the pipeline for a mint, not only at the mint.
- // A refusal at the mint arrives after the sequencer took the op,
- // where the only honest answer left is to fence the partition and
- // take the node down (`on_replicate`). Here the request has
- // entered nothing, so a transient disk fault costs the client one
- // retry instead of costing the process its life.
- //
- // The preflight answers the client itself on a refusal; see
- // [`Self::admit_reserved_send`].
- if !self.admit_reserved_send(&message, &mut reply).await {
- return;
- }
- // Two-queue: prepare slot -> project+replicate; prepare full +
- // request room -> buffer; both full -> drop+warn (client retries
- // via read-timeout).
- if consensus.pipeline_is_full() {
- let entry = consensus::RequestEntry::with_sender(message, reply.take())
- .with_consumer_offset_history(
- consumer_offset.is_some().then_some(self.poll_history),
- );
- let push_result = consensus.push_queued_request(entry);
- if let Err(mut refused) = push_result {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- "on_request: prepare and request queues both full, dropping",
- ),
- );
- // The request provably never entered either queue, so a
- // waiter can be told so instead of waiting out its
- // timeout.
- let waiter = refused.take_reply_sender();
- Self::send_partition_deny_or_log(
- consensus,
- refused.message.header(),
- IggyError::TransientNotAccepted.as_code(),
- "queues-full transient reply send failed",
- waiter,
- )
- .await;
- }
- return;
- }
-
- if let Some((kind, consumer_id, Some(_), _)) = consumer_offset
- && !self
- .admit_consumer_offset_key(message.header(), kind, consumer_id, &mut reply)
- .await
- {
- return;
- }
-
- let prepare = consensus::seal_prepare_capacity(
- message.project(consensus),
- consensus.retry_capacity(&self.dedup),
- );
- consensus.verify_pipeline();
- match reply.take() {
- Some(sender) => consensus.pipeline_message_with_sender(
- PlaneKind::Partitions,
- &prepare,
- sender,
- ),
- None => consensus.pipeline_message(PlaneKind::Partitions, &prepare),
- }
- Disposition::Replicate(prepare)
+ pub(crate) fn plan_install_io(&mut self) -> crate::PartitionIoStep {
+ match self.io_plan(crate::PartitionIoContinuation::Install) {
+ Ok(step) => step,
+ Err(error) => {
+ self.fail_install_capture(error);
+ crate::PartitionIoStep::Progress
}
- };
+ }
+ }
- match disposition {
- Disposition::Replicate(prepare) => self.on_replicate(prepare).await,
- Disposition::NoAck {
- request_header,
- kind,
- consumer_id,
- offset,
- } => {
- self.apply_consumer_offset_no_ack(
- request_header,
- kind,
- consumer_id,
- offset,
- reply.take(),
- )
- .await;
- }
+ fn io_plan(
+ &self,
+ continuation: crate::PartitionIoContinuation,
+ ) -> Result<crate::PartitionIoStep, IggyError> {
+ let materialization = continuation == crate::PartitionIoContinuation::Materialization
+ || (continuation == crate::PartitionIoContinuation::Commit
+ && self
+ .pending_commit
+ .as_ref()
+ .is_some_and(|pending| matches!(pending.phase, CommitPhase::Materialize)));
+ let mut allocation_charge = if continuation == crate::PartitionIoContinuation::Install {
+ self.install_io_charge()
+ .ok_or(IggyError::CannotWriteToFile)?
+ } else if materialization
+ && let Some(MaterializationPhase::Save(chunk)) =
+ self.materialization.as_ref().map(|pending| pending.phase)
+ {
+ chunk.allocation_charge
+ } else {
+ crate::io::file_phase_charge::<SB>(
+ std::iter::once(self.partition_dir.as_deref())
+ .chain(ConsumerKind::ALL.map(|kind| self.consumer_offsets_dir(kind)))
+ .flatten(),
+ )
+ .ok_or(IggyError::CannotWriteToFile)?
+ };
+ if continuation == crate::PartitionIoContinuation::Checkpoint
+ && let Some(PendingCheckpoint::Receipts(_, receipts)) = &self.maintenance.checkpoint
+ {
+ allocation_charge = allocation_charge
+ .checked_add(receipts.capacity())
+ .ok_or(IggyError::CannotWriteToFile)?;
+ }
+ if continuation == crate::PartitionIoContinuation::Purge
+ && let Some(PendingPartitionTransition::Purge(purge)) = &self.transition
+ && let PurgePhase::Offsets(kind) = purge.phase
+ {
+ allocation_charge = self
+ .offset_cleanup_charge(kind)
+ .and_then(|known| allocation_charge.checked_add(known))
+ .ok_or(IggyError::CannotWriteToFile)?;
}
+ if allocation_charge > self.io_bytes_max.get() {
+ return Err(IggyError::CannotWriteToFile);
+ }
+ Ok(crate::PartitionIoStep::Ready(crate::PartitionIoPlan {
+ continuation,
+ allocation_charge,
+ }))
}
- /// Promote up to `slots_freed` buffered requests into prepares post-commit.
- ///
- /// Promotion runs no DEDUP preflight: the request was classified at
- /// admission and the slice cannot have gained a higher watermark for it
- /// since (only a commit moves it, and this entry has not committed).
- ///
- /// The RESERVATION preflight is repeated per promotion, and re-derives the
- /// ceiling from the live mint frontier rather than trusting the one the
- /// request was admitted under. Several queued batches can cumulatively cross
- /// the lease while they wait, and without this the first one past it reaches
- /// the exact fence at the mint, where a refusal fences the partition and
- /// takes the node down instead of returning `TransientNotAccepted`. A refused
- /// promotion ends the drain: the ones behind it want the same claim and would
- /// each be answered with the same transient.
- ///
- /// Per-iteration `is_primary && is_normal && !is_transferring` asserts inlined
- /// (closure form's `&consensus` borrow conflicts with `&mut self`). Guards
- /// against view-change-reset flipping status across `on_replicate` await.
- ///
- /// View-change safety: `reset_view_change_state` calls
- /// [`consensus::Pipeline::clear_request_queue`]; resumed loop breaks via
- /// `else { break }`.
+ /// Caller must reserve the plan's slot and byte charge before capture.
///
- /// # Panics
- /// On mid-iteration status flip. Reachable only if `clear_request_queue`
- /// is bypassed at view-change reset.
- #[allow(clippy::future_not_send, clippy::too_many_lines)]
- pub async fn drain_request_queue_into_prepares(&mut self, slots_freed: usize) {
- self.resynchronize_consumer_offset_reservations();
- let mut promoted = 0;
- // Denials do not consume a slot, so without a budget one drain could
- // answer the whole queue, each with an awaited reply send, inside one
- // commit turn. Consecutive denials end the drain. The shard tick
- // resumes parked work even if no further operation commits.
- let mut consecutive_denials = 0usize;
- while promoted < slots_freed {
- let req = self.consensus().pop_queued_request();
- let Some(mut req) = req else { break };
- let context = req.take_auto_commit();
- // History and capacity ownership can change while the request
- // waits for a prepare slot, so admission alone is not sufficient.
- if context.as_ref().is_some_and(|context| {
- context.history != self.poll_history
- || !self
- .consumer_offset_capacity_for(context.reservation.kind())
- .owns(&context.reservation)
- }) {
- if Self::record_promotion_denial(&mut consecutive_denials) {
- break;
- }
- continue;
+ /// # Errors
+ /// Rejects exhausted identity space or a phase whose captured resources changed.
+ pub fn capture_io(
+ &mut self,
+ plan: crate::PartitionIoPlan,
+ config: &PartitionsConfig,
+ ) -> Result<Option<crate::CapturedPartitionIo<SB>>, IggyError> {
+ if self.io_active.get().is_some()
+ || self.io_active.is_deleting()
+ || (self.io_active.is_retiring()
+ && !matches!(
+ self.transition,
+ Some(PendingPartitionTransition::Quarantine { .. })
+ ))
+ {
+ return Ok(None);
+ }
+ let sequence = self
+ .io_sequence
+ .get()
+ .checked_add(1)
+ .ok_or(IggyError::CannotWriteToFile)?;
+ let mut gate = None;
+ let job = match plan.continuation {
+ crate::PartitionIoContinuation::Commit => self.capture_commit_phase(config)?,
+ crate::PartitionIoContinuation::NoAck => Some(self.capture_no_ack_phase()?),
+ crate::PartitionIoContinuation::Materialization => {
+ self.capture_materialization(config)?
}
-
- // Taken before the preflight so a refusal answers the parked waiter
- // instead of waking it with `Canceled`.
- let mut reply_sender = req.take_reply_sender();
- if !is_auto_commit_client(req.message.header().client)
- && req.message.header().metadata_watermark < self.required_metadata_frontier
- {
- if self
- .deny_queued_request(
- req.message.header(),
- IggyError::TransientNotAccepted.as_code(),
- "queued request metadata frontier is behind retirement",
- reply_sender.take(),
- &mut consecutive_denials,
- )
- .await
- {
- break;
- }
- continue;
+ crate::PartitionIoContinuation::Install => Some(self.capture_install_io(config)?),
+ crate::PartitionIoContinuation::Quarantine => {
+ let Some(PendingPartitionTransition::Quarantine { phase, .. }) =
+ self.transition.as_mut()
+ else {
+ return Ok(None);
+ };
+ *phase = QuarantinePhase::Submitted;
+ Some(crate::PartitionIoJob::Quarantine {
+ directory: self
+ .partition_dir
+ .clone()
+ .ok_or(IggyError::CannotDeleteFile)?,
+ revision: self.created_revision,
+ replicated: self.consensus.replica_count() > 1,
+ })
}
- if req
- .consumer_offset_history()
- .is_some_and(|history| history != self.poll_history)
- {
- // An old offset can fit the replacement history, and an old
- // delete can erase a replacement checkpoint. Refuse before
- // projection with a terminal error so clients do not replay it.
- let error = Self::parse_consumer_offset_request(
- req.message.header().operation,
- &req.message,
- )
- .map_or(
- IggyError::InvalidCommand,
- |(_, consumer_id, offset, _)| {
- offset.map_or_else(
- || IggyError::ConsumerOffsetNotFound(consumer_id as usize),
- IggyError::InvalidOffset,
- )
+ crate::PartitionIoContinuation::Purge => Some(self.capture_purge_phase(config)?),
+ crate::PartitionIoContinuation::Retention => self.capture_segment_removal(),
+ crate::PartitionIoContinuation::Checkpoint => self.capture_checkpoint_phase()?,
+ crate::PartitionIoContinuation::Superblock => {
+ let Some(guard) = self.superblock_lock.try_acquire_owned() else {
+ return Ok(None);
+ };
+ let Some(intent) = self.superblock_intent.take() else {
+ guard.release();
+ return Ok(None);
+ };
+ let Some(superblock) = self.superblock.as_ref() else {
+ guard.release();
+ return Ok(None);
+ };
+ let state = self.superblock_state(intent);
+ self.superblock_running
+ .set(Some((intent, self.superblock_write_is_backed_off())));
+ gate = Some(guard);
+ Some(crate::PartitionIoJob::Superblock(
+ crate::io::SuperblockIoJob {
+ superblock: Rc::clone(superblock),
+ state,
},
- );
- if self
- .deny_queued_request(
- req.message.header(),
- error.as_code(),
- "queued offset history deny reply send failed",
- reply_sender.take(),
- &mut consecutive_denials,
- )
- .await
+ ))
+ }
+ };
+ let Some(job) = job else {
+ return Ok(None);
+ };
+ let identity = crate::PartitionIoIdentity {
+ namespace: self.namespace(),
+ incarnation: self.incarnation,
+ history: self.poll_history,
+ sequence,
+ continuation: plan.continuation,
+ local_order: (plan.continuation == crate::PartitionIoContinuation::NoAck)
+ .then(|| self.no_ack.front().map(|head| head.local_order))
+ .flatten(),
+ };
+ self.io_sequence.set(sequence);
+ self.io_active.set(Some(identity));
+ Ok(Some(crate::CapturedPartitionIo {
+ identity,
+ job,
+ gate,
+ quiescence: Rc::clone(&self.io_active),
+ }))
+ }
+
+ /// # Errors
+ /// Rejects obsolete identities and propagates the phase's storage failure.
+ pub fn accept_io(
+ &mut self,
+ identity: crate::PartitionIoIdentity,
+ result: crate::PartitionIoResult,
+ ) -> Result<(), IggyError> {
+ if self.io_active.get() != Some(identity) || identity.incarnation != self.incarnation {
+ return Err(IggyError::CannotWriteToFile);
+ }
+ self.io_active.set(None);
+ if self.io_active.is_retiring()
+ && !matches!(
+ identity.continuation,
+ crate::PartitionIoContinuation::Superblock
+ | crate::PartitionIoContinuation::Quarantine
+ )
+ {
+ drop(result);
+ return Ok(());
+ }
+ if identity.history != self.poll_history {
+ self.fence_flush_failure();
+ return Err(IggyError::CannotWriteToFile);
+ }
+ let outcome = match identity.continuation {
+ crate::PartitionIoContinuation::Commit => self.accept_commit_phase(result),
+ crate::PartitionIoContinuation::Materialization => {
+ self.accept_materialization_phase(result)
+ }
+ crate::PartitionIoContinuation::NoAck => {
+ self.accept_no_ack_phase(result);
+ Ok(())
+ }
+ crate::PartitionIoContinuation::Install => {
+ self.accept_install_io(result);
+ Ok(())
+ }
+ crate::PartitionIoContinuation::Quarantine => {
+ let Some(PendingPartitionTransition::Quarantine { phase, .. }) =
+ self.transition.as_mut()
+ else {
+ return Err(IggyError::CannotDeleteFile);
+ };
+ let crate::PartitionIoResult::Quarantine(outcome) = result else {
+ return Err(IggyError::CannotDeleteFile);
+ };
+ *phase = QuarantinePhase::Done(outcome.map(Some));
+ Ok(())
+ }
+ crate::PartitionIoContinuation::Purge => {
+ self.accept_purge_phase(result);
+ Ok(())
+ }
+ crate::PartitionIoContinuation::Retention => self
+ .accept_segment_removal(&result)
+ .map(|_| ())
+ .ok_or(IggyError::CannotDeleteFile),
+ crate::PartitionIoContinuation::Checkpoint => self.accept_checkpoint_io(result),
+ crate::PartitionIoContinuation::Superblock => {
+ let Some((intent, backoff)) = self.superblock_running.take() else {
+ return Err(IggyError::CannotWriteToFile);
+ };
+ let crate::PartitionIoResult::Superblock(result) = result else {
+ return Err(IggyError::CannotWriteToFile);
+ };
+ let succeeded = self.accept_superblock(result, backoff);
+ self.superblock_completed.set(Some((intent, succeeded)));
+ if !succeeded
+ && matches!(intent, SuperblockIntent::Advance { .. })
+ && !self.io_active.is_retiring()
+ && self.superblock_intent.get().is_none()
{
- break;
+ self.superblock_intent.set(Some(intent));
}
- continue;
- }
- if !self
- .admit_reserved_send(&req.message, &mut reply_sender)
- .await
- {
- break;
+ Ok(())
}
+ };
+ if let Err(error) = &outcome {
+ self.fail_io_work(error);
+ }
+ self.notify_io();
+ outcome
+ }
- let parsed_store = (req.message.header().operation == Operation::StoreConsumerOffset)
- .then(|| {
- Self::parse_consumer_offset_request(
- Operation::StoreConsumerOffset,
- &req.message,
+ pub fn reject_io_capture(
+ &mut self,
+ continuation: crate::PartitionIoContinuation,
+ error: IggyError,
+ ) {
+ match continuation {
+ crate::PartitionIoContinuation::NoAck => {
+ let file_attempted = self.no_ack.front().is_some_and(|head| {
+ matches!(
+ head.phase,
+ NoAckPhase::Directory | NoAckPhase::DirectorySubmitted
)
});
- if let Some(parsed) = parsed_store {
- let Ok((kind, consumer_id, Some(offset), _)) = parsed else {
- if self
- .deny_queued_request(
- req.message.header(),
- IggyError::InvalidCommand.as_code(),
- "queued consumer offset parse deny reply send failed",
- reply_sender.take(),
- &mut consecutive_denials,
- )
- .await
- {
- break;
- }
- continue;
- };
- if let Some(error) = self.store_offset_range_error(kind, offset) {
- if self
- .deny_queued_request(
- req.message.header(),
- error.as_code(),
- "queued offset range deny reply send failed",
- reply_sender.take(),
- &mut consecutive_denials,
- )
- .await
- {
- break;
- }
- continue;
+ self.fail_no_ack(error, file_attempted);
+ }
+ crate::PartitionIoContinuation::Install => self.fail_install_capture(error),
+ crate::PartitionIoContinuation::Purge => {
+ if let Some(PendingPartitionTransition::Purge(purge)) = &mut self.transition {
+ purge.phase = PurgePhase::Done(Err(PurgeError::Unserviceable(error)));
}
- if !self
- .admit_consumer_offset_key(
- req.message.header(),
- kind,
- consumer_id,
- &mut reply_sender,
- )
- .await
+ }
+ crate::PartitionIoContinuation::Quarantine => {
+ if let Some(PendingPartitionTransition::Quarantine { phase, .. }) =
+ &mut self.transition
{
- if Self::record_promotion_denial(&mut consecutive_denials) {
- break;
- }
- continue;
+ *phase = QuarantinePhase::Done(Err(std::io::Error::other(error)));
}
}
- consecutive_denials = 0;
+ crate::PartitionIoContinuation::Retention => self.maintenance.retention = None,
+ _ => self.fail_io_work(&error),
+ }
+ self.notify_io();
+ }
- let prepare = {
- let consensus = self.consensus();
- assert!(
- !consensus.is_follower(),
- "drain_request_queue_into_prepares: primary only"
- );
- assert!(
- consensus.is_normal(),
- "drain_request_queue_into_prepares: status must be normal"
- );
- assert!(
- !consensus.is_transferring(),
- "drain_request_queue_into_prepares: must not be transferring state"
- );
- // The waiter parked with the request; it must travel into the
- // prepare slot or the commit has nobody to answer.
- let prepare = consensus::seal_prepare_capacity(
- req.message.project(consensus),
- consensus.retry_capacity(&self.dedup),
- );
- consensus.verify_pipeline();
- match reply_sender {
- Some(sender) => consensus.pipeline_message_with_sender(
- PlaneKind::Partitions,
- &prepare,
- sender,
- ),
- None => consensus.pipeline_message(PlaneKind::Partitions, &prepare),
- }
- prepare
- };
- promoted += 1;
- self.on_replicate(prepare).await;
- drop(context);
+ pub fn interrupt_io(&mut self, identity: crate::PartitionIoIdentity) {
+ if self.io_active.get() == Some(identity) {
+ self.fence_flush_failure();
}
}
- #[must_use]
- pub fn queued_requests_ready(&self) -> bool {
- self.fatal.is_none()
- && self.consensus.is_primary()
- && self.consensus.is_normal()
- && !self.consensus.is_transferring()
- && !self.consensus.pipeline_is_full()
- && self.consensus.request_queue_len() > 0
+ pub fn begin_shutdown_io(&mut self) {
+ if self.maintenance.shutdown.is_none() {
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::DrainAccepted);
+ }
+ self.maintenance.transfer_flush = false;
+ self.notify_io();
}
- /// Resume a bounded promotion turn without waiting for another commit.
- pub async fn resume_queued_requests(&mut self) {
- if self.queued_requests_ready() {
- self.drain_request_queue_into_prepares(1).await;
+ #[must_use]
+ pub fn shutdown_io_complete(&self) -> bool {
+ (self.io_active.get().is_none() || self.io_active.is_interrupted())
+ && ((self.maintenance.shutdown == Some(PartitionShutdownPhase::Complete)
+ && (self.fatal.is_some()
+ || self.materialization_missing
+ || self.io_active.is_retiring()
+ || self
+ .maintenance
+ .flush_completed_through
+ .is_some_and(|through| through >= self.shutdown_flush_target())))
+ || (self.io_active.is_deleting()
+ && self
+ .persistence
+ .as_ref()
+ .is_none_or(|persistence| persistence.is_quiescent())))
+ }
+
+ fn fail_io_work(&mut self, error: &IggyError) {
+ if self.pending_commit.is_none() && self.maintenance.transfer_flush {
+ self.materialization = None;
+ self.finish_flush();
+ self.maintenance.flush_completed_through = None;
+ let mut plan = self.transfer_plan.borrow_mut();
+ if matches!(*plan, Some(TransferPlanSlot::Wanted)) {
+ *plan = Some(TransferPlanSlot::Failed(
+ PartitionTransferUnavailable::FlushFailed(error.clone()),
+ ));
+ }
+ } else {
+ self.fail_pending_commit(error);
+ self.fence_flush_failure();
}
}
- async fn deny_queued_request(
- &self,
- header: &RoutedRequestHeader,
- status: u32,
- send_fail_label: &'static str,
- waiter: Option<consensus::Sender<Message<ReplyHeader>>>,
- consecutive_denials: &mut usize,
- ) -> bool {
- Self::send_partition_deny_or_log(self.consensus(), header, status, send_fail_label, waiter)
- .await;
- Self::record_promotion_denial(consecutive_denials)
+ fn defer_flush(&mut self, through: u64) {
+ self.maintenance.flush_through = Some(
+ self.maintenance
+ .flush_through
+ .map_or(through, |pending| pending.max(through)),
+ );
}
- const fn record_promotion_denial(consecutive_denials: &mut usize) -> bool {
- *consecutive_denials += 1;
- *consecutive_denials >= PROMOTION_DENIALS_MAX
+ /// A Replicated commit runs ahead of the WAL writer, so its flush takes the
+ /// written prefix and leaves the rest for a later writer completion.
+ fn flushable_through(&self, through: u64) -> u64 {
+ self.persistence
+ .as_ref()
+ .filter(|persistence| {
+ persistence.segment_checkpoint().is_some() && !self.durability().is_persisted()
+ })
+ .map_or(through, |persistence| through.min(persistence.written_op()))
}
- /// # Panics
- /// Panics on a primary when a prepare's op is ahead of the local
- /// sequencer: journaling it would make the next op assignment collide,
- /// which is unrecoverable in place.
- #[allow(clippy::future_not_send, clippy::too_many_lines)]
- pub async fn on_replicate(&mut self, message: Message<PrepareHeader>) {
- self.resynchronize_consumer_offset_reservations();
- let header = *message.header();
- // Same reason as the metadata plane: `checksum` is compared as an opaque token
- // downstream, so a corrupted frame passes whenever its flipped value satisfies
- // those comparisons.
- if let Err(reason) = verify_prepare_integrity(&header, message.as_slice()) {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions),
- "discarding prepare that failed its own integrity check",
- )
- .with_operation(header.operation)
- .with_op(header.op)
- .with_reason(reason),
- );
- return;
+ fn request_flush(&mut self, through: u64) {
+ self.defer_flush(through);
+ if self.maintenance.flush_drain.is_none()
+ && let Some(persistence) = self
+ .persistence
+ .as_ref()
+ .filter(|persistence| persistence.segment_checkpoint().is_some())
+ {
+ self.start_persistence();
+ self.maintenance.flush_drain = Some(persistence.begin_drain());
}
- let current_op = {
- let consensus = self.consensus();
- match replicate_preflight(consensus, &header) {
- Ok(current_op) => current_op,
- Err(reason) => {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- "ignoring prepare during replicate preflight",
- )
- .with_operation(header.operation)
- .with_op(header.op)
- .with_reason(reason.as_str()),
- );
- return;
- }
- }
- };
- #[allow(clippy::cast_possible_truncation)]
- let fenced_by_commit = fence_old_prepare_by_commit(self.consensus(), &header);
- if fenced_by_commit {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "received old prepare (<= commit_min), skipping replication",
- )
- .with_operation(header.operation)
- .with_op(header.op),
- );
- // Fenced by commit_min: we've already executed this op, the
- // whole chain has it committed. Safe to drop entirely.
- return;
+ self.notify_io();
+ }
+
+ fn finish_flush(&mut self) {
+ if let Some(through) = self.maintenance.flush_through.take() {
+ self.maintenance.flush_completed_through = Some(through);
+ }
+ self.maintenance.transfer_flush = false;
+ if let Some(drain) = self.maintenance.flush_drain.take()
+ && let Some(persistence) = self.persistence.as_ref()
+ {
+ persistence.finish_drain(&drain);
}
+ }
- let journal_holds_op = self.log.journal().inner.holds_op(header.op);
- if journal_holds_op {
- // Retransmit after downstream flap: durable here but commit
- // hasn't caught up. Re-forward + re-ACK so primary's view of
- // us is consistent. Both downstream and primary are idempotent
- // on duplicate (replica, op).
- emit_partition_diag(
- tracing::Level::DEBUG,
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "journal already holds prepare, re-forwarding + re-acking",
- )
- .with_operation(header.operation)
- .with_op(header.op),
- );
- let Some(journaled) = self.log.journal().inner.repair_entry(header.op) else {
- emit_partition_diag(
- tracing::Level::ERROR,
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "journal header exists without matching prepare bytes",
- )
- .with_operation(header.operation)
- .with_op(header.op),
- );
- return;
- };
- if !journaled_prepare_matches_retransmit(&journaled, &message) {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "rejecting retransmitted prepare that differs from the journaled entry",
- )
- .with_operation(header.operation)
- .with_op(header.op),
- );
- return;
+ /// The plan an offer round hashes against, or `None` while the flush that
+ /// has to come before it runs. A mounted owner pins the plan in the I/O
+ /// turn that finds every applied op in its segment, so commits landing
+ /// between rounds cannot starve it. One without an owner flushes and
+ /// captures inline.
+ pub(crate) async fn request_transfer_plan(
+ &mut self,
+ config: &PartitionsConfig,
+ ) -> Result<Option<Rc<crate::state_transfer::TransferPlan>>, PartitionTransferUnavailable> {
+ if !self.has_io_dispatcher() {
+ if !self
+ .flush_committed_messages(config)
+ .await
+ .map_err(PartitionTransferUnavailable::FlushFailed)?
+ {
+ return Ok(None);
}
- let Some(frozen_for_forward) = restamp_prepare_view(journaled, header.view) else {
- emit_partition_diag(
- tracing::Level::ERROR,
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "failed to restamp journaled prepare for retransmission",
- )
- .with_operation(header.operation)
- .with_op(header.op),
- );
- return;
- };
- let prepare_to_persist = self
- .persistence
- .as_ref()
- .map(|_| frozen_for_forward.clone());
- let consensus = self.consensus();
- if consensus.is_follower() && header.op > consensus.sequencer().current_sequence() {
- consensus.sequencer().set_sequence(header.op);
- consensus.set_last_prepare_checksum(header.checksum);
- consensus.observe_prepare_timestamp(header.timestamp);
+ return self.capture_transfer_plan().map(|plan| Some(Rc::new(plan)));
+ }
+ if self.fatal.is_some() {
+ return Err(PartitionTransferUnavailable::FlushFailed(
+ IggyError::CannotSyncFile,
+ ));
+ }
+ // A purge, an install or a quarantine rewrites the files a plan names.
+ if self.transition.is_some() || self.io_active.is_retiring() {
+ return Ok(None);
+ }
+ let slot = self.transfer_plan.borrow_mut().take();
+ match slot {
+ Some(TransferPlanSlot::Pinned(plan)) => {
+ *self.transfer_plan.borrow_mut() = Some(TransferPlanSlot::Pinned(Rc::clone(&plan)));
+ Ok(Some(plan))
}
- if let Err(error) =
- replicate_frozen_to_next_in_chain(consensus, frozen_for_forward).await
- {
- let is_transport_error = error.is_transport();
- emit_partition_diag(
- if is_transport_error {
- tracing::Level::WARN
- } else {
- tracing::Level::ERROR
- },
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "failed to re-forward retransmitted prepare to next in chain",
- )
- .with_operation(header.operation)
- .with_op(header.op)
- .with_error(error.to_string()),
- );
- if !is_transport_error {
- return;
+ Some(TransferPlanSlot::Failed(error)) => Err(error),
+ Some(TransferPlanSlot::Wanted) | None => {
+ *self.transfer_plan.borrow_mut() = Some(TransferPlanSlot::Wanted);
+ self.notify_io();
+ Ok(None)
+ }
+ }
+ }
+
+ /// Pins the plan a transfer request waits for in the turn that finds every
+ /// applied op in its segment, flushing the applied prefix first while any
+ /// of it is still resident.
+ fn drive_transfer_plan(&mut self) -> bool {
+ if !matches!(*self.transfer_plan.borrow(), Some(TransferPlanSlot::Wanted))
+ || self.history_writes_pending()
+ {
+ return false;
+ }
+ let commit_min = self.consensus.commit_min();
+ if self.log.journal().inner.committed_prefix_len(commit_min) > 0 {
+ self.maintenance.transfer_flush = true;
+ self.request_flush(commit_min);
+ return true;
+ }
+ // A flush through a committed op the walk has not applied yet leaves
+ // its bytes in the segments, past the op the plan would name.
+ if self.consensus.commit_max() > commit_min
+ && !commit_min
+ .checked_add(1)
+ .is_some_and(|next| self.log.journal().inner.holds_op(next))
+ {
+ return false;
+ }
+ let slot = match self.capture_transfer_plan() {
+ Ok(plan) => TransferPlanSlot::Pinned(Rc::new(plan)),
+ Err(error) => TransferPlanSlot::Failed(error),
+ };
+ *self.transfer_plan.borrow_mut() = Some(slot);
+ true
+ }
+
+ fn drive_shutdown_io(&mut self) -> crate::PartitionIoStep {
+ if !self.materialization_missing
+ && matches!(
+ self.maintenance.shutdown,
+ Some(
+ PartitionShutdownPhase::DrainWriters
+ | PartitionShutdownPhase::Collapse
+ | PartitionShutdownPhase::Complete
+ )
+ )
+ && self
+ .maintenance
+ .flush_completed_through
+ .is_none_or(|through| through < self.shutdown_flush_target())
+ {
+ self.maintenance.shutdown_drain = None;
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::DrainAccepted);
+ }
+ match self.maintenance.shutdown {
+ Some(PartitionShutdownPhase::DrainAccepted) => {
+ if self.pending_commit.is_some()
+ || !self.no_ack.is_empty()
+ || self.consensus.request_queue_len() > 0
+ {
+ return crate::PartitionIoStep::Pending;
}
+ // A stopping backup may never ack the uncommitted pipeline tail.
+ self.request_flush(self.shutdown_flush_target());
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::Flush);
+ crate::PartitionIoStep::Progress
}
- if let Some(prepare) = prepare_to_persist
- && !self.submit_prepare_persistence(prepare, header.operation)
+ Some(PartitionShutdownPhase::Flush)
+ if self.maintenance.flush_through.is_none() && self.materialization.is_none() =>
{
- return;
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::DrainWriters);
+ crate::PartitionIoStep::Progress
}
- self.send_prepare_ok(&header).await;
- return;
- }
-
- // A header-only StartView can announce bodies this backup still lacks.
- // Live same-view retransmits may fill its next verified slot; repair
- // frames above commit still require the elected canonical headers.
- let is_backup = self.consensus().is_follower();
- if is_backup {
- let fills_announced_gap = header.op <= current_op
- && header.op > self.purge_floor_op
- && !self.consensus().view_log_is_pending()
- && self
- .log
- .journal()
- .inner
- .repaired_window_shape(self.consensus().commit_min(), header.op - 1)
- .complete
- && if header.op == 1 {
- header.parent == 0
+ Some(PartitionShutdownPhase::DrainWriters) => {
+ match self.drain_wal_for_shutdown() {
+ None => return crate::PartitionIoStep::Pending,
+ Some(Ok(())) => {}
+ Some(Err(error)) => {
+ warn!(%error, "partition shutdown could not drain its WAL");
+ self.fence_flush_failure();
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::Complete);
+ return crate::PartitionIoStep::Progress;
+ }
+ }
+ self.maintenance.shutdown = Some(if self.materialization_missing {
+ PartitionShutdownPhase::Complete
} else {
- self.log
- .journal()
- .inner
- .repair_header(header.op - 1)
- .is_some_and(|previous| previous.checksum == header.parent)
- };
- if header.op != current_op + 1 && !fills_announced_gap {
- // `sequence` is what separates the two shapes this line covers:
- // a forward gap (op above the sequencer, the hole the repair
- // driver closes) and a retransmit of an op this replica already
- // sequenced. Without it they read identically.
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "dropping out-of-order prepare (gap)",
- )
- .with_operation(header.operation)
- .with_op(header.op)
- .with_sequence(current_op),
- );
- self.prepare_gap_drops
- .set(self.prepare_gap_drops.get().saturating_add(1));
- return;
+ PartitionShutdownPhase::Collapse
+ });
+ crate::PartitionIoStep::Progress
}
- } else {
- // Primary: `push_prepare_entry` pre-advanced the sequencer, so a
- // locally-originated prepare always satisfies
- // `header.op == current_op`. The two violation directions carry
- // very different risk:
- // - below the sequencer: a duplicate delivery (parked-frame
- // redispatch, retransmit echo) of an op this primary already
- // sequenced. Proceeding is safe only because the two gates above
- // already returned for every copy this replica can still see:
- // `fence_old_prepare_by_commit` drops the executed ops and
- // `journal_holds_op` the resident ones, so reaching here means
- // the journal lacks this op and has to be given it. Apply is not
- // idempotent on its own for a produce: `append_messages`
- // re-stamps from the local dirty counter and the journal's op
- // index is last-write-wins, so appending an op the journal
- // already holds would mint a second copy at fresh offsets and
- // orphan the first. Log loudly for diagnosis.
- // - above the sequencer: journaling an op the sequencer has not
- // assigned yet means the next local assignment would collide
- // with it. Unreachable today (view fences run first, one
- // primary per view, the chain stops before the primary), so
- // trip the invariant in debug; in release log loudly and drop
- // rather than crash a library or corrupt op assignment.
- if header.op > current_op {
- debug_assert!(
- header.op <= current_op,
- "primary: prepare op {} ahead of sequencer {}; next op assignment would collide",
- header.op,
- current_op
- );
- emit_partition_diag(
- tracing::Level::ERROR,
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "primary prepare ahead of sequencer; dropping to avoid op-assignment collision",
- )
- .with_operation(header.operation)
- .with_op(header.op),
- );
- return;
+ Some(PartitionShutdownPhase::Collapse) => {
+ let reserved = self.mint_frontier();
+ let verdict = if self.superblock.is_none()
+ || self.durable_offset_reserved.get() <= reserved
+ {
+ crate::PartitionIoVerdict::Ready
+ } else {
+ self.request_superblock_intent(SuperblockIntent::Collapse {
+ frontier: self.advanced_frontier(0),
+ reserved,
+ })
+ };
+ if verdict == crate::PartitionIoVerdict::Pending {
+ return crate::PartitionIoStep::Pending;
+ }
+ if verdict == crate::PartitionIoVerdict::Failed {
+ warn!(
+ namespace_raw = self.namespace().inner(),
+ "could not collapse the offset reservation on shutdown; the restart will retain its gap"
+ );
+ }
+ self.maintenance.shutdown = Some(PartitionShutdownPhase::Complete);
+ crate::PartitionIoStep::Progress
}
- if header.op < current_op {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "primary received prepare below sequencer; applying idempotently",
- )
- .with_operation(header.operation)
- .with_op(header.op)
- .with_reason("duplicate delivery"),
+ _ => crate::PartitionIoStep::Pending,
+ }
+ }
+
+ /// Publishes what the WAL accepted, because recovery keeps only the segment
+ /// bytes that the durable WAL frontier covers. `None` while the drain runs.
+ fn drain_wal_for_shutdown(&mut self) -> Option<std::io::Result<()>> {
+ let Some(persistence) = &self.persistence else {
+ return Some(Ok(()));
+ };
+ if self.maintenance.shutdown_drain.is_none() && persistence.segment_checkpoint().is_some() {
+ persistence.sync();
+ }
+ self.start_persistence();
+ let drain = self
+ .maintenance
+ .shutdown_drain
+ .get_or_insert_with(|| persistence.begin_drain());
+ match persistence.observe_drain(drain) {
+ Ok(false) => None,
+ Ok(true) => Some(Ok(())),
+ Err(error) => Some(Err(error)),
+ }
+ }
+
+ // Completion relies on shutdown rejecting new and repaired journal entries.
+ fn shutdown_flush_target(&self) -> u64 {
+ self.consensus
+ .commit_max()
+ .min(self.log.journal().inner.last_op().unwrap_or(0))
+ }
+
+ async fn resume_pending_prepare_ack(&self) -> bool {
+ if self.consensus.needs_superblock_persist() {
+ return false;
+ }
+ let header = self
+ .pending_persisted_acks
+ .borrow()
+ .first_key_value()
+ .map(|(_, header)| *header);
+ let Some(header) = header else {
+ return false;
+ };
+ if self
+ .log
+ .journal()
+ .inner
+ .repair_header(header.op)
+ .is_none_or(|stored| stored.checksum != header.checksum)
+ {
+ self.pending_persisted_acks.borrow_mut().remove(&header.op);
+ return true;
+ }
+ if self
+ .persistence
+ .as_ref()
+ .is_some_and(|persistence| !persistence.is_durable(&header))
+ {
+ return false;
+ }
+ if !self.send_prepare_ok(&header).await {
+ return false;
+ }
+ self.pending_persisted_acks.borrow_mut().remove(&header.op);
+ true
+ }
+
+ fn queue_committed_replies(&mut self) {
+ if let Some(committed) = self.publish_pending_commit() {
+ self.io_promotions += committed.len();
+ self.io_replies = committed.into();
+ }
+ }
+
+ pub(crate) fn invalidate_workerless_offset_files(&self) {
+ self.workerless_offset_files.clear();
+ }
+
+ /// The commit fault that fenced this partition, if one has.
+ #[must_use]
+ pub const fn fatal(&self) -> Option<&FatalCommit> {
+ self.fatal.as_ref()
+ }
+
+ /// Consecutive superblock write failures for this group, for the shard's
+ /// wedge fail-stop. A partition that cannot record its state withholds every
+ /// view-scoped send and refuses every append, so past some window it is
+ /// serving nothing and a supervisor should be handling it instead.
+ #[must_use]
+ pub const fn superblock_write_failures(&self) -> u64 {
+ self.superblock_write_failures.get()
+ }
+
+ /// Fence this partition after the shutdown flush failed to persist its
+ /// committed journal prefix: that data is cluster-committed and now lives
+ /// only in this process's memory, so the shard must not report a clean
+ /// exit over it. A fault the commit path already recorded is kept, since
+ /// it names the op that first diverged; `commit_min` here only bounds
+ /// where the unpersisted prefix ends.
+ pub fn fence_flush_failure(&mut self) {
+ if self.fatal.is_none() {
+ self.fatal = Some(FatalCommit {
+ namespace_raw: self.namespace().inner(),
+ op: self.consensus.commit_min(),
+ operation: Some(Operation::SendMessages),
+ });
+ }
+ }
+
+ pub(crate) const fn fence_install_failure(&mut self, op: u64) {
+ if self.fatal.is_none() {
+ self.fatal = Some(FatalCommit {
+ namespace_raw: self.consensus.group(),
+ op,
+ operation: Some(Operation::SendMessages),
+ });
+ }
+ }
+
+ fn partition_dir(&self) -> Option<String> {
+ if self.partition_dir.is_some() {
+ return self.partition_dir.clone();
+ }
+ // Writer-derived fallback for partitions built without
+ // `set_partition_dir`. Unreliable mid-rotation: sealed segments
+ // drop their writer, so prefer the stored path above.
+ self.log
+ .messages_writers()
+ .iter()
+ .rev()
+ .flatten()
+ .next()
+ .and_then(|writer| {
+ std::path::Path::new(&writer.path())
+ .parent()
+ .map(|dir| dir.to_string_lossy().into_owned())
+ })
+ }
+
+ /// [`Self::partition_dir`] upgraded with the reason a dir is absent, so a
+ /// disk poll can tell file-less (simulated) storage from a live partition
+ /// whose dir is transiently unresolvable mid-rotation. Storage readers,
+ /// unlike writers, survive segment sealing, so any present reader or
+ /// writer proves file-backed data exists behind the missing dir.
+ fn partition_dir_resolution(&self) -> PartitionDirResolution {
+ if let Some(dir) = self.partition_dir() {
+ return PartitionDirResolution::Resolved(dir);
+ }
+ let file_backed =
+ self.log.storages().iter().any(|storage| {
+ storage.messages_reader.is_some() || storage.messages_size.is_some()
+ });
+ if file_backed {
+ PartitionDirResolution::Unresolvable
+ } else {
+ PartitionDirResolution::NoFiles
+ }
+ }
+
+ fn has_persisted_segment_bytes(&self) -> bool {
+ self.log
+ .segments()
+ .iter()
+ .any(|segment| segment.size.as_bytes_u64() > 0)
+ }
+
+ /// Mean encoded bytes per committed message, including its share of the
+ /// batch headers, or `None` while the partition has committed nothing.
+ ///
+ /// Both counters are relaxed loads that retention also decrements, so this
+ /// is a hint and nothing reads it as a bound. Its one consumer sizes the
+ /// first read of a disk poll, where being wrong costs an extra read.
+ fn mean_encoded_message_size(&self) -> Option<u32> {
+ let messages = self.stats.messages_count_inconsistent();
+ let bytes = self.stats.size_bytes_inconsistent();
+ (messages > 0).then(|| u32::try_from(bytes / messages).unwrap_or(u32::MAX))
+ }
+
+ /// Widest observed committed batch, for the disk walk's chunk floor.
+ /// Recovered history starts unknown; each walk learns its own batch floor
+ /// when an incomplete batch requires an exact reread.
+ ///
+ /// A high-water, never lowered: retention cannot make an older batch
+ /// narrower, and the read path clamps it to the chunk ceiling anyway, so
+ /// the worst a stale value costs is the fixed-size read polls did before
+ /// they were sized at all.
+ const fn widest_committed_batch(&self) -> u64 {
+ self.widest_batch_bytes.get()
+ }
+
+ /// Starting `(segment index, byte position)` for a disk poll, resolved
+ /// via each segment's sparse index cache. An index miss starts at the
+ /// segment's first byte (the walk filters precisely).
+ fn disk_poll_start(&self, query: &MessageLookup) -> (usize, u64, Option<u64>) {
+ let segments = self.log.segments();
+ match query {
+ MessageLookup::Offset { offset, .. } => {
+ let segment_index = segments
+ .iter()
+ .rposition(|segment| segment.start_offset <= *offset)
+ .unwrap_or(0);
+ let entry = self
+ .log
+ .segment_indexes(segment_index)
+ .and_then(|cache| cache.offset_lower_bound(*offset));
+ let position = entry.map_or(0, |index| index.position);
+ let entry_offset = entry.map(|index| index.offset).or_else(|| {
+ segments
+ .get(segment_index)
+ .map(|segment| segment.start_offset)
+ });
+ (segment_index, position, entry_offset)
+ }
+ MessageLookup::Timestamp { timestamp, .. } => {
+ // Resolve the starting SEGMENT from segment metadata, not from
+ // the per-segment index caches: sealed segments drop their
+ // cache at rotation, and a cache miss must not read as "the
+ // timestamp is not in this segment" (skipping a sealed segment
+ // loses its messages). Timestamps are monotone across
+ // segments, so the first segment whose max timestamp reaches
+ // the query is the correct start; the walk filters precisely,
+ // so an early start is safe.
+ let segment_index = segments
+ .iter()
+ .position(|segment| segment.max_timestamp >= *timestamp)
+ .unwrap_or_else(|| segments.len().saturating_sub(1));
+ let position = self
+ .log
+ .segment_indexes(segment_index)
+ .and_then(|cache| cache.timestamp_lower_bound(*timestamp))
+ .map_or(0, |index| index.position);
+ (segment_index, position, None)
+ }
+ }
+ }
+
+ /// Project a client request into a prepare.
+ ///
+ /// A replay of a committed `(client, request)` is absorbed by this group's
+ /// dedup slice; anything above the watermark projects into a prepare.
+ /// Session lifecycle + eviction live on the metadata plane.
+ ///
+ /// `reply` is the in-process channel a `PartitionSubmit` carried in. When
+ /// present the committed reply fires on it instead of going to the bus:
+ /// the connection-owning shard writes it to the socket it holds, because
+ /// `header.client` is the VSR consensus id and carries no home-shard
+ /// routing. `None` keeps the bus path (auto-commit ops, tests).
+ ///
+ /// # Panics
+ /// Panics if called when this partition's consensus instance is not the
+ /// primary, is not in normal status, or is currently syncing.
+ #[allow(clippy::future_not_send, clippy::too_many_lines)]
+ #[allow(clippy::too_many_lines)]
+ pub async fn on_request(
+ &mut self,
+ message: Message<RoutedRequestHeader>,
+ reply: Option<consensus::Sender<Message<ReplyHeader>>>,
+ ) {
+ // Taken by whichever arm answers: the deny paths, the NoAck fast path,
+ // or the pipeline entry that fires it at commit. Exactly one runs.
+ let mut reply = reply;
+ self.resynchronize_consumer_offset_reservations();
+ let namespace = IggyNamespace::from_raw(message.header().group);
+ let client_id = message.header().client;
+ let request = message.header().request;
+
+ let disposition = {
+ let consensus = self.consensus();
+ emit_sim_event(
+ SimEventKind::ClientRequestReceived,
+ &RequestLogEvent {
+ replica: ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ client_id,
+ request_id: request,
+ operation: message.header().operation,
+ },
+ );
+
+ let message = if message.header().operation == Operation::SendMessages {
+ // Skip the batch-checksum pass: on the partition ingest path
+ // nothing reads it before `stamp_prepare_for_persistence`
+ // recomputes it over the stamped header. An already-canonical
+ // batch (the plane's pre-encrypt convert output) returns early
+ // inside the convert, so Skip only affects the wire-form
+ // admission, whose output goes straight to project/stamp.
+ match convert_request_message(namespace, message, ChecksumMode::Skip) {
+ Ok(message) => message,
+ Err(error) => {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ "failed to convert send_messages request",
+ )
+ .with_operation(Operation::SendMessages)
+ .with_error(error.to_string()),
+ );
+ return;
+ }
+ }
+ } else {
+ message
+ };
+
+ // Recovered commits need receipt restoration; live pipeline entries
+ // already carry the pending identity and immutable capacity.
+ let committed_history_pending = consensus.commit_min() < consensus.commit_max()
+ && consensus.with_pipeline(|pipeline| {
+ pipeline.entry_by_op(consensus.commit_min() + 1).is_none()
+ || pipeline.entry_by_op(consensus.commit_max()).is_none()
+ });
+ // State-dependent admission belongs to the primary. A backup can
+ // lag the committed offset table or message frontier and would
+ // otherwise turn a routing artifact into a terminal 404 or 400.
+ // Reject it first with the only response that proves the request
+ // was never admitted, so the caller may safely retry elsewhere.
+ if self.fatal.is_some()
+ || self.maintenance.shutdown.is_some()
+ || self.io_active.is_retiring()
+ || self.transition.is_some()
+ || self.materialization_missing
+ || consensus.is_follower()
+ || !consensus.is_normal()
+ || consensus.is_transferring()
+ || committed_history_pending
+ {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ if committed_history_pending {
+ "rejecting client request while committed history is pending"
+ } else {
+ "rejecting client request on unavailable partition replica"
+ },
+ )
+ .with_operation(message.header().operation),
+ );
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ IggyError::TransientNotAccepted.as_code(),
+ "non-primary transient reply send failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+
+ let frame_bytes = message.as_slice().len();
+ if !is_auto_commit_client(client_id)
+ && message.header().metadata_watermark < self.required_metadata_frontier
+ {
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ IggyError::TransientNotAccepted.as_code(),
+ "partition requires newer metadata",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+ if self.persistence.is_some()
+ && frame_bytes > journal::partition_journal::PREPARE_BYTES_MAX
+ {
+ let error = IggyError::InvalidMessagesSize(
+ u32::try_from(frame_bytes).unwrap_or(u32::MAX),
+ u32::try_from(journal::partition_journal::PREPARE_BYTES_MAX)
+ .expect("prepare limit fits wire size"),
+ );
+ warn!(%error, namespace_raw = self.namespace().inner(), "persisted topic rejected oversized prepare");
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ error.as_code(),
+ "oversized prepare rejection failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+
+ // Retirement and the retiring identity must never occupy the same
+ // pipeline. Otherwise a later commit could recreate its receipt.
+ let retirement_conflict = if message.header().operation == Operation::RetireSession {
+ let Ok(retirement) = RetireSessionsRequest::decode_from(message.body()) else {
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ IggyError::InvalidCommand.as_code(),
+ "invalid retirement reply failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ };
+ retirement.identities.iter().any(|identity| {
+ consensus.pending_request(identity.client_id).is_some()
+ || self.pending_session_retirement(identity.client_id, identity.session)
+ })
+ } else {
+ !is_auto_commit_client(client_id)
+ && self.pending_session_retirement(client_id, message.header().session)
+ };
+ if retirement_conflict {
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ IggyError::TransientNotAccepted.as_code(),
+ "session retirement admission reply failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+
+ // Parse once for both the delete-existence check and AckLevel dispatch.
+ let consumer_offset = match message.header().operation {
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset => {
+ match Self::parse_consumer_offset_request(message.header().operation, &message)
+ {
+ Ok(parsed) => Some(parsed),
+ Err(error) => {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(
+ consensus,
+ PlaneKind::Partitions,
+ ),
+ "failed to parse consumer offset request",
+ )
+ .with_operation(message.header().operation)
+ .with_error(error.to_string()),
+ );
+ return;
+ }
+ }
+ }
+ _ => None,
+ };
+
+ // Dedup BEFORE the admission checks below: a replay of an
+ // already-committed delete must answer the success its original
+ // earned, not the typed 404 the existence check would raise now
+ // that the offset is gone.
+ //
+ // One unresolved mutation per session/group protects the original
+ // receipt until the caller resolves it. An exact pending retry is
+ // still uncertain; a different request waits for that original
+ // instead of displacing its receipt.
+ //
+ // HTTP handlers and pipelining SDKs can deliver a client's ids out of order:
+ // a write refused transiently here is replayed after its
+ // successors committed. The slice therefore keeps a committed-id
+ // window under the watermark (`consensus::COMMITTED_WINDOW_BITS`)
+ // and admits an unmarked id inside it instead of absorbing it.
+ if !is_auto_commit_client(client_id) {
+ if let Some((pending_session, pending_request, pending_operation)) =
+ consensus.pending_request(client_id)
+ {
+ let error = if pending_request != request {
+ IggyError::TransientNotAccepted
+ } else if pending_session != message.header().session
+ || pending_operation != message.header().operation
+ {
+ IggyError::InvalidCommand
+ } else {
+ IggyError::TransientNotCommitted
+ };
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ error.as_code(),
+ "in-flight dedup transient reply send failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+ match self.dedup.check_partition_request(
+ client_id,
+ message.header().user_id,
+ message.header().session,
+ request,
+ message.header().operation,
+ ) {
+ consensus::client_table::RequestStatus::New => {}
+ consensus::client_table::RequestStatus::Duplicate(cached) => {
+ Self::deliver_reply_or_log(
+ &self.consensus,
+ message.header(),
+ cached,
+ reply.take(),
+ "duplicate reply send failed",
+ )
+ .await;
+ return;
+ }
+ status => {
+ let error = match status {
+ consensus::client_table::RequestStatus::AlreadyApplied { .. } => {
+ emit_partition_diag(
+ tracing::Level::DEBUG,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(
+ consensus,
+ PlaneKind::Partitions,
+ ),
+ "rejecting client request aged out of the dedup window",
+ )
+ .with_operation(message.header().operation)
+ .with_client(client_id)
+ .with_request(request),
+ );
+ IggyError::RequestTooOld
+ }
+ consensus::client_table::RequestStatus::OperationMismatch {
+ ..
+ } => IggyError::InvalidCommand,
+ _ => IggyError::Unauthenticated,
+ };
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ error.as_code(),
+ "dedup rejection reply failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+ }
+ if !consensus.has_retry_capacity(&self.dedup, client_id) {
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ IggyError::TransientNotAccepted.as_code(),
+ "retry protection capacity reached",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+ }
+
+ if self
+ .persistence
+ .as_ref()
+ .is_some_and(|persistence| !persistence.has_capacity(message.as_slice().len()))
+ {
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ IggyError::TransientNotAccepted.as_code(),
+ "partition WAL backpressure reply failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+
+ if matches!(message.header().operation, Operation::DeleteConsumerOffset)
+ && !self.has_unsettled_local_requests()
+ && let Some((kind, consumer_id, _, _)) = consumer_offset
+ && let Err(error) = self.ensure_consumer_offset_exists(kind, consumer_id)
+ {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ "rejecting delete_consumer_offset for missing offset",
+ )
+ .with_operation(message.header().operation)
+ .with_error(error.to_string()),
+ );
+ // Deny on the primary before the op enters the pipeline: nothing
+ // replicates, so backups never see the rejected delete, and the
+ // client gets a typed failure instead of waiting out its reply
+ // timeout. The code rides `ReplyHeader.status` (not the result
+ // body): the HTTP listener's `classify_partition_reply` reads the
+ // status field to render the typed 404.
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ error.as_code(),
+ "delete_consumer_offset deny reply send failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+
+ // Reject an out-of-range consumer-offset store at admission,
+ // mirroring the legacy `validate_partition_offset`: an empty
+ // partition accepts no offset, and a stored offset may not run ahead
+ // of the committed offset. Done here so the doomed op is never
+ // replicated. Like the delete-offset deny above, the typed
+ // `InvalidOffset` rides `ReplyHeader.status` (op=0, empty body): the
+ // status-only `classify_partition_reply` would misread a result-body
+ // code on this committed-shaped frame (op=commit_max) as success.
+ if matches!(message.header().operation, Operation::StoreConsumerOffset)
+ && !self.has_unsettled_local_requests()
+ && let Some((kind, _, Some(requested_offset), _)) = consumer_offset
+ && let Some(error) = self.store_offset_range_error(kind, requested_offset)
+ {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ "rejecting store_consumer_offset for out-of-range offset",
+ )
+ .with_operation(message.header().operation)
+ .with_error(error.to_string()),
+ );
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ error.as_code(),
+ "store_consumer_offset deny reply send failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+
+ // The node-local fast path is safe only for a single-replica
+ // partition. Every mutation in a replicated partition enters VSR
+ // regardless of the acknowledgement byte.
+ if let Some((kind, consumer_id, offset, AckLevel::NoAck)) = consumer_offset
+ && consensus.replica_count() == 1
+ && matches!(
+ message.header().operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset,
+ )
+ {
+ Disposition::NoAck {
+ request_header: Box::new(*message.header()),
+ kind,
+ consumer_id,
+ offset,
+ }
+ } else {
+ let order = match self.next_local_request_order() {
+ Ok(order) => order,
+ Err(error) => {
+ Self::send_partition_deny_or_log(
+ consensus,
+ message.header(),
+ error.as_code(),
+ "local request order exhausted reply failed",
+ reply.take(),
+ )
+ .await;
+ return;
+ }
+ };
+ // Two-queue: prepare slot -> project+replicate; prepare full +
+ // request room -> buffer; both full -> drop+warn (client retries
+ // via read-timeout).
+ let queued = consensus.pipeline_is_full()
+ || consensus.with_pipeline(|pipeline| !pipeline.request_queue_is_empty())
+ || !self.no_ack.is_empty()
+ || (consumer_offset.is_some() && self.has_unsettled_local_requests());
+ let reservation = if queued {
+ crate::PartitionIoVerdict::Pending
+ } else {
+ self.admit_reserved_send(&message).await
+ };
+ if reservation == crate::PartitionIoVerdict::Pending {
+ let entry = consensus::RequestEntry::with_sender(message, reply.take())
+ .with_consumer_offset_history(
+ consumer_offset.is_some().then_some(self.poll_history),
+ );
+ let push_result = self.queue_local_request(entry, order);
+ if let Err(mut refused) = push_result {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ "on_request: prepare and request queues both full, dropping",
+ ),
+ );
+ // The request provably never entered either queue, so a
+ // waiter can be told so instead of waiting out its
+ // timeout.
+ let waiter = refused.take_reply_sender();
+ Self::send_partition_deny_or_log(
+ consensus,
+ refused.message.header(),
+ IggyError::TransientNotAccepted.as_code(),
+ "queues-full transient reply send failed",
+ waiter,
+ )
+ .await;
+ }
+ return;
+ }
+
+ if reservation == crate::PartitionIoVerdict::Failed {
+ self.deny_unreserved_send(message.header(), reply.take())
+ .await;
+ return;
+ }
+ if let Some((kind, consumer_id, Some(_), _)) = consumer_offset
+ && !self
+ .admit_consumer_offset_key(message.header(), kind, consumer_id, &mut reply)
+ .await
+ {
+ return;
+ }
+
+ let prepare = self.pipeline_local_request(message, reply.take(), Some(order));
+ Disposition::Replicate(prepare)
+ }
+ };
+
+ match disposition {
+ Disposition::Replicate(prepare) => self.on_replicate(prepare).await,
+ Disposition::NoAck {
+ request_header,
+ kind,
+ consumer_id,
+ offset,
+ } => {
+ self.apply_consumer_offset_no_ack(
+ request_header,
+ kind,
+ consumer_id,
+ offset,
+ reply.take(),
+ )
+ .await;
+ }
+ }
+ }
+
+ /// Promote requests in local admission order after their predecessors settle.
+ /// A reservation wait retains the original queue head and its reply sender.
+ #[allow(clippy::future_not_send, clippy::too_many_lines)]
+ pub async fn drain_request_queue_into_prepares(&mut self, slots_freed: usize) {
+ if self.refuses_prepares() {
+ self.notify_io();
+ return;
+ }
+
+ self.resynchronize_consumer_offset_reservations();
+ let mut promoted = 0;
+ let mut consecutive_denials = 0usize;
+ while promoted < slots_freed && !self.consensus.pipeline_is_full() {
+ if !self.consensus.is_primary()
+ || !self.consensus.is_normal()
+ || self.consensus.is_transferring()
+ || self.fatal.is_some()
+ {
+ break;
+ }
+ let head = self.consensus.with_pipeline(|pipeline| {
+ pipeline.request_head().map(|request| {
+ (
+ *request.message.header(),
+ request.local_order,
+ self.request_mint_ceiling(&request.message),
+ )
+ })
+ });
+ let Some((header, order, ceiling)) = head else {
+ break;
+ };
+ if self.queued_request_is_blocked(&header, order) {
+ break;
+ }
+ let reservation_ready = if let Some(ceiling) = ceiling {
+ self.reserve_offsets_through_retryable(ceiling).await
+ } else {
+ crate::PartitionIoVerdict::Ready
+ };
+ if reservation_ready == crate::PartitionIoVerdict::Pending {
+ break;
+ }
+ let unchanged = self.consensus.with_pipeline(|pipeline| {
+ pipeline.request_head().is_some_and(|request| {
+ request.local_order == order
+ && bytemuck::bytes_of(request.message.header())
+ == bytemuck::bytes_of(&header)
+ })
+ });
+ if !unchanged {
+ break;
+ }
+ let Some(mut request) = self.consensus.pop_queued_request() else {
+ break;
+ };
+ let context = request.take_auto_commit();
+ if context.as_ref().is_some_and(|context| {
+ context.history != self.poll_history
+ || !self
+ .consumer_offset_capacity_for(context.reservation.kind())
+ .owns(&context.reservation)
+ }) {
+ if Self::record_promotion_denial(&mut consecutive_denials) {
+ break;
+ }
+ continue;
+ }
+ let mut reply_sender = request.take_reply_sender();
+ if !is_auto_commit_client(header.client)
+ && header.metadata_watermark < self.required_metadata_frontier
+ {
+ if self
+ .deny_queued_request(
+ &header,
+ IggyError::TransientNotAccepted.as_code(),
+ "queued request metadata frontier is behind retirement",
+ reply_sender.take(),
+ &mut consecutive_denials,
+ )
+ .await
+ {
+ break;
+ }
+ continue;
+ }
+ if let Some(error) = self.stale_offset_history_refusal(&request) {
+ // An old offset can fit the replacement history, and an old
+ // delete can erase a replacement checkpoint. Refuse before
+ // projection with a terminal error so clients do not replay it.
+ if self
+ .deny_queued_request(
+ &header,
+ error.as_code(),
+ "queued offset history deny reply send failed",
+ reply_sender.take(),
+ &mut consecutive_denials,
+ )
+ .await
+ {
+ break;
+ }
+ continue;
+ }
+ if reservation_ready == crate::PartitionIoVerdict::Failed {
+ self.deny_unreserved_send(&header, reply_sender.take())
+ .await;
+ break;
+ }
+ if matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) {
+ let parsed =
+ Self::parse_consumer_offset_request(header.operation, &request.message);
+ let (kind, consumer_id, offset, _) = match parsed {
+ Ok(parsed) => parsed,
+ Err(error) => {
+ if self
+ .deny_queued_request(
+ &header,
+ error.as_code(),
+ "queued consumer offset parse reply failed",
+ reply_sender.take(),
+ &mut consecutive_denials,
+ )
+ .await
+ {
+ break;
+ }
+ continue;
+ }
+ };
+ if is_auto_commit_client(header.client)
+ && let Some(offset) = offset
+ && self
+ .durable_consumer_offsets
+ .covers(kind, consumer_id, offset)
+ {
+ if let Some(sender) = reply_sender.take() {
+ let reply = build_reply_from_request(
+ &self.consensus,
+ &header,
+ committed_reply_body(header.operation),
+ );
+ let _ = sender.send(reply);
+ }
+ continue;
+ }
+ let validation = offset.map_or_else(
+ || self.ensure_consumer_offset_exists(kind, consumer_id),
+ |offset| {
+ self.store_offset_range_error(kind, offset)
+ .map_or(Ok(()), Err)
+ },
+ );
+ if let Err(error) = validation {
+ if self
+ .deny_queued_request(
+ &header,
+ error.as_code(),
+ "queued consumer offset validation reply failed",
+ reply_sender.take(),
+ &mut consecutive_denials,
+ )
+ .await
+ {
+ break;
+ }
+ continue;
+ }
+ if offset.is_some()
+ && !self
+ .admit_consumer_offset_key(&header, kind, consumer_id, &mut reply_sender)
+ .await
+ {
+ if Self::record_promotion_denial(&mut consecutive_denials) {
+ break;
+ }
+ continue;
+ }
+ }
+ consecutive_denials = 0;
+ let prepare = self.pipeline_local_request(request.message, reply_sender, order);
+ promoted += 1;
+ self.on_replicate(prepare).await;
+ drop(context);
+ }
+ self.drain_no_ack().await;
+ }
+
+ #[must_use]
+ pub fn queued_requests_ready(&self) -> bool {
+ self.no_ack_ready()
+ || (self.fatal.is_none()
+ && self.consensus.is_primary()
+ && self.consensus.is_normal()
+ && !self.consensus.is_transferring()
+ && !self.consensus.pipeline_is_full()
+ && self.consensus.with_pipeline(|pipeline| {
+ pipeline.request_head().is_some_and(|request| {
+ !self.queued_request_is_blocked(
+ request.message.header(),
+ request.local_order,
+ )
+ })
+ }))
+ }
+
+ fn queued_request_is_blocked(
+ &self,
+ header: &RoutedRequestHeader,
+ order: Option<LocalRequestOrder>,
+ ) -> bool {
+ self.no_ack
+ .front()
+ .is_some_and(|head| order.is_some_and(|order| order > head.local_order))
+ || (matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) && (self.consensus.pipeline_len() > 0 || self.pending_commit.is_some()))
+ }
+
+ fn no_ack_ready(&self) -> bool {
+ self.no_ack.front().is_some_and(|head| match head.phase {
+ NoAckPhase::Validate => {
+ head.poll_history != self.poll_history
+ || self.fatal.is_some()
+ || !self.consensus.is_primary()
+ || !self.consensus.is_normal()
+ || self.consensus.is_transferring()
+ || !self
+ .consumer_offset_capacity_for(head.pending.kind)
+ .owns(&head.reservation)
+ || (self.pending_commit.is_none()
+ && (self.maintenance.shutdown.is_some()
+ || !self.consensus.with_pipeline(|pipeline| {
+ pipeline.has_request_before(head.local_order)
+ })))
+ }
+ NoAckPhase::OffsetReady(_) | NoAckPhase::Directory | NoAckPhase::Reply(_) => true,
+ NoAckPhase::OffsetSubmitted(_) | NoAckPhase::DirectorySubmitted => false,
+ })
+ }
+
+ /// Shutdown promotes nothing, so answer every queued request now. None
+ /// holds an op yet, so its client can safely retry it, unless it is an
+ /// offset mutation that a history reset made stale.
+ async fn deny_queued_requests_at_shutdown(&self) {
+ while let Some(mut request) = self.consensus.pop_queued_request() {
+ let error = self
+ .stale_offset_history_refusal(&request)
+ .unwrap_or(IggyError::TransientNotAccepted);
+ let waiter = request.take_reply_sender();
+ Self::send_partition_deny_or_log(
+ &self.consensus,
+ request.message.header(),
+ error.as_code(),
+ "shutdown queued request reply failed",
+ waiter,
+ )
+ .await;
+ }
+ }
+
+ /// Resume a bounded promotion turn without waiting for another commit.
+ pub async fn resume_queued_requests(&mut self) {
+ if self.queued_requests_ready() {
+ self.drain_request_queue_into_prepares(1).await;
+ }
+ }
+
+ async fn deny_queued_request(
+ &self,
+ header: &RoutedRequestHeader,
+ status: u32,
+ send_fail_label: &'static str,
+ waiter: Option<consensus::Sender<Message<ReplyHeader>>>,
+ consecutive_denials: &mut usize,
+ ) -> bool {
+ Self::send_partition_deny_or_log(self.consensus(), header, status, send_fail_label, waiter)
+ .await;
+ Self::record_promotion_denial(consecutive_denials)
+ }
+
+ const fn record_promotion_denial(consecutive_denials: &mut usize) -> bool {
+ *consecutive_denials += 1;
+ *consecutive_denials >= PROMOTION_DENIALS_MAX
+ }
+
+ /// The terminal refusal for a queued offset mutation admitted before a
+ /// reset of the poll history, or `None` while its history is current.
+ fn stale_offset_history_refusal(&self, request: &consensus::RequestEntry) -> Option<IggyError> {
+ request
+ .consumer_offset_history()
+ .is_some_and(|history| history != self.poll_history)
+ .then(|| {
+ Self::parse_consumer_offset_request(
+ request.message.header().operation,
+ &request.message,
+ )
+ .map_or(
+ IggyError::InvalidCommand,
+ |(kind, consumer_id, offset, _)| {
+ PendingConsumerOffsetCommit::explicit(kind, consumer_id, offset)
+ .history_reset_refusal()
+ },
+ )
+ })
+ }
+
+ /// # Panics
+ /// Panics on a primary when a prepare's op is ahead of the local
+ /// sequencer: journaling it would make the next op assignment collide,
+ /// which is unrecoverable in place.
+ #[allow(clippy::future_not_send, clippy::too_many_lines)]
+ pub async fn on_replicate(&mut self, message: Message<PrepareHeader>) {
+ if self.refuses_prepares() {
+ self.notify_io();
+ return;
+ }
+
+ self.resynchronize_consumer_offset_reservations();
+ let header = *message.header();
+ // Same reason as the metadata plane: `checksum` is compared as an opaque token
+ // downstream, so a corrupted frame passes whenever its flipped value satisfies
+ // those comparisons.
+ if let Err(reason) = verify_prepare_integrity(&header, message.as_slice()) {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions),
+ "discarding prepare that failed its own integrity check",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op)
+ .with_reason(reason),
+ );
+ return;
+ }
+ let current_op = {
+ let consensus = self.consensus();
+ match replicate_preflight(consensus, &header) {
+ Ok(current_op) => current_op,
+ Err(reason) => {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ "ignoring prepare during replicate preflight",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op)
+ .with_reason(reason.as_str()),
+ );
+ return;
+ }
+ }
+ };
+ #[allow(clippy::cast_possible_truncation)]
+ let fenced_by_commit = fence_old_prepare_by_commit(self.consensus(), &header);
+ if fenced_by_commit {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "received old prepare (<= commit_min), skipping replication",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op),
+ );
+ // Fenced by commit_min: we've already executed this op, the
+ // whole chain has it committed. Safe to drop entirely.
+ return;
+ }
+
+ let journal_holds_op = self.log.journal().inner.holds_op(header.op);
+ if journal_holds_op {
+ // Retransmit after downstream flap: durable here but commit
+ // hasn't caught up. Re-forward + re-ACK so primary's view of
+ // us is consistent. Both downstream and primary are idempotent
+ // on duplicate (replica, op).
+ emit_partition_diag(
+ tracing::Level::DEBUG,
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "journal already holds prepare, re-forwarding + re-acking",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op),
+ );
+ let Some(journaled) = self.log.journal().inner.repair_entry(header.op) else {
+ emit_partition_diag(
+ tracing::Level::ERROR,
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "journal header exists without matching prepare bytes",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op),
+ );
+ return;
+ };
+ if !journaled_prepare_matches_retransmit(&journaled, &message) {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "rejecting retransmitted prepare that differs from the journaled entry",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op),
+ );
+ return;
+ }
+ let Some(frozen_for_forward) = restamp_prepare_view(journaled, header.view) else {
+ emit_partition_diag(
+ tracing::Level::ERROR,
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "failed to restamp journaled prepare for retransmission",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op),
+ );
+ return;
+ };
+ let prepare_to_persist = self
+ .persistence
+ .as_ref()
+ .map(|_| frozen_for_forward.clone());
+ let consensus = self.consensus();
+ if consensus.is_follower() && header.op > consensus.sequencer().current_sequence() {
+ consensus.sequencer().set_sequence(header.op);
+ consensus.set_last_prepare_checksum(header.checksum);
+ consensus.observe_prepare_timestamp(header.timestamp);
+ }
+ if let Err(error) =
+ replicate_frozen_to_next_in_chain(consensus, frozen_for_forward).await
+ {
+ let is_transport_error = error.is_transport();
+ emit_partition_diag(
+ if is_transport_error {
+ tracing::Level::WARN
+ } else {
+ tracing::Level::ERROR
+ },
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "failed to re-forward retransmitted prepare to next in chain",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op)
+ .with_error(error.to_string()),
+ );
+ if !is_transport_error {
+ return;
+ }
+ }
+ if let Some(prepare) = prepare_to_persist
+ && !self.submit_prepare_persistence(prepare, header.operation)
+ {
+ return;
+ }
+ self.send_prepare_ok(&header).await;
+ return;
+ }
+
+ // A header-only StartView can announce bodies this backup still lacks.
+ // Live same-view retransmits may fill its next verified slot; repair
+ // frames above commit still require the elected canonical headers.
+ let is_backup = self.consensus().is_follower();
+ if is_backup {
+ let fills_announced_gap = header.op <= current_op
+ && header.op > self.purge_floor_op
+ && !self.consensus().view_log_is_pending()
+ && self
+ .log
+ .journal()
+ .inner
+ .repaired_window_shape(self.consensus().commit_min(), header.op - 1)
+ .complete
+ && if header.op == 1 {
+ header.parent == 0
+ } else {
+ self.log
+ .journal()
+ .inner
+ .repair_header(header.op - 1)
+ .is_some_and(|previous| previous.checksum == header.parent)
+ };
+ if header.op != current_op + 1 && !fills_announced_gap {
+ // `sequence` is what separates the two shapes this line covers:
+ // a forward gap (op above the sequencer, the hole the repair
+ // driver closes) and a retransmit of an op this replica already
+ // sequenced. Without it they read identically.
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "dropping out-of-order prepare (gap)",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op)
+ .with_sequence(current_op),
+ );
+ self.prepare_gap_drops
+ .set(self.prepare_gap_drops.get().saturating_add(1));
+ return;
+ }
+ } else {
+ // Primary: `push_prepare_entry` pre-advanced the sequencer, so a
+ // locally-originated prepare always satisfies
+ // `header.op == current_op`. The two violation directions carry
+ // very different risk:
+ // - below the sequencer: a duplicate delivery (parked-frame
+ // redispatch, retransmit echo) of an op this primary already
+ // sequenced. Proceeding is safe only because the two gates above
+ // already returned for every copy this replica can still see:
+ // `fence_old_prepare_by_commit` drops the executed ops and
+ // `journal_holds_op` the resident ones, so reaching here means
+ // the journal lacks this op and has to be given it. Apply is not
+ // idempotent on its own for a produce: `append_messages`
+ // re-stamps from the local dirty counter and the journal's op
+ // index is last-write-wins, so appending an op the journal
+ // already holds would mint a second copy at fresh offsets and
+ // orphan the first. Log loudly for diagnosis.
+ // - above the sequencer: journaling an op the sequencer has not
+ // assigned yet means the next local assignment would collide
+ // with it. Unreachable today (view fences run first, one
+ // primary per view, the chain stops before the primary), so
+ // trip the invariant in debug; in release log loudly and drop
+ // rather than crash a library or corrupt op assignment.
+ if header.op > current_op {
+ debug_assert!(
+ header.op <= current_op,
+ "primary: prepare op {} ahead of sequencer {}; next op assignment would collide",
+ header.op,
+ current_op
+ );
+ emit_partition_diag(
+ tracing::Level::ERROR,
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "primary prepare ahead of sequencer; dropping to avoid op-assignment collision",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op),
+ );
+ return;
+ }
+ if header.op < current_op {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "primary received prepare below sequencer; applying idempotently",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op)
+ .with_reason("duplicate delivery"),
+ );
+ }
+ }
+ // Forward only after apply_replicated_operation journals the prepare.
+ // The journal and network share the frozen allocation, so the bytes
+ // retained for repair are exactly the bytes sent downstream.
+ let replicated_result = if is_backup && header.operation == Operation::SendMessages {
+ self.append_received_send_messages_to_journal(message).await
+ } else {
+ self.apply_replicated_operation(message).await
+ };
+ let frozen_for_forward = match replicated_result {
+ Ok(frozen) => frozen,
+ Err(error) => {
+ // A BACKUP refusing here is the design, not a fault: it rejects
+ // any prepare whose `base_offset` does not continue its own
+ // counter, which is exactly what a dropped or out-of-order
+ // prepare leaves, and withholding `PrepareOk` is the fail-closed
+ // answer. The primary retransmits, journal repair fills the gap,
+ // and the group elects around the replica if it cannot catch up.
+ // Nothing here is owed an ack this replica already skipped.
+ //
+ // On the PRIMARY the same return is unrecoverable. The op is
+ // already in the pipeline with the sequencer advanced past it
+ // (`push_prepare_entry`), and this sits ahead of
+ // `send_prepare_ok`, so it never gets its ack and `commit_max`
+ // can never pass it: every later op journals fine and queues
+ // behind it forever. Nothing lifts that -- the prepare timeout
+ // only backs off, a solo group's retransmit target is itself,
+ // this plane has no `repair_primary_self_acks`, and a solo group
+ // never starts a view change. Clients get no reply at all, since
+ // replies are generated on commit, so they wait out their read
+ // timeout, and once the queues fill so does every send after.
+ //
+ // So fence there, the way a failed local commit of a
+ // cluster-committed op does: the shard picks `fatal` up on its
+ // next tick and takes the node down. A one-second superblock
+ // backoff must not cost a partition the rest of the process's
+ // life in the dark.
+ emit_partition_diag(
+ if is_backup {
+ tracing::Level::WARN
+ } else {
+ tracing::Level::ERROR
+ },
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ if is_backup {
+ "failed to apply replicated partition operation"
+ } else {
+ "failed to apply an operation this replica sequenced; \
+ fencing the partition and shutting down"
+ },
+ )
+ .with_operation(header.operation)
+ .with_op(header.op)
+ .with_error(error.to_string()),
+ );
+ if !is_backup && self.fatal.is_none() {
+ self.fatal = Some(FatalCommit {
+ namespace_raw: self.namespace().inner(),
+ op: header.op,
+ operation: Some(header.operation),
+ });
+ }
+ return;
+ }
+ };
+
+ let prepare_to_persist = self
+ .persistence
+ .as_ref()
+ .map(|_| frozen_for_forward.clone());
+ let consensus = self.consensus();
+ // Backup only: advance sequencer + checksum after journal append.
+ // Pre-advance on failing apply would leave consensus claiming op N
+ // while the journal has nothing. Retransmit of N would silently drop
+ // as is_old_prepare (header.op <= current_sequence). The primary does
+ // not re-set here because push_prepare_entry already advanced it. A
+ // sibling request pipelined during the apply await would otherwise be
+ // rewound to a stale op + parent, projecting a duplicate next.
+ if is_backup {
+ // Filling a lower slot must not rewind the announced head or parent.
+ if header.op >= current_op {
+ consensus.sequencer().set_sequence(header.op);
+ consensus.set_last_prepare_checksum(header.checksum);
+ }
+ consensus.observe_prepare_timestamp(header.timestamp);
+ }
+ if let Err(error) = replicate_frozen_to_next_in_chain(consensus, frozen_for_forward).await {
+ let is_transport_error = error.is_transport();
+ emit_partition_diag(
+ if is_transport_error {
+ tracing::Level::WARN
+ } else {
+ tracing::Level::ERROR
+ },
+ &PartitionDiagEvent::new(
+ self.diag_ctx(),
+ "failed to replicate prepare to next in chain",
+ )
+ .with_operation(header.operation)
+ .with_op(header.op)
+ .with_error(error.to_string()),
+ );
+ if !is_transport_error {
+ return;
+ }
+ }
+
+ {
+ let consensus = self.consensus();
+ emit_namespace_progress_event(
+ SimEventKind::NamespaceProgressUpdated,
+ &ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ header.op,
+ consensus.pipeline_len(),
+ );
+ }
+
+ if let Some(prepare) = prepare_to_persist
+ && !self.submit_prepare_persistence(prepare, header.operation)
+ {
+ return;
+ }
+ self.send_prepare_ok(&header).await;
+ }
+
+ #[allow(clippy::future_not_send)]
+ pub async fn on_ack(&mut self, message: Message<PrepareOkHeader>, config: &PartitionsConfig) {
+ if self.fatal.is_some() {
+ return;
+ }
+ self.resynchronize_consumer_offset_reservations();
+ let header = *message.header();
+ {
+ let consensus = self.consensus();
+ if let Err(reason) = ack_preflight(consensus) {
+ emit_partition_diag(
+ tracing::Level::WARN,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ "ignoring ack during preflight",
+ )
+ .with_op(header.op)
+ .with_reason(reason.as_str()),
+ );
+ return;
+ }
+
+ if !consensus.pipeline_holds_entry(header.op, header.prepare_checksum) {
+ emit_partition_diag(
+ tracing::Level::DEBUG,
+ &PartitionDiagEvent::new(
+ ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ "ack target prepare not in pipeline",
+ )
+ .with_op(header.op)
+ .with_prepare_checksum(header.prepare_checksum),
+ );
+ return;
+ }
+ }
+
+ if !ack_quorum_reached(self.consensus(), PlaneKind::Partitions, &header) {
+ return;
+ }
+ if self.commit_waits_for_checkpoint() {
+ return;
+ }
+
+ if self.pending_commit.is_some() {
+ self.handle_committed_entries(Vec::new(), config, true)
+ .await;
+ return;
+ }
+
+ let drained = self.select_persistable_commits(config);
+ if drained.is_empty() {
+ return;
+ }
+
+ self.handle_committed_entries(drained, config, true).await;
+ {
+ let consensus = self.consensus();
+ emit_namespace_progress_event(
+ SimEventKind::NamespaceProgressUpdated,
+ &ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ consensus.commit_min(),
+ consensus.pipeline_len(),
+ );
+ }
+ }
+
+ /// Apply the committed prefix this replica can reach, from the pipeline if
+ /// the primary populated one and from the journal otherwise.
+ ///
+ /// The journal half is bounded by [`COMMIT_WALK_OPS_MAX`], for EVERY caller
+ /// and not just the tick sweep: `on_commit`, `StartView` adoption, the
+ /// post-transfer tail and the post-repair walk all reach this with the same
+ /// resident backlog behind them, and all four run on the shard pump. The
+ /// pipeline half is left alone, being the primary's own in-flight window,
+ /// which the pipeline depth already caps.
+ ///
+ /// Nothing is lost by stopping early: the run is re-derived from
+ /// `commit_min` on the next call, and the sweep's walk predicate stays true
+ /// until the group drains, so the next tick resumes exactly here. The one
+ /// caller with no next tick is the shutdown drain, and it is covered twice
+ /// over: [`Self::flush_committed_messages`] persists the committed prefix
+ /// by bytes rather than by walk, and what stays un-applied sits above the
+ /// `commit_min` the superblock records, which is what makes the restarted
+ /// replica ask its peers for it.
+ #[allow(clippy::future_not_send)]
+ pub async fn commit_journal(&mut self, config: &PartitionsConfig) {
+ if self.transition.is_some() || self.io_active.is_retiring() {
+ self.notify_io();
+ return;
+ }
+
+ if self.fatal.is_some()
+ || self.materialization_missing
+ || self.commit_waits_for_checkpoint()
+ {
+ return;
+ }
+ self.resynchronize_consumer_offset_reservations();
+
+ if self.pending_commit.is_some() {
+ self.handle_committed_entries(Vec::new(), config, false)
+ .await;
+ return;
+ }
+
+ // The primary commits inline via `on_ack` (it drains its own pipeline).
+ // Backups never populate the pipeline - they journal replicated prepares
+ // in `apply_replicated_operation` - so the pipeline drain is empty for
+ // them. Fall back to the journal so backups durably persist committed
+ // data. `commit_messages` then flushes only the committed prefix and
+ // keeps the uncommitted tail journal-resident, so a later commit of that
+ // tail still finds its headers here (no wedge). Pipeline-first keeps a
+ // freshly promoted primary (rebuilt pipeline) draining there, avoiding a
+ // double-count against `advance_commit_min`.
+ let mut drained = self.select_persistable_commits(config);
+ let send_client_replies = !drained.is_empty() && self.consensus.is_primary();
+ if drained.is_empty() {
+ drained = self.collect_committable_from_journal(COMMIT_WALK_OPS_MAX, config);
+ }
+ if drained.is_empty() {
+ return;
+ }
+
+ self.handle_committed_entries(drained, config, send_client_replies)
+ .await;
+ {
+ let consensus = self.consensus();
+ emit_namespace_progress_event(
+ SimEventKind::NamespaceProgressUpdated,
+ &ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
+ consensus.commit_min(),
+ consensus.pipeline_len(),
+ );
+ }
+ }
+
+ fn select_persistable_commits(&self, config: &PartitionsConfig) -> Vec<PrepareHeader> {
+ if self.transition.is_some() || self.io_active.is_retiring() {
+ return Vec::new();
+ }
+
+ if self.pending_commit.is_some() {
+ return Vec::new();
+ }
+ let Some(first) = consensus::peek_committable_head(self.consensus()) else {
+ // A repairing backup has no pipeline, and its journal walk commits
+ // only what the WAL accepted. Shutdown keeps repair history out.
+ if self.maintenance.shutdown.is_none() {
+ self.persist_repaired_prefix();
+ }
+ return Vec::new();
+ };
+ self.persist_repaired_prefix();
+ let through = self.persistence.as_ref().map_or_else(
+ || self.consensus.commit_max(),
+ |persistence| self.consensus.commit_max().min(persistence.head()),
+ );
+ let materialize = self.should_persist_messages(config);
+ self.consensus.with_pipeline(|pipeline| {
+ let mut selected = Vec::new();
+ for op in first.op..=through {
+ let Some(entry) = pipeline.entry_by_op(op) else {
+ break;
+ };
+ if !self.prepare_body_is_ready(&entry.header, materialize) {
+ break;
+ }
+ if self.no_ack.front().is_some_and(|head| {
+ entry
+ .local_order
+ .is_some_and(|order| order > head.local_order)
+ }) {
+ break;
+ }
+ selected.push(entry.header);
+ }
+ selected
+ })
+ }
+
+ /// Under a dispatcher a Replicated commit runs ahead of the WAL writer while
+ /// [`PartitionPersistence::can_run_ahead`] allows it, and its flush takes
+ /// the written prefix later, see `flushable_through`. Without one the
+ /// commit flushes inline, so a due flush waits for the written body.
+ fn prepare_body_is_ready(&self, header: &PrepareHeader, materialize: bool) -> bool {
+ let dispatched = self.has_io_dispatcher();
+ header.operation != Operation::SendMessages
+ || header.op <= self.purge_floor_op
+ || (!self.durability().is_persisted() && !materialize && !dispatched)
+ || self
+ .persistence
+ .as_ref()
+ .filter(|persistence| persistence.segment_checkpoint().is_some())
+ .is_none_or(|persistence| {
+ if self.durability().is_persisted() {
+ persistence.is_durable(header)
+ } else {
+ persistence.is_written(header)
+ || (dispatched && persistence.can_run_ahead())
+ }
+ })
+ }
+
+ /// Committable entries (ops `commit_min+1 ..= commit_max`) read from the
+ /// journal, for a backup whose pipeline is empty. Stops at the first missing
+ /// op: a replication gap must not be skipped, or `advance_commit_min`'s
+ /// sequential contract breaks.
+ ///
+ /// Reads resident headers only. The commit walk caps flushing at its last
+ /// drained op, keeping later headers resident for the next bounded walk.
+ fn collect_committable_from_journal(
+ &self,
+ max_ops: usize,
+ config: &PartitionsConfig,
+ ) -> Vec<PrepareHeader> {
+ if self.transition.is_some() || self.io_active.is_retiring() {
+ return Vec::new();
+ }
+
+ let from_op = self.consensus.commit_min() + 1;
+ // Stop below the pipeline head. The drain above holds rather than pops
+ // when the head is not the op owed next; walking that op out of the
+ // journal instead advances `commit_min` past a still-resident entry, and
+ // `on_ack` then finds the drain empty on every later ack -- no reply is
+ // ever shipped and each stranded entry leaves its awaiter parked.
+ //
+ // Only a head at or above `from_op` lowers the ceiling: an absent head is
+ // a backup's empty pipeline, and a lower head is already stranded and must
+ // not freeze the walk on top of that.
+ let commit_max = self.persistence.as_ref().map_or_else(
+ || self.consensus.commit_max(),
+ |persistence| self.consensus.commit_max().min(persistence.head()),
+ );
+ let commit_max = self
+ .consensus
+ .pipeline_head_header()
+ .filter(|head| head.op >= from_op)
+ .map_or(commit_max, |head| commit_max.min(head.op - 1));
+ let materialize = self.should_persist_messages(config);
+ self.log
+ .journal()
+ .inner
+ .committed_headers_from(from_op, commit_max, max_ops)
+ .into_iter()
+ .take_while(|header| self.prepare_body_is_ready(header, materialize))
+ .collect()
+ }
+
+ async fn apply_replicated_operation(
+ &mut self,
+ message: Message<PrepareHeader>,
+ ) -> Result<Frozen<4096>, IggyError> {
+ let header = *message.header();
+ let replica_id = self.consensus.replica();
+ let namespace_raw = self.consensus.group();
+
+ match header.operation {
+ Operation::RetireSession => {
+ let retirement = RetireSessionsRequest::decode_from(message.body())
+ .map_err(|_| IggyError::InvalidCommand)?;
+ if retirement
+ .identities
+ .iter()
+ .any(|identity| is_auto_commit_client(identity.client_id))
+ {
+ return Err(IggyError::InvalidCommand);
+ }
+ let frozen = message.into_frozen();
+ self.journal_append(frozen.clone()).await?;
+ Ok(frozen)
+ }
+ Operation::SendMessages => {
+ let frozen = self.append_send_messages_to_journal(message).await?;
+ debug!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica = replica_id,
+ op = header.op,
+ namespace_raw,
+ operation = ?header.operation,
+ "replicated send_messages appended to partition journal"
+ );
+ Ok(frozen)
+ }
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset => {
+ // Replication semantics do not depend on the acknowledgement
+ // byte. Multi-replica `NoAck` requests enter this same path.
+ let (kind, consumer_id, offset, _ack) =
+ Self::parse_staged_consumer_offset_commit(header.operation, &message)?;
+ let write_lock = self.write_lock.clone();
+ let _guard = write_lock.lock().await;
+
+ // Journal the prepare before staging so
+ // `VsrAction::RetransmitPrepares` can read this op back
+ // on a view change. Without the journal entry, the
+ // `header_by_op` lookup in `on_replicate` would miss,
+ // the gap check would drop the retransmit, and the
+ // primary's pipeline would wedge indefinitely. Skip
+ // the `journal.info` accounting: it counts SendMessages
+ // batches for segment-commit thresholds, which do not
+ // apply to offset ops.
+ let frozen = message.into_frozen();
+ self.journal_append(frozen.clone()).await?;
+
+ match header.operation {
+ Operation::StoreConsumerOffset => {
+ self.stage_consumer_offset_upsert(
+ header.op,
+ kind,
+ consumer_id,
+ offset.expect("store_consumer_offset must include offset"),
+ is_auto_commit_client(header.client),
+ );
+ }
+ Operation::DeleteConsumerOffset => {
+ self.stage_consumer_offset_delete(header.op, kind, consumer_id);
+ }
+ _ => unreachable!(),
+ }
+
+ debug!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica = replica_id,
+ op = header.op,
+ namespace_raw,
+ operation = ?header.operation,
+ consumer_kind = ?kind,
+ consumer_id,
+ offset = ?offset,
+ "replicated consumer offset journaled and staged"
+ );
+ Ok(frozen)
+ }
+ _ => {
+ warn!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica = replica_id,
+ namespace_raw,
+ op = header.op,
+ operation = ?header.operation,
+ "unexpected replicated partition operation"
+ );
+ Err(IggyError::InvalidCommand)
+ }
+ }
+ }
+
+ /// Journal one prepare and record the mutation.
+ ///
+ /// Every partition-plane append goes through here. The DVC suffix snapshot is
+ /// tagged by the head and the commit point, and a repair filling a body under
+ /// a `StartView` this replica already adopted moves neither: the head is the
+ /// announced one and the op is not committed yet. An append that skipped the
+ /// counter would leave the header-only snapshot reading as current, and the
+ /// merge takes a header no sender can serve as proof that sender never
+ /// journaled the op.
+ async fn journal_append(&self, entry: Frozen<4096>) -> Result<(), IggyError> {
+ let appended = self.log.journal().inner.append(entry).await;
+ self.consensus.note_journal_mutation();
+ appended.map_err(|_| IggyError::CannotAppendMessage)
+ }
+
+ async fn append_send_messages_to_journal(
+ &mut self,
+ message: Message<PrepareHeader>,
+ ) -> Result<Frozen<4096>, IggyError> {
+ let write_lock = self.write_lock.clone();
+ let _guard = write_lock.lock().await;
+ self.stamp_and_append_messages(message)
+ .await
+ .map(|journaled| journaled.prepare)
+ }
+
+ async fn append_received_send_messages_to_journal(
+ &mut self,
+ message: Message<PrepareHeader>,
+ ) -> Result<Frozen<4096>, IggyError> {
+ let write_lock = self.write_lock.clone();
+ let _guard = write_lock.lock().await;
+ let header = *message.header();
+ if header.operation != Operation::SendMessages {
+ return Err(IggyError::CannotAppendMessage);
+ }
+ let validated = decode_prepare_slice(message.as_slice())?.header;
+ if validated.message_count == 0 {
+ return Err(IggyError::InvalidCommand);
+ }
+ let expected_offset = if self.offset_space.append_live {
+ self.dirty_offset
+ .load(Ordering::Relaxed)
+ .checked_add(1)
+ .ok_or(IggyError::CannotAppendMessage)?
+ } else {
+ 0
+ };
+ if (validated.base_offset, validated.base_timestamp) != (expected_offset, header.timestamp)
+ {
+ return Err(IggyError::CannotAppendMessage);
+ }
+ self.append_stamped_messages(message, validated)
+ .await
+ .map(|journaled| journaled.prepare)
+ }
+
+ async fn append_stamped_messages(
+ &mut self,
+ message: Message<PrepareHeader>,
+ batch: BatchHeader,
+ ) -> Result<JournaledMessages, IggyError> {
+ let batch_messages_count = batch.message_count;
+ if batch_messages_count == 0 {
+ return Err(IggyError::CannotAppendMessage);
+ }
+
+ let batch_messages_size =
+ u64::try_from(batch.total_size()).map_err(|_| IggyError::CannotAppendMessage)?;
+ let last_dirty_offset = batch
+ .base_offset
+ .checked_add(u64::from(batch_messages_count) - 1)
+ .ok_or(IggyError::CannotAppendMessage)?;
+
+ // Admission must have covered this exact mint frontier before sequencing.
+ if !self.offset_range_is_reserved(last_dirty_offset) {
+ return Err(IggyError::CannotAppendMessage);
+ }
+
+ let segment_index = self.log.segments().len() - 1;
+ let current_position = self.log.segments()[segment_index].current_position;
+ let next_position = current_position
+ .checked_add(batch_messages_size)
+ .ok_or(IggyError::CannotAppendMessage)?;
+
+ let mut journal_info = self.log.journal().info;
+ journal_info.messages_count = journal_info
+ .messages_count
+ .checked_add(batch_messages_count)
+ .ok_or(IggyError::CannotAppendMessage)?;
+ journal_info.size = IggyByteSize::from(
+ journal_info
+ .size
+ .as_bytes_u64()
+ .checked_add(batch_messages_size)
+ .ok_or(IggyError::CannotAppendMessage)?,
+ );
+ journal_info.current_offset = last_dirty_offset;
+ if journal_info.first_timestamp == 0 {
+ journal_info.first_timestamp = batch.base_timestamp;
+ }
+ journal_info.end_timestamp = batch.base_timestamp;
+ journal_info.max_timestamp = journal_info.max_timestamp.max(batch.base_timestamp);
+
+ let frozen = message.into_frozen();
+ self.journal_append(frozen.clone()).await?;
+
+ self.note_append_live();
+ self.dirty_offset
+ .store(last_dirty_offset, Ordering::Relaxed);
+ self.log.segments_mut()[segment_index].current_position = next_position;
+ self.log.journal_mut().info = journal_info;
+
+ Ok(JournaledMessages {
+ result: AppendResult::new(batch.base_offset, last_dirty_offset, batch_messages_count),
+ prepare: frozen,
+ })
+ }
+
+ /// Drop an uncommitted view-divergent suffix and restore every append cursor
+ /// from the retained prefix as one write-locked operation.
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if a retained batch is invalid, the restored segment
+ /// position overflows, or the journal cannot truncate the suffix.
+ pub async fn truncate_uncommitted_from(&mut self, from_op: u64) -> Result<usize, IggyError> {
+ if let Some(persistence) = &self.persistence {
+ persistence.truncate_from(from_op);
+ self.start_persistence();
+ persistence
+ .drain_with_timeout()
+ .await
+ .map_err(|_| IggyError::CannotSyncFile)?;
+ }
+ self.truncate_resident_from(from_op).await
+ }
+
+ async fn truncate_resident_from(&mut self, from_op: u64) -> Result<usize, IggyError> {
+ let mut entries = self.log.journal().inner.resident_entries();
+ entries.sort_unstable_by_key(peek_op);
+ let mut retained_info = JournalInfo::default();
+ let mut retained_next_offset = 0;
+ let mut rewind_next_offset = None;
+ for entry in &entries {
+ if peek_operation(entry) != Operation::SendMessages {
+ continue;
+ }
+ let batch = decode_prepare_slice_trusted(entry.as_slice())
+ .map_err(|_| IggyError::InvalidCommand)?;
+ if batch.message_count() == 0 {
+ continue;
+ }
+ if peek_op(entry) >= from_op {
+ rewind_next_offset = Some(
+ rewind_next_offset.map_or(batch.header.base_offset, |offset: u64| {
+ offset.min(batch.header.base_offset)
+ }),
);
+ continue;
}
+ accumulate_committed_info(
+ &mut retained_info,
+ batch.header.base_offset,
+ batch.header.base_timestamp,
+ batch.header.total_size() as u64,
+ batch.message_count(),
+ );
+ retained_next_offset = retained_next_offset.max(
+ batch
+ .header
+ .base_offset
+ .saturating_add(u64::from(batch.message_count())),
+ );
}
- // Forward only after apply_replicated_operation journals the prepare.
- // The journal and network share the frozen allocation, so the bytes
- // retained for repair are exactly the bytes sent downstream.
- let replicated_result = if is_backup && header.operation == Operation::SendMessages {
- self.append_received_send_messages_to_journal(message).await
+
+ let active = self.log.active_segment();
+ let active_size = active.size.as_bytes_u64();
+ let durable_next_offset = if active_size == 0 {
+ active.start_offset
} else {
- self.apply_replicated_operation(message).await
- };
- let frozen_for_forward = match replicated_result {
- Ok(frozen) => frozen,
- Err(error) => {
- // A BACKUP refusing here is the design, not a fault: it rejects
- // any prepare whose `base_offset` does not continue its own
- // counter, which is exactly what a dropped or out-of-order
- // prepare leaves, and withholding `PrepareOk` is the fail-closed
- // answer. The primary retransmits, journal repair fills the gap,
- // and the group elects around the replica if it cannot catch up.
- // Nothing here is owed an ack this replica already skipped.
- //
- // On the PRIMARY the same return is unrecoverable. The op is
- // already in the pipeline with the sequencer advanced past it
- // (`push_prepare_entry`), and this sits ahead of
- // `send_prepare_ok`, so it never gets its ack and `commit_max`
- // can never pass it: every later op journals fine and queues
- // behind it forever. Nothing lifts that -- the prepare timeout
- // only backs off, a solo group's retransmit target is itself,
- // this plane has no `repair_primary_self_acks`, and a solo group
- // never starts a view change. Clients get no reply at all, since
- // replies are generated on commit, so they wait out their read
- // timeout, and once the queues fill so does every send after.
- //
- // So fence there, the way a failed local commit of a
- // cluster-committed op does: the shard picks `fatal` up on its
- // next tick and takes the node down. A one-second superblock
- // backoff must not cost a partition the rest of the process's
- // life in the dark.
- emit_partition_diag(
- if is_backup {
- tracing::Level::WARN
- } else {
- tracing::Level::ERROR
- },
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- if is_backup {
- "failed to apply replicated partition operation"
- } else {
- "failed to apply an operation this replica sequenced; \
- fencing the partition and shutting down"
- },
- )
- .with_operation(header.operation)
- .with_op(header.op)
- .with_error(error.to_string()),
- );
- if !is_backup && self.fatal.is_none() {
- self.fatal = Some(FatalCommit {
- namespace_raw: self.namespace().inner(),
- op: header.op,
- operation: Some(header.operation),
- });
- }
- return;
- }
+ active.end_offset.saturating_add(1)
};
+ let minimum_next_offset = durable_next_offset
+ .max(retained_next_offset)
+ .max(
+ self.recovered_durable_offset
+ .map_or(0, |offset| offset.saturating_add(1)),
+ )
+ .max(self.installed_frontier.unwrap_or(0));
+ let restored_position = active_size
+ .checked_add(retained_info.size.as_bytes_u64())
+ .ok_or(IggyError::CannotAppendMessage)?;
+ let removed = self
+ .log
+ .journal()
+ .inner
+ .truncate_from(from_op)
+ .await
+ .map_err(|_| IggyError::CannotAppendMessage)?;
- let prepare_to_persist = self
- .persistence
- .as_ref()
- .map(|_| frozen_for_forward.clone());
- let consensus = self.consensus();
- // Backup only: advance sequencer + checksum after journal append.
- // Pre-advance on failing apply would leave consensus claiming op N
- // while the journal has nothing. Retransmit of N would silently drop
- // as is_old_prepare (header.op <= current_sequence). The primary does
- // not re-set here because push_prepare_entry already advanced it. A
- // sibling request pipelined during the apply await would otherwise be
- // rewound to a stale op + parent, projecting a duplicate next.
- if is_backup {
- // Filling a lower slot must not rewind the announced head or parent.
- if header.op >= current_op {
- consensus.sequencer().set_sequence(header.op);
- consensus.set_last_prepare_checksum(header.checksum);
+ self.log.journal_mut().info = retained_info;
+ self.log.active_segment_mut().current_position = restored_position;
+ if let Some(next_offset) = rewind_next_offset {
+ let next_offset = next_offset.max(minimum_next_offset);
+ self.dirty_offset
+ .store(next_offset.saturating_sub(1), Ordering::Relaxed);
+ // The APPEND bit follows the rewound counter. The committed bit does
+ // not: this drops an uncommitted suffix, which by definition names
+ // nothing that ever committed, so raising it here would make a
+ // truncation the event that publishes offsets no quorum agreed on.
+ // A rewind to zero is the exception -- nothing is left at all.
+ self.offset_space.append_live = next_offset > 0;
+ if next_offset == 0 {
+ self.offset_space.committed_seeded = false;
}
- consensus.observe_prepare_timestamp(header.timestamp);
}
- if let Err(error) = replicate_frozen_to_next_in_chain(consensus, frozen_for_forward).await {
- let is_transport_error = error.is_transport();
- emit_partition_diag(
- if is_transport_error {
- tracing::Level::WARN
- } else {
- tracing::Level::ERROR
- },
- &PartitionDiagEvent::new(
- self.diag_ctx(),
- "failed to replicate prepare to next in chain",
- )
- .with_operation(header.operation)
- .with_op(header.op)
- .with_error(error.to_string()),
- );
- if !is_transport_error {
- return;
- }
+ self.consensus.note_journal_mutation();
+ let commit_max = self.consensus.commit_max();
+ self.pending_consumer_offset_commits
+ .retain(|op, _| *op < from_op || *op <= commit_max);
+ self.offset_reservations_need_resync.set(true);
+ Ok(removed)
+ }
+
+ #[cfg(test)]
+ async fn commit_messages(
+ &mut self,
+ config: &PartitionsConfig,
+ through_op: u64,
+ ) -> Result<bool, IggyError> {
+ #[cfg(any(test, feature = "fault-injection"))]
+ if std::mem::take(&mut self.injected_commit_failure) {
+ return Err(IggyError::CannotSaveMessagesToSegment);
}
+ self.commit_messages_inner(
+ config,
+ self.consensus.replica_count() == 1 && self.durability().is_persisted(),
+ through_op,
+ )
+ .await
+ }
+ /// Flush the committed journal prefix to segment storage regardless of
+ /// the `messages_required_to_save` thresholds.
+ ///
+ /// Shutdown-path counterpart of the commit-time persist gate: a graceful
+ /// stop must not lose committed messages that were still resident in the
+ /// in-memory journal (consumer offsets are persisted eagerly, so losing
+ /// the messages would fail recovery with an offset ahead of the data).
+ ///
+ /// Returns `false` while the mounted owner's flush is pending. The caller
+ /// retries through its existing protocol state after completion service.
+ ///
+ /// # Errors
+ ///
+ /// Returns [`IggyError`] when writing the committed batches or their
+ /// index entries to segment storage fails.
+ pub async fn flush_committed_messages(
+ &mut self,
+ config: &PartitionsConfig,
+ ) -> Result<bool, IggyError> {
+ if self.has_io_dispatcher() {
+ if self.fatal.is_some() {
+ return Err(IggyError::CannotSyncFile);
+ }
+ if self.history_is_busy() {
+ return Ok(false);
+ }
+ let through = self.consensus.commit_max();
+ if self
+ .maintenance
+ .flush_completed_through
+ .is_some_and(|completed| completed >= through)
+ && self.log.journal().inner.committed_prefix_len(through) == 0
+ {
+ return Ok(true);
+ }
+ self.request_flush(through);
+ return Ok(false);
+ }
+ if let Some(persistence) = self
+ .persistence
+ .as_ref()
+ .filter(|persistence| persistence.segment_checkpoint().is_some())
{
- let consensus = self.consensus();
- emit_namespace_progress_event(
- SimEventKind::NamespaceProgressUpdated,
- &ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- header.op,
- consensus.pipeline_len(),
- );
+ // Recovery only keeps segment bytes covered by the published WAL frontier.
+ persistence.sync();
+ self.start_persistence();
+ // Shutdown and transfer must finish the flush, but never drain while
+ // holding the lock used to append new messages.
+ persistence.drain_with_timeout().await.map_err(|error| {
+ warn!(%error, "cannot flush committed partition messages");
+ IggyError::CannotSyncFile
+ })?;
}
-
- if let Some(prepare) = prepare_to_persist
- && !self.submit_prepare_persistence(prepare, header.operation)
+ if self
+ .commit_messages_inner(config, true, self.consensus.commit_max())
+ .await?
{
- return;
+ Ok(true)
+ } else {
+ Err(IggyError::CannotSyncFile)
}
- self.send_prepare_ok(&header).await;
}
- #[allow(clippy::future_not_send)]
- pub async fn on_ack(&mut self, message: Message<PrepareOkHeader>, config: &PartitionsConfig) {
- if self.fatal.is_some() {
- return;
+ fn should_persist_messages(&self, config: &PartitionsConfig) -> bool {
+ let journal_info = self.log.journal().info;
+ // The existing thresholds include both committed and uncommitted batches.
+ let messages_due = journal_info.messages_count > 0
+ && (self.log.active_segment().is_full()
+ || journal_info.messages_count >= self.effective_messages_required_to_save(config)
+ || journal_info.size.as_bytes_u64()
+ >= self.effective_size_of_messages_required_to_save(config));
+ messages_due || self.control_ops_due(config)
+ }
+
+ /// Consumer offset ops count toward no message threshold, yet the flush
+ /// is the journal's only eviction, so they get the message-count bound of
+ /// their own: without it a consume-only partition keeps one auto-commit
+ /// op per poll resident until a producer happens to trip a flush.
+ fn control_ops_due(&self, config: &PartitionsConfig) -> bool {
+ self.log.journal().inner.resident_control_ops()
+ >= self.effective_messages_required_to_save(config) as usize
+ }
+
+ /// Returns false while the captured prefix is still pending in the WAL.
+ async fn commit_messages_inner(
+ &mut self,
+ config: &PartitionsConfig,
+ force: bool,
+ through_op: u64,
+ ) -> Result<bool, IggyError> {
+ if force && self.pending_commit.is_none() {
+ self.defer_flush(through_op);
}
- self.resynchronize_consumer_offset_reservations();
- let header = *message.header();
- {
- let consensus = self.consensus();
- if let Err(reason) = ack_preflight(consensus) {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- "ignoring ack during preflight",
- )
- .with_op(header.op)
- .with_reason(reason.as_str()),
- );
- return;
+ if self.materialization.is_none() {
+ if self.log.journal().inner.is_empty()
+ || (!force && !self.should_persist_messages(config))
+ {
+ return Ok(true);
}
-
- if !consensus.pipeline_holds_entry(header.op, header.prepare_checksum) {
- emit_partition_diag(
- tracing::Level::DEBUG,
- &PartitionDiagEvent::new(
- ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- "ack target prepare not in pipeline",
- )
- .with_op(header.op)
- .with_prepare_checksum(header.prepare_checksum),
- );
- return;
+ if !self.begin_materialization(through_op)? {
+ return Ok(false);
+ }
+ if self.materialization.is_none() {
+ return Ok(true);
}
}
-
- if !ack_quorum_reached(self.consensus(), PlaneKind::Partitions, &header) {
- return;
+ if self.io_notifier.borrow().is_some() {
+ self.notify_io();
+ return Ok(false);
}
- // Keep the earned quorum and reply slots in the pipeline while the
- // checkpoint worker synchronizes this partition's materialized files.
- if self.persistence_checkpoint_pending() {
- return;
+ loop {
+ match self.drive_materialization()? {
+ MaterializationStep::Progress => {}
+ MaterializationStep::Pending => return Ok(false),
+ MaterializationStep::Ready => {
+ if let Some(job) = self.capture_materialization(config)? {
+ let result = job.execute().await;
+ self.accept_materialization_phase(result)?;
+ }
+ }
+ MaterializationStep::Complete => return Ok(true),
+ }
}
+ }
- let drained = self.drain_persistable_commits(config);
- if drained.is_empty() {
- return;
+ fn begin_materialization(&mut self, through_op: u64) -> Result<bool, IggyError> {
+ if self.materialization.is_some() {
+ return Ok(true);
}
-
- self.handle_committed_entries(drained, config, true).await;
- {
- let consensus = self.consensus();
- emit_namespace_progress_event(
- SimEventKind::NamespaceProgressUpdated,
- &ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- consensus.commit_min(),
- consensus.pipeline_len(),
- );
+ let through_op = self.consensus.commit_max().min(through_op);
+ let prefix_len = self.log.journal().inner.committed_prefix_len(through_op);
+ if prefix_len == 0 {
+ return Ok(true);
+ }
+ if !self.materialization_prefix_is_ready(prefix_len, through_op)? {
+ return Ok(false);
}
+ self.materialization = Some(PendingMaterialization {
+ incarnation: self.incarnation,
+ poll_history: self.poll_history,
+ through_op,
+ prefix_len,
+ next_entry: 0,
+ durable_offset: None,
+ phase: MaterializationPhase::Inspect,
+ save: None,
+ });
+ Ok(true)
}
- /// Apply the committed prefix this replica can reach, from the pipeline if
- /// the primary populated one and from the journal otherwise.
- ///
- /// The journal half is bounded by [`COMMIT_WALK_OPS_MAX`], for EVERY caller
- /// and not just the tick sweep: `on_commit`, `StartView` adoption, the
- /// post-transfer tail and the post-repair walk all reach this with the same
- /// resident backlog behind them, and all four run on the shard pump. The
- /// pipeline half is left alone, being the primary's own in-flight window,
- /// which the pipeline depth already caps.
- ///
- /// Nothing is lost by stopping early: the run is re-derived from
- /// `commit_min` on the next call, and the sweep's walk predicate stays true
- /// until the group drains, so the next tick resumes exactly here. The one
- /// caller with no next tick is the shutdown drain, and it is covered twice
- /// over: [`Self::flush_committed_messages`] persists the committed prefix
- /// by bytes rather than by walk, and what stays un-applied sits above the
- /// `commit_min` the superblock records, which is what makes the restarted
- /// replica ask its peers for it.
- #[allow(clippy::future_not_send)]
- pub async fn commit_journal(&mut self, config: &PartitionsConfig) {
- if self.fatal.is_some()
- || self.materialization_missing
- || self.persistence_checkpoint_pending()
- {
- return;
+ fn drive_materialization(&mut self) -> Result<MaterializationStep, IggyError> {
+ let pending = self
+ .materialization
+ .as_ref()
+ .ok_or(IggyError::CannotWriteToFile)?;
+ if pending.incarnation != self.incarnation || pending.poll_history != self.poll_history {
+ return Err(IggyError::CannotWriteToFile);
}
- self.resynchronize_consumer_offset_reservations();
-
- // The primary commits inline via `on_ack` (it drains its own pipeline).
- // Backups never populate the pipeline - they journal replicated prepares
- // in `apply_replicated_operation` - so the pipeline drain is empty for
- // them. Fall back to the journal so backups durably persist committed
- // data. `commit_messages` then flushes only the committed prefix and
- // keeps the uncommitted tail journal-resident, so a later commit of that
- // tail still finds its headers here (no wedge). Pipeline-first keeps a
- // freshly promoted primary (rebuilt pipeline) draining there, avoiding a
- // double-count against `advance_commit_min`.
- let mut drained = self.drain_persistable_commits(config);
- let send_client_replies = !drained.is_empty() && self.consensus.is_primary();
- if drained.is_empty() {
- drained = self.collect_committable_from_journal(COMMIT_WALK_OPS_MAX, config);
+ match pending.phase {
+ MaterializationPhase::Save(_) | MaterializationPhase::Rotate(_) => {
+ return Ok(MaterializationStep::Ready);
+ }
+ MaterializationPhase::Saving(_) | MaterializationPhase::Rotating => {
+ return Ok(MaterializationStep::Pending);
+ }
+ MaterializationPhase::Inspect => {}
}
- if drained.is_empty() {
- return;
+ if !self.materialization_prefix_is_ready(pending.prefix_len, pending.through_op)? {
+ return Ok(MaterializationStep::Pending);
}
-
- self.handle_committed_entries(drained, config, send_client_replies)
- .await;
+ let segment = self.log.active_segment();
+ if segment.size.as_bytes_u64() >= segment.max_size.as_bytes_u64() {
+ let next_offset = segment
+ .end_offset
+ .checked_add(1)
+ .ok_or(IggyError::CannotWriteToFile)?;
+ self.materialization
+ .as_mut()
+ .expect("materialization exists")
+ .phase = MaterializationPhase::Rotate(next_offset);
+ return Ok(MaterializationStep::Ready);
+ }
+ if pending.next_entry == pending.prefix_len {
+ let finished = self.materialization.take().expect("materialization exists");
+ if self
+ .maintenance
+ .flush_through
+ .is_some_and(|through| through <= finished.through_op)
+ {
+ self.finish_flush();
+ }
+ self.evict_committed_prefix(finished.prefix_len);
+ if self.pending_commit.is_none()
+ && let Some(offset) = finished.durable_offset
+ {
+ self.note_committed_seeded();
+ // A Replicated flush runs behind commits that already
+ // published a higher offset.
+ let visible = self.offset.fetch_max(offset, Ordering::AcqRel).max(offset);
+ self.stats.set_current_offset(visible);
+ }
+ return Ok(MaterializationStep::Complete);
+ }
+ let chunk =
+ self.plan_materialization_chunk(pending.next_entry, pending.prefix_len, None)?;
+ if chunk.index.is_some()
+ && self.log.index_writers().last().is_none_or(Option::is_none)
+ && self
+ .log
+ .messages_writers()
+ .last()
+ .is_none_or(Option::is_none)
+ && self
+ .persistence
+ .as_ref()
+ .is_none_or(|persistence| persistence.segment_checkpoint().is_none())
{
- let consensus = self.consensus();
- emit_namespace_progress_event(
- SimEventKind::NamespaceProgressUpdated,
- &ReplicaLogContext::from_consensus(consensus, PlaneKind::Partitions),
- consensus.commit_min(),
- consensus.pipeline_len(),
+ let segment = self.log.active_segment_mut();
+ segment.size = IggyByteSize::from(
+ segment
+ .size
+ .as_bytes_u64()
+ .checked_add(chunk.info.size.as_bytes_u64())
+ .ok_or(IggyError::CannotWriteToFile)?,
);
+ self.accept_materialization_chunk(chunk)?;
+ return Ok(MaterializationStep::Progress);
+ }
+ let piece = if chunk.allocation_charge > self.io_bytes_max.get() {
+ self.plan_materialization_chunk(
+ chunk.first_entry,
+ chunk.end_entry,
+ Some(self.io_bytes_max.get()),
+ )?
+ } else {
+ chunk
+ };
+ let pending = self
+ .materialization
+ .as_mut()
+ .expect("materialization exists");
+ if chunk.index.is_none() {
+ pending.next_entry = chunk.end_entry;
+ return Ok(MaterializationStep::Progress);
}
+ pending.save = Some(PendingMaterializationSave { chunk, written: 0 });
+ pending.phase = MaterializationPhase::Save(piece);
+ Ok(MaterializationStep::Ready)
}
- fn drain_persistable_commits(&self, config: &PartitionsConfig) -> Vec<PipelineEntry> {
- let Some(persistence) = &self.persistence else {
- return drain_committable_prefix(self.consensus());
- };
- self.persist_repaired_prefix();
- let through = self.consensus.commit_max().min(persistence.head());
- let materialize = self.should_persist_messages(config);
- let mut drained = Vec::new();
- self.consensus.with_pipeline_mut(|pipeline| {
- let mut next = self.consensus.commit_min() + 1;
- while let Some(entry) = pipeline.head() {
- if entry.header.op > through {
- break;
- }
- if entry.header.op != next {
- report_uncommittable_head(
- self.consensus.replica(),
- entry.header.op,
- self.consensus.commit_min(),
- self.consensus.commit_max(),
- drained.len(),
- );
- break;
- }
- if !self.prepare_body_is_ready(&entry.header, materialize) {
- break;
+ fn materialization_prefix_is_ready(
+ &self,
+ prefix_len: usize,
+ through_op: u64,
+ ) -> Result<bool, IggyError> {
+ if let Some(persistence) = self
+ .persistence
+ .as_ref()
+ .filter(|persistence| persistence.segment_checkpoint().is_some())
+ {
+ if persistence.failure().is_some() {
+ return Err(IggyError::CannotSyncFile);
+ }
+ let last_send = self.log.journal().inner.with_entries(|entries| {
+ entries
+ .iter()
+ .take(prefix_len)
+ .rev()
+ .find(|entry| {
+ peek_operation(entry) == Operation::SendMessages
+ && peek_op(entry) > self.purge_floor_op
+ })
+ .map(peek_op)
+ });
+ if last_send.is_some_and(|op| op > through_op) {
+ return Err(IggyError::CannotWriteToFile);
+ }
+ if last_send.is_some_and(|op| {
+ if self.durability().is_persisted() {
+ !persistence.is_durable_through(op)
+ } else {
+ !persistence.is_written_through(op)
}
- drained.push(pipeline.pop().expect("pipeline head exists"));
- next += 1;
+ }) {
+ return Ok(false);
}
- });
- if !drained.is_empty() {
- self.consensus.sync_prepare_timeout();
}
- drained
- }
- fn prepare_body_is_ready(&self, header: &PrepareHeader, materialize: bool) -> bool {
- header.operation != Operation::SendMessages
- || header.op <= self.purge_floor_op
- || (!self.durability().is_persisted() && !materialize)
- || self
- .persistence
- .as_ref()
- .filter(|persistence| persistence.segment_checkpoint().is_some())
- .is_none_or(|persistence| {
- if self.durability().is_persisted() {
- persistence.is_durable(header)
- } else {
- persistence.is_written(header)
- }
- })
+ Ok(true)
}
- /// Committable entries (ops `commit_min+1 ..= commit_max`) read from the
- /// journal, for a backup whose pipeline is empty. Stops at the first missing
- /// op: a replication gap must not be skipped, or `advance_commit_min`'s
- /// sequential contract breaks.
- ///
- /// Reads resident headers only. The commit walk caps flushing at its last
- /// drained op, keeping later headers resident for the next bounded walk.
- fn collect_committable_from_journal(
+ fn plan_materialization_chunk(
&self,
- max_ops: usize,
- config: &PartitionsConfig,
- ) -> Vec<PipelineEntry> {
- let from_op = self.consensus.commit_min() + 1;
- // Stop below the pipeline head. The drain above holds rather than pops
- // when the head is not the op owed next; walking that op out of the
- // journal instead advances `commit_min` past a still-resident entry, and
- // `on_ack` then finds the drain empty on every later ack -- no reply is
- // ever shipped and each stranded entry leaves its awaiter parked.
- //
- // Only a head at or above `from_op` lowers the ceiling: an absent head is
- // a backup's empty pipeline, and a lower head is already stranded and must
- // not freeze the walk on top of that.
- let commit_max = self.persistence.as_ref().map_or_else(
- || self.consensus.commit_max(),
- |persistence| self.consensus.commit_max().min(persistence.head()),
- );
- let commit_max = self
- .consensus
- .pipeline_head_header()
- .filter(|head| head.op >= from_op)
- .map_or(commit_max, |head| commit_max.min(head.op - 1));
- let materialize = self.should_persist_messages(config);
- self.log
- .journal()
- .inner
- .committed_headers_from(from_op, commit_max, max_ops)
- .into_iter()
- .take_while(|header| self.prepare_body_is_ready(header, materialize))
- .map(PipelineEntry::new)
- .collect()
- }
-
- async fn apply_replicated_operation(
- &mut self,
- message: Message<PrepareHeader>,
- ) -> Result<Frozen<4096>, IggyError> {
- let header = *message.header();
- let replica_id = self.consensus.replica();
- let namespace_raw = self.consensus.group();
-
- match header.operation {
- Operation::RetireSession => {
- let retirement = RetireSessionsRequest::decode_from(message.body())
- .map_err(|_| IggyError::InvalidCommand)?;
- if retirement
- .identities
- .iter()
- .any(|identity| is_auto_commit_client(identity.client_id))
- {
- return Err(IggyError::InvalidCommand);
+ first_entry: usize,
+ prefix_len: usize,
+ bytes_max: Option<usize>,
+ ) -> Result<MaterializationChunk, IggyError> {
+ let segment = self.log.active_segment();
+ let position = segment.size.as_bytes_u64();
+ let persisted_end = if position == 0 {
+ segment.start_offset.checked_sub(1)
+ } else {
+ Some(segment.end_offset)
+ }
+ .max(self.recovered_durable_offset);
+ let mut chunk = MaterializationChunk {
+ first_entry,
+ end_entry: first_entry,
+ persisted_end,
+ buffers: 0,
+ allocation_charge: 0,
+ index: None,
+ info: JournalInfo::default(),
+ };
+ self.log.journal().inner.with_entries(|entries| {
+ let entries = entries
+ .get(first_entry..prefix_len)
+ .ok_or(IggyError::CannotWriteToFile)?;
+ let mut backing = 0_usize;
+ let mut file_position = position;
+ for entry in entries {
+ let Some(meta) =
+ materialization_batch_meta(entry, persisted_end, self.purge_floor_op)?
+ else {
+ chunk.end_entry += 1;
+ continue;
+ };
+ let next_backing = backing
+ .checked_add(
+ Frozen::<4096>::allocation_size(entry.backing_capacity())
+ .ok_or(IggyError::CannotWriteToFile)?,
+ )
+ .ok_or(IggyError::CannotWriteToFile)?;
+ let next_charge = crate::io::materialization_charge(
+ next_backing,
+ chunk.buffers + 1,
+ crate::iggy_index::IGGY_INDEX_SIZE,
+ )
+ .ok_or(IggyError::CannotWriteToFile)?;
+ if bytes_max.is_some_and(|maximum| next_charge > maximum) {
+ if chunk.buffers == 0 {
+ return Err(IggyError::CannotWriteToFile);
+ }
+ break;
}
- let frozen = message.into_frozen();
- self.journal_append(frozen.clone()).await?;
- Ok(frozen)
- }
- Operation::SendMessages => {
- let frozen = self.append_send_messages_to_journal(message).await?;
- debug!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- replica = replica_id,
- op = header.op,
- namespace_raw,
- operation = ?header.operation,
- "replicated send_messages appended to partition journal"
+ chunk.end_entry += 1;
+ if chunk.index.is_none() {
+ chunk.index = Some(crate::iggy_index::IggyIndex::new(
+ meta.base_offset,
+ meta.base_timestamp,
+ file_position,
+ ));
+ }
+ file_position = file_position
+ .checked_add(meta.total_size)
+ .ok_or(IggyError::CannotWriteToFile)?;
+ chunk.buffers += 1;
+ backing = next_backing;
+ accumulate_committed_info(
+ &mut chunk.info,
+ meta.base_offset,
+ meta.base_timestamp,
+ meta.total_size,
+ meta.message_count,
);
- Ok(frozen)
+ if file_position >= segment.max_size.as_bytes_u64() {
+ break;
+ }
}
- Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset => {
- // Replication semantics do not depend on the acknowledgement
- // byte. Multi-replica `NoAck` requests enter this same path.
- let (kind, consumer_id, offset, _ack) =
- Self::parse_staged_consumer_offset_commit(header.operation, &message)?;
- let write_lock = self.write_lock.clone();
- let _guard = write_lock.lock().await;
-
- // Journal the prepare before staging so
- // `VsrAction::RetransmitPrepares` can read this op back
- // on a view change. Without the journal entry, the
- // `header_by_op` lookup in `on_replicate` would miss,
- // the gap check would drop the retransmit, and the
- // primary's pipeline would wedge indefinitely. Skip
- // the `journal.info` accounting: it counts SendMessages
- // batches for segment-commit thresholds, which do not
- // apply to offset ops.
- let frozen = message.into_frozen();
- self.journal_append(frozen.clone()).await?;
+ chunk.allocation_charge = crate::io::materialization_charge(
+ backing,
+ chunk.buffers,
+ crate::iggy_index::IGGY_INDEX_SIZE,
+ )
+ .ok_or(IggyError::CannotWriteToFile)?;
+ Ok(chunk)
+ })
+ }
- match header.operation {
- Operation::StoreConsumerOffset => {
- self.stage_consumer_offset_upsert(
- header.op,
- kind,
- consumer_id,
- offset.expect("store_consumer_offset must include offset"),
- is_auto_commit_client(header.client),
- );
+ fn capture_materialization(
+ &mut self,
+ config: &PartitionsConfig,
+ ) -> Result<Option<crate::io::PartitionIoJob<SB>>, IggyError> {
+ let phase = self
+ .materialization
+ .as_ref()
+ .ok_or(IggyError::CannotWriteToFile)?
+ .phase;
+ match phase {
+ MaterializationPhase::Save(chunk) => {
+ let save = self
+ .materialization
+ .as_ref()
+ .and_then(|pending| pending.save)
+ .ok_or(IggyError::CannotWriteToFile)?;
+ let completes = chunk.end_entry == save.chunk.end_entry;
+ let batches = self.log.journal().inner.with_entries(|entries| {
+ let entries = entries
+ .get(chunk.first_entry..chunk.end_entry)
+ .ok_or(IggyError::CannotWriteToFile)?;
+ let mut batches = Vec::with_capacity(chunk.buffers);
+ for entry in entries {
+ if materialization_batch_meta(
+ entry,
+ chunk.persisted_end,
+ self.purge_floor_op,
+ )?
+ .is_some()
+ {
+ batches.push(entry.clone());
+ }
}
- Operation::DeleteConsumerOffset => {
- self.stage_consumer_offset_delete(header.op, kind, consumer_id);
+ Ok::<_, IggyError>(batches)
+ })?;
+ let index = save.chunk.index.ok_or(IggyError::CannotWriteToFile)?;
+ let index_bytes = if completes {
+ crate::iggy_index::IggyIndexCache::serialize(&index)
+ } else {
+ Vec::new()
+ };
+ let job = self.prepare_materialization(
+ batches,
+ index_bytes,
+ chunk.info.messages_count,
+ save.written,
+ completes,
+ )?;
+ if let Some(job) = job {
+ if job.allocation_charge() > chunk.allocation_charge {
+ return Err(IggyError::CannotWriteToFile);
}
- _ => unreachable!(),
+ self.materialization
+ .as_mut()
+ .expect("materialization exists")
+ .phase = MaterializationPhase::Saving(chunk);
+ Ok(Some(crate::io::PartitionIoJob::Materialize(job)))
+ } else {
+ self.accept_materialization_chunk(chunk)?;
+ Ok(None)
}
-
- debug!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- replica = replica_id,
- op = header.op,
- namespace_raw,
- operation = ?header.operation,
- consumer_kind = ?kind,
- consumer_id,
- offset = ?offset,
- "replicated consumer offset journaled and staged"
- );
- Ok(frozen)
}
- _ => {
- warn!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- replica = replica_id,
- namespace_raw,
- op = header.op,
- operation = ?header.operation,
- "unexpected replicated partition operation"
- );
- Err(IggyError::InvalidCommand)
+ MaterializationPhase::Rotate(start_offset) => {
+ let (target, job) = self.prepare_rotation(config, start_offset);
+ self.materialization
+ .as_mut()
+ .expect("materialization exists")
+ .phase = MaterializationPhase::Rotating;
+ Ok(Some(crate::io::PartitionIoJob::Rotate { target, job }))
}
+ _ => Err(IggyError::CannotWriteToFile),
}
}
- /// Journal one prepare and record the mutation.
- ///
- /// Every partition-plane append goes through here. The DVC suffix snapshot is
- /// tagged by the head and the commit point, and a repair filling a body under
- /// a `StartView` this replica already adopted moves neither: the head is the
- /// announced one and the op is not committed yet. An append that skipped the
- /// counter would leave the header-only snapshot reading as current, and the
- /// merge takes a header no sender can serve as proof that sender never
- /// journaled the op.
- async fn journal_append(&self, entry: Frozen<4096>) -> Result<(), IggyError> {
- let appended = self.log.journal().inner.append(entry).await;
- self.consensus.note_journal_mutation();
- appended.map_err(|_| IggyError::CannotAppendMessage)
+ fn accept_materialization_phase(
+ &mut self,
+ result: crate::io::PartitionIoResult,
+ ) -> Result<(), IggyError> {
+ let phase = self
+ .materialization
+ .as_ref()
+ .ok_or(IggyError::CannotWriteToFile)?
+ .phase;
+ match (phase, result) {
+ (
+ MaterializationPhase::Saving(chunk),
+ crate::io::PartitionIoResult::Materialize(result),
+ ) => {
+ if let Err(error) = self.accept_materialization(result) {
+ self.materialization
+ .as_mut()
+ .expect("materialization exists")
+ .phase = MaterializationPhase::Save(chunk);
+ return Err(error);
+ }
+ let save = self
+ .materialization
+ .as_ref()
+ .and_then(|pending| pending.save)
+ .ok_or(IggyError::CannotWriteToFile)?;
+ if chunk.end_entry < save.chunk.end_entry {
+ let written = save
+ .written
+ .checked_add(chunk.info.size.as_bytes_u64())
+ .ok_or(IggyError::CannotWriteToFile)?;
+ let next = self.plan_materialization_chunk(
+ chunk.end_entry,
+ save.chunk.end_entry,
+ Some(self.io_bytes_max.get()),
+ )?;
+ let pending = self
+ .materialization
+ .as_mut()
+ .expect("materialization exists");
+ pending.save = Some(PendingMaterializationSave {
+ chunk: save.chunk,
+ written,
+ });
+ pending.phase = MaterializationPhase::Save(next);
+ Ok(())
+ } else {
+ self.materialization
+ .as_mut()
+ .expect("materialization exists")
+ .save = None;
+ self.accept_materialization_chunk(save.chunk)
+ }
+ }
+ (
+ MaterializationPhase::Rotating,
+ crate::io::PartitionIoResult::Rotate { target, outcome },
+ ) => {
+ let start_offset = target
+ .segment_end
+ .checked_add(1)
+ .ok_or(IggyError::CannotWriteToFile)?;
+ match outcome {
+ Ok(installed) => self.accept_rotation(&target, installed)?,
+ Err(error) => {
+ self.materialization
+ .as_mut()
+ .expect("materialization exists")
+ .phase = MaterializationPhase::Rotate(start_offset);
+ return Err(error);
+ }
+ }
+ self.materialization
+ .as_mut()
+ .expect("materialization exists")
+ .phase = MaterializationPhase::Inspect;
+ Ok(())
+ }
+ _ => Err(IggyError::CannotWriteToFile),
+ }
}
- async fn append_send_messages_to_journal(
+ fn accept_materialization_chunk(
&mut self,
- message: Message<PrepareHeader>,
- ) -> Result<Frozen<4096>, IggyError> {
- let write_lock = self.write_lock.clone();
- let _guard = write_lock.lock().await;
- self.stamp_and_append_messages(message)
- .await
- .map(|journaled| journaled.prepare)
+ chunk: MaterializationChunk,
+ ) -> Result<(), IggyError> {
+ let index = chunk.index.ok_or(IggyError::CannotWriteToFile)?;
+ if let Some(persistence) = &self.persistence {
+ persistence.mark_segment_dirty(self.log.active_segment().start_offset);
+ }
+ self.log.ensure_indexes();
+ self.log
+ .active_indexes_mut()
+ .ok_or(IggyError::CannotWriteToFile)?
+ .insert(index.offset, index.timestamp, index.position);
+ let segment = self.log.active_segment_mut();
+ if segment.start_timestamp == 0 && chunk.info.first_timestamp != 0 {
+ segment.start_timestamp = chunk.info.first_timestamp;
+ }
+ segment.end_timestamp = chunk.info.end_timestamp;
+ segment.max_timestamp = segment.max_timestamp.max(chunk.info.max_timestamp);
+ segment.end_offset = chunk.info.current_offset;
+ let pending = self
+ .materialization
+ .as_mut()
+ .ok_or(IggyError::CannotWriteToFile)?;
+ pending.next_entry = chunk.end_entry;
+ pending.durable_offset = Some(chunk.info.current_offset);
+ pending.phase = MaterializationPhase::Inspect;
+ Ok(())
}
- async fn append_received_send_messages_to_journal(
- &mut self,
- message: Message<PrepareHeader>,
- ) -> Result<Frozen<4096>, IggyError> {
- let write_lock = self.write_lock.clone();
- let _guard = write_lock.lock().await;
- let header = *message.header();
- if header.operation != Operation::SendMessages {
- return Err(IggyError::CannotAppendMessage);
+ /// Evict the committed prefix (the `count` front entries read by
+ /// `committed_prefix`) and reset `journal.info` to reflect only the
+ /// uncommitted tail left resident, so the next persist threshold counts that
+ /// tail alone. Call once the prefix is durable, or when there is nothing to
+ /// persist. The retained tail's accounting is folded from the meta
+ /// `evict_prefix` surfaced during its re-append, so the tail is not decoded
+ /// a second time.
+ fn evict_committed_prefix(&mut self, count: usize) {
+ if count == 0 {
+ return;
}
- let validated = decode_prepare_slice(message.as_slice())?.header;
- if validated.message_count == 0 {
- return Err(IggyError::InvalidCommand);
+ if let Some(pending) = &mut self.pending_commit {
+ pending.evictable = pending.evictable.max(count);
+ return;
}
- let expected_offset = if self.offset_space.append_live {
- self.dirty_offset
- .load(Ordering::Relaxed)
- .checked_add(1)
- .ok_or(IggyError::CannotAppendMessage)?
- } else {
- 0
- };
- if (validated.base_offset, validated.base_timestamp) != (expected_offset, header.timestamp)
- {
- return Err(IggyError::CannotAppendMessage);
+ let retained = self.log.journal().inner.evict_prefix(count);
+ // Eviction drains the ring and re-appends the retained tail, so the slots
+ // a suffix snapshot describes move under it without the head or the
+ // commit point changing.
+ self.consensus.note_journal_mutation();
+ let mut retained_info = JournalInfo::default();
+ for (entry, meta) in &retained {
+ // Purge floor: a retained pre-purge batch must not fold its
+ // accounting back into `journal.info`, or the info would re-adopt
+ // a pre-purge `current_offset` the purge just reset.
+ if peek_op(entry) <= self.purge_floor_op {
+ continue;
+ }
+ if let Some(meta) = meta {
+ accumulate_committed_info(
+ &mut retained_info,
+ meta.base_offset,
+ meta.base_timestamp,
+ meta.total_size,
+ meta.message_count,
+ );
+ }
}
- self.append_stamped_messages(message, validated)
- .await
- .map(|journaled| journaled.prepare)
+ self.log.journal_mut().info = retained_info;
}
- async fn append_stamped_messages(
+ async fn handle_committed_entries(
&mut self,
- message: Message<PrepareHeader>,
- batch: BatchHeader,
- ) -> Result<JournaledMessages, IggyError> {
- let batch_messages_count = batch.message_count;
- if batch_messages_count == 0 {
- return Err(IggyError::CannotAppendMessage);
+ headers: Vec<PrepareHeader>,
+ config: &PartitionsConfig,
+ send_client_replies: bool,
+ ) {
+ if !self.io_replies.is_empty() {
+ self.notify_io();
+ return;
}
+ if self.pending_commit.is_none()
+ && !self.select_commit(headers, config, send_client_replies)
+ {
+ return;
+ }
+ loop {
+ let step = match self.drive_commit() {
+ Ok(step) => step,
+ Err(error) => {
+ self.fail_pending_commit(&error);
+ return;
+ }
+ };
+ match step {
+ CommitStep::Progress => {}
+ CommitStep::Pending => return,
+ CommitStep::Ready => {
+ if self.has_io_dispatcher() {
+ self.notify_io();
+ return;
+ }
+ let job = match self.capture_commit_phase(config) {
+ Ok(job) => job,
+ Err(error) => {
+ self.fail_pending_commit(&error);
+ return;
+ }
+ };
+ if let Some(job) = job
+ && let Err(error) = self.accept_commit_phase(job.execute().await)
+ {
+ self.fail_pending_commit(&error);
+ return;
+ }
+ }
+ CommitStep::Publish => {
+ let Some(committed) = self.publish_pending_commit() else {
+ return;
+ };
+ let committed_count = committed.len();
+ self.deliver_commit_replies(committed).await;
+ self.drain_no_ack().await;
+ self.drain_request_queue_into_prepares(committed_count)
+ .await;
+ return;
+ }
+ }
+ }
+ }
- let batch_messages_size =
- u64::try_from(batch.total_size()).map_err(|_| IggyError::CannotAppendMessage)?;
- let last_dirty_offset = batch
- .base_offset
- .checked_add(u64::from(batch_messages_count) - 1)
- .ok_or(IggyError::CannotAppendMessage)?;
-
- // Past this line the offsets are in the journal, hence committable,
- // pollable, confirmable and forwardable, and no later gate can take
- // them back. See [`Self::reserve_offsets_through`].
- //
- // LOCK ORDER: the caller holds `write_lock` and this takes
- // `superblock_lock` under it. The install path takes them in the
- // reverse order, safely only because `reset_offset_frontier_at` drops
- // `superblock_lock` before `try_install` takes `write_lock`. Never hold
- // `superblock_lock` across a `write_lock` acquire.
- if !self.reserve_offsets_through(last_dirty_offset).await {
- return Err(IggyError::CannotAppendMessage);
+ fn drive_commit(&mut self) -> Result<CommitStep, IggyError> {
+ if !self.pending_commit_is_current() {
+ return Err(IggyError::CannotWriteToFile);
+ }
+ let pending = self
+ .pending_commit
+ .as_ref()
+ .ok_or(IggyError::CannotWriteToFile)?;
+ match pending.phase {
+ CommitPhase::Start => self.start_commit_materialization(),
+ CommitPhase::Materialize => match self.drive_materialization()? {
+ MaterializationStep::Progress => Ok(CommitStep::Progress),
+ MaterializationStep::Pending => Ok(CommitStep::Pending),
+ MaterializationStep::Ready => Ok(CommitStep::Ready),
+ MaterializationStep::Complete => {
+ let pending = self
+ .pending_commit
+ .as_mut()
+ .expect("selected commit exists");
+
+ pending.phase = CommitPhase::Offsets;
+ Ok(CommitStep::Progress)
+ }
+ },
+ CommitPhase::Offsets => self.drive_committed_offsets(),
+ CommitPhase::OffsetReady(_) if self.offset_writes_wait_for_checkpoint() => {
+ Ok(CommitStep::Pending)
+ }
+ CommitPhase::OffsetReady(_) => Ok(CommitStep::Ready),
+ CommitPhase::SegmentDirectory => {
+ if (pending.has_sends() || pending.materialize)
+ && self.consensus.replica_count() == 1
+ && self.durability().is_persisted()
+ && self.segment_names_dirty.get()
+ && self.partition_dir.is_some()
+ {
+ return Ok(CommitStep::Ready);
+ }
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::OffsetDirectories;
+ Ok(CommitStep::Progress)
+ }
+ CommitPhase::OffsetDirectories => {
+ let dirty = self
+ .consumer_offset_dirs_touched
+ .iter()
+ .zip(&self.consumer_offset_dirs_dirty)
+ .any(|(touched, dirty)| touched.get().is_some() && dirty.get());
+ if dirty {
+ return Ok(CommitStep::Ready);
+ }
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::Publish;
+ Ok(CommitStep::Publish)
+ }
+ CommitPhase::Publish => Ok(CommitStep::Publish),
+ CommitPhase::OffsetSubmitted(_)
+ | CommitPhase::SegmentDirectorySubmitted
+ | CommitPhase::OffsetDirectoriesSubmitted => Ok(CommitStep::Pending),
}
+ }
- let segment_index = self.log.segments().len() - 1;
- let current_position = self.log.segments()[segment_index].current_position;
- let next_position = current_position
- .checked_add(batch_messages_size)
- .ok_or(IggyError::CannotAppendMessage)?;
+ fn start_commit_materialization(&mut self) -> Result<CommitStep, IggyError> {
+ let pending = self
+ .pending_commit
+ .as_ref()
+ .ok_or(IggyError::CannotWriteToFile)?;
- let mut journal_info = self.log.journal().info;
- journal_info.messages_count = journal_info
- .messages_count
- .checked_add(batch_messages_count)
- .ok_or(IggyError::CannotAppendMessage)?;
- journal_info.size = IggyByteSize::from(
- journal_info
- .size
- .as_bytes_u64()
- .checked_add(batch_messages_size)
- .ok_or(IggyError::CannotAppendMessage)?,
- );
- journal_info.current_offset = last_dirty_offset;
- if journal_info.first_timestamp == 0 {
- journal_info.first_timestamp = batch.base_timestamp;
+ #[cfg(any(test, feature = "fault-injection"))]
+ if (pending.has_sends() || pending.materialize)
+ && std::mem::take(&mut self.injected_commit_failure)
+ {
+ return Err(IggyError::CannotSaveMessagesToSegment);
}
- journal_info.end_timestamp = batch.base_timestamp;
- journal_info.max_timestamp = journal_info.max_timestamp.max(batch.base_timestamp);
-
- let frozen = message.into_frozen();
- self.journal_append(frozen.clone()).await?;
+ let has_sends = pending.has_sends();
+ if pending.materialize {
+ let through_op = pending
+ .headers
+ .last()
+ .ok_or(IggyError::CannotWriteToFile)?
+ .op;
+ if !self.begin_materialization(through_op)? {
+ if has_sends {
+ return Ok(CommitStep::Pending);
+ }
+ self.defer_flush(through_op);
+ }
+ if self.materialization.is_some() {
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::Materialize;
+ return Ok(CommitStep::Progress);
+ }
+ }
+ let pending = self
+ .pending_commit
+ .as_mut()
+ .expect("selected commit exists");
- self.note_append_live();
- self.dirty_offset
- .store(last_dirty_offset, Ordering::Relaxed);
- self.log.segments_mut()[segment_index].current_position = next_position;
- self.log.journal_mut().info = journal_info;
+ pending.phase = CommitPhase::Offsets;
+ Ok(CommitStep::Progress)
+ }
- Ok(JournaledMessages {
- result: AppendResult::new(batch.base_offset, last_dirty_offset, batch_messages_count),
- prepare: frozen,
- })
+ fn drive_committed_offsets(&mut self) -> Result<CommitStep, IggyError> {
+ let pending = self
+ .pending_commit
+ .as_ref()
+ .ok_or(IggyError::CannotWriteToFile)?;
+
+ let Some(header) = pending.headers.get(pending.next_entry).copied() else {
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::SegmentDirectory;
+ return Ok(CommitStep::Progress);
+ };
+ if header.op <= self.purge_floor_op
+ || !matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ )
+ {
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .next_entry += 1;
+ return Ok(CommitStep::Progress);
+ }
+ let offset = self
+ .pending_consumer_offset_commits
+ .get(&header.op)
+ .copied()
+ .map_or_else(|| self.restage_consumer_offset_from_journal(header.op), Ok)?;
+ let tracked = pending
+ .offset_effects
+ .iter()
+ .rev()
+ .find(|effect| {
+ effect.pending.kind == offset.kind
+ && effect.pending.consumer_id == offset.consumer_id
+ })
+ .map_or_else(
+ || {
+ self.durable_consumer_offsets
+ .get(offset.kind, offset.consumer_id)
+ },
+ |effect| effect.state,
+ );
+ let plan = self.plan_consumer_offset_effect(offset, tracked);
+ if let ConsumerOffsetAction::Ready(effect) = plan.action {
+ self.accept_committed_offset_effect(effect)?;
+ } else {
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::OffsetReady(plan);
+ }
+ Ok(CommitStep::Progress)
}
- /// Drop an uncommitted view-divergent suffix and restore every append cursor
- /// from the retained prefix as one write-locked operation.
- ///
- /// # Errors
- ///
- /// Returns an error if a retained batch is invalid, the restored segment
- /// position overflows, or the journal cannot truncate the suffix.
- pub async fn truncate_uncommitted_from(&mut self, from_op: u64) -> Result<usize, IggyError> {
- let write_lock = self.write_lock.clone();
- let _guard = write_lock.lock().await;
+ fn capture_commit_phase(
+ &mut self,
+ config: &PartitionsConfig,
+ ) -> Result<Option<crate::io::PartitionIoJob<SB>>, IggyError> {
+ if !self.pending_commit_is_current() {
+ return Err(IggyError::CannotWriteToFile);
+ }
+ let phase = self
+ .pending_commit
+ .as_ref()
+ .ok_or(IggyError::CannotWriteToFile)?
+ .phase;
+ let (job, submitted) = match phase {
+ CommitPhase::Materialize => return self.capture_materialization(config),
+ CommitPhase::OffsetReady(plan) => (
+ self.capture_consumer_offset_effect(&plan)?,
+ CommitPhase::OffsetSubmitted(plan),
+ ),
+ CommitPhase::SegmentDirectory => (
+ crate::io::PartitionIoJob::SegmentDirectory(
+ self.partition_dir
+ .clone()
+ .ok_or(IggyError::CannotSyncFile)?,
+ ),
+ CommitPhase::SegmentDirectorySubmitted,
+ ),
+ CommitPhase::OffsetDirectories => (
+ crate::io::PartitionIoJob::OffsetDirectories(
+ self.prepare_consumer_offset_directories(
+ self.consumer_offset_dirs_touched
+ .each_ref()
+ .map(|touched| touched.get().is_some()),
+ ),
+ ),
+ CommitPhase::OffsetDirectoriesSubmitted,
+ ),
+ _ => return Err(IggyError::CannotWriteToFile),
+ };
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = submitted;
+ Ok(Some(job))
+ }
- let mut entries = self.log.journal().inner.resident_entries();
- entries.sort_unstable_by_key(peek_op);
- let mut retained_info = JournalInfo::default();
- let mut retained_next_offset = 0;
- let mut rewind_next_offset = None;
- for entry in &entries {
- if peek_operation(entry) != Operation::SendMessages {
- continue;
+ fn accept_commit_phase(
+ &mut self,
+ result: crate::io::PartitionIoResult,
+ ) -> Result<(), IggyError> {
+ let phase = self
+ .pending_commit
+ .as_ref()
+ .ok_or(IggyError::CannotWriteToFile)?
+ .phase;
+ match (phase, result) {
+ (CommitPhase::Materialize, result) => self.accept_materialization_phase(result),
+ (CommitPhase::OffsetSubmitted(plan), result) => {
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::OffsetReady(plan);
+ let effect = self.accept_consumer_offset_effect(plan, result)?;
+ self.accept_committed_offset_effect(effect)
}
- let batch = decode_prepare_slice_trusted(entry.as_slice())
- .map_err(|_| IggyError::InvalidCommand)?;
- if batch.message_count() == 0 {
- continue;
+ (
+ CommitPhase::SegmentDirectorySubmitted,
+ crate::io::PartitionIoResult::SegmentDirectory(outcome),
+ ) => {
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::SegmentDirectory;
+ outcome.map_err(|_| IggyError::CannotSyncFile)?;
+ self.segment_names_dirty.set(false);
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::OffsetDirectories;
+ Ok(())
}
- if peek_op(entry) >= from_op {
- rewind_next_offset = Some(
- rewind_next_offset.map_or(batch.header.base_offset, |offset: u64| {
- offset.min(batch.header.base_offset)
- }),
- );
- continue;
+ (
+ CommitPhase::OffsetDirectoriesSubmitted,
+ crate::io::PartitionIoResult::OffsetDirectories(result),
+ ) => {
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::OffsetDirectories;
+ self.accept_consumer_offset_directories(&result);
+ if let Some((op, operation)) = result
+ .failed
+ .iter()
+ .zip(&self.consumer_offset_dirs_touched)
+ .find_map(|(failed, touched)| if *failed { touched.get() } else { None })
+ {
+ self.fatal = Some(FatalCommit {
+ namespace_raw: self.namespace().inner(),
+ op,
+ operation: Some(operation),
+ });
+ return Err(IggyError::CannotSyncFile);
+ }
+ self.pending_commit
+ .as_mut()
+ .expect("selected commit exists")
+ .phase = CommitPhase::Publish;
+ Ok(())
}
- accumulate_committed_info(
- &mut retained_info,
- batch.header.base_offset,
- batch.header.base_timestamp,
- batch.header.total_size() as u64,
- batch.message_count(),
- );
- retained_next_offset = retained_next_offset.max(
- batch
- .header
- .base_offset
- .saturating_add(u64::from(batch.message_count())),
- );
+ _ => Err(IggyError::CannotWriteToFile),
}
+ }
- let active = self.log.active_segment();
- let active_size = active.size.as_bytes_u64();
- let durable_next_offset = if active_size == 0 {
- active.start_offset
- } else {
- active.end_offset.saturating_add(1)
- };
- let minimum_next_offset = durable_next_offset
- .max(retained_next_offset)
- .max(
- self.recovered_durable_offset
- .map_or(0, |offset| offset.saturating_add(1)),
- )
- .max(self.installed_frontier.unwrap_or(0));
- let restored_position = active_size
- .checked_add(retained_info.size.as_bytes_u64())
- .ok_or(IggyError::CannotAppendMessage)?;
+ fn accept_committed_offset_effect(
+ &mut self,
+ effect: ConsumerOffsetEffect,
+ ) -> Result<(), IggyError> {
+ let pending = self
+ .pending_commit
+ .as_mut()
+ .ok_or(IggyError::CannotWriteToFile)?;
+ let header = pending
+ .headers
+ .get(pending.next_entry)
+ .ok_or(IggyError::CannotWriteToFile)?;
if let Some(persistence) = &self.persistence {
- persistence.truncate_from(from_op);
- self.start_persistence();
- persistence
- .drain_with_timeout()
- .await
- .map_err(|_| IggyError::CannotSyncFile)?;
+ persistence.mark_offset_dirty(
+ effect.pending.kind.index(),
+ effect.pending.consumer_id,
+ effect.state.is_some(),
+ );
}
- let removed = self
- .log
- .journal()
- .inner
- .truncate_from(from_op)
- .await
- .map_err(|_| IggyError::CannotAppendMessage)?;
+ self.consumer_offset_dirs_touched[effect.pending.kind.index()]
+ .set(Some((header.op, header.operation)));
+ pending.offset_effects.push(effect);
+ pending.next_entry += 1;
+ pending.phase = CommitPhase::Offsets;
+ Ok(())
+ }
- self.log.journal_mut().info = retained_info;
- self.log.active_segment_mut().current_position = restored_position;
- if let Some(next_offset) = rewind_next_offset {
- let next_offset = next_offset.max(minimum_next_offset);
- self.dirty_offset
- .store(next_offset.saturating_sub(1), Ordering::Relaxed);
- // The APPEND bit follows the rewound counter. The committed bit does
- // not: this drops an uncommitted suffix, which by definition names
- // nothing that ever committed, so raising it here would make a
- // truncation the event that publishes offsets no quorum agreed on.
- // A rewind to zero is the exception -- nothing is left at all.
- self.offset_space.append_live = next_offset > 0;
- if next_offset == 0 {
- self.offset_space.committed_seeded = false;
- }
+ fn fail_pending_commit(&mut self, error: &IggyError) {
+ warn!(namespace_raw = self.namespace().inner(), %error, "committed partition phase failed");
+ if self.fatal.is_none()
+ && let Some(pending) = &self.pending_commit
+ && let Some(header) = pending
+ .headers
+ .get(pending.next_entry)
+ .or_else(|| pending.headers.last())
+ {
+ self.fence_failed_commit(*header);
}
- self.consensus.note_journal_mutation();
- let commit_max = self.consensus.commit_max();
- self.pending_consumer_offset_commits
- .retain(|op, _| *op < from_op || *op <= commit_max);
- self.offset_reservations_need_resync.set(true);
- Ok(removed)
}
- async fn commit_messages(
+ fn select_commit(
&mut self,
+ headers: Vec<PrepareHeader>,
config: &PartitionsConfig,
- through_op: u64,
- ) -> Result<bool, IggyError> {
- #[cfg(any(test, feature = "fault-injection"))]
- if std::mem::take(&mut self.injected_commit_failure) {
- return Err(IggyError::CannotSaveMessagesToSegment);
+ send_client_replies: bool,
+ ) -> bool {
+ let applied_floor = self.consensus.commit_min();
+ if self.transition.is_some()
+ || self.consensus.is_transferring()
+ || self.io_active.is_retiring()
+ || !self.io_replies.is_empty()
+ || headers
+ .first()
+ .is_none_or(|first| Some(first.op) != applied_floor.checked_add(1))
+ || headers
+ .windows(2)
+ .any(|pair| pair[0].op.checked_add(1) != Some(pair[1].op))
+ {
+ return false;
}
- self.commit_messages_inner(
- config,
- self.consensus.replica_count() == 1 && self.durability().is_persisted(),
- through_op,
- )
- .await
+ let from_pipeline = self.consensus.pipeline_head_header().as_ref() == headers.first();
+ let journal_headers = headers
+ .iter()
+ .map(|header| self.log.journal().inner.header_by_op(header.op))
+ .collect();
+ let batch_stats = self.resolve_committed_batch_stats(&headers);
+ let Some(retirements) = self.validate_committed_entries(&headers, &batch_stats) else {
+ return false;
+ };
+ let has_sends = headers
+ .iter()
+ .any(|header| header.operation == Operation::SendMessages);
+ let flush_due = if has_sends {
+ self.should_persist_messages(config)
+ || (self.consensus.replica_count() == 1 && self.durability().is_persisted())
+ } else {
+ self.control_ops_due(config)
+ };
+ // A Replicated reply needs no local file. Under a dispatcher its flush
+ // runs as maintenance, so the reply never waits for this partition's
+ // I/O slot, which a checkpoint phase can hold for tens of milliseconds.
+ let materialize =
+ flush_due && (self.durability().is_persisted() || !self.has_io_dispatcher());
+ if flush_due
+ && !materialize
+ && let Some(last) = headers.last()
+ {
+ self.defer_flush(last.op);
+ self.notify_io();
+ }
+ self.pending_commit = Some(PendingCommit {
+ headers,
+ journal_headers,
+ batch_stats,
+ retirements,
+ offset_effects: Vec::new(),
+ incarnation: self.incarnation,
+ poll_history: self.poll_history,
+ applied_floor,
+ from_pipeline,
+ send_client_replies,
+ materialize,
+ phase: CommitPhase::Start,
+ next_entry: 0,
+ evictable: 0,
+ });
+ for touched in &self.consumer_offset_dirs_touched {
+ touched.set(None);
+ }
+ true
}
- /// Flush the committed journal prefix to segment storage regardless of
- /// the `messages_required_to_save` thresholds.
- ///
- /// Shutdown-path counterpart of the commit-time persist gate: a graceful
- /// stop must not lose committed messages that were still resident in the
- /// in-memory journal (consumer offsets are persisted eagerly, so losing
- /// the messages would fail recovery with an offset ahead of the data).
- ///
- /// # Errors
- ///
- /// Returns [`IggyError`] when writing the committed batches or their
- /// index entries to segment storage fails.
- pub async fn flush_committed_messages(
- &mut self,
- config: &PartitionsConfig,
- ) -> Result<(), IggyError> {
- if let Some(persistence) = self
- .persistence
- .as_ref()
- .filter(|persistence| persistence.segment_checkpoint().is_some())
+ fn pending_commit_is_current(&self) -> bool {
+ let Some(pending) = &self.pending_commit else {
+ return false;
+ };
+ if pending.incarnation != self.incarnation
+ || pending.poll_history != self.poll_history
+ || pending.applied_floor != self.consensus.commit_min()
{
- // Recovery only keeps segment bytes covered by the published WAL frontier.
- persistence.sync();
- self.start_persistence();
- // Shutdown and transfer must finish the flush, but never drain while
- // holding the lock used to append new messages.
- persistence.drain_with_timeout().await.map_err(|error| {
- warn!(%error, "cannot flush committed partition messages");
- IggyError::CannotSyncFile
- })?;
+ return false;
}
- if self
- .commit_messages_inner(config, true, self.consensus.commit_max())
- .await?
+ if pending
+ .headers
+ .iter()
+ .zip(&pending.journal_headers)
+ .any(|(header, captured)| self.log.journal().inner.header_by_op(header.op) != *captured)
{
- Ok(())
- } else {
- Err(IggyError::CannotSyncFile)
+ return false;
}
+ !pending.from_pipeline
+ || self.consensus.with_pipeline(|pipeline| {
+ pending.headers.iter().all(|header| {
+ pipeline
+ .entry_by_op(header.op)
+ .is_some_and(|entry| entry.header == *header)
+ })
+ })
}
- fn should_persist_messages(&self, config: &PartitionsConfig) -> bool {
- let journal_info = self.log.journal().info;
- // The existing thresholds include both committed and uncommitted batches.
- let messages_due = journal_info.messages_count > 0
- && (self.log.active_segment().is_full()
- || journal_info.messages_count >= self.effective_messages_required_to_save(config)
- || journal_info.size.as_bytes_u64()
- >= self.effective_size_of_messages_required_to_save(config));
- messages_due || self.control_ops_due(config)
- }
-
- /// Consumer offset ops count toward no message threshold, yet the flush
- /// is the journal's only eviction, so they get the message-count bound of
- /// their own: without it a consume-only partition keeps one auto-commit
- /// op per poll resident until a producer happens to trip a flush.
- fn control_ops_due(&self, config: &PartitionsConfig) -> bool {
- self.log.journal().inner.resident_control_ops()
- >= self.effective_messages_required_to_save(config) as usize
+ fn fence_failed_commit(&mut self, header: PrepareHeader) {
+ error!(
+ namespace_raw = self.namespace().inner(),
+ op = header.op, operation = ?header.operation,
+ "partition local commit failed for a cluster-committed operation"
+ );
+ self.fatal = Some(FatalCommit {
+ namespace_raw: self.namespace().inner(),
+ op: header.op,
+ operation: Some(header.operation),
+ });
}
- /// Returns false while the requested physical prefix is still pending in the WAL.
#[allow(clippy::too_many_lines)]
- async fn commit_messages_inner(
- &mut self,
- config: &PartitionsConfig,
- force: bool,
- through_op: u64,
- ) -> Result<bool, IggyError> {
- let write_lock = self.write_lock.clone();
- let _guard = write_lock.lock().await;
-
- if self.log.journal().inner.is_empty() {
- if force {
- tracing::info!(
- target: "iggy.partitions.diag",
- namespace_raw = self.namespace().inner(),
- "forced flush: journal is empty, nothing to persist"
- );
- }
- return Ok(true);
- }
-
- if !force && !self.should_persist_messages(config) {
- return Ok(true);
+ fn publish_pending_commit(&mut self) -> Option<Vec<(PipelineEntry, Option<CachedReply>)>> {
+ if !self.pending_commit_is_current() {
+ self.pending_commit = None;
+ return None;
}
+ let pending = self.pending_commit.take()?;
+ let entries = if pending.from_pipeline {
+ self.consensus.with_pipeline_mut(|pipeline| {
+ let mut entries = Vec::with_capacity(pending.headers.len());
+ for _ in &pending.headers {
+ entries.push(pipeline.pop()?);
+ }
+ Some(entries)
+ })
+ } else {
+ Some(
+ pending
+ .headers
+ .iter()
+ .copied()
+ .map(PipelineEntry::new)
+ .collect(),
+ )
+ };
+ let Some(entries) = entries else {
+ self.fence_failed_commit(*pending.headers.first()?);
+ return None;
+ };
- // Read (do NOT yet evict) ONLY the committed prefix (op <= commit_max,
- // gap-stopped). A backup journals replicated prepares ahead of the
- // commit frontier; flushing the uncommitted tail would write
- // per-replica-timing bytes to its segment (cross-replica divergence) and
- // drop the headers those ops need when their own commit later lands
- // (commit_min wedge). Eviction is deferred until the bytes are durable,
- // so a persist failure retains the prefix rather than losing a committed
- // batch (a live-process I/O fault only; the in-memory journal does not
- // survive a crash). Only the transfer-offer flush survives to re-read it;
- // the commit path panics the shard pump instead. All segment range /
- // stats / durable-offset accounting below is computed from the committed
- // entries, not the resident-journal snapshot above.
- let commit_max = self.consensus.commit_max().min(through_op);
- let committed_entries = self.log.journal().inner.committed_prefix(commit_max);
- if committed_entries.is_empty() {
- if force {
- tracing::info!(
- target: "iggy.partitions.diag",
- namespace_raw = self.namespace().inner(),
- commit_max,
- journal_messages = self.log.journal().info.messages_count,
- "forced flush: no committed entries resident"
- );
- }
- return Ok(true);
+ for effect in &pending.offset_effects {
+ self.publish_consumer_offset_effect(*effect);
+ self.apply_consumer_offset_commit(effect.pending);
}
- if let Some(persistence) = self
- .persistence
- .as_ref()
- .filter(|persistence| persistence.segment_checkpoint().is_some())
+ let mut retirements = pending.retirements.into_iter().peekable();
+ let mut committed = Vec::with_capacity(entries.len());
+ for ((entry, header), batch_stats) in entries
+ .into_iter()
+ .zip(&pending.headers)
+ .zip(&pending.batch_stats)
{
- if persistence.failure().is_some() {
- return Err(IggyError::CannotSyncFile);
- }
- // Check the entire flush before evicting any chunk. A partial pending
- // flush could evict headers that the commit walk still needs to retry.
- if committed_entries
- .iter()
- .rev()
- .find(|entry| {
- peek_operation(entry) == Operation::SendMessages
- && peek_op(entry) > self.purge_floor_op
- })
- .is_some_and(|entry| {
- if self.durability().is_persisted() {
- !persistence.is_durable_through(peek_op(entry))
- } else {
- !persistence.is_written_through(peek_op(entry))
- }
- })
- {
- return Ok(false);
+ if let Some(batch_stats) = batch_stats.filter(|_| header.op > self.purge_floor_op) {
+ self.publish_committed_batch_stats(batch_stats);
}
- }
- // Persist the prefix in segment-sized chunks: a segment seals on the
- // first flush whose committed bytes reach OR EXCEED `max_size`, no
- // matter how many entries this flush happens to cover. The batch that
- // crosses the cap is appended whole, so a sealed segment lands
- // anywhere in `[max_size, max_size + one maximum batch)` and never
- // exactly at `max_size`. A backup commits in bursts behind the
- // primary, so any grouping- or timing-sensitive roll rule
- // (like keying rotation on the journal-position `is_full` above)
- // seals segments at per-replica offsets, and the offset-keyed segment
- // GC staged by the reconciler never converges across the cluster.
- let max_segment_size = self.log.active_segment().max_size.as_bytes_u64();
- let mut entries = committed_entries.into_iter().peekable();
- let mut durable_offset = None;
- // Entries whose bytes are durable but which are still resident in the
- // journal. Evicted ONCE after the loop: `evict_prefix` drains and
- // re-appends the whole retained tail, so a per-chunk call would
- // re-walk that tail once per segment crossed, quadratic in the flush
- // span -- all under the partition write lock. On an error mid-flush
- // the accumulated prefix is evicted before propagating, so any later
- // flush attempt re-reads only what did not land.
- let mut evictable = 0usize;
- let mut skipped_control_ops = 0usize;
- while entries.peek().is_some() {
- // A recovered active segment can already sit at or past the cap
- // (crash between persist and rotation); seal it before appending.
- if self.log.active_segment().size.as_bytes_u64() >= max_segment_size
- && let Err(error) = self.rotate_segment(config).await
- {
- self.evict_committed_prefix(evictable).await;
- return Err(error);
+ if let Some(offset) = self.pending_consumer_offset_commits.remove(&header.op) {
+ self.refresh_consumer_offset_reservation(offset.kind, offset.consumer_id);
}
-
- let (frozen_batches, index_bytes, flush_index, batch_count, committed_info, chunk_len) = {
- let segment = self.log.active_segment();
- let mut file_position = segment.size.as_bytes_u64();
- let persisted_end = if file_position == 0 {
- segment.start_offset.checked_sub(1)
- } else {
- Some(segment.end_offset)
+ if let Some((_, retirement)) = retirements.next_if(|(op, _)| *op == header.op) {
+ for identity in retirement.identities {
+ self.dedup
+ .forget_session(identity.client_id, identity.session);
+ self.required_metadata_frontier = self
+ .required_metadata_frontier
+ .max(identity.metadata_watermark);
}
- .max(self.recovered_durable_offset);
- let mut flush_index = None;
- let mut frozen = Vec::with_capacity(entries.len());
- let mut batch_count = 0u32;
- let mut committed_info = JournalInfo::default();
- let mut chunk_len = 0usize;
-
- for entry in entries.by_ref() {
- chunk_len += 1;
- // Consumer-offset ops are journaled in the same prefix but carry
- // no segment bytes; they were applied when staged, so skip them.
- if peek_operation(&entry) != Operation::SendMessages {
- skipped_control_ops += 1;
- continue;
- }
- // Purge floor: a pre-purge batch committing after the
- // purge must not flush its (purged) bytes into the fresh
- // segment. It still counts into `chunk_len`, so it joins
- // the evictable prefix and commit_min advances normally.
- if peek_op(&entry) <= self.purge_floor_op {
- continue;
- }
- // Resident committed SendMessages entry: this node stamped it
- // in `append_messages` (recomputing the batch checksum over these
- // exact bytes), so a validating re-decode would only re-hash ~1
- // MiB to confirm our own write. Trust the structural decode; the
- // batch-checksum recompute belongs at network ingress (repair
- // validation + the follower receive gate), not on locally-stamped
- // bytes. Guard the invariant for a future disk read-back path that
- // could make decode fallible.
- let Ok(batch) = decode_prepare_slice_trusted(entry.as_slice()) else {
- tracing::error!(
- target: "iggy.partitions.diag",
- namespace_raw = self.namespace().inner(),
- entry_len = entry.as_slice().len(),
- "resident committed SendMessages entry failed to decode"
- );
- continue;
- };
- let message_count = batch.message_count();
- if message_count == 0 {
- continue;
- }
- // Flush can run ahead of the bounded commit walk. Repair
- // may re-journal an evicted batch above commit_min even
- // after this process persisted it. Include the current
- // segment frontier, not just the boot recovery frontier,
- // so that replay cannot append a second copy.
- let batch_end = batch.header.base_offset + u64::from(message_count) - 1;
- if let Some(durable) = persisted_end
- && batch_end <= durable
- {
- continue;
- }
-
- if flush_index.is_none() {
- // Record only; the in-mem cache insert is deferred until the
- // batch + index are durable (see post-persist below).
- flush_index = Some(crate::iggy_index::IggyIndex::new(
- batch.header.base_offset,
- batch.header.base_timestamp,
- file_position,
- ));
- }
- file_position += batch.header.total_size() as u64;
- batch_count += message_count;
- accumulate_committed_info(
- &mut committed_info,
- batch.header.base_offset,
- batch.header.base_timestamp,
- batch.header.total_size() as u64,
- message_count,
+ }
+ let reply = match self.commit_receipt(
+ &entry,
+ header,
+ *batch_stats,
+ pending.send_client_replies,
+ ) {
+ Ok(reply) => reply,
+ Err(error) => {
+ error!(
+ namespace_raw = self.namespace().inner(),
+ op = header.op,
+ %error,
+ "cannot retain committed partition receipt"
);
- frozen.push(entry);
- if file_position >= max_segment_size {
- break;
- }
+ self.fence_failed_commit(*header);
+ return None;
}
-
- let index_bytes = flush_index
- .as_ref()
- .map(crate::iggy_index::IggyIndexCache::serialize);
-
- (
- frozen,
- index_bytes,
- flush_index,
- batch_count,
- committed_info,
- chunk_len,
- )
};
-
- // No committed SendMessages batch was resident in this chunk (e.g.
- // a committed consumer-offset run that is not persisted to a
- // segment). Nothing to flush; no segment bytes are at risk, so the
- // entries just join the evictable prefix.
- let Some(index_bytes) = index_bytes else {
- evictable += chunk_len;
- continue;
+ self.consensus.advance_commit_min(header.op);
+ let event = CommitLogEvent {
+ replica: ReplicaLogContext::from_consensus(&self.consensus, PlaneKind::Partitions),
+ op: header.op,
+ client_id: header.client,
+ request_id: header.request,
+ operation: header.operation,
+ pipeline_depth: self.consensus.pipeline_len(),
};
-
- // Persist BEFORE eviction so a write failure leaves the rest of the
- // committed prefix resident instead of dropping it. On the commit
- // path a failure fences the partition and stops the shard pump; the
- // server then shuts down non-zero after one final best-effort flush
- // of every partition. A shutdown-flush failure warns and moves to
- // the next namespace, while the transfer offer turns it into
- // `FlushFailed`. Recovery reopens each writer at the on-disk length
- // boot recovery validated.
- //
- // The ordering therefore buys a non-corrupting failure, not an
- // in-process one. Write cursors advance only once both the batch and the
- // index are durable, so a later attempt, including the final
- // shutdown flush, rewrites the same positions instead of appending
- // a duplicate. Chunks already durable are evicted before the error
- // propagates, so it cannot re-read them and write them past a
- // rotation.
- if let Err(error) = self
- .persist_frozen_batches_to_disk(frozen_batches, index_bytes, batch_count)
- .await
- {
- self.evict_committed_prefix(evictable).await;
- return Err(error);
- }
- if let Some(persistence) = &self.persistence {
- persistence.mark_segment_dirty(self.log.active_segment().start_offset);
- }
- // Insert the flushed sparse-index entry into the in-mem cache only now
- // that the batch + index are durable. Inserting in the build loop (before
- // persist) re-inserts a duplicate on the next flush after a persist
- // failure, which re-reads the same prefix. The active segment has not
- // rotated yet, so this targets the segment that received the batches.
- if let Some(index) = flush_index {
- self.log.ensure_indexes();
- let indexes = self.log.active_indexes_mut().expect("indexes must exist");
- indexes.insert(index.offset, index.timestamp, index.position);
- }
- evictable += chunk_len;
-
- // Stamp range metadata on the segment that received the batches
- // BEFORE rotating: rotation seals it and derives the next segment's
- // start offset from `end_offset`, so updating after rotation would
- // tag the fresh segment with the old range and shift every
- // subsequent segment boundary off the file contents.
- let segment_index = self.log.segments().len() - 1;
- let segment = &mut self.log.segments_mut()[segment_index];
- if segment.start_timestamp == 0 && committed_info.first_timestamp != 0 {
- segment.start_timestamp = committed_info.first_timestamp;
- }
- segment.end_timestamp = committed_info.end_timestamp;
- segment.max_timestamp = segment.max_timestamp.max(committed_info.max_timestamp);
- segment.end_offset = committed_info.current_offset;
- durable_offset = Some(committed_info.current_offset);
-
- // Seal eagerly once the committed bytes cross the cap so the
- // segment becomes removable (GC skips the active segment) without
- // waiting for the next flush.
- if self.log.active_segment().size.as_bytes_u64() >= max_segment_size
- && let Err(error) = self.rotate_segment(config).await
- {
- self.evict_committed_prefix(evictable).await;
- return Err(error);
- }
- }
- self.evict_committed_prefix(evictable).await;
- if force && skipped_control_ops > 0 {
- tracing::info!(
- target: "iggy.partitions.diag",
- namespace_raw = self.namespace().inner(),
- skipped_control_ops,
- "forced flush: evicted non-send entries without segment bytes"
+ emit_sim_event(SimEventKind::OperationCommitted, &event);
+ emit_namespace_progress_event(
+ SimEventKind::NamespaceProgressUpdated,
+ &event.replica,
+ header.op,
+ event.pipeline_depth,
);
+ committed.push((entry, reply));
}
-
- // Aggregate stats (`messages_count`/`size_bytes`) advance at commit in
- // `commit_partition_entry`, not here: this persist path is threshold-
- // gated, so counting here would leave the stats lagging the visible
- // offset until a flush and would double-count once it fires.
- if let Some(durable_offset) = durable_offset {
- self.note_committed_seeded();
- self.offset.store(durable_offset, Ordering::Release);
- self.stats.set_current_offset(durable_offset);
- }
- Ok(true)
- }
-
- /// Evict the committed prefix (the `count` front entries read by
- /// `committed_prefix`) and reset `journal.info` to reflect only the
- /// uncommitted tail left resident, so the next persist threshold counts that
- /// tail alone. Call once the prefix is durable, or when there is nothing to
- /// persist. The retained tail's accounting is folded from the meta
- /// `evict_prefix` surfaced during its re-append, so the tail is not decoded
- /// a second time.
- async fn evict_committed_prefix(&mut self, count: usize) {
- if count == 0 {
- return;
+ for effect in &pending.offset_effects {
+ self.refresh_consumer_offset_reservation(
+ effect.pending.kind,
+ effect.pending.consumer_id,
+ );
}
- let retained = self.log.journal().inner.evict_prefix(count).await;
- // Eviction drains the ring and re-appends the retained tail, so the slots
- // a suffix snapshot describes move under it without the head or the
- // commit point changing.
- self.consensus.note_journal_mutation();
- let mut retained_info = JournalInfo::default();
- for (entry, meta) in &retained {
- // Purge floor: a retained pre-purge batch must not fold its
- // accounting back into `journal.info`, or the info would re-adopt
- // a pre-purge `current_offset` the purge just reset.
- if peek_op(entry) <= self.purge_floor_op {
- continue;
- }
- if let Some(meta) = meta {
- accumulate_committed_info(
- &mut retained_info,
- meta.base_offset,
- meta.base_timestamp,
- meta.total_size,
- meta.message_count,
- );
- }
+ self.consensus.sync_prepare_timeout();
+ self.evict_committed_prefix(pending.evictable);
+ for touched in &self.consumer_offset_dirs_touched {
+ touched.set(None);
}
- self.log.journal_mut().info = retained_info;
+ Some(committed)
}
- #[allow(clippy::too_many_lines)]
- async fn handle_committed_entries(
+ /// Every replica retains the committed result, so a promoted primary answers
+ /// a retry with the original append. Returns the reply this replica sends.
+ ///
+ /// # Errors
+ /// The receipt table refused the result, which fences the partition.
+ fn commit_receipt(
&mut self,
- drained: Vec<PipelineEntry>,
- config: &PartitionsConfig,
+ entry: &PipelineEntry,
+ header: &PrepareHeader,
+ batch_stats: Option<CommittedBatchStats>,
send_client_replies: bool,
- ) {
- let replica_id = self.consensus.replica();
- let namespace_raw = self.consensus.group();
- let Some(through_op) = drained.last().map(|entry| entry.header.op) else {
- return;
+ ) -> Result<Option<CachedReply>, ClientTableWireError> {
+ let auto_commit = is_auto_commit_client(header.client);
+ if auto_commit && !entry.has_reply_sender() {
+ return Ok(None);
+ }
+ let body = match header.operation {
+ Operation::SendMessages => send_messages_reply_body(header.group, batch_stats),
+ operation => committed_reply_body(operation),
};
- let drained_count = drained.len();
- if let (Some(first), Some(last)) = (drained.first(), drained.last()) {
- debug!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- replica_id,
- first_op = first.header.op,
- last_op = last.header.op,
- drained_count,
- "draining committed partition ops"
- );
+ let reply = build_reply_message(header, &body);
+ if auto_commit {
+ return Ok(Some(CachedReply::from(reply)));
}
+ let cached = self
+ .dedup
+ .commit_partition_reply(header.user_id, header.session, reply)?;
+ Ok(send_client_replies.then_some(cached))
+ }
- let mut failed_commit = false;
- // Capture receipt and retirement data before commit_messages evicts the
- // journal prefix. Missing data must fence the partition before apply;
- // an incomplete receipt cannot safely answer a retry.
- let committed_batch_stats = self.resolve_committed_batch_stats(&drained);
+ /// Captures each retirement before the walk evicts the journal prefix.
+ /// Missing retry protection or append results fence the partition before
+ /// apply, because an incomplete receipt cannot safely answer a retry.
+ fn validate_committed_entries(
+ &mut self,
+ headers: &[PrepareHeader],
+ batch_stats: &[Option<CommittedBatchStats>],
+ ) -> Option<Vec<(u64, RetireSessionsRequest)>> {
+ let namespace_raw = self.namespace().inner();
let mut retirements = Vec::new();
- for (entry, batch_stats) in drained.iter().zip(&committed_batch_stats) {
+ for (header, batch_stats) in headers.iter().zip(batch_stats) {
let configured_capacity =
(!self.dedup.capacity_committed()).then(|| self.dedup.capacity());
- let retirement = (entry.header.operation == Operation::RetireSession)
+ let retirement = (header.operation == Operation::RetireSession)
.then(|| {
- let prepare = self.log.journal().inner.repair_entry(entry.header.op)?;
+ let prepare = self.log.journal().inner.repair_entry(header.op)?;
RetireSessionsRequest::decode_from(
prepare
.as_slice()
@@ -6537,22 +9467,22 @@ where
.flatten();
if self
.dedup
- .commit_capacity(entry.header.retry_capacity as usize)
+ .commit_capacity(header.retry_capacity as usize)
.is_err()
- || (entry.header.operation == Operation::SendMessages && batch_stats.is_none())
- || (entry.header.operation == Operation::RetireSession && retirement.is_none())
+ || (header.operation == Operation::SendMessages && batch_stats.is_none())
+ || (header.operation == Operation::RetireSession && retirement.is_none())
{
error!(
namespace_raw,
- op = entry.header.op,
+ op = header.op,
"committed partition entry lacks valid retry protection or append result"
);
self.fatal = Some(FatalCommit {
namespace_raw,
- op: entry.header.op,
- operation: Some(entry.header.operation),
+ op: header.op,
+ operation: Some(header.operation),
});
- return;
+ return None;
}
if let Some(configured_capacity) = configured_capacity
&& configured_capacity != self.dedup.capacity()
@@ -6565,354 +9495,99 @@ where
);
}
if let Some(identity) = retirement {
- retirements.push((entry.header.op, identity));
- }
- }
- // Keep the threshold decision used before draining. An append during an
- // offset write or lock wait must not require an unchecked body mid-walk.
- let mut messages_committed = !self.durability().is_persisted()
- && self
- .persistence
- .as_ref()
- .is_some_and(|persistence| persistence.segment_checkpoint().is_some())
- && !self.should_persist_messages(config);
-
- // Apply the drained batch before advancing any op because directory
- // durability is shared by every delete in the batch. Until the sync
- // below succeeds, earlier applied entries intentionally remain above
- // commit_min and their replies and dedup folds stay owned by `drained`.
- // A failure at entry K fences the replica. Recovery replays the whole
- // unadvanced prefix idempotently, including entries applied before K.
- for cell in &self.consumer_offset_dirs_touched {
- cell.set(None);
- }
- for (entry, batch_stats) in drained.iter().zip(&committed_batch_stats) {
- let prepare_header = entry.header;
- if !self
- .commit_partition_entry(
- prepare_header,
- &mut messages_committed,
- batch_stats.filter(|_| prepare_header.op > self.purge_floor_op),
- &mut failed_commit,
- config,
- through_op,
- )
- .await
- {
- if !failed_commit {
- return;
- }
- // Local commit failed but cluster committed (op came from
- // drain_committable_prefix). Replica diverged, can't serve
- // reads.
- //
- // `continue` is unsafe: failed op popped, commit_min not
- // advanced; next advance_commit_min(op+1) would assert
- // op+1 == commit_min + 1, panics cryptically.
- //
- // Fatal, but NOT by panicking: this runs on the shard's
- // message pump, and `compio::runtime::spawn` swallows a panic
- // there, leaving every partition on the shard unable to
- // commit, tick or reply while the process still reports
- // healthy and answers on its other shards. Fence the
- // partition and stop draining; the shard's tick picks the
- // fault up and takes the server down through the ordinary
- // shutdown path, so the remaining partitions flush and the
- // exit code is non-zero. Operator restarts; recovery+repair
- // re-syncs.
- error!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- replica_id,
- namespace_raw,
- op = prepare_header.op,
- operation = ?prepare_header.operation,
- "partition local commit failed for a cluster-committed op; \
- replica is divergent, fencing the partition and shutting down"
- );
- self.fatal = Some(FatalCommit {
- namespace_raw,
- op: prepare_header.op,
- operation: Some(prepare_header.operation),
- });
- return;
- }
- }
-
- if messages_committed
- && self.consensus.replica_count() == 1
- && self.durability().is_persisted()
- && self.segment_names_dirty.get()
- && let Some(directory) = &self.partition_dir
- {
- if let Err(error) = crate::state_transfer::fsync_dir(directory).await {
- error!(%error, namespace_raw, "cannot publish persisted segment names");
- self.fatal = Some(FatalCommit {
- namespace_raw,
- op: drained
- .last()
- .map_or_else(|| self.consensus.commit_min(), |entry| entry.header.op),
- operation: Some(Operation::SendMessages),
- });
- return;
- }
-
- self.segment_names_dirty.set(false);
- }
-
- // Commit replies and the applied frontier must follow directory
- // durability. One sync per dirty kind covers the whole walk. A sync
- // failure is attributed to an operation of that kind in this walk.
- // Covered stores depend on previously dirty directory entries too.
- // Only a kind THIS walk uses can fence it: dirt left by a NoAck
- // request belongs to that request's kind, and fencing a walk that wrote
- // consumer offsets over the groups directory would take the node down
- // for a failure none of its ops caused.
- let touched: [_; ConsumerKind::COUNT] =
- std::array::from_fn(|index| self.consumer_offset_dirs_touched[index].replace(None));
- let failed = self
- .flush_consumer_offset_directories_for(touched.map(|entry| entry.is_some()))
- .await;
- if failed.iter().any(|failed| *failed) {
- let failed_entry = failed
- .iter()
- .zip(touched)
- .find_map(|(failed, entry)| if *failed { entry } else { None });
- error!(
- namespace_raw,
- failed_consumer = failed[0],
- failed_consumer_group = failed[1],
- failed_external_group = failed[2],
- touched_consumer = touched[0].is_some(),
- touched_consumer_group = touched[1].is_some(),
- touched_external_group = touched[2].is_some(),
- fences = failed_entry.is_some(),
- "consumer offset directory sync failed after committed operations"
- );
- if let Some((op, operation)) = failed_entry {
- self.fatal = Some(FatalCommit {
- namespace_raw,
- op,
- operation: Some(operation),
- });
- return;
- }
- }
-
- let mut retirements = retirements.into_iter().peekable();
- for (mut entry, batch_stats) in drained.into_iter().zip(committed_batch_stats) {
- let prepare_header = entry.header;
- if let Some((_, retirement)) = retirements.next_if(|(op, _)| *op == prepare_header.op) {
- for identity in retirement.identities {
- self.dedup
- .forget_session(identity.client_id, identity.session);
- self.required_metadata_frontier = self
- .required_metadata_frontier
- .max(identity.metadata_watermark);
- }
+ retirements.push((header.op, identity));
}
- // Fold the committed request into this group's dedup slice. Runs on
- // EVERY replica, not just the one that replies, so a promoted
- // primary can absorb a replay of what its predecessor committed.
- // Auto-commit ops carry the reserved sentinel client and no client
- // ever replays them.
- let reply_sender = entry.take_reply_sender();
- let auto_commit = is_auto_commit_client(prepare_header.client);
- let reply = if auto_commit && reply_sender.is_none() {
- None
- } else {
- let body = match prepare_header.operation {
- Operation::SendMessages => {
- send_messages_reply_body(prepare_header.group, batch_stats)
- }
- operation => committed_reply_body(operation),
- };
- let reply = build_reply_message(&prepare_header, &body);
- if auto_commit {
- Some(CachedReply::from(reply))
- } else {
- let cached = match self.dedup.commit_partition_reply(
- prepare_header.user_id,
- prepare_header.session,
- reply,
- ) {
- Ok(cached) => cached,
- Err(error) => {
- error!(namespace_raw, op = prepare_header.op, %error, "cannot retain committed partition receipt");
- self.fatal = Some(FatalCommit {
- namespace_raw,
- op: prepare_header.op,
- operation: Some(prepare_header.operation),
- });
- return;
- }
- };
- send_client_replies.then_some(cached)
- }
- };
- self.consensus.advance_commit_min(prepare_header.op);
-
- let pipeline_depth = self.consensus.pipeline_len();
- let event = CommitLogEvent {
- replica: ReplicaLogContext::from_consensus(&self.consensus, PlaneKind::Partitions),
- op: prepare_header.op,
- client_id: prepare_header.client,
- request_id: prepare_header.request,
- operation: prepare_header.operation,
- pipeline_depth,
- };
- emit_sim_event(SimEventKind::OperationCommitted, &event);
- emit_namespace_progress_event(
- SimEventKind::NamespaceProgressUpdated,
- &event.replica,
- prepare_header.op,
- pipeline_depth,
- );
+ }
+ Some(retirements)
+ }
+ async fn deliver_commit_replies(
+ &self,
+ committed: impl IntoIterator<Item = (PipelineEntry, Option<CachedReply>)>,
+ ) {
+ for (mut entry, reply) in committed {
let Some(reply) = reply else {
continue;
};
- // A server-generated auto-commit op (a poll's `auto_commit`,
- // replicated for failover) carries the reserved
- // `AUTO_COMMIT_CLIENT_ID`: no client ever waits on it, so skip the
- // reply. Emitting it would push an unrequested frame onto a real
- // client's lockstep reply stream if the sentinel ever routed there.
- if auto_commit {
- if let Some(sender) = reply_sender {
+ let header = entry.header;
+ if is_auto_commit_client(header.client) {
+ if let Some(sender) = entry.take_reply_sender() {
let _ = sender.send(reply.into_message());
}
- } else {
- emit_sim_event(SimEventKind::ClientReplyEmitted, &event);
-
- // An in-process waiter takes the reply instead of the bus: it
- // arrived as a `PartitionSubmit`, so `header.client` is the VSR
- // consensus id and carries no home-shard routing. The awaiting
- // shard owns the socket. A dropped receiver is ignored -- the
- // client recovers on its own read-timeout.
- //
- // Without a waiter the bus is tried, and for a TCP client it
- // cannot route the VSR id. That is the expected shape of every
- // op re-committed after a view change (the rebuilt pipeline
- // entries carry no sender), so it logs at debug: the original
- // waiter was cancelled by the view change and the client is
- // already on its read-timeout.
- if let Some(sender) = reply_sender {
- let _ = sender.send(reply.into_message());
- } else if let Err(error) = self
- .consensus
- .message_bus()
- .send_to_client(prepare_header.client, reply.into_wire_bytes())
- .await
- {
- tracing::debug!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- client = prepare_header.client,
- op = prepare_header.op,
- namespace_raw,
- %error,
- "client reply not routable by the bus; client will time out",
- );
- }
+ continue;
+ }
+ let event = CommitLogEvent {
+ replica: ReplicaLogContext::from_consensus(&self.consensus, PlaneKind::Partitions),
+ op: header.op,
+ client_id: header.client,
+ request_id: header.request,
+ operation: header.operation,
+ pipeline_depth: self.consensus.pipeline_len(),
+ };
+ emit_sim_event(SimEventKind::ClientReplyEmitted, &event);
+ if let Some(sender) = entry.take_reply_sender() {
+ let _ = sender.send(reply.into_message());
+ } else if let Err(error) = self
+ .consensus
+ .message_bus()
+ .send_to_client(header.client, reply.into_wire_bytes())
+ .await
+ {
+ debug!(
+ namespace_raw = self.namespace().inner(),
+ client = header.client, op = header.op, %error,
+ "client reply not routable by the bus; client will time out"
+ );
}
}
-
- if failed_commit {
- warn!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- replica_id,
- namespace_raw,
- "partition failed local commit handling for one or more ops"
- );
- }
-
- // Each commit frees one prepare slot, promote up to drained_count
- // buffered requests so the pipeline stays busy.
- self.drain_request_queue_into_prepares(drained_count).await;
}
- /// Sync the dirty offset directories selected by `kinds`, indexed by
- /// [`ConsumerKind::index`], then their parents. Returns which of them
- /// failed. A failed directory stays dirty for the next attempt.
+ #[cfg(test)]
async fn flush_consumer_offset_directories_for(
&self,
kinds: [bool; ConsumerKind::COUNT],
) -> [bool; ConsumerKind::COUNT] {
- let mut failed = [false; ConsumerKind::COUNT];
- let mut parents: [Option<&str>; ConsumerKind::COUNT] = [None; ConsumerKind::COUNT];
- for kind in ConsumerKind::ALL {
- let index = kind.index();
- if !kinds[index] || !self.consumer_offset_dirs_dirty[index].get() {
- continue;
+ let job = self.prepare_consumer_offset_directories(kinds);
+ let result = job.execute().await;
+ self.accept_consumer_offset_directories(&result);
+ result.failed
+ }
+
+ /// Selects the dirty offset directories of `kinds`, indexed by
+ /// [`ConsumerKind::index`].
+ fn prepare_consumer_offset_directories(
+ &self,
+ kinds: [bool; ConsumerKind::COUNT],
+ ) -> crate::io::OffsetDirectoriesIoJob {
+ let attempted = std::array::from_fn(|index| {
+ kinds[index] && self.consumer_offset_dirs_dirty[index].get()
+ });
+ let paths = std::array::from_fn(|index| {
+ if attempted[index] {
+ self.consumer_offsets_dir(ConsumerKind::ALL[index])
+ .map(str::to_owned)
+ } else {
+ None
}
+ });
+ crate::io::OffsetDirectoriesIoJob {
+ paths,
+ attempted,
#[cfg(test)]
- if self.consumer_offset_dir_sync_fault.get() == Some(index) {
- failed[index] = true;
- continue;
- }
- let Some(dir) = self.consumer_offsets_dir(kind) else {
- self.consumer_offset_dirs_dirty[index].set(false);
- continue;
- };
- match crate::state_transfer::fsync_dir(dir).await {
- Ok(()) => {}
- Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
- // The directory disappeared after the final unlink.
- // There is no remaining dirent whose durability needs
- // proving.
- }
- Err(error) => {
- warn!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- replica_id = self.consensus.replica(),
- namespace_raw = self.namespace().inner(),
- path = dir,
- error_kind = ?error.kind(),
- %error,
- "consumer offset directory sync failed"
- );
- failed[index] = true;
- continue;
- }
- }
- parents[index] = std::path::Path::new(dir)
- .parent()
- .and_then(std::path::Path::to_str);
- if parents[index].is_none() {
- self.finish_consumer_offset_dir_sync(index);
- }
+ fault: self.consumer_offset_dir_sync_fault.get(),
}
- // The kind directory is itself an entry in offsets/. Its publication
- // must survive before a local persisted offset reply. Every kind shares
- // that parent, so each distinct parent syncs once, after all of them.
- for index in 0..ConsumerKind::COUNT {
- let Some(parent) = parents[index] else {
- continue;
- };
- if parents[..index].contains(&Some(parent)) {
- continue;
- }
- let synced = match crate::state_transfer::fsync_dir(parent).await {
- Ok(()) => true,
- Err(error) => {
- warn!(%error, path = parent, "consumer offset parent directory sync failed");
- false
- }
- };
- for sibling in index..ConsumerKind::COUNT {
- if parents[sibling] != Some(parent) {
- continue;
- }
- if synced {
- self.finish_consumer_offset_dir_sync(sibling);
- } else {
- failed[sibling] = true;
- }
+ }
+
+ /// A failed directory stays dirty for the next attempt.
+ fn accept_consumer_offset_directories(&self, result: &crate::io::OffsetDirectoriesIoResult) {
+ for index in 0..result.attempted.len() {
+ if result.synced[index] {
+ self.finish_consumer_offset_dir_sync(index);
+ } else if result.attempted[index] && !result.failed[index] {
+ // A kind without a directory has no entry to make durable.
+ self.consumer_offset_dirs_dirty[index].set(false);
}
}
- failed
}
fn finish_consumer_offset_dir_sync(&self, index: usize) {
@@ -6939,17 +9614,17 @@ where
/// one batch's offsets to another entry's reply.
fn resolve_committed_batch_stats(
&self,
- drained: &[PipelineEntry],
+ drained: &[PrepareHeader],
) -> Vec<Option<CommittedBatchStats>> {
drained
.iter()
.map(|entry| {
- if entry.header.operation != Operation::SendMessages {
+ if entry.operation != Operation::SendMessages {
return None;
}
// Receipts describe the original append even after purge removes
// its messages. Only the state-apply path filters purged offsets.
- match self.committed_batch_stats_for_prepare(&entry.header) {
+ match self.committed_batch_stats_for_prepare(entry) {
Ok(batch_stats) => batch_stats,
Err(error) => {
warn!(
@@ -6957,8 +9632,8 @@ where
plane = "partitions",
replica_id = self.consensus.replica(),
namespace_raw = self.namespace().inner(),
- op = entry.header.op,
- operation = ?entry.header.operation,
+ op = entry.op,
+ operation = ?entry.operation,
%error,
"failed to resolve committed visible offset for partition entry"
);
@@ -6969,137 +9644,33 @@ where
.collect()
}
- async fn commit_partition_entry(
- &mut self,
- prepare_header: PrepareHeader,
- messages_committed: &mut bool,
- batch_stats: Option<CommittedBatchStats>,
- failed_commit: &mut bool,
- config: &PartitionsConfig,
- through_op: u64,
- ) -> bool {
- match prepare_header.operation {
- Operation::SendMessages => {
- if !*messages_committed {
- if self
- .commit_messages_for_entry(
- prepare_header,
- failed_commit,
- config,
- through_op,
- )
- .await
- != Some(true)
- {
- return false;
- }
- *messages_committed = true;
- }
-
- if let Some(batch_stats) = batch_stats {
- self.widest_batch_bytes
- .set(self.widest_batch_bytes.get().max(batch_stats.size_bytes));
- let end_offset = batch_stats.end_offset();
- // The committed counter now names data, which is what makes
- // it pollable and persistable. Outside the recovered-offset
- // guard below: that guard only skips re-counting stats a
- // previous life already persisted, and those offsets are
- // committed either way.
- self.note_committed_seeded();
- // A repaired batch at or below the boot-time recovered
- // durable offset was already counted (and persisted)
- // before the restart; skip it. Live traffic always sits
- // above the (immutable) line.
- if self
- .recovered_durable_offset
- .is_none_or(|durable| end_offset > durable)
- {
- self.offset.store(end_offset, Ordering::Release);
- self.stats.set_current_offset(end_offset);
- // Advance the aggregate stats with the visible offset. Disk
- // persistence is threshold-gated in `commit_messages`, which
- // must not also touch these counters or committed messages
- // would be double-counted once they flush.
- self.stats
- .increment_messages_count(u64::from(batch_stats.message_count));
- self.stats.increment_size_bytes(batch_stats.size_bytes);
- }
- }
- !*failed_commit
- }
- Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset => {
- if !self
- .commit_consumer_offset_entry(prepare_header, failed_commit)
- .await
- {
- return false;
- }
- // A consume-only walk holds no `SendMessages` entry, so the
- // control-op bound has to fire from here or the journal never
- // evicts. `Some(false)` (a message tail in front still pending
- // in the WAL) defers the flush to a later walk instead of
- // holding this op back: it carries no segment bytes, so its
- // commit does not depend on the flush.
- if !*messages_committed && self.control_ops_due(config) {
- if self
- .commit_messages_for_entry(
- prepare_header,
- failed_commit,
- config,
- through_op,
- )
- .await
- .is_none()
- {
- return false;
- }
- *messages_committed = true;
- }
- true
- }
- Operation::RetireSession => true,
- _ => {
... 14759 lines suppressed ...