You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "spetz (via GitHub)" <gi...@apache.org> on 2026/09/05 06:27:20 UTC
[PR] harden consumer offsets [iggy]
spetz opened a new pull request, #4063:
URL: https://github.com/apache/iggy/pull/4063
Every distinct consumer id stored one offset file and one map
entry per partition with no upper bound, so a partition could
accumulate durable state until the filesystem or memory gave out
and the replica fenced itself. Offset deletes acknowledged
locally could also free a primary slot while backups kept every
older generation, and follower-served auto-commit cursors could
be promoted into committed state across leadership changes.
Add a per-partition, per-kind admission limit enforced on the
primary before a new key enters consensus or touches disk.
Admission counts durable keys plus in-flight reservations, so
existing consumers keep updating at the limit while new keys
get a typed TooManyConsumerOffsets reply. Replicated partitions
now commit every offset store and delete through VSR regardless
of the acknowledgement byte, and consumer-group cleanup uses
replicated deletes, which keeps file counts identical on all
replicas. State transfer exports committed values only, and
recovery preserves state above a lowered limit with a warning.
Auto-commit hands its reservation to the partition pump through
an internal frame instead of a per-poll waiter, phantom cursors
are reclaimed on promotion, and the reconciler no longer runs a
full pass on every group offset advance. Missing consumer groups
return the existing not-found codes. The limit ships as
[partition] consumer_offsets_max with a default of 4096, and the
new error code is mirrored in the Go, Java, and Node SDKs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4063:
URL: https://github.com/apache/iggy/pull/4063#discussion_r3942247018
##########
core/sdk/src/tcp/tcp_client.rs:
##########
@@ -1324,7 +1324,13 @@ impl TcpClient {
// never reach the rest of the roster.
(next, true)
} else {
- return Err(IggyError::TransientNotAccepted);
+ // A one-node roster has nowhere else to walk while a
+ // freshly committed partition is still materialising.
+ // The server explicitly did not admit this request, so
+ // keep retrying the current endpoint within the existing
+ // overall deadline rather than surfacing a transient
+ // solely because the roster contains no alternative.
+ (current, false)
Review Comment:
warning: `RosterWalk::next()` returns `None` for any exhausted walk, not just a one-node roster, so on a 3-node cluster where every peer refuses we stop surfacing `TransientNotAccepted` and re-hammer the last endpoint. gate on roster size.
##########
core/sdk/src/websocket/websocket_client.rs:
##########
@@ -211,7 +211,10 @@ impl BinaryTransport for WebSocketClient {
} else if let Some(next) = roster_walk.as_mut().and_then(RosterWalk::next) {
(next, true)
} else {
Review Comment:
warning: same as the tcp client - this fires on any exhausted roster walk, not just a one-node roster, so a full-cluster refusal replays one endpoint instead of surfacing `TransientNotAccepted`. gate on roster size.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2686,130 +2917,96 @@ where
"consumer group offset key {} exceeds u32",
key.0
);
- narrowed
- .and_then(|id| self.persisted_offset_path(ConsumerKind::ConsumerGroup, id))
+ narrowed.and_then(|id| {
+ self.persisted_offset_path(ConsumerKind::ConsumerGroup, id)
+ .map(|path| (ConsumerKind::ConsumerGroup, id, path))
+ })
})
.collect();
guard.clear();
- paths.extend(strayed_offset_files(
- self.consumer_group_offsets_path.as_deref(),
- &offsets_wire.groups,
- ));
+ paths.extend(
+ strayed_offset_files(
+ self.consumer_group_offsets_path.as_deref(),
+ &offsets_wire.groups,
+ )
+ .into_iter()
+ .filter_map(|path| {
+ numeric_offset_id(&path).map(|id| (ConsumerKind::ConsumerGroup, id, path))
+ }),
+ );
paths
};
- for path in old_consumer_paths.into_iter().chain(old_group_paths) {
- if let Err(error) = delete_persisted_offset(&path).await {
- // Not fatal, but not silent either: a stranded file is an id
- // absent from the NEW table (matching ids get overwritten at
- // the same path), and boot resurrects it. Sharpest after a
- // purged origin ships `next_offset = 0`, where the clamp drops
- // every incoming entry and the whole old table survives while
- // the install still reports success.
- tracing::warn!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- namespace_raw = self.consensus().group(),
- path = %path,
- %error,
- "failed to unlink a superseded consumer-offset file during install"
- );
+ let mut offset_dirs_changed = [false; 2];
+ for (kind, consumer_id, path) in old_consumer_paths.into_iter().chain(old_group_paths) {
+ let removed = delete_persisted_offset(&path)
Review Comment:
warning: an unlinkable offset file makes install fail here, and converge hits the same dirent at line 3178 and fails too, so the partition fences and rebuilds forever. keep it non-fatal, like the purge path at `iggy_partition.rs:6410`.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2008,83 +2120,122 @@ where
// durably stored; the in-memory update is idempotent on replay
// because we look up by (kind, id).
self.persist_consumer_offset_commit(pending).await?;
- self.apply_consumer_offset_commit(pending)?;
+ self.apply_consumer_offset_commit(pending);
self.pending_consumer_offset_commits.remove(&op);
+ self.refresh_consumer_offset_reservation(pending.kind, pending.consumer_id);
Ok(())
}
async fn persist_consumer_offset_commit(
&self,
pending: PendingConsumerOffsetCommit,
) -> Result<(), IggyError> {
- let Some(path) = self.persisted_offset_path(pending.kind, pending.consumer_id) else {
- return Ok(());
- };
- let key = (pending.kind, pending.consumer_id);
+ let path = self.persisted_offset_path(pending.kind, pending.consumer_id);
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
match pending.mutation {
// A server auto-commit persists monotonically: its op offset can
// trail the durably-recorded value (disk-tier polls replicate in
// IO-completion order), so a plain overwrite would rewind the file
- // and re-deliver on restart. The `persisted_offsets` tracker keeps
+ // and re-deliver on restart. The durable offset tracker keeps
// the fold off the file: a covered offset skips the write, an
// advancing one blind-writes, and only a cold key (first commit
// after boot) reads the file once. Explicit client stores
// overwrite, so a deliberate offset reset still holds. Mirrors the
// in-memory `upsert_offset_max` vs `upsert_offset` split in the
// commit-apply.
PendingConsumerOffsetMutation::Upsert(offset) if pending.auto_commit => {
- let tracked = self.persisted_offsets.borrow().get(&key).copied();
- let persisted = match tracked {
- Some(high_water) if offset <= high_water => return Ok(()),
- Some(_) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- offset
+ let tracked = self
+ .durable_consumer_offsets
+ .get(pending.kind, pending.consumer_id);
+ let persisted_high_water = match (path.as_deref(), tracked) {
+ (None, _) => offset,
+ (Some(_), Some(state)) if offset <= state.persisted_high_water => {
+ state.persisted_high_water
}
- None => {
- persist_offset_max(&path, offset, self.consumer_offset_enforce_fsync)
- .await?
+ (Some(path), Some(state)) => {
+ let value = state.committed_offset.max(offset);
+ persist_offset(path, value, self.consumer_offset_enforce_fsync).await?;
+ value
+ }
+ (Some(path), None) => {
+ persist_offset_max(path, offset, self.consumer_offset_enforce_fsync).await?
}
};
- self.persisted_offsets.borrow_mut().insert(key, persisted);
+ self.durable_consumer_offsets.record_auto_commit(
+ pending.kind,
+ pending.consumer_id,
+ if tracked.is_none() {
+ persisted_high_water
+ } else {
+ offset
+ },
+ persisted_high_water,
+ );
+ if path.is_some() && self.consumer_offset_enforce_fsync {
+ self.mark_consumer_offset_dir_dirty(pending.kind);
+ }
+ capacity.clear_stranded(pending.consumer_id);
+ if pending.kind == ConsumerKind::ConsumerGroup && tracked.is_none() {
+ self.mark_consumer_group_offsets_need_reconcile();
+ }
Ok(())
}
PendingConsumerOffsetMutation::Upsert(offset) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- self.persisted_offsets.borrow_mut().insert(key, offset);
+ if let Some(path) = path.as_deref() {
+ persist_offset(path, offset, self.consumer_offset_enforce_fsync).await?;
+ }
+ let created = self.durable_consumer_offsets.record_explicit(
+ pending.kind,
+ pending.consumer_id,
+ offset,
+ offset,
+ );
+ if path.is_some() && self.consumer_offset_enforce_fsync {
+ self.mark_consumer_offset_dir_dirty(pending.kind);
+ }
+ capacity.clear_stranded(pending.consumer_id);
+ if pending.kind == ConsumerKind::ConsumerGroup && created {
+ self.mark_consumer_group_offsets_need_reconcile();
+ }
Ok(())
}
PendingConsumerOffsetMutation::Delete => {
- delete_persisted_offset(&path).await?;
- self.persisted_offsets.borrow_mut().remove(&key);
+ if let Some(path) = path.as_deref() {
+ if delete_persisted_offset(path).await? {
Review Comment:
warning: a committed delete whose unlink fails takes the process down via `FatalCommit`, and the file survives so boot undoes the delete anyway. strand it and return `Ok`, and suppress the map removal too or `snapshot_offset_kind` marks the partition inconsistent forever.
##########
core/sdk/src/tcp/tcp_client.rs:
##########
@@ -1324,7 +1324,13 @@ impl TcpClient {
// never reach the rest of the roster.
(next, true)
} else {
- return Err(IggyError::TransientNotAccepted);
+ // A one-node roster has nowhere else to walk while a
+ // freshly committed partition is still materialising.
+ // The server explicitly did not admit this request, so
+ // keep retrying the current endpoint within the existing
+ // overall deadline rather than surfacing a transient
+ // solely because the roster contains no alternative.
+ (current, false)
};
loop {
Review Comment:
warning: this keeps `routing_guard` held to the 30s deadline instead of dropping it when the walk exhausts, so other transient requests block at line 1284 and reconnect stalls on the same lock. drop the guard before re-issuing.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2008,83 +2120,122 @@ where
// durably stored; the in-memory update is idempotent on replay
// because we look up by (kind, id).
self.persist_consumer_offset_commit(pending).await?;
- self.apply_consumer_offset_commit(pending)?;
+ self.apply_consumer_offset_commit(pending);
self.pending_consumer_offset_commits.remove(&op);
+ self.refresh_consumer_offset_reservation(pending.kind, pending.consumer_id);
Ok(())
}
async fn persist_consumer_offset_commit(
&self,
pending: PendingConsumerOffsetCommit,
) -> Result<(), IggyError> {
- let Some(path) = self.persisted_offset_path(pending.kind, pending.consumer_id) else {
- return Ok(());
- };
- let key = (pending.kind, pending.consumer_id);
+ let path = self.persisted_offset_path(pending.kind, pending.consumer_id);
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
match pending.mutation {
// A server auto-commit persists monotonically: its op offset can
// trail the durably-recorded value (disk-tier polls replicate in
// IO-completion order), so a plain overwrite would rewind the file
- // and re-deliver on restart. The `persisted_offsets` tracker keeps
+ // and re-deliver on restart. The durable offset tracker keeps
// the fold off the file: a covered offset skips the write, an
// advancing one blind-writes, and only a cold key (first commit
// after boot) reads the file once. Explicit client stores
// overwrite, so a deliberate offset reset still holds. Mirrors the
// in-memory `upsert_offset_max` vs `upsert_offset` split in the
// commit-apply.
PendingConsumerOffsetMutation::Upsert(offset) if pending.auto_commit => {
- let tracked = self.persisted_offsets.borrow().get(&key).copied();
- let persisted = match tracked {
- Some(high_water) if offset <= high_water => return Ok(()),
- Some(_) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- offset
+ let tracked = self
+ .durable_consumer_offsets
+ .get(pending.kind, pending.consumer_id);
+ let persisted_high_water = match (path.as_deref(), tracked) {
+ (None, _) => offset,
+ (Some(_), Some(state)) if offset <= state.persisted_high_water => {
+ state.persisted_high_water
}
- None => {
- persist_offset_max(&path, offset, self.consumer_offset_enforce_fsync)
- .await?
+ (Some(path), Some(state)) => {
+ let value = state.committed_offset.max(offset);
+ persist_offset(path, value, self.consumer_offset_enforce_fsync).await?;
+ value
+ }
+ (Some(path), None) => {
+ persist_offset_max(path, offset, self.consumer_offset_enforce_fsync).await?
}
};
- self.persisted_offsets.borrow_mut().insert(key, persisted);
+ self.durable_consumer_offsets.record_auto_commit(
+ pending.kind,
+ pending.consumer_id,
+ if tracked.is_none() {
+ persisted_high_water
+ } else {
+ offset
+ },
+ persisted_high_water,
+ );
+ if path.is_some() && self.consumer_offset_enforce_fsync {
Review Comment:
warning: `path.is_some()` does not mean a file was written - line 2152 writes nothing, and line 2161 skips its write when the file already covers the value. key the flag on what persist actually returned.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, Ref, RefCell};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: u64,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+ membership_epoch: Cell<u64>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset && state.persisted_high_water >= offset
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) -> bool {
+ let created = self
+ .entries(kind)
+ .borrow_mut()
+ .insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ )
+ .is_none();
+ if created {
+ self.bump_membership_epoch();
+ }
+ created
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ match entries.entry(id) {
+ Entry::Occupied(mut entry) => {
+ let state = entry.get_mut();
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = state.persisted_high_water.max(persisted_high_water);
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ });
+ drop(entries);
+ self.bump_membership_epoch();
+ }
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ let removed = self.entries(kind).borrow_mut().remove(&id).is_some();
+ if removed {
+ self.bump_membership_epoch();
+ }
+ removed
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ self.bump_membership_epoch();
+ }
+
+ #[cfg(any(test, feature = "simulator"))]
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ pub(crate) fn snapshot_entries(
+ &self,
+ kind: ConsumerKind,
+ ) -> Ref<'_, HashMap<u32, DurableOffsetState>> {
+ self.entries(kind).borrow()
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+
+ fn bump_membership_epoch(&self) {
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Arc<ProvisionalToken>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
+ map_warned: Cell<bool>,
+ reclaim_epoch: Arc<AtomicU64>,
+ last_reclaim: Cell<Option<(u64, u64)>>,
+}
+
+impl ConsumerOffsetCapacity {
+ pub(crate) fn new(kind: ConsumerKind, limit: usize) -> Self {
+ Self {
+ kind,
+ limit: Cell::new(limit),
+ pending: RefCell::new(HashMap::new()),
+ provisional: RefCell::new(HashMap::new()),
+ stranded: RefCell::new(HashSet::new()),
+ uncertain: Cell::new(false),
+ durable_warned: Cell::new(false),
+ map_warned: Cell::new(false),
+ reclaim_epoch: Arc::new(AtomicU64::new(0)),
+ last_reclaim: Cell::new(None),
+ }
+ }
+
+ pub(crate) fn set_limit(&self, limit: usize) {
+ self.limit.set(limit);
+ }
+
+ pub(crate) const fn limit(&self) -> usize {
+ self.limit.get()
+ }
+
+ pub(crate) fn try_reserve(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ *self.pending.borrow_mut().entry(id).or_default() += 1;
+ Ok(())
+ }
+
+ pub(crate) fn check(
Review Comment:
warning: `occupied()` walks pending, provisional and stranded on every new-key admission, so ramping n consumers is quadratic on the shard thread. when the four counts sum below the limit, admit without walking - but keep the `uncertain` gate.
##########
core/partitions/src/offset_storage.rs:
##########
@@ -109,6 +112,54 @@ pub fn decode_offset_record(bytes: &[u8]) -> OffsetRecord {
/// # Errors
/// [`IggyError`] when the directory, file, or write cannot be created or completed.
pub async fn persist_offset(path: &str, offset: u64, enforce_fsync: bool) -> Result<(), IggyError> {
+ replace_file(path, encode_offset_record(offset), enforce_fsync, false).await
+}
+
+pub(crate) async fn stage_offset_replacement(path: &str, offset: u64) -> Result<(), IggyError> {
Review Comment:
nit: hardcoding `enforce_fsync = true` here is right, since install unlinks then renames and non-durable staging would lose every cursor on a crash. say so in a comment - it reads like the knob was missed.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2261,21 +2418,44 @@ where
);
if let Err(error) = self.persist_consumer_offset_commit(pending).await {
+ if offset.is_some() {
+ self.release_consumer_offset_reservation(kind, consumer_id);
+ }
emit_partition_diag(
tracing::Level::WARN,
&PartitionDiagEvent::new(self.diag_ctx(), "no_ack offset persist failed")
.with_operation(request_header.operation)
.with_error(error.to_string()),
);
+ Self::send_partition_deny_or_log(
+ &self.consensus,
+ &request_header,
+ error.as_code(),
+ "no_ack offset failure reply send failed",
+ waiter,
+ )
+ .await;
return;
}
- if let Err(error) = self.apply_consumer_offset_commit(pending) {
+ self.apply_consumer_offset_commit(pending);
Review Comment:
warning: apply runs before the dir sync, so a sync failure denies the client after the offset is already visible. a delete then retries into 3021 for a delete that took effect - document the contract or roll back.
##########
core/server/src/offset_recovery.rs:
##########
@@ -167,28 +201,43 @@ pub fn load_consumer_group_offsets(
consumer_group_offsets.push((consumer_group_id, consumer_offset));
}
- Ok(consumer_group_offsets)
+ Ok(RecoveredOffsets {
+ entries: consumer_group_offsets,
+ stranded_ids: stranded,
+ })
}
-fn read_offset_file(path: &str, offset_kind: &'static str) -> Option<AtomicU64> {
+/// A crashed atomic replacement leaves its sibling behind. The rename never
+/// landed, so the sibling carries nothing the numeric file lacks.
+fn remove_stale_replacement(path: &std::path::Path, name: &str) {
Review Comment:
nit: this unlinks with no parent fsync and only warns, while `remove_invalid_offset_file` below fsyncs and downgrades to stranded on failure. fine either way, but say why the replacement sibling needs no durable removal.
##########
core/shard/src/lib.rs:
##########
@@ -8310,18 +8354,16 @@ where
if !commit_lag && head <= commit_to_op {
return false;
}
- let missing_suffix = partition_missing_suffix(partition);
- if !commit_lag && !missing_suffix {
+ let missing_suffix = partition_missing_suffix_through(partition);
+ if !commit_lag && missing_suffix.is_none() {
return false;
}
let nonce = iggy_common::random_id::get_uuid();
let from_op = consensus.commit_min() + 1;
- // The widening is for the replica that is LEVEL with the commit
- // frontier and short of bodies above it. Widening while a commit lag
- // stands would ask for `(commit_min, head]` -- the whole committed
- // prefix this replica already holds, refetched -- and the suffix is
- // reached anyway once the lag closes, on the arm after it.
- let fetch_to_op = if commit_lag { commit_to_op } else { head };
+ // Include the adopted suffix in the same repair as the committed
+ // prefix. Otherwise a newer live prepare can advance the sequencer
+ // while an older adopted body is still missing.
+ let fetch_to_op = missing_suffix.unwrap_or(commit_to_op);
Review Comment:
nit: this widens `fetch_to_op` to the adopted head even under commit lag, reversing the comment you deleted. it is a real liveness fix and strictly additive, but nothing asserts `fetch_to_op`, and line 9751 still links the removed `partition_missing_suffix`.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2341,7 +2610,21 @@ where
}
};
- if found {
+ if self.consensus.replica_count() > 1 {
Review Comment:
nit: a replicated delete for a live-map key that is not durable answers 3021 while `GetConsumerOffset` still returns its value. the client sees a key it cannot delete.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2587,7 +3065,7 @@ where
offset: u64,
) -> Result<(), IggyError> {
let pending = PendingConsumerOffsetCommit::try_from_polling_consumer(consumer, offset)?;
- self.apply_consumer_offset_commit(pending)?;
+ self.apply_consumer_offset_commit(pending);
Review Comment:
nit: this applies without persisting, so it arms neither the reconcile flag nor the shard epoch, and skips `persist_offset` so the commit dies on restart. no production caller - delete it or route it through the staged path.
##########
core/server/src/consumer_group.rs:
##########
@@ -226,16 +225,21 @@ where
let body = request_body(&request);
// The store/delete ops differ only in the decode type; this collapses
// their identical decode -> resolve group id -> rewrite consumer id ->
- // re-encode bodies. A non-group consumer or unresolved group returns the
- // request untouched (the apply/read path handles the miss).
+ // re-encode bodies. Individual consumers pass through. A group identifier
+ // that metadata cannot resolve is rejected before it can create a raw file
+ // in the group-offset directory.
macro_rules! rewrite_group_offset {
($ty:ty) => {{
let mut wire = <$ty>::decode_from(body).map_err(|_| IggyError::InvalidCommand)?;
- let Some(group_id) =
- resolve_group_offset_id(shard, &wire.consumer, (&wire.stream_id, &wire.topic_id))
- else {
+ if wire.consumer.kind != KIND_CONSUMER_GROUP {
return Ok(request);
- };
+ }
+ let group_id = resolve_offset_group_id(
Review Comment:
nit: a store or delete against a missing group now returns 5000/5003 instead of `ResourceNotFound`. go clients matching `ErrResourceNotFound` reroute silently - name both codes in the PR body.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2341,7 +2610,21 @@ where
}
};
Review Comment:
nit: this branch is the replicated default and it throws away `found`, which cost two pins and two `contains_key` calls just above. move that computation under this early return.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -1551,6 +1635,21 @@ fn final_paths(partition_dir: &str, start_offset: u64) -> (String, String) {
/// depth against how long one partition monopolises it; matches the tick's
/// superblock pre-pass.
const OFFSET_PERSIST_CONCURRENCY: usize = 16;
+const OFFSET_IO_ATTEMPTS: usize = 3;
+
+async fn retry_offset_mutation<T, E, F: Future<Output = Result<T, E>>>(
Review Comment:
nit: three attempts with only a reactor yield between them finish in microseconds against a hard ENOSPC or EIO, and the caller only sees the last error. either back off properly or do not retry this class.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, Ref, RefCell};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: u64,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+ membership_epoch: Cell<u64>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset && state.persisted_high_water >= offset
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) -> bool {
+ let created = self
+ .entries(kind)
+ .borrow_mut()
+ .insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ )
+ .is_none();
+ if created {
+ self.bump_membership_epoch();
+ }
+ created
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ match entries.entry(id) {
+ Entry::Occupied(mut entry) => {
+ let state = entry.get_mut();
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = state.persisted_high_water.max(persisted_high_water);
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ });
+ drop(entries);
+ self.bump_membership_epoch();
+ }
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ let removed = self.entries(kind).borrow_mut().remove(&id).is_some();
+ if removed {
+ self.bump_membership_epoch();
+ }
+ removed
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ self.bump_membership_epoch();
+ }
+
+ #[cfg(any(test, feature = "simulator"))]
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ pub(crate) fn snapshot_entries(
+ &self,
+ kind: ConsumerKind,
+ ) -> Ref<'_, HashMap<u32, DurableOffsetState>> {
+ self.entries(kind).borrow()
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+
+ fn bump_membership_epoch(&self) {
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Arc<ProvisionalToken>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
Review Comment:
simplification: `rearm_map_if_below_limit` is redundant once `admit_local_map_key` clears the latch on success - its only `Ok` path is already `map_len < limit`. drop the method and its eight call sites.
##########
core/server/src/offset_recovery.rs:
##########
@@ -27,19 +27,31 @@
//! here as unchecksummed.
use iggy_common::{ConsumerGroupId, ConsumerKind, ConsumerOffset, IggyError};
-use partitions::offset_storage::{OffsetRecord, decode_offset_record};
+use partitions::offset_storage::{OFFSET_REPLACEMENT_SUFFIX, OffsetRecord, decode_offset_record};
use std::sync::atomic::AtomicU64;
use tracing::{error, trace, warn};
const COMPONENT: &str = "STREAMING_PARTITIONS";
-pub fn load_consumer_offsets(path: &str) -> Result<Vec<ConsumerOffset>, IggyError> {
+pub struct RecoveredOffsets<T> {
+ pub entries: Vec<T>,
+ pub stranded_ids: Vec<u32>,
+}
+
+enum OffsetFileLoad {
+ Loaded(AtomicU64),
+ Removed,
+ Stranded,
+}
+
+pub fn load_consumer_offsets(path: &str) -> Result<RecoveredOffsets<ConsumerOffset>, IggyError> {
Review Comment:
simplification: this and `load_consumer_group_offsets` are the same 60-line walk with a different element. one private helper taking a constructor closure saves about 55 lines - note the consumer side sorts and the group side does not.
##########
core/sdk/src/quic/quic_client.rs:
##########
@@ -218,7 +218,10 @@ impl BinaryTransport for QuicClient {
} else if let Some(next) = roster_walk.as_mut().and_then(RosterWalk::next) {
(next, true)
} else {
Review Comment:
warning: same as the tcp client - this fires on any exhausted roster walk, not just a one-node roster, so a full-cluster refusal replays one endpoint instead of surfacing `TransientNotAccepted`. gate on roster size.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2638,177 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ fn resynchronize_consumer_offset_reservations(&mut self) {
+ self.resynchronize_consumer_offset_reservations_inner(false);
+ }
+
+ /// Retry incomplete accounting at most once per shard tick, after progress.
+ pub fn retry_consumer_offset_reservations(&mut self) {
+ if self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain()
+ {
+ self.resynchronize_consumer_offset_reservations_inner(true);
+ }
+ }
+
+ #[allow(clippy::too_many_lines)]
+ fn resynchronize_consumer_offset_reservations_inner(&mut self, from_tick: bool) {
let current_view = self.consensus.view();
+ let scan_state = (
+ self.consensus.commit_min(),
+ self.consensus.commit_max(),
+ self.consensus.sequencer().current_sequence(),
+ self.log.journal().inner.last_op(),
+ );
+ let uncertain = self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain();
if current_view == self.observed_view {
- return;
+ if uncertain && !from_tick {
+ return;
+ }
+ let retry_requested = self.offset_reservations_need_resync.get()
+ || (uncertain && self.offset_reservations_scan_state != Some(scan_state));
+ if !retry_requested {
+ return;
+ }
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ }
+
+ let from_op = self
+ .consensus
+ .commit_min()
+ .max(self.purge_floor_op)
+ .saturating_add(1);
+ let commit_max = self.consensus.commit_max();
+ let to_op = self
+ .consensus
+ .sequencer()
+ .current_sequence()
+ .min(self.log.journal().inner.last_op().unwrap_or(commit_max));
+ // Committed offset prepares still need local apply, even if a message
+ // flush already evicted their journal bytes. Never drop their staging.
+ let mut rebuilt: HashMap<_, _> = self
+ .pending_consumer_offset_commits
+ .iter()
+ .filter(|(op, _)| **op >= from_op && **op <= commit_max)
+ .map(|(op, pending)| (*op, *pending))
+ .collect();
+ let headers = self.log.journal().inner.repair_headers_in(from_op..=to_op);
+ let uncommitted_from = from_op.max(commit_max.saturating_add(1));
+ let expected = to_op
+ .checked_sub(uncommitted_from)
+ .map_or(0, |span| span.saturating_add(1));
+ let mut decode_failed =
+ headers.keys().filter(|op| **op >= uncommitted_from).count() as u64 != expected;
+ for (op, header) in headers {
+ if !matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) {
+ continue;
+ }
+ match self.restage_consumer_offset_from_journal(op) {
+ Ok(pending) => {
+ rebuilt.insert(op, pending);
+ }
+ Err(error) => {
+ error!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica_id = self.consensus.replica(),
+ namespace_raw = self.namespace().inner(),
+ op,
+ %error,
+ "failed to rebuild consumer offset reservations after view change"
+ );
+ decode_failed = true;
+ break;
+ }
+ }
+ }
+ self.pending_consumer_offset_commits = rebuilt;
+ if decode_failed {
+ self.consumer_offset_capacity.mark_uncertain();
+ self.consumer_group_offset_capacity.mark_uncertain();
+ } else {
+ let consumer_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::Consumer
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, consumer_ids);
+ let group_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::ConsumerGroup
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_group_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, group_ids);
+ }
self.observed_view = current_view;
+ self.mark_consumer_group_offsets_need_reconcile();
Review Comment:
warning: bumping the shard-wide epoch unconditionally defeats the O(1) reconciler skip added at `partition_reconciler.rs:495`, for every partition on the shard, including ones holding no group offsets. bump it inside the view-change check at line 2683.
##########
core/server/src/responses.rs:
##########
@@ -295,10 +293,46 @@ where
false,
)
.map(|_| ())
- .ok_or(IggyError::ConsumerGroupPartitionNotOwned(
- client_id as u32,
- partition_id,
- ))
+ .ok_or_else(|| {
Review Comment:
simplification: the nested `ok_or_else` resolves the group twice on the error path. flatten it with a `let ... else` - do not reorder to resolve first, that adds a walk to the success path.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -4470,6 +5092,56 @@ where
self.drain_request_queue_into_prepares(drained_count).await;
}
+ async fn flush_consumer_offset_directories(&self) -> Result<(), IggyError> {
Review Comment:
warning: this returns on the first failing index and never tries the second, so a groups dir fault leaves consumer dirents unproven. attempt both, clear each on its own success, report after the loop.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2638,177 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ fn resynchronize_consumer_offset_reservations(&mut self) {
+ self.resynchronize_consumer_offset_reservations_inner(false);
+ }
+
+ /// Retry incomplete accounting at most once per shard tick, after progress.
+ pub fn retry_consumer_offset_reservations(&mut self) {
+ if self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain()
+ {
+ self.resynchronize_consumer_offset_reservations_inner(true);
+ }
+ }
+
+ #[allow(clippy::too_many_lines)]
+ fn resynchronize_consumer_offset_reservations_inner(&mut self, from_tick: bool) {
let current_view = self.consensus.view();
+ let scan_state = (
+ self.consensus.commit_min(),
+ self.consensus.commit_max(),
+ self.consensus.sequencer().current_sequence(),
+ self.log.journal().inner.last_op(),
+ );
+ let uncertain = self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain();
if current_view == self.observed_view {
- return;
+ if uncertain && !from_tick {
+ return;
+ }
+ let retry_requested = self.offset_reservations_need_resync.get()
+ || (uncertain && self.offset_reservations_scan_state != Some(scan_state));
+ if !retry_requested {
+ return;
+ }
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ }
+
+ let from_op = self
+ .consensus
+ .commit_min()
+ .max(self.purge_floor_op)
+ .saturating_add(1);
+ let commit_max = self.consensus.commit_max();
+ let to_op = self
+ .consensus
+ .sequencer()
+ .current_sequence()
+ .min(self.log.journal().inner.last_op().unwrap_or(commit_max));
+ // Committed offset prepares still need local apply, even if a message
+ // flush already evicted their journal bytes. Never drop their staging.
+ let mut rebuilt: HashMap<_, _> = self
+ .pending_consumer_offset_commits
+ .iter()
+ .filter(|(op, _)| **op >= from_op && **op <= commit_max)
+ .map(|(op, pending)| (*op, *pending))
+ .collect();
+ let headers = self.log.journal().inner.repair_headers_in(from_op..=to_op);
+ let uncommitted_from = from_op.max(commit_max.saturating_add(1));
+ let expected = to_op
+ .checked_sub(uncommitted_from)
+ .map_or(0, |span| span.saturating_add(1));
+ let mut decode_failed =
+ headers.keys().filter(|op| **op >= uncommitted_from).count() as u64 != expected;
+ for (op, header) in headers {
+ if !matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) {
+ continue;
+ }
+ match self.restage_consumer_offset_from_journal(op) {
+ Ok(pending) => {
+ rebuilt.insert(op, pending);
+ }
+ Err(error) => {
+ error!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica_id = self.consensus.replica(),
+ namespace_raw = self.namespace().inner(),
+ op,
+ %error,
+ "failed to rebuild consumer offset reservations after view change"
+ );
+ decode_failed = true;
+ break;
+ }
+ }
+ }
+ self.pending_consumer_offset_commits = rebuilt;
+ if decode_failed {
+ self.consumer_offset_capacity.mark_uncertain();
+ self.consumer_group_offset_capacity.mark_uncertain();
+ } else {
+ let consumer_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::Consumer
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, consumer_ids);
+ let group_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::ConsumerGroup
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_group_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, group_ids);
+ }
self.observed_view = current_view;
+ self.mark_consumer_group_offsets_need_reconcile();
+ self.offset_reservations_scan_state = Some(scan_state);
+ // The shard tick retries uncertainty after journal or frontier progress.
+ self.offset_reservations_need_resync.set(false);
+ }
+
+ fn reclaim_phantom_offsets(&self, kind: ConsumerKind, map_count: usize) {
+ let capacity = self.consumer_offset_capacity_for(kind);
+ if self.durable_consumer_offsets.count(kind) >= capacity.limit()
+ || !capacity.should_reclaim(&self.durable_consumer_offsets)
+ {
+ return;
+ }
+ let mut remaining = map_count.saturating_sub(capacity.limit()).saturating_add(1);
+ match kind {
Review Comment:
simplification: the two arms are the same loop over a different map. a generic helper needs only `K: Hash + Eq` - roughly line-neutral, but it gives one `break`/`remaining` protocol and one place for the eviction log.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2669,15 +2894,21 @@ where
// and a purged origin offering `next_offset = 0` drops every
// incoming entry, so a map-only sweep leaves the old table for boot
// to resurrect.
- paths.extend(strayed_offset_files(
- self.consumer_offsets_path.as_deref(),
- &offsets_wire.consumers,
- ));
+ paths.extend(
+ strayed_offset_files(
Review Comment:
warning: this excludes `offsets_wire` ids, but `clamp` drops every entry when `next_offset` is 0, so nothing renames them back and an old offset file survives unowned. boot then resurrects it - build the set from `planned_offsets` ids instead.
##########
core/partitions/src/offset_storage.rs:
##########
@@ -109,6 +112,54 @@ pub fn decode_offset_record(bytes: &[u8]) -> OffsetRecord {
/// # Errors
/// [`IggyError`] when the directory, file, or write cannot be created or completed.
pub async fn persist_offset(path: &str, offset: u64, enforce_fsync: bool) -> Result<(), IggyError> {
Review Comment:
warning: every durable offset commit now burns a fresh inode and a `renameat` instead of an in-place truncate, plus a batched dir fsync when `enforce_fsync` is on. that's the auto-commit hot path - note the trade-off here and measure it.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2638,177 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ fn resynchronize_consumer_offset_reservations(&mut self) {
+ self.resynchronize_consumer_offset_reservations_inner(false);
+ }
+
+ /// Retry incomplete accounting at most once per shard tick, after progress.
+ pub fn retry_consumer_offset_reservations(&mut self) {
+ if self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain()
+ {
+ self.resynchronize_consumer_offset_reservations_inner(true);
+ }
+ }
+
+ #[allow(clippy::too_many_lines)]
+ fn resynchronize_consumer_offset_reservations_inner(&mut self, from_tick: bool) {
let current_view = self.consensus.view();
+ let scan_state = (
+ self.consensus.commit_min(),
+ self.consensus.commit_max(),
+ self.consensus.sequencer().current_sequence(),
+ self.log.journal().inner.last_op(),
+ );
+ let uncertain = self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain();
if current_view == self.observed_view {
- return;
+ if uncertain && !from_tick {
+ return;
+ }
+ let retry_requested = self.offset_reservations_need_resync.get()
+ || (uncertain && self.offset_reservations_scan_state != Some(scan_state));
+ if !retry_requested {
+ return;
+ }
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ }
+
+ let from_op = self
+ .consensus
+ .commit_min()
+ .max(self.purge_floor_op)
+ .saturating_add(1);
+ let commit_max = self.consensus.commit_max();
+ let to_op = self
+ .consensus
+ .sequencer()
+ .current_sequence()
+ .min(self.log.journal().inner.last_op().unwrap_or(commit_max));
+ // Committed offset prepares still need local apply, even if a message
+ // flush already evicted their journal bytes. Never drop their staging.
+ let mut rebuilt: HashMap<_, _> = self
+ .pending_consumer_offset_commits
+ .iter()
+ .filter(|(op, _)| **op >= from_op && **op <= commit_max)
+ .map(|(op, pending)| (*op, *pending))
+ .collect();
+ let headers = self.log.journal().inner.repair_headers_in(from_op..=to_op);
+ let uncommitted_from = from_op.max(commit_max.saturating_add(1));
+ let expected = to_op
+ .checked_sub(uncommitted_from)
+ .map_or(0, |span| span.saturating_add(1));
+ let mut decode_failed =
+ headers.keys().filter(|op| **op >= uncommitted_from).count() as u64 != expected;
+ for (op, header) in headers {
+ if !matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) {
+ continue;
+ }
+ match self.restage_consumer_offset_from_journal(op) {
+ Ok(pending) => {
+ rebuilt.insert(op, pending);
+ }
+ Err(error) => {
+ error!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica_id = self.consensus.replica(),
+ namespace_raw = self.namespace().inner(),
+ op,
+ %error,
+ "failed to rebuild consumer offset reservations after view change"
+ );
+ decode_failed = true;
+ break;
+ }
+ }
+ }
+ self.pending_consumer_offset_commits = rebuilt;
+ if decode_failed {
+ self.consumer_offset_capacity.mark_uncertain();
+ self.consumer_group_offset_capacity.mark_uncertain();
+ } else {
+ let consumer_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::Consumer
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, consumer_ids);
+ let group_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::ConsumerGroup
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_group_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, group_ids);
+ }
self.observed_view = current_view;
+ self.mark_consumer_group_offsets_need_reconcile();
+ self.offset_reservations_scan_state = Some(scan_state);
+ // The shard tick retries uncertainty after journal or frontier progress.
+ self.offset_reservations_need_resync.set(false);
+ }
+
+ fn reclaim_phantom_offsets(&self, kind: ConsumerKind, map_count: usize) {
Review Comment:
warning: eviction drops live cursors silently and in hash order, so a hot consumer can lose its cursor and re-read the partition from 0 while a colder one survives. add a `debug!` with the kind and consumer id.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2649,18 +2872,20 @@ where
let next_offset = offsets_wire
.next_offset
.max(installed_end.map_or(0, |end| end + 1));
- let mut offsets_written = true;
// A key that fails the u32 narrowing would strand its old offset
// file's delete, which boot can then resurrect: unreachable while
// keys are minted from u32 wire ids, so assert it.
- let old_consumer_paths: Vec<String> = {
+ let old_consumer_paths: Vec<(ConsumerKind, u32, String)> = {
Review Comment:
nit: ids in both the map and the incoming table are unlinked here then re-created by rename, so a crash between the two loses the cursor. filter on ids that have a planned write, not on `offsets_wire` ids.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, Ref, RefCell};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: u64,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+ membership_epoch: Cell<u64>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset && state.persisted_high_water >= offset
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) -> bool {
+ let created = self
+ .entries(kind)
+ .borrow_mut()
+ .insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ )
+ .is_none();
+ if created {
+ self.bump_membership_epoch();
+ }
+ created
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ match entries.entry(id) {
+ Entry::Occupied(mut entry) => {
+ let state = entry.get_mut();
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = state.persisted_high_water.max(persisted_high_water);
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ });
+ drop(entries);
+ self.bump_membership_epoch();
+ }
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ let removed = self.entries(kind).borrow_mut().remove(&id).is_some();
+ if removed {
+ self.bump_membership_epoch();
+ }
+ removed
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ self.bump_membership_epoch();
+ }
+
+ #[cfg(any(test, feature = "simulator"))]
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ pub(crate) fn snapshot_entries(
Review Comment:
warning: `snapshot_entries` hands a live `Ref` out of the type and the caller runs a closure while it is held, so any future closure touching `durable_consumer_offsets` panics the pump on a double borrow. invert it to `with_entries(kind, |entries| ..)`.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2638,177 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ fn resynchronize_consumer_offset_reservations(&mut self) {
+ self.resynchronize_consumer_offset_reservations_inner(false);
+ }
+
+ /// Retry incomplete accounting at most once per shard tick, after progress.
+ pub fn retry_consumer_offset_reservations(&mut self) {
+ if self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain()
+ {
+ self.resynchronize_consumer_offset_reservations_inner(true);
+ }
+ }
+
+ #[allow(clippy::too_many_lines)]
+ fn resynchronize_consumer_offset_reservations_inner(&mut self, from_tick: bool) {
let current_view = self.consensus.view();
+ let scan_state = (
+ self.consensus.commit_min(),
+ self.consensus.commit_max(),
+ self.consensus.sequencer().current_sequence(),
+ self.log.journal().inner.last_op(),
+ );
+ let uncertain = self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain();
if current_view == self.observed_view {
- return;
+ if uncertain && !from_tick {
+ return;
+ }
+ let retry_requested = self.offset_reservations_need_resync.get()
+ || (uncertain && self.offset_reservations_scan_state != Some(scan_state));
+ if !retry_requested {
+ return;
+ }
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ }
+
+ let from_op = self
+ .consensus
+ .commit_min()
+ .max(self.purge_floor_op)
+ .saturating_add(1);
+ let commit_max = self.consensus.commit_max();
+ let to_op = self
+ .consensus
+ .sequencer()
+ .current_sequence()
+ .min(self.log.journal().inner.last_op().unwrap_or(commit_max));
+ // Committed offset prepares still need local apply, even if a message
+ // flush already evicted their journal bytes. Never drop their staging.
+ let mut rebuilt: HashMap<_, _> = self
+ .pending_consumer_offset_commits
+ .iter()
+ .filter(|(op, _)| **op >= from_op && **op <= commit_max)
+ .map(|(op, pending)| (*op, *pending))
+ .collect();
+ let headers = self.log.journal().inner.repair_headers_in(from_op..=to_op);
Review Comment:
nit: while a partition is uncertain and advancing, every tick rebuilds the whole staging window and deep-copies each offset op it finds. bounded by journal residency, but worth an op budget with a resume cursor.
##########
core/partitions/src/offset_storage.rs:
##########
@@ -325,14 +364,15 @@ async fn read_offset_record(path: &str) -> Result<Option<OffsetRecord>, IggyErro
/// Unlink a persisted consumer-offset file. A no-op if the file is absent.
Review Comment:
nit: the "returns whether a file was removed" line sits under `# Errors`, so rustdoc renders success semantics as an error note. move it into the summary.
##########
core/server/src/offset_recovery.rs:
##########
@@ -63,11 +75,15 @@ pub fn load_consumer_offsets(path: &str) -> Result<Vec<ConsumerOffset>, IggyErro
}
};
- if metadata.is_dir() {
+ if !metadata.is_file() {
continue;
}
let name = dir_entry.file_name().to_string_lossy().to_string();
+ if name.ends_with(OFFSET_REPLACEMENT_SUFFIX) {
Review Comment:
nit: this matches any `.tmp` in the offsets dir before parsing the id, so it deletes unrelated files. require the stem to parse as `u32` first.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2638,177 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ fn resynchronize_consumer_offset_reservations(&mut self) {
+ self.resynchronize_consumer_offset_reservations_inner(false);
+ }
+
+ /// Retry incomplete accounting at most once per shard tick, after progress.
+ pub fn retry_consumer_offset_reservations(&mut self) {
+ if self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain()
+ {
+ self.resynchronize_consumer_offset_reservations_inner(true);
+ }
+ }
+
+ #[allow(clippy::too_many_lines)]
+ fn resynchronize_consumer_offset_reservations_inner(&mut self, from_tick: bool) {
let current_view = self.consensus.view();
+ let scan_state = (
+ self.consensus.commit_min(),
+ self.consensus.commit_max(),
+ self.consensus.sequencer().current_sequence(),
+ self.log.journal().inner.last_op(),
+ );
+ let uncertain = self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain();
if current_view == self.observed_view {
- return;
+ if uncertain && !from_tick {
+ return;
+ }
+ let retry_requested = self.offset_reservations_need_resync.get()
+ || (uncertain && self.offset_reservations_scan_state != Some(scan_state));
+ if !retry_requested {
+ return;
+ }
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ }
+
+ let from_op = self
+ .consensus
+ .commit_min()
+ .max(self.purge_floor_op)
+ .saturating_add(1);
+ let commit_max = self.consensus.commit_max();
+ let to_op = self
+ .consensus
+ .sequencer()
+ .current_sequence()
+ .min(self.log.journal().inner.last_op().unwrap_or(commit_max));
+ // Committed offset prepares still need local apply, even if a message
+ // flush already evicted their journal bytes. Never drop their staging.
+ let mut rebuilt: HashMap<_, _> = self
+ .pending_consumer_offset_commits
+ .iter()
+ .filter(|(op, _)| **op >= from_op && **op <= commit_max)
+ .map(|(op, pending)| (*op, *pending))
+ .collect();
+ let headers = self.log.journal().inner.repair_headers_in(from_op..=to_op);
+ let uncommitted_from = from_op.max(commit_max.saturating_add(1));
+ let expected = to_op
+ .checked_sub(uncommitted_from)
+ .map_or(0, |span| span.saturating_add(1));
+ let mut decode_failed =
+ headers.keys().filter(|op| **op >= uncommitted_from).count() as u64 != expected;
+ for (op, header) in headers {
+ if !matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) {
+ continue;
+ }
+ match self.restage_consumer_offset_from_journal(op) {
+ Ok(pending) => {
+ rebuilt.insert(op, pending);
+ }
+ Err(error) => {
+ error!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica_id = self.consensus.replica(),
+ namespace_raw = self.namespace().inner(),
+ op,
+ %error,
+ "failed to rebuild consumer offset reservations after view change"
+ );
+ decode_failed = true;
+ break;
+ }
+ }
+ }
+ self.pending_consumer_offset_commits = rebuilt;
+ if decode_failed {
+ self.consumer_offset_capacity.mark_uncertain();
+ self.consumer_group_offset_capacity.mark_uncertain();
+ } else {
+ let consumer_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::Consumer
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, consumer_ids);
+ let group_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::ConsumerGroup
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_group_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, group_ids);
+ }
self.observed_view = current_view;
+ self.mark_consumer_group_offsets_need_reconcile();
+ self.offset_reservations_scan_state = Some(scan_state);
+ // The shard tick retries uncertainty after journal or frontier progress.
+ self.offset_reservations_need_resync.set(false);
+ }
+
+ fn reclaim_phantom_offsets(&self, kind: ConsumerKind, map_count: usize) {
+ let capacity = self.consumer_offset_capacity_for(kind);
+ if self.durable_consumer_offsets.count(kind) >= capacity.limit()
+ || !capacity.should_reclaim(&self.durable_consumer_offsets)
+ {
+ return;
+ }
+ let mut remaining = map_count.saturating_sub(capacity.limit()).saturating_add(1);
+ match kind {
+ ConsumerKind::Consumer => {
+ let map = self.consumer_offsets.pin();
+ for (key, _) in &map {
+ if let Ok(id) = u32::try_from(*key)
+ && !capacity.holds(id, &self.durable_consumer_offsets)
+ {
+ map.remove(key);
+ capacity.forget_inactive_provisional(id);
+ remaining -= 1;
+ if remaining == 0 {
+ break;
+ }
+ }
+ }
+ }
+ ConsumerKind::ConsumerGroup => {
+ let map = self.consumer_group_offsets.pin();
+ for (key, _) in &map {
+ if let Ok(id) = u32::try_from(key.0)
+ && !capacity.holds(id, &self.durable_consumer_offsets)
+ {
+ map.remove(key);
+ capacity.forget_inactive_provisional(id);
+ remaining -= 1;
+ if remaining == 0 {
+ break;
+ }
+ }
+ }
+ }
+ }
+ capacity.rearm_map_if_below_limit(self.consumer_offset_map_count(kind));
Review Comment:
nit: this calls papaya `len()` even when the loop evicted nothing, and it sums a per-CPU counter. `map_count` was read at line 2886 and `remaining` tracks the rest, so subtract instead.
##########
core/server/src/offset_recovery.rs:
##########
@@ -167,28 +201,43 @@ pub fn load_consumer_group_offsets(
consumer_group_offsets.push((consumer_group_id, consumer_offset));
}
- Ok(consumer_group_offsets)
+ Ok(RecoveredOffsets {
+ entries: consumer_group_offsets,
+ stranded_ids: stranded,
+ })
}
-fn read_offset_file(path: &str, offset_kind: &'static str) -> Option<AtomicU64> {
+/// A crashed atomic replacement leaves its sibling behind. The rename never
+/// landed, so the sibling carries nothing the numeric file lacks.
+fn remove_stale_replacement(path: &std::path::Path, name: &str) {
+ match std::fs::remove_file(path) {
+ Ok(()) => trace!("Removed stale offset replacement file: '{name}'."),
+ Err(e) => warn!(
+ "{COMPONENT} (error: {e}) - could not remove stale offset replacement \
+ file: '{name}', skipping."
+ ),
+ }
+}
+
+fn read_offset_file(path: &str, offset_kind: &'static str) -> OffsetFileLoad {
let bytes = match std::fs::read(path) {
Ok(bytes) => bytes,
Err(e) => {
warn!(
"{COMPONENT} (error: {e}) - failed to read offset file, \
path: {path}, skipping."
);
- return None;
+ return OffsetFileLoad::Stranded;
}
};
match decode_offset_record(&bytes) {
- OffsetRecord::Value { offset, .. } => Some(AtomicU64::new(offset)),
+ OffsetRecord::Value { offset, .. } => OffsetFileLoad::Loaded(AtomicU64::new(offset)),
OffsetRecord::Torn => {
warn!(
Review Comment:
nit: the warn still ends `skipping.` but this branch unlinks the file now. say removed.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2686,130 +2917,96 @@ where
"consumer group offset key {} exceeds u32",
key.0
);
- narrowed
- .and_then(|id| self.persisted_offset_path(ConsumerKind::ConsumerGroup, id))
+ narrowed.and_then(|id| {
+ self.persisted_offset_path(ConsumerKind::ConsumerGroup, id)
+ .map(|path| (ConsumerKind::ConsumerGroup, id, path))
+ })
})
.collect();
guard.clear();
- paths.extend(strayed_offset_files(
- self.consumer_group_offsets_path.as_deref(),
- &offsets_wire.groups,
- ));
+ paths.extend(
+ strayed_offset_files(
+ self.consumer_group_offsets_path.as_deref(),
+ &offsets_wire.groups,
+ )
+ .into_iter()
+ .filter_map(|path| {
+ numeric_offset_id(&path).map(|id| (ConsumerKind::ConsumerGroup, id, path))
+ }),
+ );
paths
};
- for path in old_consumer_paths.into_iter().chain(old_group_paths) {
- if let Err(error) = delete_persisted_offset(&path).await {
- // Not fatal, but not silent either: a stranded file is an id
- // absent from the NEW table (matching ids get overwritten at
- // the same path), and boot resurrects it. Sharpest after a
- // purged origin ships `next_offset = 0`, where the clamp drops
- // every incoming entry and the whole old table survives while
- // the install still reports success.
- tracing::warn!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- namespace_raw = self.consensus().group(),
- path = %path,
- %error,
- "failed to unlink a superseded consumer-offset file during install"
- );
+ let mut offset_dirs_changed = [false; 2];
+ for (kind, consumer_id, path) in old_consumer_paths.into_iter().chain(old_group_paths) {
+ let removed = delete_persisted_offset(&path)
+ .await
+ .map_err(|source| PartitionInstallError::OffsetPersistence { path, source })?;
+ if removed {
+ offset_dirs_changed[consumer_kind_index(kind)] = true;
}
+ self.consumer_offset_capacity_for(kind)
+ .clear_stranded(consumer_id);
}
- self.persisted_offsets.borrow_mut().clear();
+ self.durable_consumer_offsets.clear();
self.pending_consumer_offset_commits.clear();
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ self.consumer_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, std::iter::empty());
+ self.consumer_group_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, std::iter::empty());
self.last_polled_offsets.pin().clear();
- // `None` when the group's offset space is empty (`next_offset == 0`,
- // a purged origin): clamping every transferred offset to 0 would tell
- // each consumer it consumed offset 0 on a partition that never minted
- // one, so a `Next` poll skips the first message. Dropping the entries
- // is what "no offsets yet" means.
- let clamp = |offset: u64| next_offset.checked_sub(1).map(|last| offset.min(last));
- if self.consumer_offsets_path.is_none() || self.consumer_group_offsets_path.is_none() {
- // Nothing to write the transferred table into: unreachable via
- // the server boot paths (they always configure storage), but if
- // it ever fires the table was dropped and the flag must say so.
- offsets_written = false;
- }
- // Both maps are populated first (no await, so nothing borrows across
- // one), then the files are written in capped batches. One await per
- // file put a rejoin carrying thousands of consumers on the pump for
- // thousands of sequential open + write + optional fsync round trips;
- // the tick's superblock pre-pass sets the precedent for the width.
- // The dedup slice is memory-only, so it installs here with the maps
- // rather than being written anywhere. No frontier fence is needed: the
- // install lifts `commit_min` to the offer's `commit_op`, so the commit
- // walk that follows starts strictly above everything this artifact
- // covers, and `record_commit` is idempotent besides.
+ // The replacement siblings were written and data-synced before any
+ // segment mutation. Finalize only their directory entries here, then
+ // publish the matching maps and durable membership.
self.dedup_mut()
.install_watermarks(offsets_wire.dedup.iter().copied());
- let mut planned: Vec<PlannedOffsetWrite> =
- Vec::with_capacity(offsets_wire.consumers.len() + offsets_wire.groups.len());
- if let Some(dir) = self.consumer_offsets_path.clone() {
- for (id, offset) in &offsets_wire.consumers {
- let Some(value) = clamp(*offset) else {
- continue;
- };
- let entry = ConsumerOffset::default_for_consumer(*id, &dir);
- entry.offset.store(value, Ordering::Release);
- let path = entry.path.clone();
- self.consumer_offsets.pin().insert(*id as usize, entry);
- planned.push(PlannedOffsetWrite {
- kind: ConsumerKind::Consumer,
- id: *id,
- path,
- value,
- });
- }
- }
- if let Some(dir) = self.consumer_group_offsets_path.clone() {
- for (id, offset) in &offsets_wire.groups {
- let Some(value) = clamp(*offset) else {
- continue;
- };
- let group_id = ConsumerGroupId(*id as usize);
- let entry = ConsumerOffset::default_for_consumer_group(group_id, &dir);
- entry.offset.store(value, Ordering::Release);
- let path = entry.path.clone();
- self.consumer_group_offsets.pin().insert(group_id, entry);
- planned.push(PlannedOffsetWrite {
- kind: ConsumerKind::ConsumerGroup,
- id: *id,
- path,
- value,
- });
- }
- }
- let enforce_fsync = self.consumer_offset_enforce_fsync;
- for batch in planned.chunks(OFFSET_PERSIST_CONCURRENCY) {
- let writes = batch.iter().map(|write| async move {
- let written = persist_offset(&write.path, write.value, enforce_fsync)
- .await
- .is_ok();
- (written, write.kind, write.id, write.value)
- });
- for (written, kind, id, value) in futures::future::join_all(writes).await {
- if written {
- self.persisted_offsets
- .borrow_mut()
- .insert((kind, id), value);
- } else {
- offsets_written = false;
+ for write in planned_offsets {
+ commit_offset_replacement(&write.path)
Review Comment:
nit: the post-swap delete and rename are unguarded while the pre-swap staging retries, and any errno here escalates to the full wipe-and-re-pull. worth a note - a yield-only retry would not help this class.
##########
core/common/src/traits/consumer_offset_client.rs:
##########
@@ -24,6 +24,7 @@ pub trait ConsumerOffsetClient {
/// Store the consumer offset for a specific consumer or consumer group for the given stream and topic by unique IDs or names.
///
/// Authentication is required, and the permission to poll the messages.
+ /// A new key at the per-partition limit returns [`IggyError::TooManyConsumerOffsets`] (3024).
Review Comment:
nit: the 3024 note only covers `store_consumer_offset`. `delete_consumer_offset` types the same admission errors, and `get_consumer_offset` is where "missing key is `Ok(None)`" belongs.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2959,6 +3156,36 @@ where
}
self.log.journal().inner.clear_all();
self.log.journal_mut().info = crate::log::JournalInfo::default();
+ self.consumer_offsets.pin().clear();
+ self.consumer_group_offsets.pin().clear();
+ self.last_polled_offsets.pin().clear();
+ self.durable_consumer_offsets.clear();
+ self.pending_consumer_offset_commits.clear();
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ for kind in [ConsumerKind::Consumer, ConsumerKind::ConsumerGroup] {
+ self.consumer_offset_capacity_for(kind)
+ .rebuild(&self.durable_consumer_offsets, std::iter::empty());
+ }
+ for (kind, dir) in [
+ (ConsumerKind::Consumer, self.consumer_offsets_path.as_ref()),
+ (
+ ConsumerKind::ConsumerGroup,
+ self.consumer_group_offsets_path.as_ref(),
+ ),
+ ] {
+ let Some(dir) = dir else { continue };
+ for path in strayed_offset_files(Some(dir), &[]) {
Review Comment:
nit: `strayed_offset_files` parses names as `u32`, so the converge sweep skips every `<id>.tmp` sibling even though the comment says the directory converges to an empty shape.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -395,13 +457,15 @@ pub async fn dispatch_partition_request<B, MJ, S, SB>(
operation = ?header.operation,
"partition request with unresolved namespace; replying denied"
Review Comment:
nit: the gate moved from error identity to operation identity, so every namespace-resolution error on these two ops now goes to the client verbatim instead of as `ResourceNotFound`. the wire-visible set widened with no coverage.
##########
core/server/src/offset_recovery.rs:
##########
@@ -206,13 +255,55 @@ fn read_offset_file(path: &str, offset_kind: &'static str) -> Option<AtomicU64>
(offset: {offset}, expected: {expected}, found: {found}), \
path: {path}, removing it and resuming this consumer from the start."
);
- if let Err(e) = std::fs::remove_file(path) {
- error!(
- "{COMPONENT} (error: {e}) - could not remove the corrupt \
- {offset_kind} file, path: {path}; remove it manually."
- );
- }
- None
+ remove_invalid_offset_file(path, offset_kind)
}
}
}
+
+fn remove_invalid_offset_file(path: &str, offset_kind: &'static str) -> OffsetFileLoad {
Review Comment:
nit: this does a blocking `remove_file` plus parent `sync_all` inside the loader, and `load_partition` runs from the reconciler at runtime, so a torn file costs a dir fsync on a serving shard thread.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2008,83 +2120,122 @@ where
// durably stored; the in-memory update is idempotent on replay
// because we look up by (kind, id).
self.persist_consumer_offset_commit(pending).await?;
- self.apply_consumer_offset_commit(pending)?;
+ self.apply_consumer_offset_commit(pending);
self.pending_consumer_offset_commits.remove(&op);
+ self.refresh_consumer_offset_reservation(pending.kind, pending.consumer_id);
Ok(())
}
async fn persist_consumer_offset_commit(
&self,
pending: PendingConsumerOffsetCommit,
) -> Result<(), IggyError> {
- let Some(path) = self.persisted_offset_path(pending.kind, pending.consumer_id) else {
- return Ok(());
- };
- let key = (pending.kind, pending.consumer_id);
+ let path = self.persisted_offset_path(pending.kind, pending.consumer_id);
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
match pending.mutation {
// A server auto-commit persists monotonically: its op offset can
// trail the durably-recorded value (disk-tier polls replicate in
// IO-completion order), so a plain overwrite would rewind the file
- // and re-deliver on restart. The `persisted_offsets` tracker keeps
+ // and re-deliver on restart. The durable offset tracker keeps
// the fold off the file: a covered offset skips the write, an
// advancing one blind-writes, and only a cold key (first commit
// after boot) reads the file once. Explicit client stores
// overwrite, so a deliberate offset reset still holds. Mirrors the
// in-memory `upsert_offset_max` vs `upsert_offset` split in the
// commit-apply.
PendingConsumerOffsetMutation::Upsert(offset) if pending.auto_commit => {
- let tracked = self.persisted_offsets.borrow().get(&key).copied();
- let persisted = match tracked {
- Some(high_water) if offset <= high_water => return Ok(()),
- Some(_) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- offset
+ let tracked = self
+ .durable_consumer_offsets
+ .get(pending.kind, pending.consumer_id);
+ let persisted_high_water = match (path.as_deref(), tracked) {
+ (None, _) => offset,
+ (Some(_), Some(state)) if offset <= state.persisted_high_water => {
+ state.persisted_high_water
}
- None => {
- persist_offset_max(&path, offset, self.consumer_offset_enforce_fsync)
- .await?
+ (Some(path), Some(state)) => {
+ let value = state.committed_offset.max(offset);
+ persist_offset(path, value, self.consumer_offset_enforce_fsync).await?;
+ value
+ }
+ (Some(path), None) => {
Review Comment:
nit: `persist_offset_max` skips its write when the file already covers the value, so this branch can write nothing. the dirty-bit fix above needs its predicate from the return value, not the branch.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -1577,6 +1710,45 @@ where
B: MessageBus,
SB: SuperblockStore,
{
+ fn plan_transfer_offset_writes(
+ &self,
+ offsets_wire: &ConsumerOffsetsWire,
+ next_offset: u64,
+ ) -> Result<Vec<PlannedOffsetWrite>, PartitionInstallError> {
+ let consumer_dir = self.consumer_offsets_path.as_deref().ok_or_else(|| {
+ PartitionInstallError::OffsetPersistence {
+ path: "consumer offset directory".to_owned(),
+ source: iggy_common::IggyError::InvalidConfiguration,
+ }
+ })?;
+ let group_dir = self.consumer_group_offsets_path.as_deref().ok_or_else(|| {
+ PartitionInstallError::OffsetPersistence {
+ path: "consumer group offset directory".to_owned(),
+ source: iggy_common::IggyError::InvalidConfiguration,
+ }
+ })?;
+ let clamp = |offset: u64| next_offset.checked_sub(1).map(|last| offset.min(last));
+ let mut planned =
+ Vec::with_capacity(offsets_wire.consumers.len() + offsets_wire.groups.len());
+ planned.extend(offsets_wire.consumers.iter().filter_map(|(id, offset)| {
Review Comment:
simplification: the two `planned.extend(..)` blocks differ only in kind, dir and source vec. one closure applied twice.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -1577,6 +1710,45 @@ where
B: MessageBus,
SB: SuperblockStore,
{
+ fn plan_transfer_offset_writes(
+ &self,
+ offsets_wire: &ConsumerOffsetsWire,
+ next_offset: u64,
+ ) -> Result<Vec<PlannedOffsetWrite>, PartitionInstallError> {
+ let consumer_dir = self.consumer_offsets_path.as_deref().ok_or_else(|| {
Review Comment:
simplification: this fakes a path string to report a missing directory config. a `NoOffsetDir` variant like `NoPartitionDir` drops two sentinels and a `Display` that lies.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2186,38 +2323,58 @@ where
.collect()
}
- /// Reclaim every stored consumer-group offset whose group id is no longer
- /// `is_live`, returning the owned persisted-file paths the caller must unlink.
- ///
- /// Fully synchronous (no `.await`): the in-memory papaya remove happens here,
- /// the disk unlink is deferred to the caller on owned `String` data so no
- /// borrow of `self` survives across the await. This is the only safe shape
- /// for the reconciler, which runs on a sibling task to the pump that may
- /// realloc the partitions vec during that await. The remove-then-unlink
- /// ordering matches the crash-safe GC invariant (monotonic, never-reused
- /// group ids mean a recreated group never reads a dead group's offset).
+ /// Snapshot dead group keys for deletion through the partition's VSR log.
+ /// A local unlink could free the primary's quota while backups retained
+ /// every older generation, so reclamation uses the same ordered delete as
+ /// an explicit consumer-offset request.
#[must_use]
- #[allow(clippy::cast_possible_truncation)]
- pub fn reclaim_dead_group_offsets(&self, is_live: impl Fn(u64) -> bool) -> Vec<String> {
- let pinned = self.consumer_group_offsets.pin();
- let dead: Vec<u64> = pinned
- .keys()
- .map(|key| key.0 as u64)
- .filter(|group_id| !is_live(*group_id))
- .collect();
- let mut paths = Vec::with_capacity(dead.len());
- for group_id in dead {
- pinned.remove(&ConsumerGroupId(group_id as usize));
- self.persisted_offsets
- .borrow_mut()
- .remove(&(ConsumerKind::ConsumerGroup, group_id as u32));
- if let Some(path) =
- self.persisted_offset_path(ConsumerKind::ConsumerGroup, group_id as u32)
- {
- paths.push(path);
+ pub fn dead_consumer_group_offset_ids(&self, is_live: impl Fn(u64) -> bool) -> Vec<u32> {
+ if !self.consensus.is_primary() || !self.consensus.is_normal() {
+ return Vec::new();
+ }
+ let mut dead = Vec::new();
+ for key in self.consumer_group_offsets.pin().keys() {
+ let Ok(id) = u32::try_from(key.0) else {
+ continue;
+ };
+ if !is_live(u64::from(id)) {
+ dead.push(id);
}
}
- paths
+ // A stranded file already failed normal loading or unlink. Reissuing
+ // replicated deletes every reconciliation pass cannot make its
+ // filesystem writable and would create a permanent commit loop.
+ // A single-replica explicit deletion can retry after repair. On a
+ // replicated partition the file must be repaired or removed locally,
+ // because older peers do not recognize a delete for a map-missing key.
+ dead.sort_unstable();
+ dead.dedup();
+ self.consumer_group_offsets_need_reconcile
+ .set(!dead.is_empty());
+ dead
+ }
+
+ #[must_use]
Review Comment:
simplification: `consumer_group_offsets_reconcile_needed()` has no caller outside this file's tests - `reconcile_consumer_group_offsets` walks every namespace ungated, so only the shard epoch does work. drop the reader and the `Cell`.
##########
core/server/src/partition_helpers.rs:
##########
@@ -268,15 +313,21 @@ fn load_partition_consumer_offsets(
stream_id: usize,
topic_id: usize,
partition_id: usize,
-) -> Result<Vec<iggy_common::ConsumerOffset>, ServerError> {
+) -> Result<RecoveredOffsets<iggy_common::ConsumerOffset>, ServerError> {
if !Path::new(path).exists() {
Review Comment:
simplification: four copies of `RecoveredOffsets { entries: Vec::new(), stranded_ids: Vec::new() }`. a hand-written `Default` (no `T: Default` bound) removes all of them.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2649,18 +2872,20 @@ where
let next_offset = offsets_wire
Review Comment:
simplification: `next_offset` is computed here and again at line 2486. pass it in - `installed_end` has to stay, line 3027 reads it.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -4470,6 +5092,56 @@ where
self.drain_request_queue_into_prepares(drained_count).await;
}
+ async fn flush_consumer_offset_directories(&self) -> Result<(), IggyError> {
+ for (index, dir) in [
+ self.consumer_offsets_path.as_deref(),
+ self.consumer_group_offsets_path.as_deref(),
+ ]
+ .into_iter()
+ .enumerate()
+ {
+ if !self.consumer_offset_dirs_dirty[index].get() {
+ continue;
+ }
+ if let Some(dir) = dir {
+ match crate::state_transfer::fsync_dir(dir).await {
+ Ok(()) => {}
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
+ // The directory disappeared after the final unlink.
+ // There is no remaining dirent whose durability needs
+ // proving.
+ }
+ Err(error) => {
+ warn!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica_id = self.consensus.replica(),
+ namespace_raw = self.namespace().inner(),
+ path = dir,
+ error_kind = ?error.kind(),
+ %error,
+ "consumer offset directory sync failed"
+ );
+ return Err(IggyError::CannotSyncFile);
+ }
+ }
+ #[cfg(test)]
+ self.offset_dir_sync_count
+ .set(self.offset_dir_sync_count.get() + 1);
+ }
+ self.consumer_offset_dirs_dirty[index].set(false);
+ }
+ Ok(())
+ }
+
+ fn mark_consumer_offset_dir_dirty(&self, kind: ConsumerKind) {
+ let index = match kind {
Review Comment:
simplification: this repeats `consumer_kind_index` from `state_transfer.rs:1664`, and both index `[_; 2]` arrays under the same convention. make it `pub(crate)` and call it - two copies will drift.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,658 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, Ref, RefCell};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: u64,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+ membership_epoch: Cell<u64>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset && state.persisted_high_water >= offset
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) -> bool {
+ let created = self
+ .entries(kind)
+ .borrow_mut()
+ .insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ )
+ .is_none();
+ if created {
+ self.bump_membership_epoch();
+ }
+ created
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ match entries.entry(id) {
+ Entry::Occupied(mut entry) => {
+ let state = entry.get_mut();
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = state.persisted_high_water.max(persisted_high_water);
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ });
+ drop(entries);
+ self.bump_membership_epoch();
+ }
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ let removed = self.entries(kind).borrow_mut().remove(&id).is_some();
+ if removed {
+ self.bump_membership_epoch();
+ }
+ removed
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ self.bump_membership_epoch();
+ }
+
+ #[cfg(any(test, feature = "simulator"))]
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ pub(crate) fn snapshot_entries(
+ &self,
+ kind: ConsumerKind,
+ ) -> Ref<'_, HashMap<u32, DurableOffsetState>> {
+ self.entries(kind).borrow()
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+
+ fn bump_membership_epoch(&self) {
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Arc<ProvisionalToken>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
+ map_warned: Cell<bool>,
+ reclaim_epoch: Arc<AtomicU64>,
+ last_reclaim: Cell<Option<(u64, u64)>>,
+}
+
+impl ConsumerOffsetCapacity {
+ pub(crate) fn new(kind: ConsumerKind, limit: usize) -> Self {
+ Self {
+ kind,
+ limit: Cell::new(limit),
+ pending: RefCell::new(HashMap::new()),
+ provisional: RefCell::new(HashMap::new()),
+ stranded: RefCell::new(HashSet::new()),
+ uncertain: Cell::new(false),
+ durable_warned: Cell::new(false),
+ map_warned: Cell::new(false),
+ reclaim_epoch: Arc::new(AtomicU64::new(0)),
+ last_reclaim: Cell::new(None),
+ }
+ }
+
+ pub(crate) fn set_limit(&self, limit: usize) {
+ self.limit.set(limit);
+ }
+
+ pub(crate) const fn limit(&self) -> usize {
+ self.limit.get()
+ }
+
+ pub(crate) fn try_reserve(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ *self.pending.borrow_mut().entry(id).or_default() += 1;
+ Ok(())
+ }
+
+ pub(crate) fn check(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ if self.holds(id, durable) || self.stranded.borrow().contains(&id) {
+ return Ok(());
+ }
+ let limit = self.limit.get();
+ let durable_count = durable.count(self.kind);
+ // A full durable table cannot gain room by pruning provisional keys.
+ let occupied = if durable_count >= limit {
+ durable_count
+ } else {
+ let mut provisional = self.provisional.borrow_mut();
+ if provisional.len() >= limit {
+ provisional
+ .retain(|key, token| *key == id || token.active.load(Ordering::Relaxed) > 0);
+ }
+ drop(provisional);
+ self.occupied(durable)
+ };
+ if self.uncertain.get() || occupied >= limit {
+ return Err(ConsumerOffsetCapacityError {
+ kind: self.kind,
+ occupied,
+ limit,
+ first_in_episode: !self.durable_warned.replace(true),
+ uncertain: self.uncertain.get(),
+ });
+ }
+ self.durable_warned.set(false);
+ Ok(())
+ }
+
+ pub(crate) fn reserve_provisional(
+ self: &Rc<Self>,
+ id: u32,
+ durable: &Rc<DurableConsumerOffsets>,
+ ) -> Result<AutoCommitReservation, ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ let mut provisional = self.provisional.borrow_mut();
+ let token = Arc::clone(provisional.entry(id).or_insert_with(|| {
+ Arc::new(ProvisionalToken {
+ reclaim_epoch: Arc::clone(&self.reclaim_epoch),
+ active: AtomicUsize::new(0),
+ })
+ }));
+ token.active.fetch_add(1, Ordering::Relaxed);
+ Ok(AutoCommitReservation {
+ token,
+ kind: self.kind,
+ consumer_id: id,
+ })
+ }
+
+ pub(crate) fn owns(&self, reservation: &AutoCommitReservation) -> bool {
+ reservation.kind == self.kind
+ && self
+ .provisional
+ .borrow()
+ .get(&reservation.consumer_id)
+ .is_some_and(|token| Arc::ptr_eq(token, &reservation.token))
+ }
+
+ pub(crate) fn holds(&self, id: u32, durable: &DurableConsumerOffsets) -> bool {
+ durable.contains(self.kind, id)
+ || self.pending.borrow().contains_key(&id)
+ || self
+ .provisional
+ .borrow()
+ .get(&id)
+ .is_some_and(|token| token.active.load(Ordering::Relaxed) > 0)
+ }
+
+ pub(crate) fn set_pending_count(&self, id: u32, count: usize) {
+ if count == 0 {
+ if self.pending.borrow_mut().remove(&id).is_some() {
+ self.note_local_key_change();
+ }
+ } else {
+ self.pending.borrow_mut().insert(id, count);
+ }
+ }
+
+ pub(crate) fn release_reservation(&self, id: u32) {
+ let mut pending = self.pending.borrow_mut();
+ let Some(count) = pending.get_mut(&id) else {
+ return;
+ };
+ if *count == 1 {
+ pending.remove(&id);
+ self.note_local_key_change();
+ } else {
+ *count -= 1;
+ }
+ }
+
+ pub(crate) const fn is_uncertain(&self) -> bool {
+ self.uncertain.get()
+ }
+
+ pub(crate) fn rebuild(
+ &self,
+ durable: &DurableConsumerOffsets,
+ pending_ids: impl IntoIterator<Item = u32>,
+ ) {
+ let mut pending = self.pending.borrow_mut();
+ pending.clear();
+ for id in pending_ids {
+ *pending.entry(id).or_default() += 1;
+ }
+ drop(pending);
+ self.note_local_key_change();
+ self.uncertain.set(false);
+ self.rearm_if_below_limit(durable);
+ }
+
+ pub(crate) fn mark_uncertain(&self) {
+ self.pending.borrow_mut().clear();
+ self.uncertain.set(true);
+ self.note_local_key_change();
+ }
+
+ pub(crate) fn record_stranded(&self, id: u32) {
+ self.stranded.borrow_mut().insert(id);
+ }
+
+ pub(crate) fn clear_stranded(&self, id: u32) {
+ self.stranded.borrow_mut().remove(&id);
+ }
+
+ pub(crate) fn is_stranded(&self, id: u32) -> bool {
+ self.stranded.borrow().contains(&id)
+ }
+
+ pub(crate) fn rearm_if_below_limit(&self, durable: &DurableConsumerOffsets) {
+ if !self.durable_warned.get()
+ || self.uncertain.get()
+ || durable.count(self.kind) >= self.limit.get()
+ {
+ return;
+ }
+ if self.occupied(durable) < self.limit.get() {
+ self.durable_warned.set(false);
+ }
+ }
+
+ pub(crate) const fn admit_local_map_key(
+ &self,
+ map_len: usize,
+ durable_full: bool,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ let limit = self.limit.get();
+ if map_len < limit {
+ return Ok(());
+ }
+ Err(ConsumerOffsetCapacityError {
+ kind: self.kind,
+ occupied: map_len,
+ limit,
+ first_in_episode: !self.map_warned.replace(true),
+ uncertain: !durable_full,
+ })
+ }
+
+ pub(crate) fn rearm_map_if_below_limit(&self, map_len: usize) {
+ if map_len < self.limit.get() {
+ self.map_warned.set(false);
+ }
+ }
+
+ pub(crate) fn note_local_key_change(&self) {
+ self.reclaim_epoch.fetch_add(1, Ordering::Relaxed);
+ }
+
+ pub(crate) fn forget_inactive_provisional(&self, id: u32) {
+ let mut provisional = self.provisional.borrow_mut();
+ if provisional
+ .get(&id)
+ .is_some_and(|token| token.active.load(Ordering::Relaxed) == 0)
+ {
+ provisional.remove(&id);
+ }
+ }
+
+ pub(crate) fn should_reclaim(&self, durable: &DurableConsumerOffsets) -> bool {
+ if self.uncertain.get() {
+ return false;
+ }
+ let epoch = (
+ self.reclaim_epoch.load(Ordering::Relaxed),
+ durable.membership_epoch.get(),
+ );
+ self.last_reclaim.replace(Some(epoch)) != Some(epoch)
+ }
+
+ pub(crate) fn occupied(&self, durable: &DurableConsumerOffsets) -> usize {
Review Comment:
simplification: the three-way chain is a set union, so one `HashSet` pass minus durable members is about 14 lines shorter. only worth it alongside the admission fast path, since it allocates where this does not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz merged PR #4063:
URL: https://github.com/apache/iggy/pull/4063
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4063:
URL: https://github.com/apache/iggy/pull/4063#issuecomment-5550013207
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/4063?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 85.09%. Comparing base ([`63d3273`](https://app.codecov.io/gh/apache/iggy/commit/63d3273b714c168102dde52b88fe3fc50edd73b9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`0398708`](https://app.codecov.io/gh/apache/iggy/commit/039870803b8b1656ba294b6faec7159c46dfe8fe?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4063 +/- ##
============================================
- Coverage 85.51% 85.09% -0.43%
Complexity 1402 1402
============================================
Files 1240 1238 -2
Lines 186294 185730 -564
Branches 152598 152240 -358
============================================
- Hits 159316 158052 -1264
- Misses 22923 23630 +707
+ Partials 4055 4048 -7
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/4063/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/4063/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.46% <ø> (+<0.01%)` | :arrow_up: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/4063/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.29% <ø> (ø)` | |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/4063/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.33% <ø> (ø)` | |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/4063/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.06% <ø> (ø)` | |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/4063/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.65% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/4063/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `89.92% <100.00%> (-6.24%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/4063/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.31% <ø> (ø)` | |
| [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/4063?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core/binary\_protocol/src/primitives/ack\_level.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fprimitives%2Fack_level.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL3ByaW1pdGl2ZXMvYWNrX2xldmVsLnJz) | `97.61% <ø> (ø)` | |
| [...equests/consumer\_offsets/delete\_consumer\_offset.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Frequests%2Fconsumer_offsets%2Fdelete_consumer_offset.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL3JlcXVlc3RzL2NvbnN1bWVyX29mZnNldHMvZGVsZXRlX2NvbnN1bWVyX29mZnNldC5ycw==) | `100.00% <ø> (ø)` | |
| [...requests/consumer\_offsets/store\_consumer\_offset.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Frequests%2Fconsumer_offsets%2Fstore_consumer_offset.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL3JlcXVlc3RzL2NvbnN1bWVyX29mZnNldHMvc3RvcmVfY29uc3VtZXJfb2Zmc2V0LnJz) | `100.00% <ø> (ø)` | |
| [core/common/src/error/iggy\_error.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fcommon%2Fsrc%2Ferror%2Figgy_error.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb21tb24vc3JjL2Vycm9yL2lnZ3lfZXJyb3IucnM=) | `100.00% <ø> (ø)` | |
| [core/common/src/traits/message\_client.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fcommon%2Fsrc%2Ftraits%2Fmessage_client.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb21tb24vc3JjL3RyYWl0cy9tZXNzYWdlX2NsaWVudC5ycw==) | `100.00% <ø> (ø)` | |
| [core/configs/src/server\_config/defaults.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fconfigs%2Fsrc%2Fserver_config%2Fdefaults.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25maWdzL3NyYy9zZXJ2ZXJfY29uZmlnL2RlZmF1bHRzLnJz) | `100.00% <ø> (ø)` | |
| [core/configs/src/server\_config/displays.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fconfigs%2Fsrc%2Fserver_config%2Fdisplays.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25maWdzL3NyYy9zZXJ2ZXJfY29uZmlnL2Rpc3BsYXlzLnJz) | `100.00% <ø> (ø)` | |
| [core/configs/src/server\_config/partition.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fconfigs%2Fsrc%2Fserver_config%2Fpartition.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25maWdzL3NyYy9zZXJ2ZXJfY29uZmlnL3BhcnRpdGlvbi5ycw==) | `97.46% <ø> (ø)` | |
| [core/message\_bus/src/lib.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fmessage_bus%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9tZXNzYWdlX2J1cy9zcmMvbGliLnJz) | `97.57% <ø> (ø)` | |
| [core/metadata/src/impls/metadata.rs](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree&filepath=core%2Fmetadata%2Fsrc%2Fimpls%2Fmetadata.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9tZXRhZGF0YS9zcmMvaW1wbHMvbWV0YWRhdGEucnM=) | `88.68% <ø> (ø)` | |
| ... and [24 more](https://app.codecov.io/gh/apache/iggy/pull/4063?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [53 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/4063/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4063:
URL: https://github.com/apache/iggy/pull/4063#discussion_r3940241746
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2813,6 +2945,15 @@ where
}
}
+ if !offsets_written {
Review Comment:
warning: one failed offset write or dir fsync now fails the install after the journal is cleared and old files unlinked, so converge wipes the segments and the shard re-pulls from the next peer. retry the writes a few times first, like the segment open at line 2632, converge steps at 3119 and 3122 included.
##########
core/sdk/src/clients/consumer.rs:
##########
@@ -1361,9 +1366,9 @@ impl IggyConsumer {
if is_consumer_group {
joined_consumer_group.store(false, ORDERING);
}
- trace!("Retrying to poll messages in {retry_interval}...");
- sleep(retry_interval.get_duration()).await;
}
+ trace!("Retrying to poll messages in {retry_interval}...");
+ sleep(retry_interval.get_duration()).await;
Review Comment:
warning: this sleep used to sit inside the disconnect/unauthenticated/stale arm, now every poll error waits `polling_retry_interval` before it surfaces, and the docs at 469 and 551 still describe the old scope. scope it to the retryable errors, or keep it and update both docs and the PR body.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -395,13 +447,15 @@ pub async fn dispatch_partition_request<B, MJ, S, SB>(
operation = ?header.operation,
"partition request with unresolved namespace; replying denied"
);
- send_deny_reply(
- shard,
- transport_client_id,
- &header,
- IggyError::ResourceNotFound(String::new()).as_code(),
- )
- .await;
+ let status = if matches!(
Review Comment:
warning: only the two group-not-found codes get through, `ConsumerGroupPartitionNotOwned` and `InvalidIdentifier` from the same fence still collapse to `ResourceNotFound` on store and delete, while the poll path types them. forward `error.as_code()` for every resolver error.
##########
core/server/src/partition_reconciler.rs:
##########
@@ -960,47 +994,114 @@ async fn tear_down_owned_partition(
counters.removed_local += 1;
}
-/// Reclaim consumer-group offsets left behind by a `DeleteConsumerGroup` whose
-/// topic still exists (a topic/stream delete already drops the whole partition
-/// directory, offsets included). For each owned partition, any stored
-/// consumer-group offset whose group id is no longer present in the topic's
-/// committed metadata is removed (in-memory entry + persisted file). Monotonic,
-/// never-reused group ids make this purely reclamation -- a recreated group
-/// gets a fresh id and never reads a dead group's offset -- so it is safe to do
-/// lazily on the reconcile pass rather than synchronously on delete.
-async fn reconcile_consumer_group_offsets(ctx: &ReconcilerCtx, counters: &mut PassCounters) {
+/// Reclaim deleted groups through ordered offset deletes. Replicas must see
+/// each delete before a replacement store can reuse its durable slot.
+fn reconcile_consumer_group_offsets(ctx: &ReconcilerCtx, counters: &mut PassCounters) {
let live_groups = snapshot_topic_live_groups(ctx);
let partitions = ctx.shard.plane.partitions();
let owned: Vec<IggyNamespace> = partitions.namespaces().copied().collect();
- for ns in owned {
- let live = live_groups.get(&(ns.stream_id(), ns.topic_id()));
- // Take the in-memory removes + owned unlink paths under a closure-scoped
- // borrow that cannot escape into the await below. Holding a raw
- // `&IggyPartition` across `delete_persisted_offset().await` would let the
- // pump task realloc the partitions vec underneath us (a UAF).
- let paths = partitions.with_partition(&ns, |partition| {
- partition.reclaim_dead_group_offsets(|group_id| {
- live.is_some_and(|set| set.contains(&group_id))
- })
- });
- let Some(paths) = paths else {
+ for namespace in owned {
+ if ctx
+ .group_offset_cleanup_inflight
+ .borrow()
+ .contains(&namespace)
+ {
+ continue;
+ }
+ let Some((next_group_id, live)) =
+ live_groups.get(&(namespace.stream_id(), namespace.topic_id()))
+ else {
continue;
};
- for path in paths {
- if let Err(err) = delete_persisted_offset(&path).await {
- warn!(
- shard = ctx.shard.id,
- ns_raw = ns.inner(),
- error = %err,
- "reconciler failed to reclaim deleted consumer-group offset"
- );
- continue;
+ let dead = partitions
+ .with_partition(&namespace, |partition| {
+ partition.dead_consumer_group_offset_ids(|group_id| {
+ // A lagging metadata replica cannot prove a group deleted
+ // until it has applied the allocation of that group's id.
+ group_id >= *next_group_id || live.contains(&group_id)
+ })
+ })
+ .unwrap_or_default();
+ // Bound work per pass so a historical directory cannot monopolize the
+ // reconciler. Unprocessed keys keep the partition's dirty flag armed.
+ let mut tickets = Vec::with_capacity(dead.len().min(GROUP_OFFSET_DELETES_PER_PASS));
+ for consumer_id in dead.into_iter().take(GROUP_OFFSET_DELETES_PER_PASS) {
Review Comment:
warning: a committed delete for a stranded id whose unlink fails again (numeric dir, `chattr +i`, EROFS) ends in `FatalCommit`, and boot replays the op, so the replica stays down until someone fixes the fs. treat unlink failure on a committed delete as strand-and-ok, and skip ids that already failed or this turns into a delete loop.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -4320,13 +4859,13 @@ where
let committed_batch_stats = self.resolve_committed_visible_offsets(&drained);
let mut messages_committed = false;
- for (mut entry, batch_stats) in drained.into_iter().zip(committed_batch_stats) {
+ for (entry, batch_stats) in drained.iter().zip(&committed_batch_stats) {
Review Comment:
nit: the walk is now apply-all then advance-and-reply-all, so a `FatalCommit` at entry k leaves k-1 ops applied but never advanced, waiters dropped, dedup fold at 4935 skipped, and the flush failure at 4913 blames `drained.last()` instead of the failing op. comment the widened window, or advance and reply per entry.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,615 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, RefCell};
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, Weak};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: Option<u64>,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+ membership_epoch: Cell<u64>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset
+ && state
+ .persisted_high_water
+ .is_some_and(|persisted| persisted >= offset)
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: Option<u64>,
+ ) {
+ self.entries(kind).borrow_mut().insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ );
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ let state = entries.entry(id).or_insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water: None,
+ });
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = Some(
+ state
+ .persisted_high_water
+ .unwrap_or(0)
+ .max(persisted_high_water),
+ );
+ }
+
+ pub(crate) fn mark_persisted(&self, kind: ConsumerKind, id: u32, high_water: u64) {
+ if let Some(state) = self.entries(kind).borrow_mut().get_mut(&id) {
+ state.persisted_high_water = Some(high_water);
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ let removed = self.entries(kind).borrow_mut().remove(&id).is_some();
+ if removed {
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+ removed
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Weak<ProvisionalToken>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
+ map_warned: Cell<bool>,
+ reclaim_epoch: Arc<AtomicU64>,
+ last_reclaim: Cell<Option<(u64, u64)>>,
+}
+
+impl ConsumerOffsetCapacity {
+ pub(crate) fn new(kind: ConsumerKind, limit: usize) -> Self {
+ Self {
+ kind,
+ limit: Cell::new(limit),
+ pending: RefCell::new(HashMap::new()),
+ provisional: RefCell::new(HashMap::new()),
+ stranded: RefCell::new(HashSet::new()),
+ uncertain: Cell::new(false),
+ durable_warned: Cell::new(false),
+ map_warned: Cell::new(false),
+ reclaim_epoch: Arc::new(AtomicU64::new(0)),
+ last_reclaim: Cell::new(None),
+ }
+ }
+
+ pub(crate) fn set_limit(&self, limit: usize) {
+ self.limit.set(limit);
+ }
+
+ pub(crate) const fn limit(&self) -> usize {
+ self.limit.get()
+ }
+
+ pub(crate) fn try_reserve(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ *self.pending.borrow_mut().entry(id).or_default() += 1;
+ Ok(())
+ }
+
+ pub(crate) fn check(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ if durable.contains(self.kind, id)
+ || self.pending.borrow().contains_key(&id)
+ || self
+ .provisional
+ .borrow()
+ .get(&id)
+ .is_some_and(|token| token.strong_count() > 0)
+ || self.stranded.borrow().contains(&id)
+ {
+ return Ok(());
+ }
+ let limit = self.limit.get();
+ let durable_count = durable.count(self.kind);
+ // A full durable table cannot gain room by pruning provisional keys.
+ let occupied = if durable_count >= limit {
+ durable_count
+ } else {
+ self.provisional
+ .borrow_mut()
+ .retain(|_, token| token.strong_count() > 0);
+ self.occupied(durable)
+ };
+ if self.uncertain.get() || occupied >= limit {
+ return Err(ConsumerOffsetCapacityError {
+ kind: self.kind,
+ occupied,
+ limit,
+ first_in_episode: !self.durable_warned.replace(true),
+ uncertain: self.uncertain.get(),
+ });
+ }
+ self.durable_warned.set(false);
+ Ok(())
+ }
+
+ pub(crate) fn reserve_provisional(
+ self: &Rc<Self>,
+ id: u32,
+ durable: &Rc<DurableConsumerOffsets>,
+ ) -> Result<AutoCommitReservation, ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ let mut provisional = self.provisional.borrow_mut();
+ let token = provisional
+ .get(&id)
+ .and_then(Weak::upgrade)
+ .unwrap_or_else(|| {
+ let token = Arc::new(ProvisionalToken {
Review Comment:
nit: this mints a fresh `Arc<ProvisionalToken>` per auto-commit poll past durable, and the token is dead again by the next poll, so it is a malloc and free per poll. a strong `Arc` in the map fixes it only if the reclaim-epoch bump moves out of `ProvisionalToken::drop` into a `Drop` for the reservation, or reclaim stops firing.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -1919,53 +1968,93 @@ where
self.transfer_offer_cache.borrow_mut().take();
}
- /// Snapshot the live offset maps + purge generation into the wire shape.
- /// Eagerly auto-committed offsets can run slightly ahead of committed
- /// state; that is safe because their covering ops sit in
- /// `(commit_op, commit_max]`, which the receiver's tail repair replays,
- /// and offset applies converge (monotone auto-commit, verbatim stores).
- fn offsets_wire_snapshot(&self) -> ConsumerOffsetsWire {
- // Every key is minted from a u32 wire id, so the narrowing filter is
- // an invariant, not a policy: say so out loud instead of silently
- // shrinking the snapshot when it ever breaks.
- let mut consumers: Vec<(u32, u64)> = self
- .consumer_offsets
- .pin()
- .iter()
- .filter_map(|(id, offset)| {
- let narrowed = u32::try_from(*id).ok();
- debug_assert!(narrowed.is_some(), "consumer offset key {id} exceeds u32");
- narrowed.map(|id| (id, offset.offset.load(Ordering::Acquire)))
- })
- .collect();
+ fn validate_consumer_offset_transfer_counts(&self) -> Result<(), PartitionTransferUnavailable> {
+ for kind in [ConsumerKind::Consumer, ConsumerKind::ConsumerGroup] {
+ let count = self.durable_consumer_offsets.count(kind);
+ if let Err(error) = validate_consumer_offset_transfer_count(
+ kind,
+ count,
+ CONSUMER_OFFSETS_ENTRIES_MAX as usize,
+ ) {
+ tracing::error!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ namespace_raw = self.consensus().group(),
+ ?kind,
+ count,
+ max = CONSUMER_OFFSETS_ENTRIES_MAX,
+ "consumer offset state exceeds the transfer ceiling"
+ );
+ return Err(error);
+ }
+ }
+ Ok(())
+ }
+
+ /// Snapshot committed durable offsets only. Eager auto-commit progress and
+ /// follower-local cursor entries stay in the live maps until a replicated
+ /// store commits them, so neither can be promoted by state transfer.
+ fn offsets_wire_snapshot(&self) -> Result<ConsumerOffsetsWire, PartitionTransferUnavailable> {
+ self.validate_consumer_offset_transfer_counts()?;
+ let mut consumers = self
Review Comment:
simplification: `committed_entries()` walks the map and then `durable.get()` re-probes each id, and the 16-line body is repeated for both kinds. one `snapshot_kind(kind, map_contains)` over the durable map does it in one pass, it just needs a `pub(crate)` `entries` accessor.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2260,30 +2425,56 @@ where
|value| PendingConsumerOffsetCommit::upsert(kind, consumer_id, value),
);
- if let Err(error) = self.persist_consumer_offset_commit(pending).await {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(self.diag_ctx(), "no_ack offset persist failed")
- .with_operation(request_header.operation)
- .with_error(error.to_string()),
- );
- return;
+ let persisted = async {
+ self.persist_consumer_offset_commit(pending).await?;
+ self.flush_consumer_offset_directories().await
}
- if let Err(error) = self.apply_consumer_offset_commit(pending) {
+ .await;
+ if let Err(error) = persisted {
+ if offset.is_some() {
+ self.release_consumer_offset_reservation(kind, consumer_id);
+ if !self.durable_consumer_offsets.contains(kind, consumer_id)
+ && let Some(path) = self.persisted_offset_path(kind, consumer_id)
+ {
+ if delete_persisted_offset(&path).await.is_ok() {
+ self.consumer_offset_capacity_for(kind)
+ .clear_stranded(consumer_id);
+ } else {
+ self.consumer_offset_capacity_for(kind)
+ .record_stranded(consumer_id);
+ }
+ }
+ }
+ if offset.is_none() {
Review Comment:
simplification: `if offset.is_some() {..}` followed by `if offset.is_none() {..}` is an if/else. make it `} else {`.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2421,6 +2830,29 @@ where
}
};
+ if args.auto_commit
+ && let Ok(pending) = PendingConsumerOffsetCommit::try_from_polling_consumer(consumer, 0)
+ {
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
+ if !capacity.is_uncertain()
+ && self.consumer_offset_map_count(pending.kind) >= capacity.limit()
Review Comment:
warning: `consumer_offset_map_count()` is papaya `len()`, a sum over one counter per cpu, and it runs on every auto-commit poll before we know the key is absent. check `contains_key` first and only call `len()` for absent keys.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2959,6 +3100,30 @@ where
}
self.log.journal().inner.clear_all();
self.log.journal_mut().info = crate::log::JournalInfo::default();
+ self.consumer_offsets.pin().clear();
+ self.consumer_group_offsets.pin().clear();
+ self.last_polled_offsets.pin().clear();
+ self.durable_consumer_offsets.clear();
+ self.pending_consumer_offset_commits.clear();
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ for kind in [ConsumerKind::Consumer, ConsumerKind::ConsumerGroup] {
+ self.consumer_offset_capacity_for(kind)
+ .rebuild(&self.durable_consumer_offsets, std::iter::empty());
+ }
+ for dir in self
+ .consumer_offsets_path
+ .iter()
+ .chain(self.consumer_group_offsets_path.iter())
+ {
+ for path in strayed_offset_files(Some(dir), &[]) {
Review Comment:
warning: converge unlinks the file but never calls `clear_stranded`, so an id the install failed to unlink keeps a capacity slot until restart and standalone consumer ids are never reclaimed. call `clear_stranded(numeric_offset_id(path))` per unlinked path, like 2826 and the purge path do.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2635,143 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ #[allow(clippy::too_many_lines)]
+ fn resynchronize_consumer_offset_reservations(&mut self) {
let current_view = self.consensus.view();
- if current_view == self.observed_view {
+ let scan_state = (
+ self.consensus.commit_min(),
+ self.consensus.commit_max(),
+ self.consensus.sequencer().current_sequence(),
+ self.log.journal().inner.last_op(),
+ );
+ let retry_uncertain = (self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain())
+ && self.offset_reservations_scan_state != Some(scan_state);
Review Comment:
warning: while a capacity is uncertain this retries on every op, and each retry is a full `repair_headers_in` scan of the headers and the evicted ring, on the replica that is already behind. retry once per consensus tick or when `last_op`/`commit_max` cross the failed scan's `to_op`. the comment at 5311 saying this never runs per commit is now stale.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2635,143 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ #[allow(clippy::too_many_lines)]
+ fn resynchronize_consumer_offset_reservations(&mut self) {
let current_view = self.consensus.view();
- if current_view == self.observed_view {
+ let scan_state = (
+ self.consensus.commit_min(),
+ self.consensus.commit_max(),
+ self.consensus.sequencer().current_sequence(),
+ self.log.journal().inner.last_op(),
+ );
+ let retry_uncertain = (self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain())
+ && self.offset_reservations_scan_state != Some(scan_state);
+ if current_view == self.observed_view
+ && !self.offset_reservations_need_resync.get()
+ && !retry_uncertain
+ {
return;
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ }
+
+ let from_op = self
+ .consensus
+ .commit_min()
+ .max(self.purge_floor_op)
+ .saturating_add(1);
+ let commit_max = self.consensus.commit_max();
+ let to_op = self
+ .consensus
+ .sequencer()
+ .current_sequence()
+ .min(self.log.journal().inner.last_op().unwrap_or(commit_max));
+ // Committed offset prepares still need local apply, even if a message
+ // flush already evicted their journal bytes. Never drop their staging.
+ let mut rebuilt: HashMap<_, _> = self
+ .pending_consumer_offset_commits
+ .iter()
+ .filter(|(op, _)| **op >= from_op && **op <= commit_max)
+ .map(|(op, pending)| (*op, *pending))
+ .collect();
+ let headers = self.log.journal().inner.repair_headers_in(from_op..=to_op);
+ let uncommitted_from = from_op.max(commit_max.saturating_add(1));
+ let expected = to_op
+ .checked_sub(uncommitted_from)
+ .map_or(0, |span| span.saturating_add(1));
+ let mut decode_failed =
+ headers.keys().filter(|op| **op >= uncommitted_from).count() as u64 != expected;
+ for (op, header) in headers {
+ if !matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) {
+ continue;
+ }
+ match self.restage_consumer_offset_from_journal(op) {
+ Ok(pending) => {
+ rebuilt.insert(op, pending);
+ }
+ Err(error) => {
+ error!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica_id = self.consensus.replica(),
+ namespace_raw = self.namespace().inner(),
+ op,
+ %error,
+ "failed to rebuild consumer offset reservations after view change"
+ );
+ decode_failed = true;
+ break;
+ }
+ }
+ }
+ self.pending_consumer_offset_commits = rebuilt;
+ if decode_failed {
+ self.consumer_offset_capacity.mark_uncertain();
+ self.consumer_group_offset_capacity.mark_uncertain();
+ } else {
+ let consumer_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::Consumer
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, consumer_ids);
+ let group_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::ConsumerGroup
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_group_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, group_ids);
+ }
self.observed_view = current_view;
+ self.consumer_group_offsets_need_reconcile.set(true);
+ self.offset_reservations_scan_state = Some(scan_state);
+ // Retry uncertainty on journal or frontier progress, not every tick.
+ self.offset_reservations_need_resync.set(false);
+ if !decode_failed {
+ self.reclaim_phantom_offsets(ConsumerKind::Consumer);
+ self.reclaim_phantom_offsets(ConsumerKind::ConsumerGroup);
+ }
+ }
+
+ fn reclaim_phantom_offsets(&self, kind: ConsumerKind) {
+ let capacity = self.consumer_offset_capacity_for(kind);
+ if self.durable_consumer_offsets.count(kind) >= capacity.limit()
+ || !capacity.should_reclaim(&self.durable_consumer_offsets)
+ {
+ return;
+ }
+ let keep = |id| capacity.protects(id, &self.durable_consumer_offsets);
+ match kind {
+ ConsumerKind::Consumer => self
Review Comment:
warning: this retain drops every unprotected follower-local cursor at once when one new consumer hits a full map, and each evicted consumer restarts at 0 on its next poll. evict only enough to admit the new key, or deny it with the transient error.
##########
core/server/src/partition_reconciler.rs:
##########
@@ -464,7 +480,21 @@ async fn reconcile_once(ctx: &ReconcilerCtx) -> bool {
if ctx.last_revision.get() == Some(revision)
&& ctx.last_pass_noop.get()
&& ctx.failure_state.borrow().is_empty()
+ && ctx.group_offset_cleanup_completed.get() == 0
&& !ctx.shard.has_parked_partition_frames()
+ && !ctx.shard.plane.partitions().namespaces().any(|namespace| {
Review Comment:
nit: this scan is O(owned partitions) on every wake, the 1s tick and every metadata commit tick, where the old guard was four O(1) reads. a shard-level dirty flag armed where `consumer_group_offsets_need_reconcile` is set keeps the skip O(1), it just has to honour the inflight exclusion at 491.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2260,30 +2425,56 @@ where
|value| PendingConsumerOffsetCommit::upsert(kind, consumer_id, value),
);
- if let Err(error) = self.persist_consumer_offset_commit(pending).await {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(self.diag_ctx(), "no_ack offset persist failed")
- .with_operation(request_header.operation)
- .with_error(error.to_string()),
- );
- return;
+ let persisted = async {
Review Comment:
warning: persist and flush are folded into one result, so a flush failure after a successful store skips the in-memory apply at 2468 and `offsets_wire_snapshot` then answers `ConsumerOffsetStateInconsistent`, which is non-transient - one dead consumer blocks state transfer from this replica. keep the two results apart and apply the commit whenever persist succeeded.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2008,54 +2118,100 @@ where
// durably stored; the in-memory update is idempotent on replay
// because we look up by (kind, id).
self.persist_consumer_offset_commit(pending).await?;
- self.apply_consumer_offset_commit(pending)?;
+ self.apply_consumer_offset_commit(pending);
self.pending_consumer_offset_commits.remove(&op);
+ self.refresh_consumer_offset_reservation(pending.kind, pending.consumer_id);
Ok(())
}
async fn persist_consumer_offset_commit(
&self,
pending: PendingConsumerOffsetCommit,
) -> Result<(), IggyError> {
- let Some(path) = self.persisted_offset_path(pending.kind, pending.consumer_id) else {
- return Ok(());
- };
- let key = (pending.kind, pending.consumer_id);
+ let path = self.persisted_offset_path(pending.kind, pending.consumer_id);
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
+ let creates_group = pending.kind == ConsumerKind::ConsumerGroup
Review Comment:
nit: `creates_group` probes durable and then `record_*` probes the same key again, and the auto-commit arm already has the answer in `tracked` at 2149. derive it from `tracked.is_none()` there and have `record_explicit` return `created` for the explicit arm.
also at line 2256.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2798,10 +3238,27 @@ where
message: Message<RoutedRequestHeader>,
reply: Option<consensus::Sender<Message<ReplyHeader>>>,
) {
+ self.on_request_with_reservation(message, reply, None).await;
+ }
+
+ #[allow(clippy::too_many_lines)]
+ pub(crate) async fn on_request_with_reservation(
+ &mut self,
+ message: Message<RoutedRequestHeader>,
+ reply: Option<consensus::Sender<Message<ReplyHeader>>>,
+ mut reservation: Option<crate::AutoCommitReservation>,
+ ) {
+ if reservation.as_ref().is_some_and(|reservation| {
Review Comment:
nit: this drops the auto-commit frame with a bare return, no log, so a stale-partition drop is invisible. add a `debug!` here and at the tombstoned/missing drops in iggy_partitions.rs:638.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2260,30 +2425,56 @@ where
|value| PendingConsumerOffsetCommit::upsert(kind, consumer_id, value),
);
- if let Err(error) = self.persist_consumer_offset_commit(pending).await {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(self.diag_ctx(), "no_ack offset persist failed")
- .with_operation(request_header.operation)
- .with_error(error.to_string()),
- );
- return;
+ let persisted = async {
+ self.persist_consumer_offset_commit(pending).await?;
+ self.flush_consumer_offset_directories().await
}
- if let Err(error) = self.apply_consumer_offset_commit(pending) {
+ .await;
+ if let Err(error) = persisted {
+ if offset.is_some() {
+ self.release_consumer_offset_reservation(kind, consumer_id);
+ if !self.durable_consumer_offsets.contains(kind, consumer_id)
Review Comment:
warning: `persist_offset` truncates then writes, so a failed NoAck store on a durable key leaves a 0-byte file, this guard skips cleanup for durable keys, and the next boot reads it as torn and loses the committed position. pre-existing, but this branch is new: temp-and-rename in `persist_offset`, or reset the persisted state here so the next commit blind-writes.
##########
core/common/src/traits/message_client.rs:
##########
@@ -31,6 +31,10 @@ pub trait MessageClient {
/// Polling a consumer group the client is not (or no longer) a member of fails with `ConsumerGroupMemberNotFound` rather than returning an empty batch, so the caller can rejoin.
/// A member that holds no partitions gets an empty batch whose `partition_id` is [`NO_ASSIGNED_PARTITION`](crate::NO_ASSIGNED_PARTITION).
#[allow(clippy::too_many_arguments)]
+ /// With server-side auto-commit enabled, a new consumer offset key can be
Review Comment:
nit: this paragraph sits after the `#[allow]` with no blank `///`, so rustdoc glues it onto the previous paragraph. move it above the attribute and add a separator.
##########
core/sdk/src/clients/consumer.rs:
##########
@@ -618,6 +618,11 @@ unsafe impl Sync for IggyConsumer {}
/// [`topic()`]: crate::prelude::IggyConsumerBuilder::topic
/// [`without_encryptor()`]: crate::prelude::IggyConsumerBuilder::without_encryptor
/// [`without_poll_interval()`]: crate::prelude::IggyConsumerBuilder::without_poll_interval
+///
+/// A server-side auto-commit poll can fail with `TooManyConsumerOffsets` when
Review Comment:
warning: true for the raw poll only; every `AutoCommit` variant except `Disabled` commits the same key client-side, and a 3024 there is logged at 237 and dropped at 1075 and 1102, so the stream keeps yielding with no durable progress. say only `AutoCommit::Disabled` avoids the key and that client-side commits fail the same way, logged not surfaced.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2075,16 +2231,35 @@ where
consumer_id: u32,
Review Comment:
simplification: `is_auto_commit_offset_covered` has no production caller left now that dispatch uses `reserve_durable`, only tests. delete it and assert on `durable_consumer_offsets.covers` in the tests.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -111,14 +111,27 @@ where
spawn_poll_io(Rc::clone(&shard), namespace, plan, reply);
}
Some(plan) => {
- let (fragments, current_offset, auto_commit) = plan.execute_resident();
- if let Some(applied) = auto_commit {
- submit_auto_commit(&shard, namespace, &applied);
- }
- let _ = reply.try_send(PartitionReadReply::Poll {
- fragments,
- current_offset,
- });
+ let result = match plan.execute_resident() {
+ Ok((fragments, current_offset, auto_commit)) => {
+ if let Some(applied) = auto_commit
+ && let Err(error) =
+ submit_auto_commit(&shard, namespace, &applied)
+ {
+ PartitionReadReply::Rejected(error)
+ } else {
+ PartitionReadReply::Poll {
+ fragments,
+ current_offset,
+ }
+ }
+ }
+ Err(error) => {
+ shard.metrics().record_consumer_offset_denied(error.kind);
Review Comment:
nit: this counts `uncertain: true` denials too, which the client sees as `TransientNotAccepted`, while the help text says admission limit and the relay at 633 already gates on the permanent variant. gate on `!error.uncertain`.
also at lines 229, 296.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -569,6 +624,17 @@ async fn relay_partition_reply<B, MJ, S, SB>(
// commits moments later. The client's read-timeout is the recovery.
return;
};
+ if reply
+ .as_slice()
+ .get(..size_of::<iggy_binary_protocol::ReplyHeader>())
+ .and_then(|bytes| {
+ bytemuck::checked::try_from_bytes::<iggy_binary_protocol::ReplyHeader>(bytes).ok()
Review Comment:
nit: the `ReplyHeader` parse runs before `consumer_kind` is checked, so every relayed write reply pays it though only `StoreConsumerOffset` has a kind. test `consumer_kind` first.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -3137,9 +3614,28 @@ where
/// is bypassed at view-change reset.
#[allow(clippy::future_not_send)]
pub async fn drain_request_queue_into_prepares(&mut self, slots_freed: usize) {
- for _ in 0..slots_freed {
+ self.resynchronize_consumer_offset_reservations();
+ let mut promoted = 0;
+ while promoted < slots_freed {
let req = self.consensus().pop_queued_request();
let Some(mut req) = req else { break };
+ let _reservation =
Review Comment:
simplification: the store body is parsed here and again at 3651, two `decode_from` and two name hashes per promoted store. parse once above the preflight, gated on `StoreConsumerOffset`, and reuse `(kind, id, offset)` in both blocks.
##########
core/server/config.toml:
##########
@@ -1024,6 +1024,22 @@ prepare_queue_depth = 32
# actually sees, not the node's client total.
dedup_clients_max = 4096
+# Distinct durable consumer-offset keys a partition primary admits per kind.
+# Standalone consumers and consumer groups are counted separately. Existing
+# keys remain writable at the limit. A new key is rejected before consensus.
+# A non-empty auto_commit poll that needs a new offset key is also rejected
+# with TooManyConsumerOffsets and returns no messages. Polls with auto_commit
+# disabled do not allocate offset keys and remain available.
+# UPGRADE: existing files are loaded even above this limit, but new keys then
+# remain blocked until offsets are explicitly deleted or this limit is raised.
+# Standalone offsets have no automatic expiry. Reuse stable consumer ids and
+# count existing numeric offset files per partition and kind before upgrading.
+# The environment override is IGGY_PARTITION_CONSUMER_OFFSETS_MAX.
+# On replicated partitions, offset stores and deletes carrying NoAck now wait
Review Comment:
nit: "now" is a changelog tell in shipped config, and the UPGRADE line above covers the quota, not NoAck. drop "now", and give the NoAck change its own UPGRADE line if operators need the before and after.
##########
core/common/src/error/iggy_error.rs:
##########
@@ -333,6 +333,8 @@ pub enum IggyError {
NotResolvedConsumer(Identifier) = 3022,
#[error("Cannot open consumer offsets file for path: {0}")]
CannotOpenConsumerOffsetsFile(String) = 3023,
+ #[error("Per-partition consumer offset limit reached (see [partition] consumer_offsets_max)")]
Review Comment:
nit: only error string here that embeds a server config key, mirrored into the go and node tables, so SDK users get an operator hint and a reword is four edits. use "Consumer offset limit reached for partition" and keep the config pointer in the server warn.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2008,54 +2118,100 @@ where
// durably stored; the in-memory update is idempotent on replay
// because we look up by (kind, id).
self.persist_consumer_offset_commit(pending).await?;
- self.apply_consumer_offset_commit(pending)?;
+ self.apply_consumer_offset_commit(pending);
self.pending_consumer_offset_commits.remove(&op);
+ self.refresh_consumer_offset_reservation(pending.kind, pending.consumer_id);
Ok(())
}
async fn persist_consumer_offset_commit(
&self,
pending: PendingConsumerOffsetCommit,
) -> Result<(), IggyError> {
- let Some(path) = self.persisted_offset_path(pending.kind, pending.consumer_id) else {
- return Ok(());
- };
- let key = (pending.kind, pending.consumer_id);
+ let path = self.persisted_offset_path(pending.kind, pending.consumer_id);
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
+ let creates_group = pending.kind == ConsumerKind::ConsumerGroup
+ && !self
+ .durable_consumer_offsets
+ .contains(pending.kind, pending.consumer_id);
match pending.mutation {
// A server auto-commit persists monotonically: its op offset can
// trail the durably-recorded value (disk-tier polls replicate in
// IO-completion order), so a plain overwrite would rewind the file
- // and re-deliver on restart. The `persisted_offsets` tracker keeps
+ // and re-deliver on restart. The durable offset tracker keeps
// the fold off the file: a covered offset skips the write, an
// advancing one blind-writes, and only a cold key (first commit
// after boot) reads the file once. Explicit client stores
// overwrite, so a deliberate offset reset still holds. Mirrors the
// in-memory `upsert_offset_max` vs `upsert_offset` split in the
// commit-apply.
PendingConsumerOffsetMutation::Upsert(offset) if pending.auto_commit => {
- let tracked = self.persisted_offsets.borrow().get(&key).copied();
- let persisted = match tracked {
- Some(high_water) if offset <= high_water => return Ok(()),
- Some(_) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- offset
+ let tracked = self
+ .durable_consumer_offsets
+ .get(pending.kind, pending.consumer_id);
+ let persisted_high_water = match (path.as_deref(), tracked) {
+ (None, _) => offset,
+ (Some(_), Some(state))
+ if state
+ .persisted_high_water
+ .is_some_and(|high_water| offset <= high_water) =>
+ {
+ state.persisted_high_water.expect("covered persisted value")
+ }
+ (Some(path), Some(state)) => {
+ let value = state.committed_offset.max(offset);
+ persist_offset(path, value, self.consumer_offset_enforce_fsync).await?;
+ value
}
- None => {
- persist_offset_max(&path, offset, self.consumer_offset_enforce_fsync)
- .await?
+ (Some(path), None) => {
+ persist_offset_max(path, offset, self.consumer_offset_enforce_fsync).await?
}
};
- self.persisted_offsets.borrow_mut().insert(key, persisted);
+ self.durable_consumer_offsets.record_auto_commit(
+ pending.kind,
+ pending.consumer_id,
+ if tracked.is_none() {
+ persisted_high_water
+ } else {
+ offset
+ },
+ persisted_high_water,
+ );
+ capacity.clear_stranded(pending.consumer_id);
+ if creates_group {
+ self.consumer_group_offsets_need_reconcile.set(true);
+ }
Ok(())
}
PendingConsumerOffsetMutation::Upsert(offset) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- self.persisted_offsets.borrow_mut().insert(key, offset);
+ if let Some(path) = path.as_deref() {
+ persist_offset(path, offset, self.consumer_offset_enforce_fsync).await?;
+ }
+ self.durable_consumer_offsets.record_explicit(
+ pending.kind,
+ pending.consumer_id,
+ offset,
+ Some(offset),
+ );
+ capacity.clear_stranded(pending.consumer_id);
+ if creates_group {
+ self.consumer_group_offsets_need_reconcile.set(true);
+ }
Ok(())
}
PendingConsumerOffsetMutation::Delete => {
- delete_persisted_offset(&path).await?;
- self.persisted_offsets.borrow_mut().remove(&key);
+ if let Some(path) = path.as_deref() {
+ delete_persisted_offset(path).await?;
+ self.consumer_offset_dirs_dirty[match pending.kind {
+ ConsumerKind::Consumer => 0,
Review Comment:
nit: a `match` inlined inside the index subscript reads badly. hoist the index into a local.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,615 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, RefCell};
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, Weak};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: Option<u64>,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+ membership_epoch: Cell<u64>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset
+ && state
+ .persisted_high_water
+ .is_some_and(|persisted| persisted >= offset)
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: Option<u64>,
+ ) {
+ self.entries(kind).borrow_mut().insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ );
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ let state = entries.entry(id).or_insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water: None,
+ });
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = Some(
+ state
+ .persisted_high_water
+ .unwrap_or(0)
+ .max(persisted_high_water),
+ );
+ }
+
+ pub(crate) fn mark_persisted(&self, kind: ConsumerKind, id: u32, high_water: u64) {
+ if let Some(state) = self.entries(kind).borrow_mut().get_mut(&id) {
+ state.persisted_high_water = Some(high_water);
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ let removed = self.entries(kind).borrow_mut().remove(&id).is_some();
+ if removed {
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+ removed
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Weak<ProvisionalToken>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
+ map_warned: Cell<bool>,
+ reclaim_epoch: Arc<AtomicU64>,
+ last_reclaim: Cell<Option<(u64, u64)>>,
+}
+
+impl ConsumerOffsetCapacity {
+ pub(crate) fn new(kind: ConsumerKind, limit: usize) -> Self {
+ Self {
+ kind,
+ limit: Cell::new(limit),
+ pending: RefCell::new(HashMap::new()),
+ provisional: RefCell::new(HashMap::new()),
+ stranded: RefCell::new(HashSet::new()),
+ uncertain: Cell::new(false),
+ durable_warned: Cell::new(false),
+ map_warned: Cell::new(false),
+ reclaim_epoch: Arc::new(AtomicU64::new(0)),
+ last_reclaim: Cell::new(None),
+ }
+ }
+
+ pub(crate) fn set_limit(&self, limit: usize) {
+ self.limit.set(limit);
+ }
+
+ pub(crate) const fn limit(&self) -> usize {
+ self.limit.get()
+ }
+
+ pub(crate) fn try_reserve(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ *self.pending.borrow_mut().entry(id).or_default() += 1;
+ Ok(())
+ }
+
+ pub(crate) fn check(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ if durable.contains(self.kind, id)
+ || self.pending.borrow().contains_key(&id)
+ || self
+ .provisional
+ .borrow()
+ .get(&id)
+ .is_some_and(|token| token.strong_count() > 0)
+ || self.stranded.borrow().contains(&id)
+ {
+ return Ok(());
+ }
+ let limit = self.limit.get();
+ let durable_count = durable.count(self.kind);
+ // A full durable table cannot gain room by pruning provisional keys.
+ let occupied = if durable_count >= limit {
+ durable_count
+ } else {
+ self.provisional
+ .borrow_mut()
+ .retain(|_, token| token.strong_count() > 0);
+ self.occupied(durable)
+ };
+ if self.uncertain.get() || occupied >= limit {
+ return Err(ConsumerOffsetCapacityError {
+ kind: self.kind,
+ occupied,
+ limit,
+ first_in_episode: !self.durable_warned.replace(true),
+ uncertain: self.uncertain.get(),
+ });
+ }
+ self.durable_warned.set(false);
+ Ok(())
+ }
+
+ pub(crate) fn reserve_provisional(
+ self: &Rc<Self>,
+ id: u32,
+ durable: &Rc<DurableConsumerOffsets>,
+ ) -> Result<AutoCommitReservation, ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ let mut provisional = self.provisional.borrow_mut();
+ let token = provisional
+ .get(&id)
+ .and_then(Weak::upgrade)
+ .unwrap_or_else(|| {
+ let token = Arc::new(ProvisionalToken {
+ reclaim_epoch: Arc::clone(&self.reclaim_epoch),
+ });
+ provisional.insert(id, Arc::downgrade(&token));
+ token
+ });
+ Ok(AutoCommitReservation {
+ token,
+ kind: self.kind,
+ consumer_id: id,
+ })
+ }
+
+ pub(crate) fn owns(&self, reservation: &AutoCommitReservation) -> bool {
+ reservation.kind == self.kind
+ && self
+ .provisional
+ .borrow()
+ .get(&reservation.consumer_id)
+ .is_some_and(|token| std::ptr::eq(token.as_ptr(), Arc::as_ptr(&reservation.token)))
+ }
+
+ pub(crate) fn protects(&self, id: u32, durable: &DurableConsumerOffsets) -> bool {
+ durable.contains(self.kind, id)
Review Comment:
simplification: `protects` is the first arm of `check` minus `stranded`. share a `holds(id, durable)` helper and keep the deliberate `stranded` difference in the callers.
##########
core/server/src/offset_recovery.rs:
##########
@@ -33,13 +33,16 @@ use tracing::{error, trace, warn};
const COMPONENT: &str = "STREAMING_PARTITIONS";
-pub fn load_consumer_offsets(path: &str) -> Result<Vec<ConsumerOffset>, IggyError> {
+pub type RecoveredOffsets<T> = (Vec<T>, Vec<u32>);
Review Comment:
simplification: the tuple alias is destructured by position in partition_helpers.rs:190 and 231, and the `is_file()` re-stat at 89 and 164 only tells corrupt-and-removed from torn. have `read_offset_file` return `Loaded | Removed | Stranded` and a named struct, which drops the re-stat too.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,615 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, RefCell};
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, Weak};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: Option<u64>,
Review Comment:
simplification: `None` only exists inside `install_state_transfer` between `record_explicit(.., None)` and `mark_persisted`, and converge clears the map before anything can observe it, so this can be a plain `u64` and the `(Some, Some)` arm at iggy_partition.rs:2154 becomes one compare. it drops a defence-in-depth check, so only if the install stays fail-closed.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -111,14 +111,27 @@ where
spawn_poll_io(Rc::clone(&shard), namespace, plan, reply);
}
Some(plan) => {
- let (fragments, current_offset, auto_commit) = plan.execute_resident();
- if let Some(applied) = auto_commit {
- submit_auto_commit(&shard, namespace, &applied);
- }
- let _ = reply.try_send(PartitionReadReply::Poll {
- fragments,
- current_offset,
- });
+ let result = match plan.execute_resident() {
Review Comment:
simplification: this match is byte-identical to the one at 215. one `poll_reply(shard, namespace, result)` keeps a single copy of the deny sequence, the line saving is small but there is one place to fix.
##########
core/server/src/consumer_group.rs:
##########
@@ -226,15 +226,28 @@ where
let body = request_body(&request);
// The store/delete ops differ only in the decode type; this collapses
// their identical decode -> resolve group id -> rewrite consumer id ->
- // re-encode bodies. A non-group consumer or unresolved group returns the
- // request untouched (the apply/read path handles the miss).
+ // re-encode bodies. Individual consumers pass through. A group identifier
+ // that metadata cannot resolve is rejected before it can create a raw file
+ // in the group-offset directory.
macro_rules! rewrite_group_offset {
($ty:ty) => {{
let mut wire = <$ty>::decode_from(body).map_err(|_| IggyError::InvalidCommand)?;
+ if wire.consumer.kind != KIND_CONSUMER_GROUP {
+ return Ok(request);
+ }
let Some(group_id) =
Review Comment:
simplification: this branch and `topic_exists` repeat `fence_group_offset` in responses.rs:295 character for character, and the fence already runs first on every transport. one shared classifier for both, and the comment at dispatch/partition.rs:528 saying this is unreachable is now wrong - it is reachable through the await at 501.
##########
core/integration/tests/server/consumer_offset_quota_vsr.rs:
##########
@@ -0,0 +1,493 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy::prelude::*;
+use iggy_binary_protocol::codec::WireEncode;
+use iggy_binary_protocol::consensus::Operation;
+use iggy_binary_protocol::requests::consumer_offsets::StoreConsumerOffsetRequest;
+use iggy_binary_protocol::{AckLevel, WireConsumer, WireIdentifier};
+use iggy_common::store_consumer_offset::StoreConsumerOffset;
+use integration::harness::TestBinary;
+use integration::iggy_harness;
+use reqwest::StatusCode;
+use std::collections::BTreeMap;
+use std::fs;
+
+use super::http_client::HttpClient;
+use super::raw_tcp;
+
+const STREAM_NAME: &str = "consumer-offset-quota-stream";
+const TOPIC_NAME: &str = "consumer-offset-quota-topic";
+const PARTITION_ID: u32 = 0;
+const LIMIT: u32 = 4;
+
+#[iggy_harness(
+ cluster_nodes = 1,
+ server(partition.consumer_offsets_max = "4")
+)]
+async fn given_full_consumer_offset_table_when_creating_another_should_reject_without_new_file(
+ harness: &TestHarness,
+) {
+ let client = harness.tcp_root_client().await.expect("TCP root client");
+ let stream = Identifier::named(STREAM_NAME).expect("stream identifier");
+ let topic = Identifier::named(TOPIC_NAME).expect("topic identifier");
+ let stream_details = client
+ .create_stream(STREAM_NAME)
+ .await
+ .expect("create stream");
+ let topic_details = client
+ .create_topic(
+ &stream,
+ TOPIC_NAME,
+ &TopicCreateOptions {
+ partitions_count: Some(1),
+ message_expiry: Some(IggyExpiry::NeverExpire),
+ ..TopicCreateOptions::default()
+ },
+ )
+ .await
+ .expect("create topic");
+ let mut messages = vec![
+ IggyMessage::builder()
+ .payload("offset-quota".into())
+ .build()
+ .expect("build message"),
+ ];
+ client
+ .send_messages(
+ &stream,
+ &topic,
+ &Partitioning::partition_id(PARTITION_ID),
+ &mut messages,
+ )
+ .await
+ .expect("seed non-empty partition");
+
+ client
+ .create_user(
+ "offset-poll-only",
+ "password123",
+ UserStatus::Active,
+ Some(Permissions {
+ global: GlobalPermissions::default(),
+ streams: Some(BTreeMap::from([(
+ stream_details.id as usize,
+ StreamPermissions {
+ topics: Some(BTreeMap::from([(
+ topic_details.id as usize,
+ TopicPermissions {
+ poll_messages: true,
+ ..Default::default()
+ },
+ )])),
+ ..Default::default()
+ },
+ )])),
+ }),
+ )
+ .await
+ .expect("create a topic-scoped consumer");
+ let client = harness.tcp_new_client().await.expect("consumer TCP client");
+ client
+ .login_user("offset-poll-only", "password123")
+ .await
+ .expect("consumer login");
+
+ let first_consumer = Consumer::new(Identifier::numeric(1).unwrap());
+ let polled = client
+ .poll_messages(
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ &first_consumer,
+ &PollingStrategy::first(),
+ 1,
+ true,
+ )
+ .await
+ .expect("new auto-commit consumer fits");
+ assert_eq!(polled.messages.len(), 1);
+ let first_file = harness.server().data_path().join(format!(
+ "streams/{}/topics/{}/partitions/{PARTITION_ID}/offsets/consumers/1",
+ stream_details.id, topic_details.id
+ ));
+ let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(10);
+ while !first_file.is_file() {
+ assert!(
+ tokio::time::Instant::now() < deadline,
+ "auto-commit never reached its file"
+ );
+ tokio::time::sleep(std::time::Duration::from_millis(10)).await;
+ }
+ assert!(
+ client
+ .poll_messages(
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ &first_consumer,
+ &PollingStrategy::next(),
+ 1,
+ true
+ )
+ .await
+ .expect("next poll")
+ .messages
+ .is_empty()
+ );
+
+ for consumer_id in 1..=LIMIT {
+ client
+ .store_consumer_offset(
+ &Consumer::new(Identifier::numeric(consumer_id).expect("consumer identifier")),
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ 0,
+ )
+ .await
+ .expect("store offset within limit");
+ }
+
+ let rejected_consumer =
+ Consumer::new(Identifier::numeric(LIMIT + 1).expect("consumer identifier"));
+ let rejected = client
+ .store_consumer_offset(&rejected_consumer, &stream, &topic, Some(PARTITION_ID), 0)
+ .await;
+ assert!(
+ matches!(rejected, Err(IggyError::TooManyConsumerOffsets)),
+ "the first key above the limit must receive the typed capacity error"
+ );
+
+ client
+ .store_consumer_offset(
+ &Consumer::new(Identifier::numeric(1).expect("consumer identifier")),
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ 0,
+ )
+ .await
+ .expect("existing key remains writable at the limit");
+
+ let poll_rejected = client
+ .poll_messages(
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ &rejected_consumer,
+ &PollingStrategy::first(),
+ 1,
+ true,
+ )
+ .await;
+ assert!(
+ matches!(poll_rejected, Err(IggyError::TooManyConsumerOffsets)),
+ "auto-commit must not return data when its new key cannot be admitted"
+ );
+ client
+ .poll_messages(
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ &rejected_consumer,
+ &PollingStrategy::first(),
+ 1,
+ false,
+ )
+ .await
+ .expect("the same poll succeeds when auto-commit is disabled");
+
+ client
+ .delete_consumer_offset(
+ &Consumer::new(Identifier::numeric(1).expect("consumer identifier")),
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ )
+ .await
+ .expect("delete one accepted offset");
+ client
+ .store_consumer_offset(&rejected_consumer, &stream, &topic, Some(PARTITION_ID), 0)
+ .await
+ .expect("delete releases one durable slot");
+
+ let mut raw = raw_tcp::connect(harness).await;
+ let raw_client_id = 0xC0FF_EE03;
+ let session = raw_tcp::register_root(&mut raw, raw_client_id).await;
+ let unresolved_group = StoreConsumerOffsetRequest {
+ consumer: WireConsumer::consumer_group(WireIdentifier::Numeric(999)),
+ stream_id: WireIdentifier::Numeric(stream_details.id),
+ topic_id: WireIdentifier::Numeric(topic_details.id),
+ partition_id: Some(PARTITION_ID),
+ offset: 0,
+ ack: AckLevel::Quorum,
+ }
+ .to_bytes();
+ let header = raw_tcp::request_header(
+ Operation::StoreConsumerOffset,
+ raw_client_id,
+ session,
+ 1,
+ unresolved_group.len(),
+ );
+ let (reply, _) = raw_tcp::exchange(&mut raw, &header, &unresolved_group).await;
+ assert_eq!(
+ raw_tcp::reply_status(&reply),
+ IggyError::ConsumerGroupIdNotFound(Identifier::numeric(999).unwrap(), topic.clone())
+ .as_code()
+ );
+
+ let offsets_dir = harness.server().data_path().join(format!(
+ "streams/{}/topics/{}/partitions/{PARTITION_ID}/offsets/consumers",
+ stream_details.id, topic_details.id
+ ));
+ let file_count = integration::harness::disk::consumer_offset_file_ids(
+ &harness.server().data_path(),
+ stream_details.id,
+ topic_details.id,
+ PARTITION_ID,
+ ConsumerKind::Consumer,
+ )
+ .expect("consumer offsets directory")
+ .len();
+ assert_eq!(file_count, LIMIT as usize);
+ let groups_dir = offsets_dir
Review Comment:
simplification: this hand-rolls what `disk::consumer_offset_file_ids(.., ConsumerGroup)` already does nine lines up. call it with `.map_or(0, |ids| ids.len())` to keep absent-dir as 0, and the `offsets_dir` binding at 255 goes too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4063:
URL: https://github.com/apache/iggy/pull/4063#discussion_r3947244629
##########
core/shard/src/lib.rs:
##########
@@ -9830,8 +9872,16 @@ fn rotate_sweep_to_cursor(namespaces: &mut [IggyNamespace], cursor: Option<IggyN
namespaces.rotate_left(namespaces.partition_point(|namespace| *namespace < cursor));
}
-/// Whether this replica holds adopted suffix HEADERS above `commit_max` whose
-/// bodies never arrived.
+fn partition_repair_fetch_to_op(
+ commit_min: u64,
+ commit_max: u64,
+ missing_suffix: Option<u64>,
Review Comment:
Warning: Returning the adopted suffix head while `commit_min < commit_max` makes the repair request span the committed prefix this replica already holds. This VSR change is unrelated to consumer offset quotas; split it into its own PR.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -4075,6 +4675,12 @@ where
let (frozen_batches, index_bytes, flush_index, batch_count, committed_info, chunk_len) = {
let segment = self.log.active_segment();
let mut file_position = segment.size.as_bytes_u64();
+ let persisted_end = if file_position == 0 {
Review Comment:
Nit: The `persisted_end` fix is segment-flush idempotence for a repaired batch, unrelated to consumer offset quotas. Split it out so it can be reverted independently.
##########
core/bench/src/benchmarks/common.rs:
##########
@@ -44,7 +44,11 @@ pub async fn create_consumer(
"Consumer #{} → joining consumer group #{}",
consumer_id, consumer_group_id
);
- let cg_identifier = Identifier::try_from(*consumer_group_id).unwrap();
+ // By name: the group was created by name and the server assigns its
Review Comment:
Nit: The consumer group addressing fixes here and at line 216 are unrelated to consumer offset quotas. Split them into their own commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4063:
URL: https://github.com/apache/iggy/pull/4063#discussion_r3941072550
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2260,30 +2425,55 @@ where
|value| PendingConsumerOffsetCommit::upsert(kind, consumer_id, value),
);
- if let Err(error) = self.persist_consumer_offset_commit(pending).await {
- emit_partition_diag(
- tracing::Level::WARN,
- &PartitionDiagEvent::new(self.diag_ctx(), "no_ack offset persist failed")
- .with_operation(request_header.operation)
- .with_error(error.to_string()),
- );
- return;
+ let persisted = async {
+ self.persist_consumer_offset_commit(pending).await?;
+ self.flush_consumer_offset_directories().await
}
- if let Err(error) = self.apply_consumer_offset_commit(pending) {
+ .await;
+ if let Err(error) = persisted {
+ if offset.is_some() {
+ self.release_consumer_offset_reservation(kind, consumer_id);
+ if !self.durable_consumer_offsets.contains(kind, consumer_id)
+ && let Some(path) = self.persisted_offset_path(kind, consumer_id)
+ {
+ if delete_persisted_offset(&path).await.is_ok() {
Review Comment:
**Rollback unlink is never directory-fsynced — silent message skip on single-replica.**
The committed-delete path now arms `consumer_offset_dirs_dirty` (`:2205`) and flushes before replies, but this rollback unlink does not, and `:2205` is the only arming site.
This rolls back a store the client was told **failed** (`send_partition_deny_or_log` below). A crash after the unlink resurrects the dirent, boot re-seeds the live map and `durable` from it, and the consumer's stored offset becomes the value the *failed* store carried — ahead of the last acknowledged one. The next `next()` poll resumes above it and the intervening messages are never delivered.
Worse in kind than the delete-resurrection this fix targeted: that one produced re-delivery (at-least-once, which the system documents), this produces loss. Reachable only when `replica_count() == 1`, which is the topology with no peer to heal from.
Arming the bit here (and on the create in `persist_consumer_offset_commit`) lets the existing flush at `:2430` cover it. Same three lines also cover the permanent `record_stranded` at `:2443` — a failed dir-fsync there burns a quota slot with no clearing path but a later op for that same id.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2808,11 +2969,20 @@ where
.into_iter()
.chain(self.consumer_group_offsets_path.clone())
{
- if fsync_dir(&dir).await.is_err() {
+ if retry_offset_io(|| fsync_dir(&dir)).await.is_err() {
Review Comment:
**Retrying a bare fsync is not a retry — it can report success over a real failure.**
`fsync_dir` opens the directory fresh on every call (`:1654`), and `retry_offset_io` invokes the closure up to `OFFSET_IO_ATTEMPTS` times discarding each error (`:1631-1636`). Linux reports a writeback/metadata error to the first fsync that observes it and then clears it; since 4.13 the `errseq_t` counter seeds a newly opened file's `f_wb_err` from the *current* value at open time, so an fd opened after the error is never told about it. Attempt 1 fails on fd A, attempt 2 opens fd B seeded post-error and returns 0.
Consequence here: a masked failure leaves `offsets_written == true`, so the `:2977` guard never fires and the install reports success with the old offset files' unlinks not durable — which is the resurrection shape this PR added that guard to close. Same masking at `:3151` lets converge return `Ok` instead of fencing.
The neighbouring `retry_offset_io(|| persist_offset(...))` at `:2941` is correct to retry, because `persist_offset` re-opens with `truncate(true)` and rewrites the record before its `sync_data()` — the retried fsync has freshly dirtied pages. Retry the mutation+fsync as a unit, or treat a standalone `fsync_dir` error as terminal.
Related: `retry_offset_io` discards the error entirely (`:1633`), so no caller can distinguish `NotFound` from `ENOSPC` from `EIO`.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -4470,6 +5065,30 @@ where
self.drain_request_queue_into_prepares(drained_count).await;
}
+ async fn flush_consumer_offset_directories(&self) -> Result<(), IggyError> {
+ for (index, dir) in [
+ self.consumer_offsets_path.as_deref(),
+ self.consumer_group_offsets_path.as_deref(),
+ ]
+ .into_iter()
+ .enumerate()
+ {
+ if !self.consumer_offset_dirs_dirty[index].get() {
+ continue;
+ }
+ if let Some(dir) = dir {
+ crate::state_transfer::fsync_dir(dir)
+ .await
+ .map_err(|_| IggyError::CannotSyncFile)?;
Review Comment:
**`map_err` discards `io::ErrorKind`, and the fatal escalates process-wide.**
Every failure here collapses to `IggyError::CannotSyncFile`, so the caller cannot distinguish `NotFound` — nothing left to make durable — from `EIO`, a real divergence. The error then becomes `FatalCommit`, which the shard tick escalates to a whole-process exit, so one partition's offsets-dir open failure takes down every partition on every shard.
Fail-closed is right for a genuine I/O error: this flush gates the delete's reply and `advance_commit_min`, and acking a delete whose unlink is not durable is the bug this fix closes. But the same call is WARN-only at the purge site (`:6395`) and `.exists()`-guarded in converge (`state_transfer.rs:3150`) — three handlings of one fault in one PR.
Propagating the kind would allow a `NotFound` carve-out and a per-partition fence with a typed deny, instead of a process exit.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2008,54 +2118,100 @@ where
// durably stored; the in-memory update is idempotent on replay
// because we look up by (kind, id).
self.persist_consumer_offset_commit(pending).await?;
- self.apply_consumer_offset_commit(pending)?;
+ self.apply_consumer_offset_commit(pending);
self.pending_consumer_offset_commits.remove(&op);
+ self.refresh_consumer_offset_reservation(pending.kind, pending.consumer_id);
Ok(())
}
async fn persist_consumer_offset_commit(
&self,
pending: PendingConsumerOffsetCommit,
) -> Result<(), IggyError> {
- let Some(path) = self.persisted_offset_path(pending.kind, pending.consumer_id) else {
- return Ok(());
- };
- let key = (pending.kind, pending.consumer_id);
+ let path = self.persisted_offset_path(pending.kind, pending.consumer_id);
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
+ let creates_group = pending.kind == ConsumerKind::ConsumerGroup
+ && !self
+ .durable_consumer_offsets
+ .contains(pending.kind, pending.consumer_id);
match pending.mutation {
// A server auto-commit persists monotonically: its op offset can
// trail the durably-recorded value (disk-tier polls replicate in
// IO-completion order), so a plain overwrite would rewind the file
- // and re-deliver on restart. The `persisted_offsets` tracker keeps
+ // and re-deliver on restart. The durable offset tracker keeps
// the fold off the file: a covered offset skips the write, an
// advancing one blind-writes, and only a cold key (first commit
// after boot) reads the file once. Explicit client stores
// overwrite, so a deliberate offset reset still holds. Mirrors the
// in-memory `upsert_offset_max` vs `upsert_offset` split in the
// commit-apply.
PendingConsumerOffsetMutation::Upsert(offset) if pending.auto_commit => {
- let tracked = self.persisted_offsets.borrow().get(&key).copied();
- let persisted = match tracked {
- Some(high_water) if offset <= high_water => return Ok(()),
- Some(_) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- offset
+ let tracked = self
+ .durable_consumer_offsets
+ .get(pending.kind, pending.consumer_id);
+ let persisted_high_water = match (path.as_deref(), tracked) {
+ (None, _) => offset,
+ (Some(_), Some(state))
+ if state
+ .persisted_high_water
+ .is_some_and(|high_water| offset <= high_water) =>
+ {
+ state.persisted_high_water.expect("covered persisted value")
+ }
+ (Some(path), Some(state)) => {
+ let value = state.committed_offset.max(offset);
+ persist_offset(path, value, self.consumer_offset_enforce_fsync).await?;
+ value
}
- None => {
- persist_offset_max(&path, offset, self.consumer_offset_enforce_fsync)
- .await?
+ (Some(path), None) => {
+ persist_offset_max(path, offset, self.consumer_offset_enforce_fsync).await?
}
};
- self.persisted_offsets.borrow_mut().insert(key, persisted);
+ self.durable_consumer_offsets.record_auto_commit(
+ pending.kind,
+ pending.consumer_id,
+ if tracked.is_none() {
+ persisted_high_water
+ } else {
+ offset
+ },
+ persisted_high_water,
+ );
+ capacity.clear_stranded(pending.consumer_id);
+ if creates_group {
+ self.consumer_group_offsets_need_reconcile.set(true);
+ }
Ok(())
}
PendingConsumerOffsetMutation::Upsert(offset) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- self.persisted_offsets.borrow_mut().insert(key, offset);
+ if let Some(path) = path.as_deref() {
+ persist_offset(path, offset, self.consumer_offset_enforce_fsync).await?;
+ }
+ self.durable_consumer_offsets.record_explicit(
+ pending.kind,
+ pending.consumer_id,
+ offset,
+ Some(offset),
+ );
+ capacity.clear_stranded(pending.consumer_id);
+ if creates_group {
+ self.consumer_group_offsets_need_reconcile.set(true);
+ }
Ok(())
}
PendingConsumerOffsetMutation::Delete => {
- delete_persisted_offset(&path).await?;
- self.persisted_offsets.borrow_mut().remove(&key);
+ if let Some(path) = path.as_deref() {
+ delete_persisted_offset(path).await?;
+ self.consumer_offset_dirs_dirty[match pending.kind {
Review Comment:
**Dirty bit armed even when the unlink removed nothing.**
`delete_persisted_offset` tolerates `NotFound`, so a committed delete for a key that has no file on disk still arms the bit and buys a directory fsync in the flush. Backup-created phantoms are exactly that shape, and the reconciler submits them in batches of `GROUP_OFFSET_DELETES_PER_PASS`.
On its own that is wasted work. Combined with the error handling at `:5082`, it also manufactures the precondition for a process exit on a directory that is legitimately gone.
Arming only when a file was actually removed fixes both.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2808,11 +2969,20 @@ where
.into_iter()
.chain(self.consumer_group_offsets_path.clone())
{
- if fsync_dir(&dir).await.is_err() {
+ if retry_offset_io(|| fsync_dir(&dir)).await.is_err() {
offsets_written = false;
}
}
+ if !offsets_written {
Review Comment:
**One failed 8-byte offset write discards the entire installed partition and re-pulls it.**
This `Err` routes to `converge_to_empty_after_failed_install` (`:2453-2473`), which sweeps every `.log`/`.index`/staging/anchor file out of the partition dir (`:3169-3207`), retires every log storage, and re-seeds the counters. The original error then propagates to the catch-all arm in `shard/src/lib.rs:8960`, which re-arms the transfer against the next peer.
So a deterministic local fault — ENOSPC, EIO — loops transfer → wipe → transfer at the backoff rate, re-reading the serving primary's whole segment chain every cycle. That is the amplifier `abandon_or_rearm_partition_transfer` exists to prevent, and the offset files are 8 bytes each and independent of the segment swap.
Writing and fsyncing the offset files *before* the segment rename would make this failure abort pre-mutation, so the partition keeps what it had instead of discarding a good chain.
(The converge itself is correct and thorough — `durable_consumer_offsets` is cleared at `:3135`, so no file-less membership survives. The issue is only how much it destroys for how small a fault.)
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2341,7 +2616,11 @@ where
}
};
- if found {
+ if found
+ || self
Review Comment:
**Replicated stranded-key deletes can crash-loop a pre-upgrade peer during a rolling restart.**
Admitting a delete on `is_stranded` alone means no replica holds a live-map entry for it. `dead_consumer_group_offset_ids` now extends with `stranded_ids()`, and the reconciler submits those (`partition_reconciler.rs:1027`), so these replicated deletes are generated with no operator action — a torn group-offset file for a deleted group is enough.
A pre-PR peer replaying that committed op as non-follower still runs `if !removed && !self.consensus.is_follower() { return Err(ConsumerOffsetNotFound) }` (`6c47193fe:iggy_partition.rs:2154`, `:2168`) → `FatalCommit` → node exit.
Why this is not narrow: there is no replica-plane version gate anywhere — the only version negotiation is client↔server in `binary_protocol/src/version.rs`, and grepping `core/consensus` and `core/server` finds no peer handshake version or op-set gate. The Helm chart defaults to RollingUpdate (`helm/charts/iggy/README.md:551`), so a version-skewed group is the normal upgrade state, and during it the pre-upgrade pod is the *stable* one and so a likely election winner. The reconciler re-submits from `stranded_ids()` every pass, so the trigger is regenerated rather than consumed until each delete commits.
Reclaiming stranded files by local unlink would avoid this entirely — a stranded file is per-replica disk state and arguably has no business in the replicated log. Failing that, this needs the coordinated-restart note the chart already uses for `server.cluster.auth` (`README.md:494`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz commented on PR #4063:
URL: https://github.com/apache/iggy/pull/4063#issuecomment-5552346878
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4063:
URL: https://github.com/apache/iggy/pull/4063#discussion_r3939951642
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2421,6 +2776,30 @@ where
}
};
+ if args.auto_commit
+ && self.consensus.is_primary()
+ && let Ok(pending) = PendingConsumerOffsetCommit::try_from_polling_consumer(consumer, 0)
+ {
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
+ if !capacity.is_uncertain()
+ && self.consumer_offset_map_count(pending.kind) >= capacity.limit()
+ {
Review Comment:
**`papaya::HashMap::len()` on every auto-commit poll.**
`len()` is `Counter::sum()` over `available_parallelism().next_power_of_two()` `CachePadded<AtomicIsize>` shards, so on a 64-128 core host this is 4-16 KiB of cache-line loads per poll to compare against a limit that is almost never reached.
The `contains_key` probe at :2787-2796 is the cheaper test and is already written — on a hit neither ordering does anything, so probing first is semantics-preserving and free.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2613,111 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ fn resynchronize_consumer_offset_reservations(&mut self) {
let current_view = self.consensus.view();
- if current_view == self.observed_view {
+ if current_view == self.observed_view && !self.offset_reservations_need_resync.get() {
return;
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ }
+
+ let from_op = self
Review Comment:
**Scan floors at `commit_min`, but eviction is bounded by `commit_max`.**
`flush_committed_messages` evicts `committed_prefix(commit_max())` (:4516), so ops in `(commit_min, commit_max]` lose their resident header and survive only in the evicted ring — capped at `EVICTED_RING_CAPACITY = 4096` *or* `EVICTED_RING_BYTES_MAX = 16 MiB`, whichever trips first, which is a handful of batches on a produce-heavy partition. `repair_headers_in` can then legitimately return fewer than `expected`, latching the `uncertain` flag at :2643.
This fires at promotion: a new primary runs resync from `on_commit`/`on_replicate`/`on_request` before `VsrAction::CommitJournal` has caught `commit_min` up. `consensus/src/impls.rs:2880-2883` states the rule for the identical problem — "`commit_max`, not `commit_min` … `commit_min` can lag far enough to overflow the rebuild".
Note a second trigger that flooring at `commit_max` does **not** fix: `truncate_uncommitted_from` (:4353) drops entries without rewinding the sequencer and arms the resync at :4436, leaving `to_op > journal head`. Bounding `to_op` by `journal().inner.last_op()` covers both.
##########
core/server/src/partition_helpers.rs:
##########
@@ -253,15 +269,51 @@ pub fn configure_consumer_offsets(
.enforce_fsync
.unwrap_or(iggy_common::DEFAULT_ENFORCE_FSYNC);
partition.configure_consumer_offset_storage(
- consumer_offsets_path,
- consumer_group_offsets_path,
+ consumer_offsets_path.clone(),
+ consumer_group_offsets_path.clone(),
consumer_offsets,
consumer_group_offsets,
enforce_fsync,
);
+ for consumer_id in numeric_offset_file_ids(&consumer_offsets_path) {
+ if partition.seed_stranded_consumer_offset(ConsumerKind::Consumer, consumer_id) {
+ warn!(stream_id, topic_id, partition_id, consumer_id, path = %consumer_offsets_path,
+ "unloaded consumer offset file retains its capacity slot until updated or deleted");
+ }
+ }
+ for group_id in numeric_offset_file_ids(&consumer_group_offsets_path) {
+ if partition.seed_stranded_consumer_offset(ConsumerKind::ConsumerGroup, group_id) {
+ warn!(stream_id, topic_id, partition_id, group_id, path = %consumer_group_offsets_path,
+ "unloaded group offset file retains its capacity slot until repaired or reclaimed");
+ }
+ }
+ for kind in [ConsumerKind::Consumer, ConsumerKind::ConsumerGroup] {
+ let count = partition.occupied_consumer_offset_count(kind);
+ if count > config.partition.consumer_offsets_max {
+ warn!(
+ stream_id,
+ topic_id,
+ partition_id,
+ ?kind,
+ count,
+ limit = config.partition.consumer_offsets_max,
+ "recovered consumer offsets exceed the configured admission limit"
+ );
+ }
+ }
Ok(())
}
+fn numeric_offset_file_ids(path: &str) -> Vec<u32> {
+ let Ok(entries) = std::fs::read_dir(path) else {
+ return Vec::new();
+ };
+ entries
+ .filter_map(Result::ok)
+ .filter_map(|entry| entry.file_name().to_str()?.parse().ok())
Review Comment:
**No `is_file()` filter, and this is a second walk of a directory already read.**
A numerically-named *subdirectory* under the offsets dir parses fine here and is seeded as a stranded key, permanently burning a quota slot that nothing can clear — `clear_stranded` only fires on a successful persist or unlink for that id. The sibling loader skips dirs (`offset_recovery.rs:132`), and this PR's own harness helper checks `is_file()` (`integration/src/harness/disk.rs:60`), so it is a three-way asymmetry.
It is also a second blocking `std::fs::read_dir` over the two dirs `load_consumer_offsets` just walked, and not boot-only — `load_partition` runs from the reconciler at runtime (`partition_reconciler.rs:734`), so it blocks a serving shard thread.
Returning the id set (filtered) from the existing walk closes both.
Worth noting this is also the no-fault way to populate `stranded`, which is what makes the widened delete admission at `iggy_partition.rs:2597-2601` reachable on a mixed-version cluster.
##########
core/server/src/partition_reconciler.rs:
##########
@@ -960,47 +994,108 @@ async fn tear_down_owned_partition(
counters.removed_local += 1;
}
-/// Reclaim consumer-group offsets left behind by a `DeleteConsumerGroup` whose
-/// topic still exists (a topic/stream delete already drops the whole partition
-/// directory, offsets included). For each owned partition, any stored
-/// consumer-group offset whose group id is no longer present in the topic's
-/// committed metadata is removed (in-memory entry + persisted file). Monotonic,
-/// never-reused group ids make this purely reclamation -- a recreated group
-/// gets a fresh id and never reads a dead group's offset -- so it is safe to do
-/// lazily on the reconcile pass rather than synchronously on delete.
-async fn reconcile_consumer_group_offsets(ctx: &ReconcilerCtx, counters: &mut PassCounters) {
+/// Reclaim deleted groups through ordered offset deletes. Replicas must see
+/// each delete before a replacement store can reuse its durable slot.
+fn reconcile_consumer_group_offsets(ctx: &ReconcilerCtx, counters: &mut PassCounters) {
let live_groups = snapshot_topic_live_groups(ctx);
let partitions = ctx.shard.plane.partitions();
let owned: Vec<IggyNamespace> = partitions.namespaces().copied().collect();
- for ns in owned {
- let live = live_groups.get(&(ns.stream_id(), ns.topic_id()));
- // Take the in-memory removes + owned unlink paths under a closure-scoped
- // borrow that cannot escape into the await below. Holding a raw
- // `&IggyPartition` across `delete_persisted_offset().await` would let the
- // pump task realloc the partitions vec underneath us (a UAF).
- let paths = partitions.with_partition(&ns, |partition| {
- partition.reclaim_dead_group_offsets(|group_id| {
- live.is_some_and(|set| set.contains(&group_id))
- })
- });
- let Some(paths) = paths else {
+ for namespace in owned {
+ if ctx
+ .group_offset_cleanup_inflight
+ .borrow()
+ .contains(&namespace)
+ {
continue;
- };
- for path in paths {
- if let Err(err) = delete_persisted_offset(&path).await {
- warn!(
- shard = ctx.shard.id,
- ns_raw = ns.inner(),
- error = %err,
- "reconciler failed to reclaim deleted consumer-group offset"
- );
- continue;
+ }
+ let live = live_groups.get(&(namespace.stream_id(), namespace.topic_id()));
Review Comment:
**`live_groups` miss conflates "no live groups" with "STM not caught up" — and now issues replicated deletes.**
`snapshot_topic_live_groups` skips topics whose `consumer_groups` map is empty, so a topic absent from the snapshot and a topic with zero groups are the same `None` here. `live.is_some_and(...)` at :1015 then answers false for *every* group, and the pass enqueues a delete for every group offset on the partition.
Pre-PR that cost a node-local unlink other replicas could restore. Post-PR it submits durable, quorum-committed deletes (:1024), so a shard whose metadata STM is still replaying group creates — the boot window — can permanently destroy every live group's committed offset cluster-wide. Nothing surfaces to the client; consumers silently re-consume from the retention floor.
Distinguishing "topic absent from this snapshot" from "topic present with zero groups" before any delete is submitted would close it.
##########
core/server/src/consumer_group.rs:
##########
@@ -226,15 +226,28 @@ where
let body = request_body(&request);
// The store/delete ops differ only in the decode type; this collapses
// their identical decode -> resolve group id -> rewrite consumer id ->
- // re-encode bodies. A non-group consumer or unresolved group returns the
- // request untouched (the apply/read path handles the miss).
+ // re-encode bodies. Individual consumers pass through. A group identifier
+ // that metadata cannot resolve is rejected before it can create a raw file
+ // in the group-offset directory.
macro_rules! rewrite_group_offset {
($ty:ty) => {{
let mut wire = <$ty>::decode_from(body).map_err(|_| IggyError::InvalidCommand)?;
+ if wire.consumer.kind != KIND_CONSUMER_GROUP {
+ return Ok(request);
+ }
let Some(group_id) =
resolve_group_offset_id(shard, &wire.consumer, (&wire.stream_id, &wire.topic_id))
else {
- return Ok(request);
+ // An unknown stream or topic is not a missing group: let the
+ // namespace resolution below answer it with the same
+ // not-found every other partition op reports.
+ if !topic_exists(shard, &wire.stream_id, &wire.topic_id) {
+ return Ok(request);
+ }
+ return Err(missing_consumer_group_error(
Review Comment:
**Wire-visible error-code change, undocumented.**
Store/delete consumer offset against a missing group returned `ResourceNotFound` (20); this now returns `ConsumerGroupIdNotFound` (5000) or `ConsumerGroupNameNotFound` (5003) depending on the identifier form. Unknown stream/topic still returns 20 (:244-245), which is the discriminator.
Any client branching on 20 for this op breaks. Go is the sharpest case — `ierror.ErrResourceNotFound` is a typed error matched with `errors.Is`, so this reroutes control flow silently.
Only the Go BDD was updated (`bdd/go/tests/tcp_test/offset_feature_delete.go:133`) out of seven SDKs, and there is no changelog entry or coverage for the name form (5003).
##########
core/common/src/error/iggy_error.rs:
##########
@@ -333,6 +333,8 @@ pub enum IggyError {
NotResolvedConsumer(Identifier) = 3022,
#[error("Cannot open consumer offsets file for path: {0}")]
CannotOpenConsumerOffsetsFile(String) = 3023,
+ #[error("Too many consumer offsets for partition")]
Review Comment:
**Message names neither the limit nor the knob.**
Since 3024 is terminal in every SDK retry classifier, this string is the operator's entire remediation surface — and it does not say what the limit is or that `[partition] consumer_offsets_max` controls it. `config.toml` is the only doc surface in the repo, so nothing else closes the gap.
Something like "per-partition consumer offset limit reached (see `[partition] consumer_offsets_max`)" would make it actionable.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2613,111 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ fn resynchronize_consumer_offset_reservations(&mut self) {
let current_view = self.consensus.view();
- if current_view == self.observed_view {
+ if current_view == self.observed_view && !self.offset_reservations_need_resync.get() {
return;
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ }
+
+ let from_op = self
+ .consensus
+ .commit_min()
+ .max(self.purge_floor_op)
+ .saturating_add(1);
+ let to_op = self.consensus.sequencer().current_sequence();
+ let mut rebuilt = HashMap::new();
+ let headers = self.log.journal().inner.repair_headers_in(from_op..=to_op);
+ let expected = to_op
+ .checked_sub(from_op)
+ .map_or(0, |span| span.saturating_add(1));
+ let mut decode_failed = headers.len() as u64 != expected;
+ for (op, header) in headers {
+ if !matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) {
+ continue;
+ }
+ match self.restage_consumer_offset_from_journal(op) {
+ Ok(pending) => {
+ rebuilt.insert(op, pending);
+ }
+ Err(error) => {
+ error!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica_id = self.consensus.replica(),
+ namespace_raw = self.namespace().inner(),
+ op,
+ %error,
+ "failed to rebuild consumer offset reservations after view change"
+ );
+ decode_failed = true;
+ break;
+ }
+ }
+ }
+ self.pending_consumer_offset_commits = rebuilt;
+ if decode_failed {
+ self.consumer_offset_capacity.mark_uncertain();
+ self.consumer_group_offset_capacity.mark_uncertain();
+ } else {
+ let consumer_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::Consumer
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, consumer_ids);
+ let group_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::ConsumerGroup
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_group_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, group_ids);
+ }
self.observed_view = current_view;
+ self.consumer_group_offsets_need_reconcile.set(true);
+ // Repair and truncation rearm this flag when journal contents change.
+ // A failed decode alone must not cause a full scan on every tick.
+ self.offset_reservations_need_resync.set(false);
+ if !decode_failed && self.consensus.is_primary() {
+ self.reclaim_phantom_offsets(ConsumerKind::Consumer);
Review Comment:
**Phantom reclaim is primary-gated; phantom *creation* is not.**
Backup-served auto-commit polls create live-map entries (`poll_plan.rs`) via the `primary == Some(Ok(false))` → `mark_served()` branch — never replicated, never reclaimed, and both reclaim call sites (here and :2780) are behind `is_primary()`.
A backup's map therefore saturates at `consumer_offsets_max` and never drains. Every new consumer polling that backup then gets terminal `TooManyConsumerOffsets` while the leader still has room, healed only by promotion. 3024 is terminal in every first-party SDK classifier (Go `tcp_core.go:859`, Java `:786`, Node `:1039`, .NET `:292`; Rust has no arm) — no retry, no failover, no leader redirect.
Reclaim is node-local and touches no replicated state, so it is safe on a backup. On a backup `provisional` is provably empty, which makes the predicate strictly cheaper there.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2613,111 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ fn resynchronize_consumer_offset_reservations(&mut self) {
let current_view = self.consensus.view();
- if current_view == self.observed_view {
+ if current_view == self.observed_view && !self.offset_reservations_need_resync.get() {
return;
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ }
+
+ let from_op = self
+ .consensus
+ .commit_min()
+ .max(self.purge_floor_op)
+ .saturating_add(1);
+ let to_op = self.consensus.sequencer().current_sequence();
+ let mut rebuilt = HashMap::new();
+ let headers = self.log.journal().inner.repair_headers_in(from_op..=to_op);
+ let expected = to_op
+ .checked_sub(from_op)
+ .map_or(0, |span| span.saturating_add(1));
+ let mut decode_failed = headers.len() as u64 != expected;
Review Comment:
**`uncertain` latches permanently — nothing re-arms it.**
`decode_failed` sets the latch, and this same pass then stamps `observed_view = current_view` (:2696) and clears `offset_reservations_need_resync` (:2700). Both are the gate conditions at :2624, so every later call early-returns and only `rebuild()` clears `uncertain` — reachable solely from a *next* view change, truncate, reanchor, install or purge.
While latched, every new offset key answers `TransientNotAccepted` and every auto-commit poll for a non-durable consumer returns zero messages. Clients treat 58 as replayable, so each request burns a full roster walk and then fails; admission is primary-only, so re-targeting can never succeed.
Suggest re-arming `offset_reservations_need_resync` on journal growth (or retrying the rebuild per tick) rather than clearing it after a failed decode.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2008,54 +2110,95 @@ where
// durably stored; the in-memory update is idempotent on replay
// because we look up by (kind, id).
self.persist_consumer_offset_commit(pending).await?;
- self.apply_consumer_offset_commit(pending)?;
+ self.apply_consumer_offset_commit(pending);
self.pending_consumer_offset_commits.remove(&op);
+ self.refresh_consumer_offset_reservation(pending.kind, pending.consumer_id);
Ok(())
}
async fn persist_consumer_offset_commit(
&self,
pending: PendingConsumerOffsetCommit,
) -> Result<(), IggyError> {
- let Some(path) = self.persisted_offset_path(pending.kind, pending.consumer_id) else {
- return Ok(());
- };
- let key = (pending.kind, pending.consumer_id);
+ let path = self.persisted_offset_path(pending.kind, pending.consumer_id);
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
+ let creates_group = pending.kind == ConsumerKind::ConsumerGroup
+ && !self
+ .durable_consumer_offsets
+ .contains(pending.kind, pending.consumer_id);
match pending.mutation {
// A server auto-commit persists monotonically: its op offset can
// trail the durably-recorded value (disk-tier polls replicate in
// IO-completion order), so a plain overwrite would rewind the file
- // and re-deliver on restart. The `persisted_offsets` tracker keeps
+ // and re-deliver on restart. The durable offset tracker keeps
// the fold off the file: a covered offset skips the write, an
// advancing one blind-writes, and only a cold key (first commit
// after boot) reads the file once. Explicit client stores
// overwrite, so a deliberate offset reset still holds. Mirrors the
// in-memory `upsert_offset_max` vs `upsert_offset` split in the
// commit-apply.
PendingConsumerOffsetMutation::Upsert(offset) if pending.auto_commit => {
- let tracked = self.persisted_offsets.borrow().get(&key).copied();
- let persisted = match tracked {
- Some(high_water) if offset <= high_water => return Ok(()),
- Some(_) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- offset
+ let tracked = self
+ .durable_consumer_offsets
+ .get(pending.kind, pending.consumer_id);
+ let persisted_high_water = match (path.as_deref(), tracked) {
+ (None, _) => offset,
+ (Some(_), Some(state))
+ if state
+ .persisted_high_water
+ .is_some_and(|high_water| offset <= high_water) =>
+ {
+ state.persisted_high_water.expect("covered persisted value")
+ }
+ (Some(path), Some(state)) => {
+ let value = state.committed_offset.max(offset);
+ persist_offset(path, value, self.consumer_offset_enforce_fsync).await?;
+ value
}
- None => {
- persist_offset_max(&path, offset, self.consumer_offset_enforce_fsync)
- .await?
+ (Some(path), None) => {
+ persist_offset_max(path, offset, self.consumer_offset_enforce_fsync).await?
}
};
- self.persisted_offsets.borrow_mut().insert(key, persisted);
+ self.durable_consumer_offsets.record_auto_commit(
+ pending.kind,
+ pending.consumer_id,
+ if tracked.is_none() {
+ persisted_high_water
+ } else {
+ offset
+ },
+ persisted_high_water,
+ );
+ capacity.clear_stranded(pending.consumer_id);
+ if creates_group {
+ self.consumer_group_offsets_need_reconcile.set(true);
+ }
Ok(())
}
PendingConsumerOffsetMutation::Upsert(offset) => {
- persist_offset(&path, offset, self.consumer_offset_enforce_fsync).await?;
- self.persisted_offsets.borrow_mut().insert(key, offset);
+ if let Some(path) = path.as_deref() {
+ persist_offset(path, offset, self.consumer_offset_enforce_fsync).await?;
+ }
+ self.durable_consumer_offsets.record_explicit(
+ pending.kind,
+ pending.consumer_id,
+ offset,
+ Some(offset),
+ );
+ capacity.clear_stranded(pending.consumer_id);
+ if creates_group {
+ self.consumer_group_offsets_need_reconcile.set(true);
+ }
Ok(())
}
PendingConsumerOffsetMutation::Delete => {
- delete_persisted_offset(&path).await?;
- self.persisted_offsets.borrow_mut().remove(&key);
+ if let Some(path) = path.as_deref() {
+ delete_persisted_offset(path).await?;
Review Comment:
**Committed offset delete unlinks with no parent-directory fsync.**
A crash after a quorum-acked delete resurrects the file at boot, and `configure_consumer_offsets` re-seeds both the live map and `durable` from it — so the cursor *and* its capacity slot come back. Nothing re-issues the delete: the op is committed and gone, and plain consumer keys have no reaper.
The crate already does this everywhere else for the stated reason: purge at :6263-6269 ("a crash right after the purge otherwise resurrects the offset files at boot") and install at `state_transfer.rs:2928-2935`.
It also does not stay local — the resurrected key is re-exported through `committed_entries`, so serving one state transfer undoes the deletion cluster-wide, which contradicts the PR's "keeps file counts identical on all replicas".
Worth matching the existing precedents' *batch* shape rather than fsyncing per delete: the reconciler submits up to `GROUP_OFFSET_DELETES_PER_PASS = 32` deletes per pass, so a per-delete dir fsync is 3-30 ms of pump stall per pass. Marking the dir dirty and issuing one `fsync_dir` per commit-walk batch gives the same guarantee — but the flush has to be ordered *before* that batch's replies, or the client is told "deleted" before the unlink is durable.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,530 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, RefCell};
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::{Arc, Weak};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: Option<u64>,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset
+ && state
+ .persisted_high_water
+ .is_some_and(|persisted| persisted >= offset)
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: Option<u64>,
+ ) {
+ self.entries(kind).borrow_mut().insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ );
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ let state = entries.entry(id).or_insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water: None,
+ });
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = Some(
+ state
+ .persisted_high_water
+ .unwrap_or(0)
+ .max(persisted_high_water),
+ );
+ }
+
+ pub(crate) fn mark_persisted(&self, kind: ConsumerKind, id: u32, high_water: u64) {
+ if let Some(state) = self.entries(kind).borrow_mut().get_mut(&id) {
+ state.persisted_high_water = Some(high_water);
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow_mut().remove(&id).is_some()
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ }
+
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Weak<()>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
+ map_warned: Cell<bool>,
+}
+
+impl ConsumerOffsetCapacity {
+ pub(crate) fn new(kind: ConsumerKind, limit: usize) -> Self {
+ Self {
+ kind,
+ limit: Cell::new(limit),
+ pending: RefCell::new(HashMap::new()),
+ provisional: RefCell::new(HashMap::new()),
+ stranded: RefCell::new(HashSet::new()),
+ uncertain: Cell::new(false),
+ durable_warned: Cell::new(false),
+ map_warned: Cell::new(false),
+ }
+ }
+
+ pub(crate) fn set_limit(&self, limit: usize) {
+ self.limit.set(limit);
+ }
+
+ pub(crate) const fn limit(&self) -> usize {
+ self.limit.get()
+ }
+
+ pub(crate) fn try_reserve(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ *self.pending.borrow_mut().entry(id).or_default() += 1;
+ Ok(())
+ }
+
+ pub(crate) fn check(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.rearm_if_below_limit(durable);
Review Comment:
**Admission gate degrades exactly when it engages.**
`check()` calls `rearm_if_below_limit` unconditionally. Once a partition has denied once, `durable_warned` is set (:216), so that call reaches `occupied()` (:356) on **every** offset store and auto-commit poll — an O(|pending| + |provisional| + |stranded|) walk with a `Weak::strong_count` atomic per element. The latch clears only when `occupied() < limit`, which never happens while the table is full, so the state the feature exists to handle is the state that makes it O(n). ~4k at the default, 262144 at the ceiling, on the shared pump — one saturated partition degrades every partition on that shard.
`d9` compounds it: the dead-`Weak` prune at :206 sits *past* the early return at :195-204, so a workload of already-known ids never prunes and `provisional` fills to the distinct-id count.
Cheapest correct first step is to delete this call and drive the rearm from the mutators that can lower occupancy (`release_reservation`, `set_pending_count(0)`, `DurableConsumerOffsets::remove`/`clear`, `clear_stranded`) — it removes the unconditional walk without touching the accounting.
If an O(1) count is wanted later, note a scalar counter is unsound here: `occupied()` is a set *union* minus `durable`, so a scalar drifts on the first overlap, and the drift direction is a partition that silently refuses every new key forever. It needs per-id set membership (one bit per set, count moved only on 0↔nonzero transitions) plus a real `Drop` on the reservation — releasing at named call sites misses the frame-dropped-by-a-closing-inbox path.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,530 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, RefCell};
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::{Arc, Weak};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: Option<u64>,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset
+ && state
+ .persisted_high_water
+ .is_some_and(|persisted| persisted >= offset)
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: Option<u64>,
+ ) {
+ self.entries(kind).borrow_mut().insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ );
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ let state = entries.entry(id).or_insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water: None,
+ });
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = Some(
+ state
+ .persisted_high_water
+ .unwrap_or(0)
+ .max(persisted_high_water),
+ );
+ }
+
+ pub(crate) fn mark_persisted(&self, kind: ConsumerKind, id: u32, high_water: u64) {
+ if let Some(state) = self.entries(kind).borrow_mut().get_mut(&id) {
+ state.persisted_high_water = Some(high_water);
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow_mut().remove(&id).is_some()
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ }
+
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Weak<()>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
+ map_warned: Cell<bool>,
+}
+
+impl ConsumerOffsetCapacity {
+ pub(crate) fn new(kind: ConsumerKind, limit: usize) -> Self {
+ Self {
+ kind,
+ limit: Cell::new(limit),
+ pending: RefCell::new(HashMap::new()),
+ provisional: RefCell::new(HashMap::new()),
+ stranded: RefCell::new(HashSet::new()),
+ uncertain: Cell::new(false),
+ durable_warned: Cell::new(false),
+ map_warned: Cell::new(false),
+ }
+ }
+
+ pub(crate) fn set_limit(&self, limit: usize) {
+ self.limit.set(limit);
+ }
+
+ pub(crate) const fn limit(&self) -> usize {
+ self.limit.get()
+ }
+
+ pub(crate) fn try_reserve(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ *self.pending.borrow_mut().entry(id).or_default() += 1;
+ Ok(())
+ }
+
+ pub(crate) fn check(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.rearm_if_below_limit(durable);
+ if durable.contains(self.kind, id)
+ || self.pending.borrow().contains_key(&id)
+ || self
+ .provisional
+ .borrow()
+ .get(&id)
+ .is_some_and(|token| token.strong_count() > 0)
+ || self.stranded.borrow().contains(&id)
+ {
+ return Ok(());
+ }
+ self.provisional
+ .borrow_mut()
+ .retain(|_, token| token.strong_count() > 0);
+ let occupied = self.occupied(durable);
+ let limit = self.limit.get();
+ if self.uncertain.get() || occupied >= limit {
+ return Err(ConsumerOffsetCapacityError {
+ kind: self.kind,
+ occupied,
+ limit,
+ first_in_episode: !self.durable_warned.replace(true),
+ uncertain: self.uncertain.get(),
+ });
+ }
+ Ok(())
+ }
+
+ pub(crate) fn reserve_provisional(
+ self: &Rc<Self>,
+ id: u32,
+ durable: &Rc<DurableConsumerOffsets>,
+ ) -> Result<AutoCommitReservation, ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ let mut provisional = self.provisional.borrow_mut();
+ let token = provisional
+ .get(&id)
+ .and_then(Weak::upgrade)
+ .unwrap_or_else(|| {
+ let token = Arc::new(());
+ provisional.insert(id, Arc::downgrade(&token));
+ token
+ });
+ Ok(AutoCommitReservation {
+ token,
+ kind: self.kind,
+ consumer_id: id,
+ })
+ }
+
+ pub(crate) fn owns(&self, reservation: &AutoCommitReservation) -> bool {
+ reservation.kind == self.kind
+ && self
+ .provisional
+ .borrow()
+ .get(&reservation.consumer_id)
+ .is_some_and(|token| std::ptr::eq(token.as_ptr(), Arc::as_ptr(&reservation.token)))
+ }
+
+ pub(crate) fn protects(&self, id: u32, durable: &DurableConsumerOffsets) -> bool {
+ durable.contains(self.kind, id)
+ || self.pending.borrow().contains_key(&id)
+ || self
+ .provisional
+ .borrow()
+ .get(&id)
+ .is_some_and(|token| token.strong_count() > 0)
+ }
+
+ pub(crate) fn set_pending_count(&self, id: u32, count: usize) {
+ if count == 0 {
+ self.pending.borrow_mut().remove(&id);
+ } else {
+ self.pending.borrow_mut().insert(id, count);
+ }
+ }
+
+ pub(crate) fn release_reservation(&self, id: u32) {
+ let mut pending = self.pending.borrow_mut();
+ let Some(count) = pending.get_mut(&id) else {
+ return;
+ };
+ if *count == 1 {
+ pending.remove(&id);
+ } else {
+ *count -= 1;
+ }
+ }
+
+ pub(crate) const fn is_uncertain(&self) -> bool {
+ self.uncertain.get()
+ }
+
+ pub(crate) fn rebuild(
+ &self,
+ durable: &DurableConsumerOffsets,
+ pending_ids: impl IntoIterator<Item = u32>,
+ ) {
+ let mut pending = self.pending.borrow_mut();
+ pending.clear();
+ for id in pending_ids {
+ *pending.entry(id).or_default() += 1;
+ }
+ drop(pending);
+ self.uncertain.set(false);
+ self.rearm_if_below_limit(durable);
+ }
+
+ pub(crate) fn mark_uncertain(&self) {
+ self.pending.borrow_mut().clear();
+ self.uncertain.set(true);
+ }
+
+ pub(crate) fn record_stranded(&self, id: u32) {
+ self.stranded.borrow_mut().insert(id);
+ }
+
+ pub(crate) fn clear_stranded(&self, id: u32) {
+ self.stranded.borrow_mut().remove(&id);
+ }
+
+ pub(crate) fn stranded_ids(&self) -> Vec<u32> {
+ self.stranded.borrow().iter().copied().collect()
+ }
+
+ pub(crate) fn is_stranded(&self, id: u32) -> bool {
+ self.stranded.borrow().contains(&id)
+ }
+
+ pub(crate) fn rearm_if_below_limit(&self, durable: &DurableConsumerOffsets) {
+ if !self.durable_warned.get() || self.uncertain.get() {
+ return;
+ }
+ if self.occupied(durable) < self.limit.get() {
+ self.durable_warned.set(false);
+ }
+ }
+
+ pub(crate) const fn admit_local_map_key(
Review Comment:
**Map-full denial is terminal, but the condition is transient.**
`uncertain: false` here makes this `TooManyConsumerOffsets`, which every SDK classifier treats as non-retryable. The condition is not terminal though: the phantoms are reclaimable, and pre-reclaim runs at plan build while `apply_local_offset` runs later on the detached disk task — requests landing in that window get a permanent code for a condition that would have cleared.
Worth pairing with the SDK backoff: `sdk/src/clients/consumer.rs:1358-1366` sleeps only for `Disconnected | Unauthenticated | StaleClient`, so reclassifying this to 58 alone would relocate the hot-loop rather than fix it — and 58 additionally triggers a full-roster walk (`leader_aware.rs:352` marks it "safe to re-issue anywhere"), turning one tight loop into an N× fan-out. The SDK change wants to land first.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2754,6 +2861,12 @@ where
entry.offset.store(value, Ordering::Release);
let path = entry.path.clone();
self.consumer_offsets.pin().insert(*id as usize, entry);
+ self.durable_consumer_offsets.record_explicit(
Review Comment:
**Durable membership is recorded before the files land.**
`record_explicit(..., None)` runs here, but `mark_persisted` only runs for writes that succeeded (:2911). On the degraded `offsets_written == false` path the partition holds durable membership for keys with no file on disk.
The safe half works — `covers()` is false for `None`, so the next auto-commit blind-writes and repairs. The unsafe half: a key that never commits again is still counted by `occupied()` and still exported by `committed_entries`, so this node advertises in its own transfer artifact an offset it will lose on its own restart.
The guard at :1998-2015 tests *live-map* membership rather than file durability, so it cannot catch this.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2421,6 +2776,30 @@ where
}
};
+ if args.auto_commit
+ && self.consensus.is_primary()
+ && let Ok(pending) = PendingConsumerOffsetCommit::try_from_polling_consumer(consumer, 0)
+ {
+ let capacity = self.consumer_offset_capacity_for(pending.kind);
+ if !capacity.is_uncertain()
+ && self.consumer_offset_map_count(pending.kind) >= capacity.limit()
+ {
+ let exists = match pending.kind {
+ ConsumerKind::Consumer => self
+ .consumer_offsets
+ .pin()
+ .contains_key(&(pending.consumer_id as usize)),
+ ConsumerKind::ConsumerGroup => self
+ .consumer_group_offsets
+ .pin()
+ .contains_key(&ConsumerGroupId(pending.consumer_id as usize)),
+ };
+ if !exists {
Review Comment:
**Unmemoized O(limit) sweep on the poll path, and the SDK closes the loop.**
`reclaim_phantom_offsets` is a full papaya `retain` whose predicate does up to 3 hash lookups plus an atomic per entry. The guard above (`map_count >= limit && !exists`) is exactly the saturated steady state, so while the map is full every poll from a non-member repeats the whole scan — on the pump that serves every partition on the shard.
It self-amplifies: refusal → `sdk/src/clients/consumer.rs:1358-1366` backs off only for `Disconnected | Unauthenticated | StaleClient`, so 3024/58 fall through with no sleep → immediate re-poll under `without_poll_interval()` → this sweep again. Any client with poll rights and fresh consumer ids arms it; no rate limit anywhere in the loop.
A `Cell<u64>` epoch bumped only by protection-losing events (`durable.remove`, `release_reservation`, `set_pending_count`, `rollback_created`, view change), skipping the `retain` when nothing changed since the last sweep that reclaimed nothing, makes this O(1) amortized — and cheap enough to also run on backups, which is what :2702 needs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz commented on PR #4063:
URL: https://github.com/apache/iggy/pull/4063#issuecomment-5550681304
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz commented on PR #4063:
URL: https://github.com/apache/iggy/pull/4063#issuecomment-5553458038
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz commented on PR #4063:
URL: https://github.com/apache/iggy/pull/4063#issuecomment-5557868015
/ready
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(partitions): bound consumer offset state per partition [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4063:
URL: https://github.com/apache/iggy/pull/4063#discussion_r3947244611
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2186,38 +2346,46 @@ where
.collect()
}
- /// Reclaim every stored consumer-group offset whose group id is no longer
- /// `is_live`, returning the owned persisted-file paths the caller must unlink.
- ///
- /// Fully synchronous (no `.await`): the in-memory papaya remove happens here,
- /// the disk unlink is deferred to the caller on owned `String` data so no
- /// borrow of `self` survives across the await. This is the only safe shape
- /// for the reconciler, which runs on a sibling task to the pump that may
- /// realloc the partitions vec during that await. The remove-then-unlink
- /// ordering matches the crash-safe GC invariant (monotonic, never-reused
- /// group ids mean a recreated group never reads a dead group's offset).
+ /// Snapshot dead group keys for deletion through the partition's VSR log.
+ /// A local unlink could free the primary's quota while backups retained
+ /// every older generation, so reclamation uses the same ordered delete as
+ /// an explicit consumer-offset request.
#[must_use]
- #[allow(clippy::cast_possible_truncation)]
- pub fn reclaim_dead_group_offsets(&self, is_live: impl Fn(u64) -> bool) -> Vec<String> {
- let pinned = self.consumer_group_offsets.pin();
- let dead: Vec<u64> = pinned
- .keys()
- .map(|key| key.0 as u64)
- .filter(|group_id| !is_live(*group_id))
- .collect();
- let mut paths = Vec::with_capacity(dead.len());
- for group_id in dead {
- pinned.remove(&ConsumerGroupId(group_id as usize));
- self.persisted_offsets
- .borrow_mut()
- .remove(&(ConsumerKind::ConsumerGroup, group_id as u32));
- if let Some(path) =
- self.persisted_offset_path(ConsumerKind::ConsumerGroup, group_id as u32)
- {
- paths.push(path);
+ pub fn dead_consumer_group_offset_ids(&self, is_live: impl Fn(u64) -> bool) -> Vec<u32> {
+ if !self.consensus.is_primary() || !self.consensus.is_normal() {
+ return Vec::new();
+ }
+ let mut dead = Vec::new();
+ for key in self.consumer_group_offsets.pin().keys() {
+ let Ok(id) = u32::try_from(key.0) else {
+ continue;
+ };
+ if !is_live(u64::from(id)) {
+ dead.push(id);
}
}
- paths
+ // A stranded file already failed normal loading or unlink. Reissuing
Review Comment:
Warning: The comment claims stranded keys are skipped, but the body only sorts and dedups. Add `&& !capacity.is_stranded(id)` to the push, or state why a stranded key cannot be in `consumer_group_offsets`.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -4367,7 +4979,26 @@ where
});
return;
}
+ }
+
+ // Commit replies and the applied frontier must follow directory
+ // durability. One sync per touched kind covers the whole walk. A sync
+ // failure is attributed to the batch boundary because one directory
+ // sync covers every delete in that kind, not one uniquely failing op.
+ if let Err(error) = self.flush_consumer_offset_directories().await {
+ if let Some(entry) = drained.last() {
+ error!(namespace_raw, %error, "consumer offset directory sync failed after committed operations");
+ self.fatal = Some(FatalCommit {
Review Comment:
Critical: One sync failure flag covers both offset kinds, so a failed sync of the groups directory sets `fatal` on a walk that only wrote consumer offsets, shutting the node down. Attribute the failure per kind.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8006,49 +9590,233 @@ mod tests {
let _ = std::fs::remove_dir_all(&dir);
}
- /// `reclaim_dead_group_offsets` must drop exactly the not-`is_live` groups
- /// from the in-memory map and hand back their owned persisted-file paths,
- /// leaving live groups untouched. The returned `Vec<String>` is what the
- /// reconciler unlinks off-borrow, so it carries no partition reference.
- ///
- /// Scope: the synchronous removal contract the off-borrow split relies on.
- /// The cross-task interleave it enables -- a pump mutating the partitions vec
- /// while a sibling task is parked mid-await -- is covered on the simulator's
- /// deterministic executor, against the debug borrow tripwire, by
- /// `simulator::tests::shell_detects_partition_borrow_held_across_await`
- /// (`swap_remove`) and
- /// `shell_detects_partition_borrow_held_across_a_pump_realloc` (a growing
- /// `push`, which relocates every element).
#[compio::test]
- async fn reclaim_dead_group_offsets_drops_dead_keeps_live() {
- let mut partition = test_partition();
- let group_offsets_path = "/iggy-test-cg-offsets".to_owned();
- partition.consumer_group_offsets_path = Some(group_offsets_path.clone());
-
- let dead: u32 = 1;
- let live: u32 = 2;
+ async fn given_dead_group_when_reclaimed_should_keep_capacity_until_replicated_delete() {
+ let (mut partition, _) = recording_partition();
partition.consumer_group_offsets.pin().insert(
- ConsumerGroupId(dead as usize),
- ConsumerOffset::new(ConsumerKind::ConsumerGroup, dead, 7, String::new()),
+ ConsumerGroupId(7),
+ ConsumerOffset::new(ConsumerKind::ConsumerGroup, 7, 11, String::new()),
+ );
+ partition.seed_recovered_consumer_offset(ConsumerKind::ConsumerGroup, 7, 11, 11);
+ assert_eq!(partition.dead_consumer_group_offset_ids(|_| false), vec![7]);
+ assert_eq!(partition.consumer_group_offset_ids(), vec![7]);
+ assert_eq!(
+ partition.occupied_consumer_offset_count(ConsumerKind::ConsumerGroup),
+ 1
+ );
+ partition.stage_consumer_offset_delete(1, ConsumerKind::ConsumerGroup, 7);
+ partition
+ .apply_staged_consumer_offset_commit(1)
+ .await
+ .expect("delete commits");
+ assert_eq!(
+ partition.occupied_consumer_offset_count(ConsumerKind::ConsumerGroup),
+ 0
+ );
+ assert!(partition.consumer_group_offset_ids().is_empty());
+ }
+
+ #[compio::test]
+ async fn given_known_stranded_group_file_when_reconciling_should_not_submit_delete_loop() {
Review Comment:
Warning: This test passes with or without a stranded filter, since id 7 is never inserted into `consumer_group_offsets` and the function iterates only that map. Insert the id before seeding it as stranded.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2261,21 +2429,50 @@ where
);
if let Err(error) = self.persist_consumer_offset_commit(pending).await {
+ if offset.is_some() {
+ self.release_consumer_offset_reservation(kind, consumer_id);
+ }
emit_partition_diag(
tracing::Level::WARN,
&PartitionDiagEvent::new(self.diag_ctx(), "no_ack offset persist failed")
.with_operation(request_header.operation)
.with_error(error.to_string()),
);
+ Self::send_partition_deny_or_log(
+ &self.consensus,
+ &request_header,
+ error.as_code(),
+ "no_ack offset failure reply send failed",
+ waiter,
+ )
+ .await;
return;
}
- if let Err(error) = self.apply_consumer_offset_commit(pending) {
+ self.apply_consumer_offset_commit(pending);
+ if let Err(error) = self.flush_consumer_offset_directories().await {
Review Comment:
Warning: `apply_consumer_offset_commit` runs before this flush, so on a sync error the client gets `CannotSyncFile` while the key is already durable and in the live map. Reply success and log, or sync only the kind this request touched.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2355,14 +2649,193 @@ where
ReplicaLogContext::from_consensus(self.consensus(), PlaneKind::Partitions)
}
- fn clear_pending_consumer_offset_commits_if_view_changed(&mut self) {
+ fn store_offset_range_error(&self, offset: u64) -> Option<IggyError> {
+ let current = self.stats.current_offset();
+ (offset > current || (current == 0 && self.stats.messages_count_inconsistent() == 0))
+ .then_some(IggyError::InvalidOffset(offset))
+ }
+
+ fn resynchronize_consumer_offset_reservations(&mut self) {
+ self.resynchronize_consumer_offset_reservations_inner(false);
+ }
+
+ /// Retry incomplete accounting at most once per shard tick, after progress.
+ pub fn retry_consumer_offset_reservations(&mut self) {
+ if self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain()
+ {
+ self.resynchronize_consumer_offset_reservations_inner(true);
+ }
+ }
+
+ #[allow(clippy::too_many_lines)]
+ fn resynchronize_consumer_offset_reservations_inner(&mut self, from_tick: bool) {
let current_view = self.consensus.view();
+ let scan_state = (
+ self.consensus.commit_min(),
+ self.consensus.commit_max(),
+ self.consensus.sequencer().current_sequence(),
+ self.log.journal().inner.last_op(),
+ );
+ let uncertain = self.consumer_offset_capacity.is_uncertain()
+ || self.consumer_group_offset_capacity.is_uncertain();
if current_view == self.observed_view {
- return;
+ if uncertain && !from_tick {
+ return;
+ }
+ let retry_requested = self.offset_reservations_need_resync.get()
+ || (uncertain && self.offset_reservations_scan_state != Some(scan_state));
+ if !retry_requested {
+ return;
+ }
}
- self.pending_consumer_offset_commits.clear();
+ if current_view != self.observed_view {
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ self.mark_consumer_group_offsets_need_reconcile();
+ }
+
+ let from_op = self
+ .consensus
+ .commit_min()
+ .max(self.purge_floor_op)
+ .saturating_add(1);
+ let commit_max = self.consensus.commit_max();
+ let to_op = self
+ .consensus
+ .sequencer()
+ .current_sequence()
+ .min(self.log.journal().inner.last_op().unwrap_or(commit_max));
+ // Committed offset prepares still need local apply, even if a message
+ // flush already evicted their journal bytes. Never drop their staging.
+ // Within one view an op is assigned once, so uncommitted staging is
+ // kept too; only a view change can replace what sits at those ops, and
+ // only then is the tail decoded again.
+ let same_view = current_view == self.observed_view;
+ let mut rebuilt: HashMap<_, _> = self
+ .pending_consumer_offset_commits
+ .iter()
+ .filter(|(op, _)| **op >= from_op && (**op <= commit_max || same_view))
+ .map(|(op, pending)| (*op, *pending))
+ .collect();
+ let headers = self.log.journal().inner.repair_headers_in(from_op..=to_op);
+ let uncommitted_from = from_op.max(commit_max.saturating_add(1));
+ let expected = to_op
+ .checked_sub(uncommitted_from)
+ .map_or(0, |span| span.saturating_add(1));
+ let mut decode_failed =
+ headers.keys().filter(|op| **op >= uncommitted_from).count() as u64 != expected;
+ for (op, header) in headers {
+ if !matches!(
+ header.operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) {
+ continue;
+ }
+ if rebuilt.contains_key(&op) {
+ continue;
+ }
+ match self.restage_consumer_offset_from_journal(op) {
+ Ok(pending) => {
+ rebuilt.insert(op, pending);
+ }
+ Err(error) => {
+ error!(
+ target: "iggy.partitions.diag",
+ plane = "partitions",
+ replica_id = self.consensus.replica(),
+ namespace_raw = self.namespace().inner(),
+ op,
+ %error,
+ "failed to rebuild consumer offset reservations after view change"
+ );
+ decode_failed = true;
+ break;
+ }
+ }
+ }
+ self.pending_consumer_offset_commits = rebuilt;
+ if decode_failed {
+ self.consumer_offset_capacity.mark_uncertain();
+ self.consumer_group_offset_capacity.mark_uncertain();
+ } else {
+ let consumer_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::Consumer
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, consumer_ids);
+ let group_ids = self
+ .pending_consumer_offset_commits
+ .values()
+ .filter(|pending| {
+ pending.kind == ConsumerKind::ConsumerGroup
+ && matches!(pending.mutation, PendingConsumerOffsetMutation::Upsert(_))
+ })
+ .map(|pending| pending.consumer_id);
+ self.consumer_group_offset_capacity
+ .rebuild(&self.durable_consumer_offsets, group_ids);
+ }
self.observed_view = current_view;
+ self.offset_reservations_scan_state = Some(scan_state);
+ // The shard tick retries uncertainty after journal or frontier progress.
+ self.offset_reservations_need_resync.set(false);
+ }
+
+ fn reclaim_phantom_offsets(&self, kind: ConsumerKind, map_count: usize) {
+ let capacity = self.consumer_offset_capacity_for(kind);
+ if self.durable_consumer_offsets.count(kind) >= capacity.limit()
+ || !capacity.should_reclaim(&self.durable_consumer_offsets)
+ {
+ return;
+ }
+ let needed = map_count.saturating_sub(capacity.limit()).saturating_add(1);
+ match kind {
+ ConsumerKind::Consumer => {
+ self.reclaim_phantom_offset_keys(&self.consumer_offsets, kind, needed, |key| {
+ u32::try_from(*key).ok()
+ });
+ }
+ ConsumerKind::ConsumerGroup => self.reclaim_phantom_offset_keys(
+ &self.consumer_group_offsets,
+ kind,
+ needed,
+ |key| u32::try_from(key.0).ok(),
+ ),
+ }
+ }
+
+ fn reclaim_phantom_offset_keys<K: Hash + Eq>(
+ &self,
+ offsets: &papaya::HashMap<K, ConsumerOffset>,
+ kind: ConsumerKind,
+ mut remaining: usize,
+ consumer_id: impl Fn(&K) -> Option<u32>,
+ ) {
+ let capacity = self.consumer_offset_capacity_for(kind);
+ let map = offsets.pin();
+ for (key, _) in &map {
+ if let Some(id) = consumer_id(key)
+ && !capacity.holds(id, &self.durable_consumer_offsets)
+ && map.remove(key).is_some()
+ {
+ capacity.forget_inactive_provisional(id);
+ debug!(
Review Comment:
Warning: Evicting a live-map cursor with no durable backing sends that consumer back to offset 0 on its next `Next` poll, a full redelivery. Raise this to `warn!` and document it in the poll contract.
##########
core/sdk/src/leader_aware.rs:
##########
@@ -406,6 +406,10 @@ impl RosterWalk {
self.attempted.push(endpoint.clone());
Some(endpoint)
}
+
+ pub(crate) fn is_single_endpoint(&self) -> bool {
Review Comment:
Warning: `is_single_endpoint` also returns true for an empty roster, since `RosterWalk::new` always seeds `attempted` with the current address. Gate on a non-empty roster so a client whose roster discovery failed does not replay one node.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -199,26 +196,55 @@ fn spawn_poll_io<B, MJ, S, SB>(
"slow partition poll; gather side may have timed out"
);
}
- // Fire-and-forget: the poll reply is not gated on the offset commit.
- if let Some(applied) = auto_commit {
- submit_auto_commit(&shard, namespace, &applied);
- }
- let _ = reply.try_send(PartitionReadReply::Poll {
- fragments,
- current_offset,
- });
+ let result = poll_reply(&shard, namespace, result);
+ let _ = reply.try_send(result);
});
}
+fn poll_reply<B, MJ, S, SB>(
+ shard: &Rc<ShellShard<B, MJ, S, SB>>,
+ namespace: IggyNamespace,
+ result: Result<(PollFragments, u64, Option<AutoCommitApplied>), ConsumerOffsetCapacityError>,
+) -> PartitionReadReply
+where
+ B: ShellBus,
+ MJ: JournalHandle + 'static,
+ MJ::Target: Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>,
+ S: 'static,
+ SB: SuperblockStore + 'static,
+{
+ match result {
+ Ok((fragments, current_offset, auto_commit)) => {
+ if let Some(applied) = auto_commit
+ && let Err(error) = submit_auto_commit(shard, namespace, &applied)
+ {
+ PartitionReadReply::Rejected(error)
Review Comment:
Warning: A poll that fails auto-commit now returns `Rejected` and drops the fragments already read, and inbox pressure reaches this with `TransientNotAccepted`, not only the quota limit. Document that case alongside `TooManyConsumerOffsets`.
##########
core/shard/src/lib.rs:
##########
@@ -9830,8 +9872,16 @@ fn rotate_sweep_to_cursor(namespaces: &mut [IggyNamespace], cursor: Option<IggyN
namespaces.rotate_left(namespaces.partition_point(|namespace| *namespace < cursor));
}
-/// Whether this replica holds adopted suffix HEADERS above `commit_max` whose
-/// bodies never arrived.
+fn partition_repair_fetch_to_op(
+ commit_min: u64,
+ commit_max: u64,
+ missing_suffix: Option<u64>,
Review Comment:
Warning: Returning the adopted suffix head while `commit_min < commit_max` makes the repair request span the committed prefix this replica already holds. This VSR change is unrelated to consumer offset quotas; split it into its own PR.
##########
core/sdk/src/tcp/tcp_client.rs:
##########
@@ -1323,6 +1327,15 @@ impl TcpClient {
// hops would bounce the request between two nodes and
// never reach the rest of the roster.
(next, true)
+ } else if roster_walk
Review Comment:
Warning: Dropping the guard here means line 1283 re-acquires `routing_lock` on the next turn, so the lock is held for roughly half the retry window. Keep the guard across the retry instead.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -3137,9 +3674,33 @@ where
/// is bypassed at view-change reset.
#[allow(clippy::future_not_send)]
pub async fn drain_request_queue_into_prepares(&mut self, slots_freed: usize) {
- for _ in 0..slots_freed {
+ self.resynchronize_consumer_offset_reservations();
+ let mut promoted = 0;
+ while promoted < slots_freed {
Review Comment:
Warning: `promoted` advances only on success and every denial path uses `continue`, so one call can drain all 64 queued requests with an awaited reply send each. Break after a few consecutive denials.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,679 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, RefCell};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: u64,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+ membership_epoch: Cell<u64>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset && state.persisted_high_water >= offset
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) -> bool {
+ let created = self
+ .entries(kind)
+ .borrow_mut()
+ .insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ )
+ .is_none();
+ if created {
+ self.bump_membership_epoch();
+ }
+ created
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ match entries.entry(id) {
+ Entry::Occupied(mut entry) => {
+ let state = entry.get_mut();
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = state.persisted_high_water.max(persisted_high_water);
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ });
+ drop(entries);
+ self.bump_membership_epoch();
+ }
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ let removed = self.entries(kind).borrow_mut().remove(&id).is_some();
+ if removed {
+ self.bump_membership_epoch();
+ }
+ removed
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ self.bump_membership_epoch();
+ }
+
+ #[cfg(any(test, feature = "simulator"))]
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ pub(crate) fn with_entries<T>(
+ &self,
+ kind: ConsumerKind,
+ read: impl FnOnce(&HashMap<u32, DurableOffsetState>) -> T,
+ ) -> T {
+ read(&self.entries(kind).borrow())
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+
+ fn bump_membership_epoch(&self) {
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Arc<ProvisionalToken>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
+ map_warned: Cell<bool>,
+ reclaim_epoch: Arc<AtomicU64>,
+ last_reclaim: Cell<Option<(u64, u64)>>,
+}
+
+impl ConsumerOffsetCapacity {
+ pub(crate) fn new(kind: ConsumerKind, limit: usize) -> Self {
+ Self {
+ kind,
+ limit: Cell::new(limit),
+ pending: RefCell::new(HashMap::new()),
+ provisional: RefCell::new(HashMap::new()),
+ stranded: RefCell::new(HashSet::new()),
+ uncertain: Cell::new(false),
+ durable_warned: Cell::new(false),
+ map_warned: Cell::new(false),
+ reclaim_epoch: Arc::new(AtomicU64::new(0)),
+ last_reclaim: Cell::new(None),
+ }
+ }
+
+ pub(crate) fn set_limit(&self, limit: usize) {
+ self.limit.set(limit);
+ }
+
+ pub(crate) const fn limit(&self) -> usize {
+ self.limit.get()
+ }
+
+ pub(crate) fn try_reserve(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ *self.pending.borrow_mut().entry(id).or_default() += 1;
+ Ok(())
+ }
+
+ pub(crate) fn check(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ if self.holds(id, durable) || self.stranded.borrow().contains(&id) {
+ return Ok(());
+ }
+ let limit = self.limit.get();
+ let durable_count = durable.count(self.kind);
+ let upper_bound = durable_count
+ .saturating_add(self.pending.borrow().len())
+ .saturating_add(self.provisional.borrow().len())
+ .saturating_add(self.stranded.borrow().len());
+ if !self.uncertain.get() && upper_bound < limit {
+ self.durable_warned.set(false);
+ return Ok(());
+ }
+ // A full durable table cannot gain room by pruning provisional keys.
+ let occupied = if durable_count >= limit {
+ durable_count
+ } else {
+ let mut provisional = self.provisional.borrow_mut();
+ if provisional.len() >= limit {
+ provisional
+ .retain(|key, token| *key == id || token.active.load(Ordering::Relaxed) > 0);
+ }
+ drop(provisional);
+ self.occupied(durable)
+ };
+ if self.uncertain.get() || occupied >= limit {
+ return Err(ConsumerOffsetCapacityError {
+ kind: self.kind,
+ occupied,
+ limit,
+ first_in_episode: !self.durable_warned.replace(true),
+ uncertain: self.uncertain.get(),
+ });
+ }
+ self.durable_warned.set(false);
+ Ok(())
+ }
+
+ pub(crate) fn reserve_provisional(
+ self: &Rc<Self>,
+ id: u32,
+ durable: &Rc<DurableConsumerOffsets>,
+ ) -> Result<AutoCommitReservation, ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ let mut provisional = self.provisional.borrow_mut();
+ let token = Arc::clone(provisional.entry(id).or_insert_with(|| {
+ Arc::new(ProvisionalToken {
+ reclaim_epoch: Arc::clone(&self.reclaim_epoch),
+ active: AtomicUsize::new(0),
+ })
+ }));
+ token.active.fetch_add(1, Ordering::Relaxed);
+ Ok(AutoCommitReservation {
+ token,
+ kind: self.kind,
+ consumer_id: id,
+ })
+ }
+
+ pub(crate) fn owns(&self, reservation: &AutoCommitReservation) -> bool {
+ reservation.kind == self.kind
+ && self
+ .provisional
+ .borrow()
+ .get(&reservation.consumer_id)
+ .is_some_and(|token| Arc::ptr_eq(token, &reservation.token))
+ }
+
+ pub(crate) fn holds(&self, id: u32, durable: &DurableConsumerOffsets) -> bool {
+ durable.contains(self.kind, id)
+ || self.pending.borrow().contains_key(&id)
+ || self
+ .provisional
+ .borrow()
+ .get(&id)
+ .is_some_and(|token| token.active.load(Ordering::Relaxed) > 0)
+ }
+
+ pub(crate) fn set_pending_count(&self, id: u32, count: usize) {
+ if count == 0 {
+ if self.pending.borrow_mut().remove(&id).is_some() {
+ self.note_local_key_change();
+ }
+ } else {
+ self.pending.borrow_mut().insert(id, count);
+ }
+ }
+
+ pub(crate) fn release_reservation(&self, id: u32) {
+ let mut pending = self.pending.borrow_mut();
+ let Some(count) = pending.get_mut(&id) else {
+ return;
+ };
+ if *count == 1 {
+ pending.remove(&id);
+ self.note_local_key_change();
+ } else {
+ *count -= 1;
+ }
+ }
+
+ pub(crate) const fn is_uncertain(&self) -> bool {
+ self.uncertain.get()
+ }
+
+ pub(crate) fn rebuild(
+ &self,
+ durable: &DurableConsumerOffsets,
+ pending_ids: impl IntoIterator<Item = u32>,
+ ) {
+ let mut pending = self.pending.borrow_mut();
+ pending.clear();
+ for id in pending_ids {
+ *pending.entry(id).or_default() += 1;
+ }
+ drop(pending);
+ self.note_local_key_change();
+ self.uncertain.set(false);
+ self.rearm_if_below_limit(durable);
+ }
+
+ pub(crate) fn mark_uncertain(&self) {
+ self.pending.borrow_mut().clear();
+ self.uncertain.set(true);
+ self.note_local_key_change();
+ }
+
+ pub(crate) fn record_stranded(&self, id: u32) {
Review Comment:
Warning: `stranded` is cleared only by `clear_stranded`, while `rebuild` and `mark_uncertain` leave it, so an unlinkable file latches permanent refusal of new keys with no metric. Export stranded cardinality.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2959,6 +3195,59 @@ where
}
self.log.journal().inner.clear_all();
self.log.journal_mut().info = crate::log::JournalInfo::default();
+ self.consumer_offsets.pin().clear();
+ self.consumer_group_offsets.pin().clear();
+ self.last_polled_offsets.pin().clear();
+ self.durable_consumer_offsets.clear();
+ self.pending_consumer_offset_commits.clear();
+ self.queued_auto_commit_reservations.borrow_mut().clear();
+ for kind in [ConsumerKind::Consumer, ConsumerKind::ConsumerGroup] {
+ self.consumer_offset_capacity_for(kind)
+ .rebuild(&self.durable_consumer_offsets, std::iter::empty());
+ }
+ for (kind, dir) in [
+ (ConsumerKind::Consumer, self.consumer_offsets_path.as_ref()),
+ (
+ ConsumerKind::ConsumerGroup,
+ self.consumer_group_offsets_path.as_ref(),
+ ),
+ ] {
+ let Some(dir) = dir else { continue };
+ for entry in offset_dir_entries(dir) {
+ match entry {
+ OffsetDirEntry::Replacement(path) => {
+ if let Err(error) = compio::fs::remove_file(&path).await {
+ tracing::warn!(
+ path,
+ %error,
+ "could not remove an abandoned offset replacement"
+ );
+ }
+ }
+ OffsetDirEntry::Offset { id, path } => {
+ // Same policy as install: an unlinkable file strands its
+ // key instead of failing the converge and looping.
+ match retry_offset_mutation(|| delete_persisted_offset(&path)).await {
+ Ok(_) => self.consumer_offset_capacity_for(kind).clear_stranded(id),
+ Err(error) => {
+ self.consumer_offset_capacity_for(kind).record_stranded(id);
+ tracing::warn!(
+ path,
+ consumer_id = id,
+ %error,
+ "converge could not remove a consumer offset file"
+ );
+ }
+ }
+ }
+ }
+ }
+ if std::path::Path::new(dir).exists() {
Review Comment:
Warning: `Path::new(dir).exists()` is a blocking stat on the pump. Use the `NotFound` tolerance already used in `flush_consumer_offset_directories`, which is why `create_parent_dir` avoids the probe.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -1606,6 +1792,7 @@ where
if self.repair.is_some() {
return Err(PartitionTransferUnavailable::RepairInProgress);
}
+ self.validate_consumer_offset_transfer_counts()?;
Review Comment:
Simplification: `validate_consumer_offset_transfer_counts` runs again as the first statement of `offsets_wire_snapshot`. Drop this call.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -4075,6 +4675,12 @@ where
let (frozen_batches, index_bytes, flush_index, batch_count, committed_info, chunk_len) = {
let segment = self.log.active_segment();
let mut file_position = segment.size.as_bytes_u64();
+ let persisted_end = if file_position == 0 {
Review Comment:
Nit: The `persisted_end` fix is segment-flush idempotence for a repaired batch, unrelated to consumer offset quotas. Split it out so it can be reverted independently.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,679 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, RefCell};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: u64,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+ membership_epoch: Cell<u64>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset && state.persisted_high_water >= offset
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) -> bool {
+ let created = self
+ .entries(kind)
+ .borrow_mut()
+ .insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ )
+ .is_none();
+ if created {
+ self.bump_membership_epoch();
+ }
+ created
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ match entries.entry(id) {
+ Entry::Occupied(mut entry) => {
+ let state = entry.get_mut();
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = state.persisted_high_water.max(persisted_high_water);
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ });
+ drop(entries);
+ self.bump_membership_epoch();
+ }
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ let removed = self.entries(kind).borrow_mut().remove(&id).is_some();
+ if removed {
+ self.bump_membership_epoch();
+ }
+ removed
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ self.bump_membership_epoch();
+ }
+
+ #[cfg(any(test, feature = "simulator"))]
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ pub(crate) fn with_entries<T>(
+ &self,
+ kind: ConsumerKind,
+ read: impl FnOnce(&HashMap<u32, DurableOffsetState>) -> T,
+ ) -> T {
+ read(&self.entries(kind).borrow())
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+
+ fn bump_membership_epoch(&self) {
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Arc<ProvisionalToken>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
+ map_warned: Cell<bool>,
+ reclaim_epoch: Arc<AtomicU64>,
+ last_reclaim: Cell<Option<(u64, u64)>>,
+}
+
+impl ConsumerOffsetCapacity {
+ pub(crate) fn new(kind: ConsumerKind, limit: usize) -> Self {
+ Self {
+ kind,
+ limit: Cell::new(limit),
+ pending: RefCell::new(HashMap::new()),
+ provisional: RefCell::new(HashMap::new()),
+ stranded: RefCell::new(HashSet::new()),
+ uncertain: Cell::new(false),
+ durable_warned: Cell::new(false),
+ map_warned: Cell::new(false),
+ reclaim_epoch: Arc::new(AtomicU64::new(0)),
+ last_reclaim: Cell::new(None),
+ }
+ }
+
+ pub(crate) fn set_limit(&self, limit: usize) {
+ self.limit.set(limit);
+ }
+
+ pub(crate) const fn limit(&self) -> usize {
+ self.limit.get()
+ }
+
+ pub(crate) fn try_reserve(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ *self.pending.borrow_mut().entry(id).or_default() += 1;
+ Ok(())
+ }
+
+ pub(crate) fn check(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ if self.holds(id, durable) || self.stranded.borrow().contains(&id) {
+ return Ok(());
+ }
+ let limit = self.limit.get();
+ let durable_count = durable.count(self.kind);
+ let upper_bound = durable_count
+ .saturating_add(self.pending.borrow().len())
+ .saturating_add(self.provisional.borrow().len())
Review Comment:
Warning: `provisional.len()` counts inactive tokens, so once the token map reaches `limit` the fast path never fires and every `check` falls through to `occupied`, allocating a `HashSet` per call. Count only active tokens.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -4367,7 +4979,26 @@ where
});
return;
}
+ }
+
+ // Commit replies and the applied frontier must follow directory
+ // durability. One sync per touched kind covers the whole walk. A sync
+ // failure is attributed to the batch boundary because one directory
+ // sync covers every delete in that kind, not one uniquely failing op.
+ if let Err(error) = self.flush_consumer_offset_directories().await {
+ if let Some(entry) = drained.last() {
Review Comment:
Nit: With an empty `drained` the sync error is dropped with no log and no fence. Log unconditionally and keep only the fence inside the `if let`.
##########
core/partitions/src/offset_storage.rs:
##########
@@ -48,7 +51,7 @@ pub enum OffsetRecord {
/// A usable offset. `checksummed` is false for a bare offset predating the
/// checksum, read as-is and upgraded by the next write.
Value { offset: u64, checksummed: bool },
- /// Shorter than the value: a crash between `persist_offset`'s truncate and write.
+ /// Shorter than the value, such as a legacy interrupted in-place write.
Review Comment:
Nit: In-place writes are not legacy. `write_in_place` is the default path whenever `consumer_offset_enforce_fsync` is false, which is the shipped default.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2280,6 +2498,16 @@ where
}
}
+ let installed_end = staged.last().map(|meta| meta.end_offset);
+ let next_offset = offsets_wire
+ .next_offset
+ .max(installed_end.map_or(0, |end| end + 1));
+ let planned_offsets = self.plan_transfer_offset_writes(&offsets_wire, next_offset)?;
+ // Write and data-sync every small offset record before the destructive
+ // segment swap. Only ignored, nonnumeric replacement siblings exist at
+ // this point, so a write fault leaves the old partition serviceable.
+ stage_offset_writes(&planned_offsets).await?;
Review Comment:
Nit: `stage_offset_writes` writes and syncs `.tmp` files here, below a marker that says nothing may mutate. Reword the marker to say nothing may mutate live state.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -2341,7 +2627,15 @@ where
}
};
- if found {
+ // The live maps are what `get_consumer_offset` answers from, so a key
Review Comment:
Nit: The `replica_count() > 1` gate is inert, since every path that records durable also inserts into the live map. Drop it so a single replica cannot strand a durable slot.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -1551,6 +1642,28 @@ fn final_paths(partition_dir: &str, start_offset: u64) -> (String, String) {
/// depth against how long one partition monopolises it; matches the tick's
/// superblock pre-pass.
const OFFSET_PERSIST_CONCURRENCY: usize = 16;
+const OFFSET_IO_ATTEMPTS: usize = 3;
+/// First retry delay of [`retry_offset_mutation`]; each further retry doubles it.
+const OFFSET_IO_BACKOFF_BASE: std::time::Duration = std::time::Duration::from_millis(10);
+
+async fn retry_offset_mutation<T, E: fmt::Debug, F: Future<Output = Result<T, E>>>(
Review Comment:
Nit: The first two attempts log at `debug!`, but the final `operation().await` returns its error unlogged. Log the last attempt too.
##########
core/partitions/src/consumer_offset_capacity.rs:
##########
@@ -0,0 +1,679 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy_common::ConsumerKind;
+use std::cell::{Cell, RefCell};
+use std::collections::hash_map::Entry;
+use std::collections::{HashMap, HashSet};
+use std::rc::Rc;
+use std::sync::Arc;
+use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct DurableOffsetState {
+ pub(crate) committed_offset: u64,
+ pub(crate) persisted_high_water: u64,
+}
+
+#[derive(Debug, Default)]
+pub struct DurableConsumerOffsets {
+ consumers: RefCell<HashMap<u32, DurableOffsetState>>,
+ groups: RefCell<HashMap<u32, DurableOffsetState>>,
+ membership_epoch: Cell<u64>,
+}
+
+impl DurableConsumerOffsets {
+ pub(crate) fn get(&self, kind: ConsumerKind, id: u32) -> Option<DurableOffsetState> {
+ self.entries(kind).borrow().get(&id).copied()
+ }
+
+ pub(crate) fn contains(&self, kind: ConsumerKind, id: u32) -> bool {
+ self.entries(kind).borrow().contains_key(&id)
+ }
+
+ pub(crate) fn count(&self, kind: ConsumerKind) -> usize {
+ self.entries(kind).borrow().len()
+ }
+
+ pub(crate) fn covers(&self, kind: ConsumerKind, id: u32, offset: u64) -> bool {
+ self.get(kind, id).is_some_and(|state| {
+ state.committed_offset >= offset && state.persisted_high_water >= offset
+ })
+ }
+
+ pub(crate) fn record_explicit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) -> bool {
+ let created = self
+ .entries(kind)
+ .borrow_mut()
+ .insert(
+ id,
+ DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ },
+ )
+ .is_none();
+ if created {
+ self.bump_membership_epoch();
+ }
+ created
+ }
+
+ pub(crate) fn record_auto_commit(
+ &self,
+ kind: ConsumerKind,
+ id: u32,
+ committed_offset: u64,
+ persisted_high_water: u64,
+ ) {
+ let mut entries = self.entries(kind).borrow_mut();
+ match entries.entry(id) {
+ Entry::Occupied(mut entry) => {
+ let state = entry.get_mut();
+ state.committed_offset = state.committed_offset.max(committed_offset);
+ state.persisted_high_water = state.persisted_high_water.max(persisted_high_water);
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(DurableOffsetState {
+ committed_offset,
+ persisted_high_water,
+ });
+ drop(entries);
+ self.bump_membership_epoch();
+ }
+ }
+ }
+
+ pub(crate) fn remove(&self, kind: ConsumerKind, id: u32) -> bool {
+ let removed = self.entries(kind).borrow_mut().remove(&id).is_some();
+ if removed {
+ self.bump_membership_epoch();
+ }
+ removed
+ }
+
+ pub(crate) fn clear(&self) {
+ self.consumers.borrow_mut().clear();
+ self.groups.borrow_mut().clear();
+ self.bump_membership_epoch();
+ }
+
+ #[cfg(any(test, feature = "simulator"))]
+ pub(crate) fn committed_entries(&self, kind: ConsumerKind) -> Vec<(u32, u64)> {
+ self.entries(kind)
+ .borrow()
+ .iter()
+ .map(|(id, state)| (*id, state.committed_offset))
+ .collect()
+ }
+
+ pub(crate) fn with_entries<T>(
+ &self,
+ kind: ConsumerKind,
+ read: impl FnOnce(&HashMap<u32, DurableOffsetState>) -> T,
+ ) -> T {
+ read(&self.entries(kind).borrow())
+ }
+
+ const fn entries(&self, kind: ConsumerKind) -> &RefCell<HashMap<u32, DurableOffsetState>> {
+ match kind {
+ ConsumerKind::Consumer => &self.consumers,
+ ConsumerKind::ConsumerGroup => &self.groups,
+ }
+ }
+
+ fn bump_membership_epoch(&self) {
+ self.membership_epoch
+ .set(self.membership_epoch.get().wrapping_add(1));
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerOffsetCapacityError {
+ pub kind: ConsumerKind,
+ pub occupied: usize,
+ pub limit: usize,
+ pub first_in_episode: bool,
+ pub uncertain: bool,
+}
+
+impl From<ConsumerOffsetCapacityError> for iggy_common::IggyError {
+ fn from(error: ConsumerOffsetCapacityError) -> Self {
+ if error.uncertain {
+ Self::TransientNotAccepted
+ } else {
+ Self::TooManyConsumerOffsets
+ }
+ }
+}
+
+#[derive(Debug)]
+pub struct ConsumerOffsetCapacity {
+ kind: ConsumerKind,
+ limit: Cell<usize>,
+ pending: RefCell<HashMap<u32, usize>>,
+ provisional: RefCell<HashMap<u32, Arc<ProvisionalToken>>>,
+ stranded: RefCell<HashSet<u32>>,
+ uncertain: Cell<bool>,
+ durable_warned: Cell<bool>,
+ map_warned: Cell<bool>,
+ reclaim_epoch: Arc<AtomicU64>,
+ last_reclaim: Cell<Option<(u64, u64)>>,
+}
+
+impl ConsumerOffsetCapacity {
+ pub(crate) fn new(kind: ConsumerKind, limit: usize) -> Self {
+ Self {
+ kind,
+ limit: Cell::new(limit),
+ pending: RefCell::new(HashMap::new()),
+ provisional: RefCell::new(HashMap::new()),
+ stranded: RefCell::new(HashSet::new()),
+ uncertain: Cell::new(false),
+ durable_warned: Cell::new(false),
+ map_warned: Cell::new(false),
+ reclaim_epoch: Arc::new(AtomicU64::new(0)),
+ last_reclaim: Cell::new(None),
+ }
+ }
+
+ pub(crate) fn set_limit(&self, limit: usize) {
+ self.limit.set(limit);
+ }
+
+ pub(crate) const fn limit(&self) -> usize {
+ self.limit.get()
+ }
+
+ pub(crate) fn try_reserve(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ *self.pending.borrow_mut().entry(id).or_default() += 1;
+ Ok(())
+ }
+
+ pub(crate) fn check(
+ &self,
+ id: u32,
+ durable: &DurableConsumerOffsets,
+ ) -> Result<(), ConsumerOffsetCapacityError> {
+ if self.holds(id, durable) || self.stranded.borrow().contains(&id) {
+ return Ok(());
+ }
+ let limit = self.limit.get();
+ let durable_count = durable.count(self.kind);
+ let upper_bound = durable_count
+ .saturating_add(self.pending.borrow().len())
+ .saturating_add(self.provisional.borrow().len())
+ .saturating_add(self.stranded.borrow().len());
+ if !self.uncertain.get() && upper_bound < limit {
+ self.durable_warned.set(false);
+ return Ok(());
+ }
+ // A full durable table cannot gain room by pruning provisional keys.
+ let occupied = if durable_count >= limit {
+ durable_count
+ } else {
+ let mut provisional = self.provisional.borrow_mut();
+ if provisional.len() >= limit {
+ provisional
+ .retain(|key, token| *key == id || token.active.load(Ordering::Relaxed) > 0);
+ }
+ drop(provisional);
+ self.occupied(durable)
+ };
+ if self.uncertain.get() || occupied >= limit {
+ return Err(ConsumerOffsetCapacityError {
+ kind: self.kind,
+ occupied,
+ limit,
+ first_in_episode: !self.durable_warned.replace(true),
+ uncertain: self.uncertain.get(),
+ });
+ }
+ self.durable_warned.set(false);
+ Ok(())
+ }
+
+ pub(crate) fn reserve_provisional(
+ self: &Rc<Self>,
+ id: u32,
+ durable: &Rc<DurableConsumerOffsets>,
+ ) -> Result<AutoCommitReservation, ConsumerOffsetCapacityError> {
+ self.check(id, durable)?;
+ let mut provisional = self.provisional.borrow_mut();
+ let token = Arc::clone(provisional.entry(id).or_insert_with(|| {
+ Arc::new(ProvisionalToken {
+ reclaim_epoch: Arc::clone(&self.reclaim_epoch),
+ active: AtomicUsize::new(0),
+ })
+ }));
+ token.active.fetch_add(1, Ordering::Relaxed);
+ Ok(AutoCommitReservation {
+ token,
+ kind: self.kind,
+ consumer_id: id,
+ })
+ }
+
+ pub(crate) fn owns(&self, reservation: &AutoCommitReservation) -> bool {
+ reservation.kind == self.kind
+ && self
+ .provisional
+ .borrow()
+ .get(&reservation.consumer_id)
+ .is_some_and(|token| Arc::ptr_eq(token, &reservation.token))
+ }
+
+ pub(crate) fn holds(&self, id: u32, durable: &DurableConsumerOffsets) -> bool {
+ durable.contains(self.kind, id)
+ || self.pending.borrow().contains_key(&id)
+ || self
+ .provisional
+ .borrow()
+ .get(&id)
+ .is_some_and(|token| token.active.load(Ordering::Relaxed) > 0)
+ }
+
+ pub(crate) fn set_pending_count(&self, id: u32, count: usize) {
Review Comment:
Nit: `set_pending_count` assigns while `release_reservation` decrements the same counter, and both take `&self`. Document that the partition's `&mut self` call sites serialize them.
##########
core/shard/src/lib.rs:
##########
@@ -2009,6 +2024,32 @@ where
})
}
+ /// Submit an auto-commit back to the partition-owning shard's pump.
+ ///
+ /// # Errors
+ /// Returns a refusal if the local inbox cannot accept the frame.
+ pub fn submit_auto_commit_offset(
+ &self,
+ request: Message<RoutedRequestHeader>,
+ reservation: partitions::AutoCommitReservation,
+ ) -> Result<(), PartitionSubmitRefused> {
+ let frame = ShardFrame::lifecycle(LifecycleFrame::AutoCommitSubmit {
+ request,
+ reservation,
+ });
+ let sender = self
+ .senders
+ .get(usize::from(self.id))
+ .ok_or(PartitionSubmitRefused)?;
+ sender.try_send(frame).map_err(|error| {
+ self.metrics.record_frame_drop(
+ crate::metrics::frame_drop_variant::PARTITION,
Review Comment:
Nit: A poll-side auto-commit refusal now lands in the same partition frame-drop series as genuinely dropped frames. Use a distinct reason label.
##########
core/partitions/src/poll_plan.rs:
##########
@@ -861,55 +888,146 @@ impl AutoCommitCtx {
/// newer explicit store; the maps are lock-free (`papaya`), so this is
/// sound off the pump task.
#[allow(clippy::cast_possible_truncation)]
- pub(crate) fn apply(&self, offset: u64) {
- match &self.target {
+ pub(crate) fn apply(
+ self,
+ offset: u64,
+ ) -> Result<AutoCommitApplied, ConsumerOffsetCapacityError> {
+ let (kind, consumer_id) = self.kind_and_id();
+ let create = |path: Option<&str>| {
+ ConsumerOffset::new(
+ kind,
+ consumer_id,
+ offset,
+ path.map_or_else(String::new, |path| format!("{path}/{consumer_id}")),
+ )
+ };
+ let previous_offset = match &self.target {
AutoCommitTarget::Consumer {
offsets,
consumer_id,
create_path,
- } => {
- let consumer_id = *consumer_id;
- let map: &ConsumerOffsets = offsets;
- upsert_offset_max(map, consumer_id as usize, offset, || {
- create_path.as_deref().map_or_else(
- || {
- ConsumerOffset::new(
- ConsumerKind::Consumer,
- consumer_id,
- 0,
- String::new(),
- )
- },
- |path| ConsumerOffset::default_for_consumer(consumer_id, path),
- )
- });
- }
+ } => apply_local_offset(
+ offsets,
+ *consumer_id as usize,
+ offset,
+ &self.capacity,
+ self.durable.count(kind) >= self.capacity.limit(),
+ || create(create_path.as_deref()),
+ )?,
AutoCommitTarget::ConsumerGroup {
offsets,
group_id,
create_path,
- } => {
- let group_id = *group_id;
- let key = ConsumerGroupId(group_id as usize);
- let map: &ConsumerGroupOffsets = offsets;
- upsert_offset_max(map, key, offset, || {
- create_path.as_deref().map_or_else(
- || {
- ConsumerOffset::new(
- ConsumerKind::ConsumerGroup,
- group_id,
- 0,
- String::new(),
- )
- },
- |path| ConsumerOffset::default_for_consumer_group(key, path),
- )
- });
- }
+ } => apply_local_offset(
+ offsets,
+ ConsumerGroupId(*group_id as usize),
+ offset,
+ &self.capacity,
+ self.durable.count(kind) >= self.capacity.limit(),
+ || create(create_path.as_deref()),
+ )?,
+ };
+ Ok(AutoCommitApplied {
+ kind,
+ consumer_id,
+ offset,
+ previous_offset,
+ target: self.target,
+ capacity: self.capacity,
+ durable: self.durable,
+ last_polled: None,
+ })
+ }
+}
+
+impl AutoCommitApplied {
+ /// Record the group handoff frontier only after poll admission succeeds.
+ pub fn mark_served(&self) {
+ if let Some(last_polled) = &self.last_polled {
+ last_polled.record(self.offset);
+ }
+ }
+ /// Reserve a durable key before the synthetic store is submitted.
+ /// Returns `None` when committed state already covers this offset.
+ ///
+ /// # Errors
+ /// Returns a capacity error when this is a new durable key and the
+ /// partition's per-kind limit has been reached.
+ pub fn reserve_durable(
+ &self,
+ ) -> Result<Option<AutoCommitReservation>, ConsumerOffsetCapacityError> {
+ if self
+ .durable
+ .covers(self.kind, self.consumer_id, self.offset)
+ {
+ return Ok(None);
+ }
+ self.capacity
+ .reserve_provisional(self.consumer_id, &self.durable)
+ .map(Some)
+ }
+
+ pub(crate) fn belongs_to(&self, durable: &Rc<DurableConsumerOffsets>) -> bool {
+ Rc::ptr_eq(&self.durable, durable)
+ }
+
+ /// Undo this poll's eager update after synchronous admission fails. The
Review Comment:
Nit: `rollback_local_offset` does a bare non-monotone `store(previous)`, so this no-yield contract is load bearing. Move the rollback decision inside `execute` so a future caller cannot break it.
##########
core/partitions/src/offset_storage.rs:
##########
@@ -106,9 +109,40 @@ pub fn decode_offset_record(bytes: &[u8]) -> OffsetRecord {
/// Overwrite a consumer-offset file with `offset` and a checksum over it.
///
+/// Without `enforce_fsync` the file is rewritten in place. With it, the record
+/// goes to a sibling inode, is data-synced and renamed over the prior file, so a
+/// failed write leaves the prior cursor intact. The replacement is tied to the
+/// same knob as the sync: without the sync neither the write nor the rename is
+/// ordered against a crash, so the extra inode and rename buy nothing. The
+/// caller syncs the parent directory afterwards.
Review Comment:
Nit: The caller syncs the parent directory only on the `enforce_fsync` branch; the in-place branch never marks the directory dirty. Split the sentence per branch.
##########
core/partitions/src/poll_plan.rs:
##########
@@ -861,55 +888,146 @@ impl AutoCommitCtx {
/// newer explicit store; the maps are lock-free (`papaya`), so this is
/// sound off the pump task.
#[allow(clippy::cast_possible_truncation)]
- pub(crate) fn apply(&self, offset: u64) {
- match &self.target {
+ pub(crate) fn apply(
+ self,
+ offset: u64,
+ ) -> Result<AutoCommitApplied, ConsumerOffsetCapacityError> {
+ let (kind, consumer_id) = self.kind_and_id();
+ let create = |path: Option<&str>| {
+ ConsumerOffset::new(
+ kind,
+ consumer_id,
+ offset,
+ path.map_or_else(String::new, |path| format!("{path}/{consumer_id}")),
+ )
+ };
+ let previous_offset = match &self.target {
AutoCommitTarget::Consumer {
offsets,
consumer_id,
create_path,
- } => {
- let consumer_id = *consumer_id;
- let map: &ConsumerOffsets = offsets;
- upsert_offset_max(map, consumer_id as usize, offset, || {
- create_path.as_deref().map_or_else(
- || {
- ConsumerOffset::new(
- ConsumerKind::Consumer,
- consumer_id,
- 0,
- String::new(),
- )
- },
- |path| ConsumerOffset::default_for_consumer(consumer_id, path),
- )
- });
- }
+ } => apply_local_offset(
+ offsets,
+ *consumer_id as usize,
+ offset,
+ &self.capacity,
+ self.durable.count(kind) >= self.capacity.limit(),
+ || create(create_path.as_deref()),
+ )?,
AutoCommitTarget::ConsumerGroup {
offsets,
group_id,
create_path,
- } => {
- let group_id = *group_id;
- let key = ConsumerGroupId(group_id as usize);
- let map: &ConsumerGroupOffsets = offsets;
- upsert_offset_max(map, key, offset, || {
- create_path.as_deref().map_or_else(
- || {
- ConsumerOffset::new(
- ConsumerKind::ConsumerGroup,
- group_id,
- 0,
- String::new(),
- )
- },
- |path| ConsumerOffset::default_for_consumer_group(key, path),
- )
- });
- }
+ } => apply_local_offset(
+ offsets,
+ ConsumerGroupId(*group_id as usize),
+ offset,
+ &self.capacity,
+ self.durable.count(kind) >= self.capacity.limit(),
+ || create(create_path.as_deref()),
+ )?,
+ };
+ Ok(AutoCommitApplied {
+ kind,
+ consumer_id,
+ offset,
+ previous_offset,
+ target: self.target,
+ capacity: self.capacity,
+ durable: self.durable,
+ last_polled: None,
+ })
+ }
+}
+
+impl AutoCommitApplied {
+ /// Record the group handoff frontier only after poll admission succeeds.
+ pub fn mark_served(&self) {
+ if let Some(last_polled) = &self.last_polled {
+ last_polled.record(self.offset);
+ }
+ }
+ /// Reserve a durable key before the synthetic store is submitted.
+ /// Returns `None` when committed state already covers this offset.
+ ///
+ /// # Errors
+ /// Returns a capacity error when this is a new durable key and the
+ /// partition's per-kind limit has been reached.
+ pub fn reserve_durable(
+ &self,
+ ) -> Result<Option<AutoCommitReservation>, ConsumerOffsetCapacityError> {
+ if self
+ .durable
+ .covers(self.kind, self.consumer_id, self.offset)
+ {
+ return Ok(None);
+ }
+ self.capacity
+ .reserve_provisional(self.consumer_id, &self.durable)
+ .map(Some)
+ }
+
+ pub(crate) fn belongs_to(&self, durable: &Rc<DurableConsumerOffsets>) -> bool {
+ Rc::ptr_eq(&self.durable, durable)
+ }
+
+ /// Undo this poll's eager update after synchronous admission fails. The
+ /// caller must not yield between execution and this rollback.
+ pub fn rollback_created(&self) {
+ match &self.target {
+ AutoCommitTarget::Consumer {
+ offsets,
+ consumer_id,
+ ..
+ } => rollback_local_offset(offsets, *consumer_id as usize, self.previous_offset),
+ AutoCommitTarget::ConsumerGroup {
+ offsets, group_id, ..
+ } => rollback_local_offset(
+ offsets,
+ ConsumerGroupId(*group_id as usize),
+ self.previous_offset,
+ ),
+ }
+ if self.previous_offset.is_none() {
+ self.capacity.note_local_key_change();
+ self.capacity.forget_inactive_provisional(self.consumer_id);
}
}
}
+fn rollback_local_offset<K: Hash + Eq + Send + Sync + Copy>(
+ map: &papaya::HashMap<K, ConsumerOffset>,
+ key: K,
+ previous: Option<u64>,
+) {
+ let guard = map.pin();
+ if let Some(previous) = previous {
+ if let Some(entry) = guard.get(&key) {
+ entry.offset.store(previous, Ordering::Relaxed);
+ }
+ } else {
+ guard.remove(&key);
+ }
+}
+
+fn apply_local_offset<K: Hash + Eq + Clone + Send + Sync>(
+ map: &papaya::HashMap<K, ConsumerOffset>,
+ key: K,
+ offset: u64,
+ capacity: &ConsumerOffsetCapacity,
+ durable_full: bool,
+ create: impl FnOnce() -> ConsumerOffset,
+) -> Result<Option<u64>, ConsumerOffsetCapacityError> {
+ let guard = map.pin();
+ if let Some(existing) = guard.get(&key) {
+ return Ok(Some(existing.offset.fetch_max(offset, Ordering::Relaxed)));
+ }
+ capacity.admit_local_map_key(guard.len(), durable_full)?;
Review Comment:
Nit: The `len()` read and the insert are not atomic on this lock-free map. Note that the bound holds only because polls for a partition run on its own shard thread.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -231,60 +257,96 @@ fn submit_auto_commit<B, MJ, S, SB>(
shard: &Rc<ShellShard<B, MJ, S, SB>>,
namespace: IggyNamespace,
applied: &AutoCommitApplied,
-) where
+) -> Result<(), IggyError>
+where
B: ShellBus,
MJ: JournalHandle + 'static,
MJ::Target: Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>,
S: 'static,
SB: SuperblockStore + 'static,
{
- enum AutoCommitGate {
- Submit,
- Covered,
- NotPrimary,
- }
- let gate = shard
+ let primary = shard
Review Comment:
Nit: `record_consumer_offset_denied` covers only the `reserve_durable` path, so pump-side denials for auto-commit clients are never counted. Count them, or state that the metric is best effort.
##########
core/bench/src/benchmarks/common.rs:
##########
@@ -44,7 +44,11 @@ pub async fn create_consumer(
"Consumer #{} → joining consumer group #{}",
consumer_id, consumer_group_id
);
- let cg_identifier = Identifier::try_from(*consumer_group_id).unwrap();
+ // By name: the group was created by name and the server assigns its
Review Comment:
Nit: The consumer group addressing fixes here and at line 216 are unrelated to consumer offset quotas. Split them into their own commit.
##########
core/sdk/src/clients/consumer.rs:
##########
@@ -1361,9 +1370,9 @@ impl IggyConsumer {
if is_consumer_group {
joined_consumer_group.store(false, ORDERING);
}
- trace!("Retrying to poll messages in {retry_interval}...");
- sleep(retry_interval.get_duration()).await;
}
+ trace!("Retrying to poll messages in {retry_interval}...");
Review Comment:
Nit: Every poll error now costs a full `polling_retry_interval`, one second by default, before the caller sees it. Use a shorter delay for errors the caller handles itself.
##########
core/server/src/boot/recovery.rs:
##########
@@ -331,6 +332,14 @@ const _: () = assert!(
const _: () = assert!(
configs::partition::PARTITION_DEDUP_CLIENTS_DEFAULT == consensus::PARTITION_DEDUP_CLIENTS_MAX
);
+const _: () = assert!(
+ configs::partition::PARTITION_CONSUMER_OFFSETS_DEFAULT
+ == partitions::DEFAULT_CONSUMER_OFFSETS_MAX
+);
+const _: () = assert!(
+ 4 * configs::partition::PARTITION_CONSUMER_OFFSETS_CEILING
Review Comment:
Nit: `4 * PARTITION_CONSUMER_OFFSETS_CEILING` equals `CONSUMER_OFFSETS_ENTRIES_MAX` exactly, so no headroom is left. Say so in the message, or lower the ceiling.
##########
core/integration/tests/server/consumer_offset_quota_vsr.rs:
##########
@@ -0,0 +1,490 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use iggy::prelude::*;
+use iggy_binary_protocol::codec::WireEncode;
+use iggy_binary_protocol::consensus::Operation;
+use iggy_binary_protocol::requests::consumer_offsets::StoreConsumerOffsetRequest;
+use iggy_binary_protocol::{AckLevel, WireConsumer, WireIdentifier};
+use iggy_common::store_consumer_offset::StoreConsumerOffset;
+use integration::harness::TestBinary;
+use integration::iggy_harness;
+use reqwest::StatusCode;
+use std::collections::BTreeMap;
+use std::fs;
+
+use super::http_client::HttpClient;
+use super::raw_tcp;
+
+const STREAM_NAME: &str = "consumer-offset-quota-stream";
+const TOPIC_NAME: &str = "consumer-offset-quota-topic";
+const PARTITION_ID: u32 = 0;
+const LIMIT: u32 = 4;
+
+#[iggy_harness(
+ cluster_nodes = 1,
+ server(partition.consumer_offsets_max = "4")
+)]
+async fn given_full_consumer_offset_table_when_creating_another_should_reject_without_new_file(
+ harness: &TestHarness,
+) {
+ let client = harness.tcp_root_client().await.expect("TCP root client");
+ let stream = Identifier::named(STREAM_NAME).expect("stream identifier");
+ let topic = Identifier::named(TOPIC_NAME).expect("topic identifier");
+ let stream_details = client
+ .create_stream(STREAM_NAME)
+ .await
+ .expect("create stream");
+ let topic_details = client
+ .create_topic(
+ &stream,
+ TOPIC_NAME,
+ &TopicCreateOptions {
+ partitions_count: Some(1),
+ message_expiry: Some(IggyExpiry::NeverExpire),
+ ..TopicCreateOptions::default()
+ },
+ )
+ .await
+ .expect("create topic");
+ let mut messages = vec![
+ IggyMessage::builder()
+ .payload("offset-quota".into())
+ .build()
+ .expect("build message"),
+ ];
+ client
+ .send_messages(
+ &stream,
+ &topic,
+ &Partitioning::partition_id(PARTITION_ID),
+ &mut messages,
+ )
+ .await
+ .expect("seed non-empty partition");
+
+ client
+ .create_user(
+ "offset-poll-only",
+ "password123",
+ UserStatus::Active,
+ Some(Permissions {
+ global: GlobalPermissions::default(),
+ streams: Some(BTreeMap::from([(
+ stream_details.id as usize,
+ StreamPermissions {
+ topics: Some(BTreeMap::from([(
+ topic_details.id as usize,
+ TopicPermissions {
+ poll_messages: true,
+ ..Default::default()
+ },
+ )])),
+ ..Default::default()
+ },
+ )])),
+ }),
+ )
+ .await
+ .expect("create a topic-scoped consumer");
+ let client = harness.tcp_new_client().await.expect("consumer TCP client");
+ client
+ .login_user("offset-poll-only", "password123")
+ .await
+ .expect("consumer login");
+
+ let first_consumer = Consumer::new(Identifier::numeric(1).unwrap());
+ let polled = client
+ .poll_messages(
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ &first_consumer,
+ &PollingStrategy::first(),
+ 1,
+ true,
+ )
+ .await
+ .expect("new auto-commit consumer fits");
+ assert_eq!(polled.messages.len(), 1);
+ let first_file = harness.server().data_path().join(format!(
+ "streams/{}/topics/{}/partitions/{PARTITION_ID}/offsets/consumers/1",
+ stream_details.id, topic_details.id
+ ));
+ let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(10);
+ while !first_file.is_file() {
+ assert!(
+ tokio::time::Instant::now() < deadline,
+ "auto-commit never reached its file"
+ );
+ tokio::time::sleep(std::time::Duration::from_millis(10)).await;
+ }
+ assert!(
+ client
+ .poll_messages(
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ &first_consumer,
+ &PollingStrategy::next(),
+ 1,
+ true
+ )
+ .await
+ .expect("next poll")
+ .messages
+ .is_empty()
+ );
+
+ for consumer_id in 1..=LIMIT {
+ client
+ .store_consumer_offset(
+ &Consumer::new(Identifier::numeric(consumer_id).expect("consumer identifier")),
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ 0,
+ )
+ .await
+ .expect("store offset within limit");
+ }
+
+ let rejected_consumer =
+ Consumer::new(Identifier::numeric(LIMIT + 1).expect("consumer identifier"));
+ let rejected = client
+ .store_consumer_offset(&rejected_consumer, &stream, &topic, Some(PARTITION_ID), 0)
+ .await;
+ assert!(
+ matches!(rejected, Err(IggyError::TooManyConsumerOffsets)),
+ "the first key above the limit must receive the typed capacity error"
+ );
+
+ client
+ .store_consumer_offset(
+ &Consumer::new(Identifier::numeric(1).expect("consumer identifier")),
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ 0,
+ )
+ .await
+ .expect("existing key remains writable at the limit");
+
+ let poll_rejected = client
+ .poll_messages(
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ &rejected_consumer,
+ &PollingStrategy::first(),
+ 1,
+ true,
+ )
+ .await;
+ assert!(
+ matches!(poll_rejected, Err(IggyError::TooManyConsumerOffsets)),
+ "auto-commit must not return data when its new key cannot be admitted"
+ );
+ client
+ .poll_messages(
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ &rejected_consumer,
+ &PollingStrategy::first(),
+ 1,
+ false,
+ )
+ .await
+ .expect("the same poll succeeds when auto-commit is disabled");
+
+ client
+ .delete_consumer_offset(
+ &Consumer::new(Identifier::numeric(1).expect("consumer identifier")),
+ &stream,
+ &topic,
+ Some(PARTITION_ID),
+ )
+ .await
+ .expect("delete one accepted offset");
+ client
+ .store_consumer_offset(&rejected_consumer, &stream, &topic, Some(PARTITION_ID), 0)
+ .await
+ .expect("delete releases one durable slot");
+
+ let mut raw = raw_tcp::connect(harness).await;
+ let raw_client_id = 0xC0FF_EE03;
+ let session = raw_tcp::register_root(&mut raw, raw_client_id).await;
+ let unresolved_group = StoreConsumerOffsetRequest {
+ consumer: WireConsumer::consumer_group(WireIdentifier::Numeric(999)),
+ stream_id: WireIdentifier::Numeric(stream_details.id),
+ topic_id: WireIdentifier::Numeric(topic_details.id),
+ partition_id: Some(PARTITION_ID),
+ offset: 0,
+ ack: AckLevel::Quorum,
+ }
+ .to_bytes();
+ let header = raw_tcp::request_header(
+ Operation::StoreConsumerOffset,
+ raw_client_id,
+ session,
+ 1,
+ unresolved_group.len(),
+ );
+ let (reply, _) = raw_tcp::exchange(&mut raw, &header, &unresolved_group).await;
+ assert_eq!(
+ raw_tcp::reply_status(&reply),
+ IggyError::ConsumerGroupIdNotFound(Identifier::numeric(999).unwrap(), topic.clone())
+ .as_code()
+ );
+
+ let file_count = integration::harness::disk::consumer_offset_file_ids(
+ &harness.server().data_path(),
+ stream_details.id,
+ topic_details.id,
+ PARTITION_ID,
+ ConsumerKind::Consumer,
+ )
+ .expect("consumer offsets directory")
+ .len();
+ assert_eq!(file_count, LIMIT as usize);
+ let group_file_count = integration::harness::disk::consumer_offset_file_ids(
+ &harness.server().data_path(),
+ stream_details.id,
+ topic_details.id,
+ PARTITION_ID,
+ ConsumerKind::ConsumerGroup,
+ )
+ .map_or(0, |ids| ids.len());
+ assert_eq!(group_file_count, 0);
+
+ let named_group = StoreConsumerOffsetRequest {
+ consumer: WireConsumer::consumer_group(WireIdentifier::named("unknown-group").unwrap()),
+ stream_id: WireIdentifier::Numeric(stream_details.id),
+ topic_id: WireIdentifier::Numeric(topic_details.id),
+ partition_id: Some(PARTITION_ID),
+ offset: 0,
+ ack: AckLevel::Quorum,
+ }
+ .to_bytes();
+ let header = raw_tcp::request_header(
+ Operation::StoreConsumerOffset,
+ raw_client_id,
+ session,
+ 2,
+ named_group.len(),
+ );
+ let (reply, _) = raw_tcp::exchange(&mut raw, &header, &named_group).await;
+ assert_eq!(
+ raw_tcp::reply_status(&reply),
+ IggyError::ConsumerGroupNameNotFound("unknown-group".to_owned(), topic.clone()).as_code()
+ );
+
+ let unknown_stream = StoreConsumerOffsetRequest {
+ consumer: WireConsumer::consumer_group(WireIdentifier::Numeric(999)),
+ stream_id: WireIdentifier::Numeric(999_999),
+ topic_id: WireIdentifier::Numeric(topic_details.id),
+ partition_id: Some(PARTITION_ID),
+ offset: 0,
+ ack: AckLevel::Quorum,
+ }
+ .to_bytes();
+ let header = raw_tcp::request_header(
+ Operation::StoreConsumerOffset,
+ raw_client_id,
+ session,
+ 3,
+ unknown_stream.len(),
+ );
+ let (reply, _) = raw_tcp::exchange(&mut raw, &header, &unknown_stream).await;
+ assert_eq!(
+ raw_tcp::reply_status(&reply),
+ IggyError::ResourceNotFound(String::new()).as_code(),
+ "a missing stream is not reported as a missing group"
+ );
+
+ let http = HttpClient::login_root(harness).await;
+ let response = http
+ .client
+ .put(http.url(&format!(
+ "/streams/{STREAM_NAME}/topics/{TOPIC_NAME}/consumer-offsets"
+ )))
+ .bearer_auth(&http.token)
+ .json(&StoreConsumerOffset {
+ consumer: Consumer::new(Identifier::numeric(6).expect("consumer identifier")),
+ partition_id: Some(PARTITION_ID),
+ offset: 0,
+ })
+ .send()
+ .await
+ .expect("HTTP capacity request");
+ assert_eq!(response.status(), StatusCode::BAD_REQUEST);
+ let body: serde_json::Value = response.json().await.expect("HTTP error body");
+ assert_eq!(body["id"], 3024);
+ assert_eq!(body["code"], "too_many_consumer_offsets");
+
+ let metrics = http
+ .client
+ .get(http.url("/metrics"))
+ .bearer_auth(&http.token)
+ .send()
+ .await
+ .expect("metrics response")
+ .text()
+ .await
+ .expect("metrics text");
+ let denied: u64 = metrics
Review Comment:
Nit: This sums every denied-total series and expects 3, so a denial added anywhere else in the test breaks it confusingly. Assert the consumer-kind series instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org