You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by hg...@apache.org on 2026/07/27 09:20:33 UTC
(iggy) branch feat/iggy-server-hardening updated (9ce4dd0fa -> b2cccceec)
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a change to branch feat/iggy-server-hardening
in repository https://gitbox.apache.org/repos/asf/iggy.git
discard 9ce4dd0fa fix(server-ng): source TLS grace from config, align listener backlog
discard 931bde2aa feat(server-ng): make VSR repair pacing and eviction ring configurable
discard 29be858a7 feat(server-ng): make VSR clients table capacity configurable
discard 646e3bc0a fix(server-ng): scale recovery barrier deadline with status timeout
discard 90d66a13c feat(server-ng): make VSR view-change timing configurable
discard 4cbba2a0b fix(server-ng): forward iggy/vsr so the bare vsr feature builds
discard d8d19fef4 feat(server-ng): add partition.prepare_queue_depth for pipeline sizing
discard c94172fa5 chore(repo): retry flaky tests twice in just nextest recipes
discard c93fa9c64 fix(server-ng): fail reads loud when restart recovery outruns its deadline
discard b3ea6f84c feat(server-ng): make HTTP write admission caps configurable
discard ca21c2bc5 feat(server-ng): add commit broadcast and prepare retransmit intervals
discard 12df22e92 fix(consensus): bound pipeline asserts by configured depth, not const
add 64ae1e9cb feat(connectors): use TCP client for Flink sink (#3657)
add d9635b6d7 feat(server-ng): forward follower HTTP requests to VSR primary (#3744)
add c5fec1ff6 test(integration): spec clients-table durability across node restart (#3736)
new b2cccceec feat(server-ng): make VSR timing, capacity, and admission configurable
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (9ce4dd0fa)
\
N -- N -- N refs/heads/feat/iggy-server-hardening (b2cccceec)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
Cargo.lock | 1 +
Cargo.toml | 2 +-
bdd/docker-compose.vsr.yml | 7 +
core/configs/src/server_ng_config/cluster.rs | 19 +
core/configs/src/server_ng_config/validators.rs | 95 ++++
.../tests/cluster/client_table_restart.rs | 382 +++++++++++++
core/integration/tests/cluster/mod.rs | 22 +-
core/integration/tests/mod.rs | 5 +-
core/integration/tests/server/http_vsr.rs | 4 +-
core/server-ng/Cargo.toml | 1 +
core/server-ng/config.toml | 17 +-
core/server-ng/src/http.rs | 104 +++-
core/server-ng/src/http/error.rs | 131 ++++-
core/server-ng/src/http/extractor.rs | 20 +-
core/server-ng/src/http/forward.rs | 632 +++++++++++++++++++++
core/server-ng/src/http/handlers.rs | 6 +-
core/server-ng/src/http/jwt.rs | 47 +-
core/server-ng/src/http/reply.rs | 51 +-
core/server-ng/src/http/state.rs | 36 +-
core/server-ng/src/http/submit.rs | 43 +-
core/server-ng/src/server_error.rs | 2 +
foreign/java/.gitignore | 1 +
.../iggy-connector-library/README.md | 33 ++
.../apache/iggy/connector/config/TcpEndpoint.java | 92 +++
.../apache/iggy/connector/flink/sink/IggySink.java | 91 ++-
.../iggy/connector/flink/sink/IggySinkWriter.java | 39 +-
.../iggy/connector/flink/source/IggySource.java | 24 +-
.../iggy/connector/config/TcpEndpointTest.java | 130 +++++
28 files changed, 1818 insertions(+), 219 deletions(-)
create mode 100644 core/integration/tests/cluster/client_table_restart.rs
create mode 100644 core/server-ng/src/http/forward.rs
create mode 100644 foreign/java/external-processors/iggy-connector-flink/iggy-connector-library/src/main/java/org/apache/iggy/connector/config/TcpEndpoint.java
create mode 100644 foreign/java/external-processors/iggy-connector-flink/iggy-connector-library/src/test/java/org/apache/iggy/connector/config/TcpEndpointTest.java
(iggy) 01/01: feat(server-ng): make VSR timing, capacity, and admission configurable
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch feat/iggy-server-hardening
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit b2cccceec059ef43f0b936bdd7785293fad65323
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Mon Jul 27 11:04:39 2026 +0200
feat(server-ng): make VSR timing, capacity, and admission configurable
The consensus and HTTP planes ran on compile-time constants:
view-change and retransmit cadence, commit broadcast interval,
pipeline prepare depth, clients table capacity, repair pacing,
eviction ring caps, and HTTP write admission had no operator
knobs, so tuning any of them for a deployment meant a rebuild.
Surface them following the cluster.heartbeat_timeout pattern:
knobs live in server_ng_config sections ([cluster], [metadata],
[partition], [http_admission]) with defaults in the embedded
config.toml, converted to consensus ticks and applied at
bootstrap, validated at boot. Defaults stay byte-identical to
the old constants, pinned by lockstep tests and static asserts;
overrides apply through boot-time setters so the simulator and
tests keep compile-time defaults and determinism is unaffected.
The HTTP session cap now derives as half the configured clients
table at runtime, preserving the non-HTTP headroom invariant.
Beyond pure knob-adds: restart recovery now fails reads loud
with a retryable 503 instead of serving rolled-back state when
the barrier outlives its deadline, and that deadline scales as
max(15s, 3x heartbeat_timeout, 3x view_change_status_timeout)
instead of a fixed 15s; validation newly rejects
repair_chunk_max >= peer_queue_capacity (a config that silently
wedges repair) and floors peer_queue_capacity for the first
time, so some previously bootable configs now fail boot; the
glibc client listener backlog aligns with the replica listener
at SOMAXCONN (was 128); the HTTP TLS handshake budget follows
message_bus.handshake_grace instead of a duplicated local
const; the dead VSR ping mechanism is deleted (liveness rides
the commit broadcast).
---
core/configs/src/lib.rs | 5 +-
core/configs/src/server_ng_config/cluster.rs | 470 ++++++++++++++++++++-
core/configs/src/server_ng_config/defaults.rs | 64 +++
core/configs/src/server_ng_config/displays.rs | 4 +-
.../configs/src/server_ng_config/http_admission.rs | 184 ++++++++
core/configs/src/server_ng_config/metadata.rs | 89 +++-
core/configs/src/server_ng_config/mod.rs | 2 +
core/configs/src/server_ng_config/partition.rs | 212 ++++++++++
core/configs/src/server_ng_config/server_ng.rs | 4 +
core/configs/src/server_ng_config/validators.rs | 45 ++
core/consensus/src/client_table.rs | 54 +++
core/consensus/src/impls.rs | 163 ++++++-
core/consensus/src/lib.rs | 5 +
core/consensus/src/plane_helpers.rs | 74 +++-
core/consensus/src/vsr_timeout.rs | 83 +++-
.../scenarios/reconnect_after_restart_scenario.rs | 9 +
core/message_bus/src/cache/connection.rs | 2 -
core/message_bus/src/client_listener/mod.rs | 2 +-
core/metadata/src/impls/metadata.rs | 8 +
core/partitions/src/journal.rs | 25 +-
core/partitions/src/lib.rs | 1 +
core/server-ng/Cargo.toml | 5 +-
core/server-ng/config.toml | 96 ++++-
core/server-ng/src/bootstrap.rs | 440 ++++++++++++++++++-
core/server-ng/src/http.rs | 15 +-
core/server-ng/src/http/admission.rs | 92 ++--
core/server-ng/src/http/error.rs | 58 ++-
core/server-ng/src/http/handlers.rs | 15 +-
core/server-ng/src/http/reads.rs | 114 ++++-
core/server-ng/src/http/session.rs | 67 ++-
core/server-ng/src/http/state.rs | 23 +-
core/server-ng/src/http/submit.rs | 14 +-
core/server-ng/src/http/tls.rs | 29 +-
core/server-ng/src/partition_helpers.rs | 20 +-
core/shard/src/lib.rs | 59 ++-
justfile | 4 +-
36 files changed, 2351 insertions(+), 205 deletions(-)
diff --git a/core/configs/src/lib.rs b/core/configs/src/lib.rs
index 5c82dbd3a..8af9f174d 100644
--- a/core/configs/src/lib.rs
+++ b/core/configs/src/lib.rs
@@ -30,6 +30,7 @@ pub use server_config::{
tcp, validators, websocket,
};
pub use server_ng_config::{
- COMPONENT_NG, cluster as ng_cluster, message_bus, metadata as ng_metadata, quic as ng_quic,
- server_ng, sharding as ng_sharding, tcp as ng_tcp, websocket as ng_websocket,
+ COMPONENT_NG, cluster as ng_cluster, http_admission as ng_http_admission, message_bus,
+ metadata as ng_metadata, partition as ng_partition, quic as ng_quic, server_ng,
+ sharding as ng_sharding, tcp as ng_tcp, websocket as ng_websocket,
};
diff --git a/core/configs/src/server_ng_config/cluster.rs b/core/configs/src/server_ng_config/cluster.rs
index 44b565f58..f619cca23 100644
--- a/core/configs/src/server_ng_config/cluster.rs
+++ b/core/configs/src/server_ng_config/cluster.rs
@@ -31,10 +31,48 @@ use serde::{Deserialize, Serialize};
use serde_with::{DisplayFromStr, serde_as};
use std::time::Duration;
-/// The primary heartbeats roughly every second (`PING_TICKS`); a window at
-/// or below one ping interval would elect on every scheduling hiccup.
+/// Absolute floor for the backup liveness window, independent of the
+/// commit-broadcast rate. The primary signals liveness through its commit
+/// broadcast (`commit_broadcast_interval`, 500ms by default); 2s spans several
+/// broadcasts, so a single delayed one never elects. The per-config
+/// `MIN_HEARTBEAT_TO_COMMIT_BROADCAST_RATIO` check scales the same headroom
+/// when the broadcast interval is retuned.
pub const MIN_CLUSTER_HEARTBEAT_TIMEOUT: Duration = Duration::from_secs(2);
+/// The backup liveness window (`heartbeat_timeout`) must span at least this
+/// many commit broadcasts (`commit_broadcast_interval`), so one dropped or
+/// delayed broadcast never trips a view change on a healthy primary.
+const MIN_HEARTBEAT_TO_COMMIT_BROADCAST_RATIO: u32 = 4;
+
+/// The view-change status backstop (`view_change_status_timeout`) must span at
+/// least this many retransmit intervals (`view_change_retransmit_interval`), so
+/// a few dropped `StartViewChange` / `DoViewChange` messages retransmit rather
+/// than escalating a progressing view change into a fresh cluster-wide election.
+const MIN_STATUS_TO_RETRANSMIT_RATIO: u32 = 4;
+
+/// Default recovering-replica probe-attempt ceiling. Duplicated here rather
+/// than imported so `core/configs` keeps off a build-time edge onto
+/// `core/consensus` (mirroring [`super::partition`]); `core/server-ng`'s
+/// bootstrap static-asserts it equal to `consensus::PROBE_ATTEMPTS_MAX`.
+pub const DEFAULT_VIEW_PROBE_ATTEMPTS_MAX: u32 = 5;
+
+/// Upper bound on `view_probe_attempts_max`. A recovering replica probes once
+/// per `request_start_view_retransmit_interval`, so hundreds of attempts would
+/// stall the election fallback for minutes on a full-cluster restart; this is a
+/// typo guard, not a sizing endorsement.
+const MAX_VIEW_PROBE_ATTEMPTS: u32 = 100;
+
+/// Default per-round repair-serving chunk. Duplicated here rather than imported
+/// so `core/configs` keeps off a build-time edge onto `core/shard` (mirroring
+/// [`DEFAULT_VIEW_PROBE_ATTEMPTS_MAX`]); `core/server-ng`'s bootstrap
+/// static-asserts it equal to `shard::REPAIR_CHUNK_MAX`.
+pub const DEFAULT_REPAIR_CHUNK_MAX: usize = 128;
+
+/// Upper bound on `repair_chunk_max`. A chunk rides the per-peer bus queue, so
+/// the load-bearing rule is `repair_chunk_max < message_bus.peer_queue_capacity`
+/// (enforced at the top level); this standalone ceiling is a typo guard.
+const MAX_REPAIR_CHUNK_MAX: usize = 1024;
+
/// Length floor for the replica-auth PSK, in raw bytes. The 32-byte MAC key
/// is KDF-derived from these bytes at use-site, so any encoding clearing this
/// length is accepted.
@@ -46,6 +84,78 @@ fn default_heartbeat_timeout() -> IggyDuration {
SERVER_NG_CONFIG.cluster.heartbeat_timeout.parse().unwrap()
}
+/// serde fallback for configs written before the field existed; the value
+/// itself lives in `core/server-ng/config.toml` like every other default.
+fn default_commit_broadcast_interval() -> IggyDuration {
+ SERVER_NG_CONFIG
+ .cluster
+ .commit_broadcast_interval
+ .parse()
+ .unwrap()
+}
+
+/// serde fallback for configs written before the field existed; the value
+/// itself lives in `core/server-ng/config.toml` like every other default.
+fn default_prepare_retransmit_interval() -> IggyDuration {
+ SERVER_NG_CONFIG
+ .cluster
+ .prepare_retransmit_interval
+ .parse()
+ .unwrap()
+}
+
+/// serde fallback for configs written before the field existed; the value
+/// itself lives in `core/server-ng/config.toml` like every other default.
+fn default_view_change_retransmit_interval() -> IggyDuration {
+ SERVER_NG_CONFIG
+ .cluster
+ .view_change_retransmit_interval
+ .parse()
+ .unwrap()
+}
+
+/// serde fallback for configs written before the field existed; the value
+/// itself lives in `core/server-ng/config.toml` like every other default.
+fn default_view_change_status_timeout() -> IggyDuration {
+ SERVER_NG_CONFIG
+ .cluster
+ .view_change_status_timeout
+ .parse()
+ .unwrap()
+}
+
+/// serde fallback for configs written before the field existed; the value
+/// itself lives in `core/server-ng/config.toml` like every other default.
+fn default_request_start_view_retransmit_interval() -> IggyDuration {
+ SERVER_NG_CONFIG
+ .cluster
+ .request_start_view_retransmit_interval
+ .parse()
+ .unwrap()
+}
+
+/// serde fallback for configs written before the field existed; the value
+/// itself lives in `core/server-ng/config.toml` like every other default.
+fn default_view_probe_attempts_max() -> u32 {
+ SERVER_NG_CONFIG.cluster.view_probe_attempts_max as u32
+}
+
+/// serde fallback for configs written before the field existed; the value
+/// itself lives in `core/server-ng/config.toml` like every other default.
+fn default_repair_retry_interval() -> IggyDuration {
+ SERVER_NG_CONFIG
+ .cluster
+ .repair_retry_interval
+ .parse()
+ .unwrap()
+}
+
+/// serde fallback for configs written before the field existed; the value
+/// itself lives in `core/server-ng/config.toml` like every other default.
+fn default_repair_chunk_max() -> usize {
+ SERVER_NG_CONFIG.cluster.repair_chunk_max as usize
+}
+
#[serde_as]
#[derive(Debug, Deserialize, Serialize, Clone, ConfigEnv)]
#[serde(deny_unknown_fields)]
@@ -62,6 +172,80 @@ pub struct ClusterConfig {
#[serde_as(as = "DisplayFromStr")]
#[config_env(leaf)]
pub heartbeat_timeout: IggyDuration,
+ /// How often the primary broadcasts its commit point to every backup, the
+ /// cluster's primary-liveness signal. Each broadcast resets the backups'
+ /// `heartbeat_timeout` window, so that window must span several broadcasts:
+ /// boot rejects `heartbeat_timeout < MIN_HEARTBEAT_TO_COMMIT_BROADCAST_RATIO
+ /// * commit_broadcast_interval`. Sizes the consensus `CommitMessage` timer.
+ /// Zero (and the `0` / `disabled` / `unlimited` sentinels, which all parse
+ /// to zero) is rejected at boot.
+ #[serde(default = "default_commit_broadcast_interval")]
+ #[serde_as(as = "DisplayFromStr")]
+ #[config_env(leaf)]
+ pub commit_broadcast_interval: IggyDuration,
+ /// How often the primary retransmits prepares a backup has not yet acked.
+ /// Lower recovers faster from a dropped prepare at the cost of replica
+ /// traffic. Sizes the consensus `Prepare` timer. Zero (and the `0` /
+ /// `disabled` / `unlimited` sentinels, which all parse to zero) is rejected
+ /// at boot.
+ #[serde(default = "default_prepare_retransmit_interval")]
+ #[serde_as(as = "DisplayFromStr")]
+ #[config_env(leaf)]
+ pub prepare_retransmit_interval: IggyDuration,
+ /// How often a plane retransmits its `StartViewChange` / `DoViewChange`
+ /// while a view change is running. Lower converges a healthy election
+ /// faster at the cost of replica traffic. Sizes both consensus view-change
+ /// retransmit timers, which are deliberately equal. Zero (and the `0` /
+ /// `disabled` / `unlimited` sentinels, which all parse to zero) is rejected
+ /// at boot.
+ #[serde(default = "default_view_change_retransmit_interval")]
+ #[serde_as(as = "DisplayFromStr")]
+ #[config_env(leaf)]
+ pub view_change_retransmit_interval: IggyDuration,
+ /// Backstop for a stalled view change: one that does not conclude within
+ /// this window escalates to a fresh cluster-wide election. Must span
+ /// several `view_change_retransmit_interval`s so a few dropped view-change
+ /// messages retransmit rather than escalate: boot rejects
+ /// `view_change_status_timeout < MIN_STATUS_TO_RETRANSMIT_RATIO *
+ /// view_change_retransmit_interval`. Zero (and the `0` / `disabled` /
+ /// `unlimited` sentinels, which all parse to zero) is rejected at boot.
+ #[serde(default = "default_view_change_status_timeout")]
+ #[serde_as(as = "DisplayFromStr")]
+ #[config_env(leaf)]
+ pub view_change_status_timeout: IggyDuration,
+ /// How often a recovering or view-change backup re-requests the current
+ /// view's `StartView` from its primary (`RequestStartView`). Sizes the
+ /// consensus `RequestStartView` timer. Zero (and the `0` / `disabled` /
+ /// `unlimited` sentinels, which all parse to zero) is rejected at boot.
+ #[serde(default = "default_request_start_view_retransmit_interval")]
+ #[serde_as(as = "DisplayFromStr")]
+ #[config_env(leaf)]
+ pub request_start_view_retransmit_interval: IggyDuration,
+ /// How many consecutive unanswered `RequestStartView` probes a recovering
+ /// replica tolerates before it falls back to an election (a full-cluster
+ /// restart leaves nobody settled to answer). Must be >= 1 and <=
+ /// `MAX_VIEW_PROBE_ATTEMPTS`.
+ #[serde(default = "default_view_probe_attempts_max")]
+ pub view_probe_attempts_max: u32,
+ /// How long a stalled journal-repair stream waits before re-requesting its
+ /// remaining window from the serving peer. Repair frames are
+ /// fire-and-forget over the lossy bus, so a session with no retry wedges
+ /// forever on a single dropped frame. Paces both the metadata and
+ /// partition repair loops. Sizes the retry threshold in consensus ticks.
+ /// Zero (and the `0` / `disabled` / `unlimited` sentinels, which all parse
+ /// to zero) is rejected at boot.
+ #[serde(default = "default_repair_retry_interval")]
+ #[serde_as(as = "DisplayFromStr")]
+ #[config_env(leaf)]
+ pub repair_retry_interval: IggyDuration,
+ /// Prepares a peer serves per repair round before the requester walks to
+ /// the next chunk. Each frame rides the per-peer message-bus queue, so this
+ /// must stay below `message_bus.peer_queue_capacity` or a full round
+ /// overruns the queue and silently drops frames (enforced at the top
+ /// level). Applies to both the metadata and partition repair planes. Must
+ /// be > 0 and <= `MAX_REPAIR_CHUNK_MAX`.
+ #[serde(default = "default_repair_chunk_max")]
+ pub repair_chunk_max: usize,
/// Full roster of cluster members. Intended to be byte-identical across
/// every node so operators ship one config. The running node's identity
/// is supplied out-of-band via the `--replica-id` CLI flag, which
@@ -199,13 +383,145 @@ impl Validatable<ConfigurationError> for ClusterConfig {
if self.heartbeat_timeout.get_duration() < MIN_CLUSTER_HEARTBEAT_TIMEOUT {
eprintln!(
"Invalid cluster configuration: cluster.heartbeat_timeout '{}' must be at least {}s \
- (the primary heartbeats every second; a shorter window elects on every hiccup)",
+ (the primary signals liveness through its commit broadcast; a shorter window \
+ elects on every scheduling hiccup)",
self.heartbeat_timeout,
MIN_CLUSTER_HEARTBEAT_TIMEOUT.as_secs()
);
return Err(ConfigurationError::InvalidConfigurationValue);
}
+ // The commit broadcast is the cluster's liveness feed and the prepare
+ // retransmit its recovery timer; both size consensus timers that have
+ // to advance. `0` / `disabled` / `unlimited` all collapse to a zero
+ // duration, which would stall the timer - reject them.
+ if self.commit_broadcast_interval.get_duration().is_zero() {
+ eprintln!(
+ "Invalid cluster configuration: cluster.commit_broadcast_interval must be nonzero \
+ (it drives the primary's liveness broadcast)"
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.prepare_retransmit_interval.get_duration().is_zero() {
+ eprintln!(
+ "Invalid cluster configuration: cluster.prepare_retransmit_interval must be \
+ nonzero (it drives prepare retransmission)"
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+
+ // The liveness window must span several commit broadcasts so a single
+ // delayed broadcast never trips a view change on a healthy primary.
+ let min_heartbeat = self
+ .commit_broadcast_interval
+ .get_duration()
+ .saturating_mul(MIN_HEARTBEAT_TO_COMMIT_BROADCAST_RATIO);
+ if self.heartbeat_timeout.get_duration() < min_heartbeat {
+ eprintln!(
+ "Invalid cluster configuration: cluster.heartbeat_timeout '{}' must be at least \
+ {}x cluster.commit_broadcast_interval '{}' so the liveness window spans several \
+ broadcasts",
+ self.heartbeat_timeout,
+ MIN_HEARTBEAT_TO_COMMIT_BROADCAST_RATIO,
+ self.commit_broadcast_interval
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+
+ // The three view-change timers each size a consensus timer that has to
+ // advance; `0` / `disabled` / `unlimited` all collapse to zero and
+ // would stall it - reject them.
+ if self
+ .view_change_retransmit_interval
+ .get_duration()
+ .is_zero()
+ {
+ eprintln!(
+ "Invalid cluster configuration: cluster.view_change_retransmit_interval must be \
+ nonzero (it drives StartViewChange / DoViewChange retransmission)"
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.view_change_status_timeout.get_duration().is_zero() {
+ eprintln!(
+ "Invalid cluster configuration: cluster.view_change_status_timeout must be nonzero \
+ (it backstops a stalled view change)"
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self
+ .request_start_view_retransmit_interval
+ .get_duration()
+ .is_zero()
+ {
+ eprintln!(
+ "Invalid cluster configuration: cluster.request_start_view_retransmit_interval \
+ must be nonzero (it drives RequestStartView retransmission)"
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+
+ // The status backstop must span several retransmits so a few dropped
+ // view-change messages retransmit rather than escalating a progressing
+ // view change into a fresh cluster-wide election.
+ let min_status = self
+ .view_change_retransmit_interval
+ .get_duration()
+ .saturating_mul(MIN_STATUS_TO_RETRANSMIT_RATIO);
+ if self.view_change_status_timeout.get_duration() < min_status {
+ eprintln!(
+ "Invalid cluster configuration: cluster.view_change_status_timeout '{}' must be at \
+ least {}x cluster.view_change_retransmit_interval '{}' so a stalled view change \
+ retransmits before it escalates to an election",
+ self.view_change_status_timeout,
+ MIN_STATUS_TO_RETRANSMIT_RATIO,
+ self.view_change_retransmit_interval
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+
+ // A recovering replica needs at least one probe before it may give up
+ // and elect; the ceiling is a typo guard (see MAX_VIEW_PROBE_ATTEMPTS).
+ if self.view_probe_attempts_max == 0 {
+ eprintln!(
+ "Invalid cluster configuration: cluster.view_probe_attempts_max must be >= 1"
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.view_probe_attempts_max > MAX_VIEW_PROBE_ATTEMPTS {
+ eprintln!(
+ "Invalid cluster configuration: cluster.view_probe_attempts_max ({}) exceeds the \
+ maximum ({MAX_VIEW_PROBE_ATTEMPTS})",
+ self.view_probe_attempts_max
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+
+ // The repair retry interval sizes a tick threshold that has to advance;
+ // `0` / `disabled` / `unlimited` all collapse to zero and would wedge
+ // every stalled repair stream - reject them. The chunk is a per-round
+ // count with a standalone typo ceiling; the load-bearing rule against
+ // message_bus.peer_queue_capacity is enforced at the top level.
+ if self.repair_retry_interval.get_duration().is_zero() {
+ eprintln!(
+ "Invalid cluster configuration: cluster.repair_retry_interval must be nonzero \
+ (it paces stalled-repair retries)"
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.repair_chunk_max == 0 {
+ eprintln!("Invalid cluster configuration: cluster.repair_chunk_max must be > 0");
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.repair_chunk_max > MAX_REPAIR_CHUNK_MAX {
+ eprintln!(
+ "Invalid cluster configuration: cluster.repair_chunk_max ({}) exceeds the maximum \
+ ({MAX_REPAIR_CHUNK_MAX})",
+ self.repair_chunk_max
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+
if self.nodes.is_empty() {
eprintln!(
"Invalid cluster configuration: cluster.nodes must contain at least one entry when cluster is enabled"
@@ -363,6 +679,15 @@ mod tests {
enabled: true,
name: "iggy-cluster".to_owned(),
heartbeat_timeout: default_heartbeat_timeout(),
+ commit_broadcast_interval: default_commit_broadcast_interval(),
+ prepare_retransmit_interval: default_prepare_retransmit_interval(),
+ view_change_retransmit_interval: default_view_change_retransmit_interval(),
+ view_change_status_timeout: default_view_change_status_timeout(),
+ request_start_view_retransmit_interval: default_request_start_view_retransmit_interval(
+ ),
+ view_probe_attempts_max: default_view_probe_attempts_max(),
+ repair_retry_interval: default_repair_retry_interval(),
+ repair_chunk_max: default_repair_chunk_max(),
nodes: Vec::new(),
auth: ClusterAuthConfig {
enabled: true,
@@ -400,6 +725,15 @@ mod cluster_validate_tests {
enabled: true,
name: "iggy-cluster".to_string(),
heartbeat_timeout: default_heartbeat_timeout(),
+ commit_broadcast_interval: default_commit_broadcast_interval(),
+ prepare_retransmit_interval: default_prepare_retransmit_interval(),
+ view_change_retransmit_interval: default_view_change_retransmit_interval(),
+ view_change_status_timeout: default_view_change_status_timeout(),
+ request_start_view_retransmit_interval: default_request_start_view_retransmit_interval(
+ ),
+ view_probe_attempts_max: default_view_probe_attempts_max(),
+ repair_retry_interval: default_repair_retry_interval(),
+ repair_chunk_max: default_repair_chunk_max(),
nodes,
auth: ClusterAuthConfig::default(),
tls: ClusterTlsConfig::default(),
@@ -417,6 +751,136 @@ mod cluster_validate_tests {
assert!(c.validate().is_err());
}
+ #[test]
+ fn validate_rejects_zero_commit_broadcast_interval() {
+ // `0` / `disabled` / `unlimited` all collapse to zero and stall the
+ // liveness broadcast.
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.commit_broadcast_interval = IggyDuration::new(Duration::ZERO);
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_rejects_zero_prepare_retransmit_interval() {
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.prepare_retransmit_interval = IggyDuration::new(Duration::ZERO);
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_rejects_heartbeat_below_commit_broadcast_ratio() {
+ // 3s clears the absolute 2s floor but is still < 4x the 1s broadcast,
+ // so the ratio rule is what rejects here, not the floor.
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.heartbeat_timeout = IggyDuration::new(Duration::from_secs(3));
+ c.commit_broadcast_interval = IggyDuration::new(Duration::from_secs(1));
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_accepts_heartbeat_at_commit_broadcast_ratio() {
+ // Exactly 4x the broadcast (and above the 2s floor) must pass.
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.heartbeat_timeout = IggyDuration::new(Duration::from_secs(4));
+ c.commit_broadcast_interval = IggyDuration::new(Duration::from_secs(1));
+ assert!(c.validate().is_ok());
+ }
+
+ #[test]
+ fn validate_rejects_zero_view_change_retransmit_interval() {
+ // `0` / `disabled` / `unlimited` all collapse to zero and stall the
+ // view-change retransmit timers.
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.view_change_retransmit_interval = IggyDuration::new(Duration::ZERO);
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_rejects_zero_view_change_status_timeout() {
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.view_change_status_timeout = IggyDuration::new(Duration::ZERO);
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_rejects_zero_request_start_view_retransmit_interval() {
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.request_start_view_retransmit_interval = IggyDuration::new(Duration::ZERO);
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_rejects_view_change_status_below_retransmit_ratio() {
+ // 3s is nonzero but still < 4x the 1s retransmit, so the ratio rule is
+ // what rejects here, not the zero check.
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.view_change_retransmit_interval = IggyDuration::new(Duration::from_secs(1));
+ c.view_change_status_timeout = IggyDuration::new(Duration::from_secs(3));
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_accepts_view_change_status_at_retransmit_ratio() {
+ // Exactly 4x the retransmit interval must pass.
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.view_change_retransmit_interval = IggyDuration::new(Duration::from_secs(1));
+ c.view_change_status_timeout = IggyDuration::new(Duration::from_secs(4));
+ assert!(c.validate().is_ok());
+ }
+
+ #[test]
+ fn validate_rejects_zero_view_probe_attempts_max() {
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.view_probe_attempts_max = 0;
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_rejects_view_probe_attempts_above_ceiling() {
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.view_probe_attempts_max = MAX_VIEW_PROBE_ATTEMPTS + 1;
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_accepts_view_probe_attempts_at_ceiling() {
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.view_probe_attempts_max = MAX_VIEW_PROBE_ATTEMPTS;
+ assert!(c.validate().is_ok());
+ }
+
+ #[test]
+ fn validate_rejects_zero_repair_retry_interval() {
+ // `0` / `disabled` / `unlimited` all collapse to zero and would wedge
+ // stalled repair streams.
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.repair_retry_interval = IggyDuration::new(Duration::ZERO);
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_rejects_zero_repair_chunk_max() {
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.repair_chunk_max = 0;
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_rejects_repair_chunk_max_above_ceiling() {
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.repair_chunk_max = MAX_REPAIR_CHUNK_MAX + 1;
+ assert!(c.validate().is_err());
+ }
+
+ #[test]
+ fn validate_accepts_repair_chunk_max_at_ceiling() {
+ // Section-level validate only; the cross-section rule against
+ // message_bus.peer_queue_capacity lives in the top-level validate.
+ let mut c = cfg(vec![node("n1", 0), node("n2", 1)]);
+ c.repair_chunk_max = MAX_REPAIR_CHUNK_MAX;
+ assert!(c.validate().is_ok());
+ }
+
#[test]
fn validate_rejects_empty_nodes() {
let c = cfg(vec![]);
diff --git a/core/configs/src/server_ng_config/defaults.rs b/core/configs/src/server_ng_config/defaults.rs
index 58887a70f..560602751 100644
--- a/core/configs/src/server_ng_config/defaults.rs
+++ b/core/configs/src/server_ng_config/defaults.rs
@@ -30,8 +30,10 @@
use super::cluster::{
ClusterAuthConfig, ClusterConfig, ClusterNodeConfig, ClusterTlsConfig, TransportPorts,
};
+use super::http_admission::HttpAdmissionConfig;
use super::message_bus::MessageBusConfig;
use super::metadata::MetadataConfig;
+use super::partition::PartitionConfig;
use super::quic::{QuicCertificateConfig, QuicConfig, QuicSocketConfig};
use super::server_ng::NgSystemConfig;
use super::server_ng::{ExtraConfig, ServerNgConfig};
@@ -63,9 +65,11 @@ impl Default for ServerNgConfig {
tcp: TcpConfig::default(),
websocket: WebSocketConfig::default(),
http: HttpConfig::default(),
+ http_admission: HttpAdmissionConfig::default(),
telemetry: TelemetryConfig::default(),
cluster: ClusterConfig::default(),
metadata: MetadataConfig::default(),
+ partition: PartitionConfig::default(),
message_bus: MessageBusConfig::default(),
}
}
@@ -77,6 +81,38 @@ impl Default for ClusterConfig {
enabled: SERVER_NG_CONFIG.cluster.enabled,
name: SERVER_NG_CONFIG.cluster.name.parse().unwrap(),
heartbeat_timeout: SERVER_NG_CONFIG.cluster.heartbeat_timeout.parse().unwrap(),
+ commit_broadcast_interval: SERVER_NG_CONFIG
+ .cluster
+ .commit_broadcast_interval
+ .parse()
+ .unwrap(),
+ prepare_retransmit_interval: SERVER_NG_CONFIG
+ .cluster
+ .prepare_retransmit_interval
+ .parse()
+ .unwrap(),
+ view_change_retransmit_interval: SERVER_NG_CONFIG
+ .cluster
+ .view_change_retransmit_interval
+ .parse()
+ .unwrap(),
+ view_change_status_timeout: SERVER_NG_CONFIG
+ .cluster
+ .view_change_status_timeout
+ .parse()
+ .unwrap(),
+ request_start_view_retransmit_interval: SERVER_NG_CONFIG
+ .cluster
+ .request_start_view_retransmit_interval
+ .parse()
+ .unwrap(),
+ view_probe_attempts_max: SERVER_NG_CONFIG.cluster.view_probe_attempts_max as u32,
+ repair_retry_interval: SERVER_NG_CONFIG
+ .cluster
+ .repair_retry_interval
+ .parse()
+ .unwrap(),
+ repair_chunk_max: SERVER_NG_CONFIG.cluster.repair_chunk_max as usize,
nodes: SERVER_NG_CONFIG
.cluster
.nodes
@@ -126,6 +162,34 @@ impl Default for MetadataConfig {
MetadataConfig {
prepare_queue_depth: metadata.prepare_queue_depth as usize,
journal_slots: metadata.journal_slots as usize,
+ clients_table_max: metadata.clients_table_max as usize,
+ }
+ }
+}
+
+impl Default for PartitionConfig {
+ fn default() -> PartitionConfig {
+ // Read from the embedded TOML so the Default impl and the on-disk
+ // schema cannot drift (same pattern as MetadataConfig above).
+ let partition = &SERVER_NG_CONFIG.partition;
+ PartitionConfig {
+ prepare_queue_depth: partition.prepare_queue_depth as usize,
+ evicted_ring_capacity: partition.evicted_ring_capacity as usize,
+ evicted_ring_bytes_max: partition.evicted_ring_bytes_max.parse().unwrap(),
+ }
+ }
+}
+
+impl Default for HttpAdmissionConfig {
+ fn default() -> HttpAdmissionConfig {
+ // Read from the embedded TOML so the Default impl and the on-disk
+ // schema cannot drift; the lockstep test pins these to the canonical
+ // caps.
+ let http_admission = &SERVER_NG_CONFIG.http_admission;
+ HttpAdmissionConfig {
+ max_in_flight_writes: http_admission.max_in_flight_writes as u32,
+ max_in_flight_writes_per_session: http_admission.max_in_flight_writes_per_session
+ as u32,
}
}
}
diff --git a/core/configs/src/server_ng_config/displays.rs b/core/configs/src/server_ng_config/displays.rs
index 022e4e311..14f8e2497 100644
--- a/core/configs/src/server_ng_config/displays.rs
+++ b/core/configs/src/server_ng_config/displays.rs
@@ -56,8 +56,8 @@ impl Display for MetadataConfig {
fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
write!(
f,
- "{{ prepare_queue_depth: {}, journal_slots: {} }}",
- self.prepare_queue_depth, self.journal_slots,
+ "{{ prepare_queue_depth: {}, journal_slots: {}, clients_table_max: {} }}",
+ self.prepare_queue_depth, self.journal_slots, self.clients_table_max,
)
}
}
diff --git a/core/configs/src/server_ng_config/http_admission.rs b/core/configs/src/server_ng_config/http_admission.rs
new file mode 100644
index 000000000..5e99de1d5
--- /dev/null
+++ b/core/configs/src/server_ng_config/http_admission.rs
@@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! On-disk schema for shard-0 HTTP write admission.
+//!
+//! Two in-flight caps previously hardcoded in the server-ng HTTP write
+//! path, bounding awaited partition writes (produce / consumer-offset)
+//! that each park a handler and pin a buffered body while its
+//! decode/encode/HS256 CPU runs on the single core that also pumps
+//! consensus:
+//!
+//! - `max_in_flight_writes` - shard-0-global budget across all sessions.
+//! - `max_in_flight_writes_per_session` - one credential's slice of that
+//! budget, so a session that outruns its own commits saturates itself
+//! (429) before it can starve the shared budget.
+//!
+//! The defaults here are the canonical caps; `core/server-ng`'s HTTP
+//! layer reads the configured values at boot, and the lockstep test
+//! below pins the shipped `config.toml` defaults to these constants so a
+//! default deployment keeps the historical behavior.
+
+use super::COMPONENT_NG;
+use crate::ConfigurationError;
+use configs::ConfigEnv;
+use iggy_common::Validatable;
+use serde::{Deserialize, Serialize};
+
+/// Canonical shard-0-global in-flight write budget.
+pub const DEFAULT_MAX_IN_FLIGHT_WRITES: u32 = 128;
+
+/// Canonical per-session in-flight write cap.
+pub const DEFAULT_MAX_IN_FLIGHT_WRITES_PER_SESSION: u32 = 32;
+
+/// Upper bound on `max_in_flight_writes`. Every awaited write pins a
+/// buffered request body (budget x `http.max_request_size`) on the single
+/// shard-0 core; four thousand in-flight writes is far past any sane
+/// deployment and a likely unit typo.
+pub const MAX_IN_FLIGHT_WRITES_CEILING: u32 = 4096;
+
+/// Admission caps for shard-0 HTTP awaited partition writes.
+#[derive(Debug, Deserialize, Serialize, Clone, ConfigEnv)]
+pub struct HttpAdmissionConfig {
+ /// Shard-0-global budget for concurrently awaited partition writes
+ /// across every session. Bounds the worst-case buffered bytes
+ /// (budget x `http.max_request_size`) and how far admitted HTTP work
+ /// can delay the consensus pump (budget x per-request CPU). Refusals
+ /// past it are the server-busy 503.
+ pub max_in_flight_writes: u32,
+
+ /// Per-session cap on concurrently awaited partition writes, bounding
+ /// how much of `max_in_flight_writes` one credential may occupy.
+ /// Refusals past it are the too-many-in-flight 429, a session's own
+ /// backpressure signal before it can starve the shared budget. Must
+ /// not exceed `max_in_flight_writes`.
+ pub max_in_flight_writes_per_session: u32,
+}
+
+impl Validatable<ConfigurationError> for HttpAdmissionConfig {
+ fn validate(&self) -> Result<(), ConfigurationError> {
+ if self.max_in_flight_writes == 0 {
+ eprintln!("{COMPONENT_NG} http_admission.max_in_flight_writes must be > 0");
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ // One ceiling suffices: per_session may not exceed the global budget
+ // (checked below), so this bounds it transitively.
+ if self.max_in_flight_writes > MAX_IN_FLIGHT_WRITES_CEILING {
+ eprintln!(
+ "{COMPONENT_NG} http_admission.max_in_flight_writes ({}) exceeds the maximum ({MAX_IN_FLIGHT_WRITES_CEILING})",
+ self.max_in_flight_writes
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.max_in_flight_writes_per_session == 0 {
+ eprintln!("{COMPONENT_NG} http_admission.max_in_flight_writes_per_session must be > 0");
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ // A per-session cap above the global budget can never bind: the
+ // session would always hit the shared 503 first, so the 429 signal
+ // is dead. Reject the nonsensical ordering (equal is fine - a
+ // single session may then fill the whole budget).
+ if self.max_in_flight_writes_per_session > self.max_in_flight_writes {
+ eprintln!(
+ "{COMPONENT_NG} http_admission.max_in_flight_writes_per_session ({}) must not exceed http_admission.max_in_flight_writes ({})",
+ self.max_in_flight_writes_per_session, self.max_in_flight_writes
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ Ok(())
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn cfg(global: u32, per_session: u32) -> HttpAdmissionConfig {
+ HttpAdmissionConfig {
+ max_in_flight_writes: global,
+ max_in_flight_writes_per_session: per_session,
+ }
+ }
+
+ #[test]
+ fn embedded_defaults_match_canonical_caps() {
+ // Behavior lockstep: `Default` reads the shipped config.toml, which must
+ // carry the caps the HTTP admission path historically hardcoded so a
+ // default deployment is unaffected by surfacing them as config.
+ let config = HttpAdmissionConfig::default();
+ assert_eq!(config.max_in_flight_writes, DEFAULT_MAX_IN_FLIGHT_WRITES);
+ assert_eq!(
+ config.max_in_flight_writes_per_session,
+ DEFAULT_MAX_IN_FLIGHT_WRITES_PER_SESSION
+ );
+ }
+
+ #[test]
+ fn default_impl_validates() {
+ assert!(HttpAdmissionConfig::default().validate().is_ok());
+ }
+
+ #[test]
+ fn rejects_zero_global() {
+ assert!(
+ cfg(0, DEFAULT_MAX_IN_FLIGHT_WRITES_PER_SESSION)
+ .validate()
+ .is_err()
+ );
+ }
+
+ #[test]
+ fn rejects_zero_per_session() {
+ assert!(cfg(DEFAULT_MAX_IN_FLIGHT_WRITES, 0).validate().is_err());
+ }
+
+ #[test]
+ fn rejects_per_session_above_global() {
+ assert!(cfg(32, 33).validate().is_err());
+ }
+
+ #[test]
+ fn accepts_per_session_equal_global() {
+ // A single session may fill the whole budget; equal is the boundary.
+ assert!(cfg(32, 32).validate().is_ok());
+ }
+
+ #[test]
+ fn rejects_above_ceiling() {
+ assert!(
+ cfg(
+ MAX_IN_FLIGHT_WRITES_CEILING + 1,
+ DEFAULT_MAX_IN_FLIGHT_WRITES_PER_SESSION
+ )
+ .validate()
+ .is_err()
+ );
+ }
+
+ #[test]
+ fn accepts_at_ceiling() {
+ // Inclusive ceiling; per-session rides under it via per_session <= global.
+ assert!(
+ cfg(
+ MAX_IN_FLIGHT_WRITES_CEILING,
+ DEFAULT_MAX_IN_FLIGHT_WRITES_PER_SESSION
+ )
+ .validate()
+ .is_ok()
+ );
+ }
+}
diff --git a/core/configs/src/server_ng_config/metadata.rs b/core/configs/src/server_ng_config/metadata.rs
index bfcb25336..f38840f89 100644
--- a/core/configs/src/server_ng_config/metadata.rs
+++ b/core/configs/src/server_ng_config/metadata.rs
@@ -18,7 +18,7 @@
//! On-disk schema for the metadata consensus plane (shard 0's VSR
//! replica: users, streams, topics, sessions).
//!
-//! Two capacity knobs previously hardcoded in the runtime crates:
+//! Three capacity knobs previously hardcoded in the runtime crates:
//!
//! - `prepare_queue_depth` -> `consensus::PIPELINE_PREPARE_QUEUE_MAX`
//! (the pipeline's in-flight prepare bound; submits beyond it bounce
@@ -26,8 +26,11 @@
//! - `journal_slots` -> `journal::prepare_journal::DEFAULT_SLOT_COUNT`
//! (the WAL's in-memory index; committed-but-unsnapshotted headroom
//! between forced checkpoints)
+//! - `clients_table_max` -> `consensus::CLIENTS_TABLE_MAX` (the VSR
+//! client-table slot count; independent of the two above). The
+//! server-ng HTTP session cap tracks it at half.
//!
-//! The two interlock through the forced-checkpoint margin
+//! The first two interlock through the forced-checkpoint margin
//! (`max(64, prepare_queue_depth)` at bootstrap): while a checkpoint
//! runs, up to a full prepare queue of already-pipelined ops appends
//! into that margin, and `validate` keeps `journal_slots` far enough
@@ -37,7 +40,7 @@
//! `core/configs` does not grow build-time edges onto `core/consensus`
//! and `core/journal` (the runtime crates are the consumers of this
//! config, mirroring the `IOV_MAX_LIMIT_NG` precedent in
-//! [`super::message_bus`]). `core/server-ng`'s bootstrap pins both
+//! [`super::message_bus`]). `core/server-ng`'s bootstrap pins these
//! literals against the runtime constants with static asserts.
use super::COMPONENT_NG;
@@ -67,6 +70,20 @@ pub const MAX_METADATA_PREPARE_QUEUE_DEPTH: usize = 4096;
/// ceiling, not a tuning target.
pub const MAX_METADATA_JOURNAL_SLOTS: usize = 1 << 20;
+/// Mirrors `consensus::CLIENTS_TABLE_MAX`, the VSR client-table slot count.
+pub const DEFAULT_METADATA_CLIENTS_TABLE_MAX: usize = 8192;
+
+/// Floor on `clients_table_max`. The server-ng HTTP session cap derives as
+/// `clients_table_max / 2`; below two that floors to zero and HTTP could
+/// register no sessions at all.
+pub const MIN_METADATA_CLIENTS_TABLE_MAX: usize = 2;
+
+/// Upper bound on `clients_table_max`. Every slot is preallocated for the
+/// table's whole lifetime whether or not it holds a live client, so this caps
+/// fixed per-shard table memory; 8x the default is far past any real client
+/// population and a likely unit typo.
+pub const MAX_METADATA_CLIENTS_TABLE_MAX: usize = 1 << 16;
+
/// Capacity tunables for the metadata consensus plane.
#[derive(Debug, Deserialize, Serialize, Clone, ConfigEnv)]
pub struct MetadataConfig {
@@ -80,6 +97,13 @@ pub struct MetadataConfig {
/// checkpoints; more slots = rarer checkpoints, more memory, larger
/// per-checkpoint WAL rewrites.
pub journal_slots: usize,
+
+ /// Slot count of the VSR client table: how many distinct clients
+ /// (TCP/QUIC/WS virtual clients and HTTP sessions together) hold live
+ /// session state before the oldest-committed entry is evicted. The
+ /// server-ng HTTP session cap tracks this at half, so raising it lifts
+ /// both.
+ pub clients_table_max: usize,
}
impl MetadataConfig {
@@ -127,6 +151,20 @@ impl Validatable<ConfigurationError> for MetadataConfig {
);
return Err(ConfigurationError::InvalidConfigurationValue);
}
+ if self.clients_table_max < MIN_METADATA_CLIENTS_TABLE_MAX {
+ eprintln!(
+ "{COMPONENT_NG} metadata.clients_table_max ({}) must be >= {MIN_METADATA_CLIENTS_TABLE_MAX}",
+ self.clients_table_max
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.clients_table_max > MAX_METADATA_CLIENTS_TABLE_MAX {
+ eprintln!(
+ "{COMPONENT_NG} metadata.clients_table_max ({}) exceeds the maximum ({MAX_METADATA_CLIENTS_TABLE_MAX})",
+ self.clients_table_max
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
Ok(())
}
}
@@ -140,6 +178,7 @@ mod tests {
let config = MetadataConfig {
prepare_queue_depth: DEFAULT_METADATA_PREPARE_QUEUE_DEPTH,
journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS,
+ clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX,
};
assert!(config.validate().is_ok());
assert_eq!(config.checkpoint_margin(), METADATA_CHECKPOINT_MARGIN_FLOOR);
@@ -150,6 +189,7 @@ mod tests {
let config = MetadataConfig {
prepare_queue_depth: 256,
journal_slots: 4096,
+ clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX,
};
assert!(config.validate().is_ok());
assert_eq!(config.checkpoint_margin(), 256);
@@ -162,12 +202,14 @@ mod tests {
let boundary = MetadataConfig {
prepare_queue_depth: 256,
journal_slots: 1024,
+ clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX,
};
assert!(boundary.validate().is_ok());
// ...one slot fewer is refused.
let starved = MetadataConfig {
prepare_queue_depth: 256,
journal_slots: 1023,
+ clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX,
};
assert!(starved.validate().is_err());
}
@@ -177,6 +219,47 @@ mod tests {
let config = MetadataConfig {
prepare_queue_depth: 0,
journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS,
+ clients_table_max: DEFAULT_METADATA_CLIENTS_TABLE_MAX,
+ };
+ assert!(config.validate().is_err());
+ }
+
+ // The shipped config.toml default is the canonical slot count, so a
+ // pristine deployment sizes the table exactly as the consensus constant.
+ #[test]
+ fn embedded_default_matches_canonical_clients_table_max() {
+ assert_eq!(
+ MetadataConfig::default().clients_table_max,
+ DEFAULT_METADATA_CLIENTS_TABLE_MAX
+ );
+ }
+
+ #[test]
+ fn clients_table_max_below_floor_is_refused() {
+ let config = MetadataConfig {
+ prepare_queue_depth: DEFAULT_METADATA_PREPARE_QUEUE_DEPTH,
+ journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS,
+ clients_table_max: MIN_METADATA_CLIENTS_TABLE_MAX - 1,
+ };
+ assert!(config.validate().is_err());
+ }
+
+ #[test]
+ fn clients_table_max_at_floor_is_accepted() {
+ let config = MetadataConfig {
+ prepare_queue_depth: DEFAULT_METADATA_PREPARE_QUEUE_DEPTH,
+ journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS,
+ clients_table_max: MIN_METADATA_CLIENTS_TABLE_MAX,
+ };
+ assert!(config.validate().is_ok());
+ }
+
+ #[test]
+ fn clients_table_max_above_ceiling_is_refused() {
+ let config = MetadataConfig {
+ prepare_queue_depth: DEFAULT_METADATA_PREPARE_QUEUE_DEPTH,
+ journal_slots: DEFAULT_METADATA_JOURNAL_SLOTS,
+ clients_table_max: MAX_METADATA_CLIENTS_TABLE_MAX + 1,
};
assert!(config.validate().is_err());
}
diff --git a/core/configs/src/server_ng_config/mod.rs b/core/configs/src/server_ng_config/mod.rs
index 159d0c2f5..dc2a736b2 100644
--- a/core/configs/src/server_ng_config/mod.rs
+++ b/core/configs/src/server_ng_config/mod.rs
@@ -30,8 +30,10 @@
pub mod cluster;
pub mod defaults;
pub mod displays;
+pub mod http_admission;
pub mod message_bus;
pub mod metadata;
+pub mod partition;
pub mod quic;
pub mod server_ng;
pub mod sharding;
diff --git a/core/configs/src/server_ng_config/partition.rs b/core/configs/src/server_ng_config/partition.rs
new file mode 100644
index 000000000..580668e4b
--- /dev/null
+++ b/core/configs/src/server_ng_config/partition.rs
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! On-disk schema for the per-partition consensus plane.
+//!
+//! Capacity knobs previously hardcoded in the runtime crates:
+//!
+//! - `prepare_queue_depth` -> `consensus::PIPELINE_PREPARE_QUEUE_MAX`
+//! (the pipeline's in-flight prepare bound; submits beyond it bounce
+//! with the transient prepare-queue-full path the SDK retries)
+//! - `evicted_ring_capacity` -> `partitions::EVICTED_RING_CAPACITY` and
+//! `evicted_ring_bytes_max` -> `partitions::EVICTED_RING_BYTES_MAX`
+//! (the per-partition journal-repair retention ring's dual ceilings)
+//!
+//! Distinct from `[metadata]` (a single, shard-0-global VSR plane) because
+//! partition pipelines exist PER PARTITION. The default mirrors the runtime
+//! constant so a default deployment is byte-identical; the ceiling is far
+//! below metadata's because the request queue (`depth * 2` slots) pins full
+//! inbound produce batches, so pinned memory scales with the partition count
+//! (see [`MAX_PARTITION_PREPARE_QUEUE_DEPTH`]).
+//!
+//! The default is a duplicated literal rather than an import so
+//! `core/configs` does not grow a build-time edge onto `core/consensus`
+//! (mirroring [`super::metadata`]). `core/server-ng`'s bootstrap pins the
+//! literal against the runtime constant with a static assert.
+
+use super::COMPONENT_NG;
+use crate::ConfigurationError;
+use configs::ConfigEnv;
+use iggy_common::{IggyByteSize, Validatable};
+use serde::{Deserialize, Serialize};
+
+/// Mirrors `consensus::PIPELINE_PREPARE_QUEUE_MAX`.
+pub const DEFAULT_PARTITION_PREPARE_QUEUE_DEPTH: usize = 32;
+
+/// Upper bound on `prepare_queue_depth`. Unlike the single metadata pipeline,
+/// a pipeline exists per partition, and each queued request pins a full
+/// inbound produce batch (a 4 KiB floor up to megabytes). Worst-case pinned
+/// memory therefore scales as `depth * 2 * partition_count * batch_size`, so
+/// this ceiling sits far below metadata's 4096: it is a typo guard, not a
+/// sizing endorsement.
+pub const MAX_PARTITION_PREPARE_QUEUE_DEPTH: usize = 256;
+
+/// Mirrors `partitions::EVICTED_RING_CAPACITY`.
+pub const DEFAULT_EVICTED_RING_CAPACITY: usize = 4096;
+
+/// Upper bound on `evicted_ring_capacity`. The ring exists per multi-replica
+/// partition and each retained entry pins a full committed batch, so
+/// worst-case pinned memory scales with the partition count; a typo guard,
+/// not a sizing endorsement.
+pub const MAX_EVICTED_RING_CAPACITY: usize = 65536;
+
+/// Mirrors `partitions::EVICTED_RING_BYTES_MAX`.
+pub const DEFAULT_EVICTED_RING_BYTES_MAX: u64 = 16 * 1024 * 1024;
+
+/// Upper bound on `evicted_ring_bytes_max`, per partition. Whichever ring cap
+/// trips first evicts; this byte ceiling is the second typo guard.
+pub const MAX_EVICTED_RING_BYTES: u64 = 256 * 1024 * 1024;
+
+/// Capacity tunables for the per-partition consensus plane.
+#[derive(Debug, Deserialize, Serialize, Clone, ConfigEnv)]
+pub struct PartitionConfig {
+ /// Depth of a partition's prepare queue: how many uncommitted produce /
+ /// consumer-offset ops may be in flight at once for that partition.
+ /// Submits beyond it are rejected with the transient prepare-queue-full
+ /// path the SDK retries. Applies to every partition; raising it multiplies
+ /// pinned request-buffer memory by the partition count.
+ pub prepare_queue_depth: usize,
+
+ /// Entries the evicted ring retains per multi-replica partition for
+ /// journal repair after a peer rejoins. Larger widens the window a
+ /// restarting peer can be served from the ring before falling back to
+ /// bulk sync, at the cost of pinned memory per partition. Must be > 0 and
+ /// <= [`MAX_EVICTED_RING_CAPACITY`]. Single-replica partitions retain
+ /// nothing regardless.
+ pub evicted_ring_capacity: usize,
+
+ /// Byte ceiling for the evicted ring per partition; whichever ring cap
+ /// (this or [`Self::evicted_ring_capacity`]) trips first evicts. Bounds
+ /// the ring memory a burst of large batches can pin. Must be > 0 and <=
+ /// [`MAX_EVICTED_RING_BYTES`].
+ #[config_env(leaf)]
+ pub evicted_ring_bytes_max: IggyByteSize,
+}
+
+impl Validatable<ConfigurationError> for PartitionConfig {
+ fn validate(&self) -> Result<(), ConfigurationError> {
+ if self.prepare_queue_depth == 0 {
+ eprintln!("{COMPONENT_NG} partition.prepare_queue_depth must be > 0");
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.prepare_queue_depth > MAX_PARTITION_PREPARE_QUEUE_DEPTH {
+ eprintln!(
+ "{COMPONENT_NG} partition.prepare_queue_depth ({}) exceeds the maximum ({MAX_PARTITION_PREPARE_QUEUE_DEPTH})",
+ self.prepare_queue_depth
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.evicted_ring_capacity == 0 {
+ eprintln!("{COMPONENT_NG} partition.evicted_ring_capacity must be > 0");
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if self.evicted_ring_capacity > MAX_EVICTED_RING_CAPACITY {
+ eprintln!(
+ "{COMPONENT_NG} partition.evicted_ring_capacity ({}) exceeds the maximum ({MAX_EVICTED_RING_CAPACITY})",
+ self.evicted_ring_capacity
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ let ring_bytes = self.evicted_ring_bytes_max.as_bytes_u64();
+ if ring_bytes == 0 {
+ eprintln!("{COMPONENT_NG} partition.evicted_ring_bytes_max must be > 0");
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ if ring_bytes > MAX_EVICTED_RING_BYTES {
+ eprintln!(
+ "{COMPONENT_NG} partition.evicted_ring_bytes_max ({ring_bytes} bytes) exceeds the maximum ({MAX_EVICTED_RING_BYTES} bytes)"
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+ Ok(())
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn default_impl_validates() {
+ // `Default` reads the shipped config.toml; the pristine deployment
+ // must validate.
+ assert!(PartitionConfig::default().validate().is_ok());
+ }
+
+ #[test]
+ fn rejects_zero_prepare_queue_depth() {
+ let config = PartitionConfig {
+ prepare_queue_depth: 0,
+ ..PartitionConfig::default()
+ };
+ assert!(config.validate().is_err());
+ }
+
+ #[test]
+ fn rejects_prepare_queue_depth_above_ceiling() {
+ let config = PartitionConfig {
+ prepare_queue_depth: MAX_PARTITION_PREPARE_QUEUE_DEPTH + 1,
+ ..PartitionConfig::default()
+ };
+ assert!(config.validate().is_err());
+ }
+
+ #[test]
+ fn accepts_prepare_queue_depth_at_ceiling() {
+ let config = PartitionConfig {
+ prepare_queue_depth: MAX_PARTITION_PREPARE_QUEUE_DEPTH,
+ ..PartitionConfig::default()
+ };
+ assert!(config.validate().is_ok());
+ }
+
+ #[test]
+ fn rejects_zero_evicted_ring_capacity() {
+ let config = PartitionConfig {
+ evicted_ring_capacity: 0,
+ ..PartitionConfig::default()
+ };
+ assert!(config.validate().is_err());
+ }
+
+ #[test]
+ fn rejects_evicted_ring_capacity_above_ceiling() {
+ let config = PartitionConfig {
+ evicted_ring_capacity: MAX_EVICTED_RING_CAPACITY + 1,
+ ..PartitionConfig::default()
+ };
+ assert!(config.validate().is_err());
+ }
+
+ #[test]
+ fn rejects_zero_evicted_ring_bytes_max() {
+ let config = PartitionConfig {
+ evicted_ring_bytes_max: IggyByteSize::from(0_u64),
+ ..PartitionConfig::default()
+ };
+ assert!(config.validate().is_err());
+ }
+
+ #[test]
+ fn rejects_evicted_ring_bytes_max_above_ceiling() {
+ let config = PartitionConfig {
+ evicted_ring_bytes_max: IggyByteSize::from(MAX_EVICTED_RING_BYTES + 1),
+ ..PartitionConfig::default()
+ };
+ assert!(config.validate().is_err());
+ }
+}
diff --git a/core/configs/src/server_ng_config/server_ng.rs b/core/configs/src/server_ng_config/server_ng.rs
index 3cd8f5376..77be95249 100644
--- a/core/configs/src/server_ng_config/server_ng.rs
+++ b/core/configs/src/server_ng_config/server_ng.rs
@@ -17,8 +17,10 @@
use super::COMPONENT_NG;
use super::cluster::ClusterConfig;
+use super::http_admission::HttpAdmissionConfig;
use super::message_bus::MessageBusConfig;
use super::metadata::MetadataConfig;
+use super::partition::PartitionConfig;
use super::quic::QuicConfig;
use super::tcp::TcpConfig;
use super::websocket::WebSocketConfig;
@@ -74,10 +76,12 @@ pub struct ServerNgConfig {
pub quic: QuicConfig,
pub tcp: TcpConfig,
pub http: HttpConfig,
+ pub http_admission: HttpAdmissionConfig,
pub websocket: WebSocketConfig,
pub telemetry: TelemetryConfig,
pub cluster: ClusterConfig,
pub metadata: MetadataConfig,
+ pub partition: PartitionConfig,
pub message_bus: MessageBusConfig,
}
diff --git a/core/configs/src/server_ng_config/validators.rs b/core/configs/src/server_ng_config/validators.rs
index 4736d9055..ccc3d9641 100644
--- a/core/configs/src/server_ng_config/validators.rs
+++ b/core/configs/src/server_ng_config/validators.rs
@@ -84,6 +84,14 @@ impl Validatable<ConfigurationError> for ServerNgConfig {
self.metadata.validate().error(|e: &ConfigurationError| {
format!("{COMPONENT_NG} (error: {e}) - failed to validate metadata config")
})?;
+ self.partition.validate().error(|e: &ConfigurationError| {
+ format!("{COMPONENT_NG} (error: {e}) - failed to validate partition config")
+ })?;
+ self.http_admission
+ .validate()
+ .error(|e: &ConfigurationError| {
+ format!("{COMPONENT_NG} (error: {e}) - failed to validate http admission config")
+ })?;
self.system
.logging
.validate()
@@ -172,6 +180,20 @@ impl Validatable<ConfigurationError> for ServerNgConfig {
format!("{COMPONENT_NG} (error: {e}) - failed to validate message_bus config")
})?;
+ // Repair frames ride the bounded per-peer message-bus queue. A repair
+ // round of cluster.repair_chunk_max frames that meets or overruns
+ // message_bus.peer_queue_capacity drops its own tail silently, wedging
+ // the repair loop into slow retries. Keep the chunk strictly below the
+ // queue; this also floors peer_queue_capacity, which is otherwise only
+ // checked for > 0.
+ if self.cluster.repair_chunk_max >= self.message_bus.peer_queue_capacity {
+ eprintln!(
+ "{COMPONENT_NG} cluster.repair_chunk_max ({}) must be < message_bus.peer_queue_capacity ({}): repair frames ride the per-peer bus queue, so a chunk that fills or overruns it drops frames and wedges repair",
+ self.cluster.repair_chunk_max, self.message_bus.peer_queue_capacity
+ );
+ return Err(ConfigurationError::InvalidConfigurationValue);
+ }
+
self.quic.validate().error(|e: &ConfigurationError| {
format!("{COMPONENT_NG} (error: {e}) - failed to validate quic config")
})?;
@@ -352,6 +374,29 @@ mod tests {
assert!(config.validate().is_err());
}
+ #[test]
+ fn given_peer_queue_capacity_not_above_repair_chunk_max_when_validating_should_reject() {
+ // The default repair_chunk_max (128) must stay strictly below
+ // peer_queue_capacity; shrinking the queue to the chunk size is the
+ // silent wedged-repair footgun this cross-section guard closes.
+ let config = config_with_override("[message_bus]\npeer_queue_capacity = 128\n");
+ assert!(config.validate().is_err());
+ }
+
+ #[test]
+ fn given_repair_chunk_max_at_peer_queue_capacity_when_validating_should_reject() {
+ let config = config_with_override("[cluster]\nrepair_chunk_max = 256\n");
+ assert!(config.validate().is_err());
+ }
+
+ #[test]
+ fn given_repair_chunk_max_below_peer_queue_capacity_when_validating_should_pass() {
+ let config = config_with_override("[cluster]\nrepair_chunk_max = 255\n");
+ config
+ .validate()
+ .expect("a chunk below the peer queue capacity must validate");
+ }
+
/// The warn-helper baseline is [`ServerNgConfig::default`], but the reused
/// legacy sections source that default from the legacy server config.toml,
/// not this NG file. Pin the knobs the helper compares so any drift between
diff --git a/core/consensus/src/client_table.rs b/core/consensus/src/client_table.rs
index 6b0a8526f..aed6fc576 100644
--- a/core/consensus/src/client_table.rs
+++ b/core/consensus/src/client_table.rs
@@ -156,6 +156,21 @@ impl ClientTable {
}
}
+ /// Resize the table to `max_clients` slots. Boot-only: reallocating a
+ /// populated table would silently drop live sessions, so this must run
+ /// before any client registers (server-ng bootstrap applies the configured
+ /// `[metadata] clients_table_max` here).
+ ///
+ /// # Panics
+ /// If the table already holds a client.
+ pub fn set_capacity(&mut self, max_clients: usize) {
+ assert!(
+ self.index.is_empty(),
+ "set_capacity must run before any client registers"
+ );
+ *self = Self::new(max_clients);
+ }
+
/// Check request against table. Session first, then request progression.
/// For Register, use [`check_register`].
///
@@ -870,6 +885,45 @@ mod tests {
assert!(table.get_reply(300).is_some());
}
+ // Capacity resize (boot-only)
+
+ // Resizing an empty table swaps its slot count in: a smaller cap then
+ // evicts once the new bound is reached.
+ #[test]
+ fn set_capacity_resizes_empty_table() {
+ let mut table = ClientTable::new(10);
+ table.set_capacity(2);
+ table.commit_register(
+ 100,
+ TEST_USER_ID,
+ make_register_reply(100, 10),
+ no_in_flight(),
+ );
+ table.commit_register(
+ 200,
+ TEST_USER_ID,
+ make_register_reply(200, 20),
+ no_in_flight(),
+ );
+ table.commit_register(
+ 300,
+ TEST_USER_ID,
+ make_register_reply(300, 30),
+ no_in_flight(),
+ );
+ assert_eq!(table.count(), 2, "resized cap of 2 evicts the oldest");
+ assert!(table.get_reply(100).is_none());
+ }
+
+ // The empty-table contract is asserted, not silently honored: resizing a
+ // populated table would drop live sessions, so it must panic.
+ #[test]
+ #[should_panic(expected = "before any client registers")]
+ fn set_capacity_rejects_a_populated_table() {
+ let (mut table, _session) = table_with_client();
+ table.set_capacity(2);
+ }
+
// Edge cases
#[test]
diff --git a/core/consensus/src/impls.rs b/core/consensus/src/impls.rs
index 8869f870f..f969cb022 100644
--- a/core/consensus/src/impls.rs
+++ b/core/consensus/src/impls.rs
@@ -37,6 +37,7 @@ use server_common::sharding::{IggyNamespace, METADATA_CONSENSUS_NAMESPACE};
use std::cell::{Cell, RefCell};
use std::collections::VecDeque;
use std::rc::Rc;
+use std::time::Duration;
/// Injected time source for primary-stamped prepare timestamps.
///
@@ -122,16 +123,18 @@ impl Sequencer for LocalSequencer {
}
}
-/// TODO The below numbers need to be added a consensus config
-/// TODO understand how to configure these numbers.
-/// Maximum number of prepares that can be in-flight in the pipeline.
+/// Default in-flight prepare-queue depth.
///
-/// Sized to absorb a synchronized client burst (e.g. the 20-way
-/// concurrent-creation race tests across TCP/QUIC/WebSocket) without
-/// `PipelineFull`-rejecting and disconnecting clients that cannot replay in
-/// time. At depth 8 the QUIC burst wedges the metadata consensus even in
-/// release. Stays well under the journal's `SLOT_COUNT` (1024) and the inbox
-/// capacity headroom.
+/// [`LocalPipeline::new`] uses it, and the server-ng config default
+/// (`DEFAULT_METADATA_PREPARE_QUEUE_DEPTH`) is static-asserted equal to it at
+/// bootstrap. Operators raise the running bound via `[metadata]
+/// prepare_queue_depth`; the pipeline then carries its own capacity (see
+/// [`LocalPipeline::with_capacities`]).
+///
+/// Sized to absorb a synchronized client burst (the 20-way concurrent-creation
+/// race tests across TCP/QUIC/WebSocket) without `PipelineFull`-rejecting
+/// clients that cannot replay in time; a depth of 8 wedged the QUIC burst even
+/// in release. Stays well under the journal's slot count and the inbox headroom.
pub const PIPELINE_PREPARE_QUEUE_MAX: usize = 32;
/// Max accepted-but-not-yet-prepared requests buffered behind a full
@@ -447,7 +450,7 @@ impl LocalPipeline {
}
/// Find a message by op number and checksum (immutable).
- // Pipeline bounded at PIPELINE_PREPARE_QUEUE_MAX (8) entries; index always fits in usize.
+ // op - head_op is bounded by the configured prepare-queue depth; index always fits in usize.
#[must_use]
#[allow(clippy::cast_possible_truncation)]
pub fn message_by_op_and_checksum(&self, op: u64, checksum: u128) -> Option<&PipelineEntry> {
@@ -478,7 +481,7 @@ impl LocalPipeline {
}
/// Find a message by op number only.
- // Pipeline bounded at PIPELINE_PREPARE_QUEUE_MAX (8) entries; index always fits in usize.
+ // op - head_op is bounded by the configured prepare-queue depth; index always fits in usize.
#[must_use]
#[allow(clippy::cast_possible_truncation)]
pub fn message_by_op(&self, op: u64) -> Option<&PipelineEntry> {
@@ -494,7 +497,7 @@ impl LocalPipeline {
/// Get mutable reference to a message entry by op number.
/// Returns None if op is not in the pipeline.
- // Pipeline bounded at PIPELINE_PREPARE_QUEUE_MAX (8) entries; index always fits in usize.
+ // op - head_op is bounded by the configured prepare-queue depth; index always fits in usize.
#[allow(clippy::cast_possible_truncation)]
pub fn message_by_op_mut(&mut self, op: u64) -> Option<&mut PipelineEntry> {
let head_op = self.prepare_queue.front()?.header.op;
@@ -531,8 +534,8 @@ impl LocalPipeline {
/// If any invariant is violated.
pub fn verify(&self) {
// Check capacity limits
- assert!(self.prepare_queue.len() <= PIPELINE_PREPARE_QUEUE_MAX);
- assert!(self.request_queue.len() <= PIPELINE_REQUEST_QUEUE_MAX);
+ assert!(self.prepare_queue.len() <= self.prepare_queue_max);
+ assert!(self.request_queue.len() <= self.request_queue_max);
// Verify prepare queue hash chain
if let Some(head) = self.prepare_queue.front() {
@@ -622,6 +625,10 @@ impl Pipeline for LocalPipeline {
self.prepare_count()
}
+ fn prepare_queue_max(&self) -> usize {
+ self.prepare_queue_max
+ }
+
fn verify(&self) {
Self::verify(self);
}
@@ -784,6 +791,10 @@ where
/// Commit point the recovered WAL suffix must re-reach before admitting
/// client requests as primary (`0` = no recovered suffix pending).
recovery_barrier: Cell<u64>,
+ /// Wall-clock budget the recovered suffix has to re-commit before a waiter
+ /// on [`Self::recovery_barrier`] gives up. Armed together with the barrier;
+ /// `ZERO` when no suffix is pending (barrier `0`), which no waiter reads.
+ recovery_deadline: Cell<Duration>,
/// True while this replica declines the primaryship its (stale) recovered
/// view assigns it (see `init_as_backup`). `is_primary()` is pure view
/// math, so without this flag a restarted view-N primary would still pass
@@ -809,14 +820,24 @@ where
last_prepare_checksum: Cell<u128>,
pipeline: RefCell<P>,
+ /// Snapshot of the pipeline's in-flight prepare capacity, taken at
+ /// construction. Bounds the loopback queue and the view-change rebuild
+ /// range without re-borrowing `pipeline`.
+ prepare_queue_max: usize,
message_bus: B,
loopback_queue: RefCell<VecDeque<Message<GenericHeader>>>,
/// Tracks start view change messages received from all replicas (including self)
start_view_change_from_all_replicas: RefCell<BitSet<u32>>,
/// Consecutive unanswered `RequestStartView` probes while Recovering;
- /// at [`PROBE_ATTEMPTS_MAX`] the replica falls back to an election.
+ /// at the `probe_attempts_max` ceiling the replica falls back to an
+ /// election.
probe_attempts: Cell<u32>,
+ /// Probe-attempt ceiling backing the fall-back-to-election decision,
+ /// seeded from [`PROBE_ATTEMPTS_MAX`] and overridable by the runtime via
+ /// `[cluster] view_probe_attempts_max`. The simulator and tests keep the
+ /// built-in default.
+ probe_attempts_max: Cell<u32>,
/// Tracks DVC messages received (only used by primary candidate)
/// Stores metadata; actual log comes from message
@@ -896,6 +917,7 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
// across groups. Consider using a proper hash (e.g., Murmur3) of
// (replica_id, namespace) for production.
let timeout_seed = u128::from(replica) ^ u128::from(namespace);
+ let prepare_queue_max = pipeline.prepare_queue_max();
Self {
cluster,
replica,
@@ -904,6 +926,7 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
view: Cell::new(0),
log_view: Cell::new(0),
recovery_barrier: Cell::new(0),
+ recovery_deadline: Cell::new(Duration::ZERO),
ceded_primaryship: Cell::new(false),
status: Cell::new(Status::Recovering),
sequencer: LocalSequencer::new(0),
@@ -912,10 +935,12 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
last_timestamp: Cell::new(0),
last_prepare_checksum: Cell::new(0),
pipeline: RefCell::new(pipeline),
+ prepare_queue_max,
message_bus,
- loopback_queue: RefCell::new(VecDeque::with_capacity(PIPELINE_PREPARE_QUEUE_MAX)),
+ loopback_queue: RefCell::new(VecDeque::with_capacity(prepare_queue_max)),
start_view_change_from_all_replicas: RefCell::new(BitSet::with_capacity(REPLICAS_MAX)),
probe_attempts: Cell::new(0),
+ probe_attempts_max: Cell::new(PROBE_ATTEMPTS_MAX),
do_view_change_from_all_replicas: RefCell::new(dvc_quorum_array_empty()),
do_view_change_quorum: Cell::new(false),
sent_own_start_view_change: Cell::new(false),
@@ -934,6 +959,61 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
self.timeouts.borrow_mut().set_normal_heartbeat_ticks(ticks);
}
+ /// Override the primary's commit-broadcast interval, in consensus ticks.
+ /// Sized from `[cluster] commit_broadcast_interval` by the runtime. Must
+ /// run before `init` / `init_as_backup`: the override discards any
+ /// countdown already in flight.
+ pub fn set_commit_message_ticks(&self, ticks: u64) {
+ self.timeouts.borrow_mut().set_commit_message_ticks(ticks);
+ }
+
+ /// Override the primary's prepare-retransmit interval, in consensus ticks.
+ /// Sized from `[cluster] prepare_retransmit_interval` by the runtime. Must
+ /// run before `init` / `init_as_backup`: the override discards any
+ /// countdown already in flight.
+ pub fn set_prepare_ticks(&self, ticks: u64) {
+ self.timeouts.borrow_mut().set_prepare_ticks(ticks);
+ }
+
+ /// Override the view-change retransmit interval (`StartViewChange` and
+ /// `DoViewChange`, kept equal), in consensus ticks. Sized from `[cluster]
+ /// view_change_retransmit_interval` by the runtime. Must run before `init`
+ /// / `init_as_backup`: the override discards any countdown already in
+ /// flight.
+ pub fn set_view_change_retransmit_ticks(&self, ticks: u64) {
+ self.timeouts
+ .borrow_mut()
+ .set_view_change_retransmit_ticks(ticks);
+ }
+
+ /// Override the view-change status backstop, in consensus ticks. Sized from
+ /// `[cluster] view_change_status_timeout` by the runtime. Must run before
+ /// `init` / `init_as_backup`: the override discards any countdown already
+ /// in flight.
+ pub fn set_view_change_status_ticks(&self, ticks: u64) {
+ self.timeouts
+ .borrow_mut()
+ .set_view_change_status_ticks(ticks);
+ }
+
+ /// Override the request-start-view retransmit interval, in consensus ticks.
+ /// Sized from `[cluster] request_start_view_retransmit_interval` by the
+ /// runtime. Must run before `init` / `init_as_backup`: the override
+ /// discards any countdown already in flight.
+ pub fn set_request_start_view_ticks(&self, ticks: u64) {
+ self.timeouts
+ .borrow_mut()
+ .set_request_start_view_ticks(ticks);
+ }
+
+ /// Override the recovering-replica probe-attempt ceiling before it falls
+ /// back to an election. Sized from `[cluster] view_probe_attempts_max` by
+ /// the runtime; the simulator and tests keep [`PROBE_ATTEMPTS_MAX`]. Must
+ /// run before `init` / `init_as_backup`.
+ pub fn set_probe_attempts_max(&self, max: u32) {
+ self.probe_attempts_max.set(max);
+ }
+
pub fn init(&self) {
self.status.set(Status::Normal);
let mut timeouts = self.timeouts.borrow_mut();
@@ -1107,6 +1187,17 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
self.recovery_barrier.set(required_commit);
}
+ /// Deadline paired with [`Self::recovery_barrier`]; only meaningful while the
+ /// barrier is armed (non-zero).
+ #[must_use]
+ pub const fn recovery_deadline(&self) -> Duration {
+ self.recovery_deadline.get()
+ }
+
+ pub fn set_recovery_deadline(&self, deadline: Duration) {
+ self.recovery_deadline.set(deadline);
+ }
+
pub fn set_view(&mut self, view: u32) {
self.view.set(view);
}
@@ -1436,7 +1527,7 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
// primary answers well before the fallback fires.
let attempts = self.probe_attempts.get() + 1;
self.probe_attempts.set(attempts);
- if attempts >= PROBE_ATTEMPTS_MAX {
+ if attempts >= self.probe_attempts_max.get() {
self.finish_view_probe();
actions.extend(
self.start_election(plane, ViewChangeReason::ViewProbeUnanswered),
@@ -2436,13 +2527,13 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
// incoming PrepareOk messages can be matched and commits can proceed.
if max_commit < new_op {
assert!(
- (new_op - max_commit) <= PIPELINE_PREPARE_QUEUE_MAX as u64,
+ (new_op - max_commit) <= self.prepare_queue_max as u64,
"view change: uncommitted range {}..={} ({} ops) exceeds pipeline capacity ({}); \
DVC winner claims more in-flight ops than the pipeline can hold",
max_commit + 1,
new_op,
new_op - max_commit,
- PIPELINE_PREPARE_QUEUE_MAX,
+ self.prepare_queue_max,
);
actions.push(VsrAction::RebuildPipeline {
from_op: max_commit + 1,
@@ -2554,7 +2645,7 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
// TODO: Route SVC/DVC self-messages through loopback once VsrAction dispatch is implemented.
pub(crate) fn push_loopback(&self, message: Message<GenericHeader>) {
assert!(
- self.loopback_queue.borrow().len() < PIPELINE_PREPARE_QUEUE_MAX,
+ self.loopback_queue.borrow().len() < self.prepare_queue_max,
"loopback queue overflow: {} items",
self.loopback_queue.borrow().len()
);
@@ -2949,6 +3040,38 @@ mod pipeline_entry_tests {
let mut entry = PipelineEntry::new(header);
assert!(entry.take_reply_sender().is_none());
}
+
+ /// A pipeline configured deeper than [`PIPELINE_PREPARE_QUEUE_MAX`] must
+ /// verify a full queue instead of tripping the capacity assert: the bound
+ /// tracks the configured depth, not the default const.
+ #[test]
+ #[allow(clippy::cast_possible_truncation)]
+ fn given_prepare_queue_depth_above_default_when_verify_should_not_panic() {
+ let depth = PIPELINE_PREPARE_QUEUE_MAX * 2;
+ let mut pipeline = LocalPipeline::with_capacities(depth, depth * 2);
+
+ let mut parent = 0u128;
+ for op in 1..=depth as u64 {
+ let checksum = u128::from(op);
+ let header = PrepareHeader {
+ command: Command2::Prepare,
+ size: std::mem::size_of::<PrepareHeader>() as u32,
+ op,
+ parent,
+ checksum,
+ ..Default::default()
+ };
+ pipeline.push(PipelineEntry::new(header));
+ parent = checksum;
+ }
+
+ assert!(
+ pipeline.prepare_queue_full(),
+ "queue filled to the configured depth"
+ );
+ // Would panic on the old `len() <= PIPELINE_PREPARE_QUEUE_MAX` assert.
+ pipeline.verify();
+ }
}
#[cfg(test)]
diff --git a/core/consensus/src/lib.rs b/core/consensus/src/lib.rs
index 9c68f17ac..50d9e2260 100644
--- a/core/consensus/src/lib.rs
+++ b/core/consensus/src/lib.rs
@@ -52,6 +52,11 @@ pub trait Pipeline {
fn len(&self) -> usize;
+ /// In-flight prepare-queue capacity. `VsrConsensus` snapshots it at
+ /// construction to size the loopback queue and to bound the uncommitted
+ /// range a new primary may rebuild after a view change.
+ fn prepare_queue_max(&self) -> usize;
+
fn verify(&self);
/// True iff either queue carries `client_id`. Used by metadata-plane
diff --git a/core/consensus/src/plane_helpers.rs b/core/consensus/src/plane_helpers.rs
index e385566a4..f9b36e554 100644
--- a/core/consensus/src/plane_helpers.rs
+++ b/core/consensus/src/plane_helpers.rs
@@ -683,9 +683,9 @@ mod tests {
consensus.init();
// Diverge the frontiers: applied (commit_min=5) lags known-committed
- // (commit_max=13) by more than PIPELINE_PREPARE_QUEUE_MAX (8). op is at
- // 13 (>= commit_max), so the op clamp on the DVC commit is a no-op here
- // and the carried value is commit_max. The clamp itself is covered by
+ // (commit_max=13). op is at 13 (>= commit_max), so the op clamp on the
+ // DVC commit is a no-op here and the carried value is commit_max. The
+ // clamp itself is covered by
// `do_view_change_commit_clamped_to_op_when_commit_max_exceeds_op`.
consensus.advance_commit_max(13);
consensus.sequencer().set_sequence(13);
@@ -911,6 +911,74 @@ mod tests {
);
}
+ /// A DVC winner may claim an uncommitted range up to the *configured*
+ /// prepare depth. With a pipeline deeper than the default const, the new
+ /// primary schedules the rebuild rather than panicking on the old
+ /// `PIPELINE_PREPARE_QUEUE_MAX` bound.
+ #[test]
+ #[allow(clippy::cast_possible_truncation)]
+ fn given_view_change_range_above_default_when_complete_as_primary_should_rebuild() {
+ use iggy_binary_protocol::{DoViewChangeHeader, StartViewChangeHeader};
+
+ let depth = crate::PIPELINE_PREPARE_QUEUE_MAX * 2;
+ // Strictly above the default const, still within the configured depth.
+ let winner_op = (crate::PIPELINE_PREPARE_QUEUE_MAX + 8) as u64;
+
+ // 3 replicas, replica 0 is primary for view 3 (3 % 3 = 0).
+ let consensus = VsrConsensus::new(
+ 1,
+ 0,
+ 3,
+ 0,
+ NoopBus,
+ LocalPipeline::with_capacities(depth, depth * 2),
+ );
+ consensus.init();
+
+ // SVC from replica 1 moves replica 0 into view 3 and records its own DVC.
+ let svc = StartViewChangeHeader {
+ checksum: 0,
+ checksum_body: 0,
+ cluster: 0,
+ size: 0,
+ view: 3,
+ release: 0,
+ command: Command2::StartViewChange,
+ replica: 1,
+ reserved_frame: [0; 66],
+ namespace: 0,
+ reserved: [0; 120],
+ };
+ let _ = consensus.handle_start_view_change(PlaneKind::Metadata, &svc);
+
+ // DVC from replica 2 claims a log head far past commit, forming quorum.
+ let dvc = DoViewChangeHeader {
+ checksum: 0,
+ checksum_body: 0,
+ cluster: 0,
+ size: 0,
+ view: 3,
+ release: 0,
+ command: Command2::DoViewChange,
+ replica: 2,
+ reserved_frame: [0; 66],
+ op: winner_op,
+ commit: 0,
+ namespace: 0,
+ log_view: 0,
+ reserved: [0; 100],
+ };
+ let actions = consensus.handle_do_view_change(PlaneKind::Metadata, &dvc);
+
+ assert!(
+ actions.iter().any(|action| matches!(
+ action,
+ VsrAction::RebuildPipeline { from_op: 1, to_op } if *to_op == winner_op
+ )),
+ "expected RebuildPipeline over the full uncommitted range"
+ );
+ }
+
#[test]
fn send_prepare_ok_sends_to_bus_when_not_primary() {
// Replica 1, view 0; primary=0, so send_or_loopback takes bus path.
diff --git a/core/consensus/src/vsr_timeout.rs b/core/consensus/src/vsr_timeout.rs
index 50e6482c8..aa5d5f6e0 100644
--- a/core/consensus/src/vsr_timeout.rs
+++ b/core/consensus/src/vsr_timeout.rs
@@ -99,7 +99,6 @@ impl Timeout {
#[allow(unused)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum TimeoutKind {
- Ping,
Prepare,
CommitMessage,
NormalHeartbeat,
@@ -115,7 +114,6 @@ pub enum TimeoutKind {
#[allow(unused)]
#[derive(Debug)]
pub struct TimeoutManager {
- ping: Timeout,
prepare: Timeout,
commit_message: Timeout,
normal_heartbeat: Timeout,
@@ -130,21 +128,36 @@ pub struct TimeoutManager {
impl TimeoutManager {
// Timeout durations in ticks (10ms per tick).
// TODO define 10ms per tick in a separate constant.
- const PING_TICKS: u64 = 100;
- const PREPARE_TICKS: u64 = 25;
- const COMMIT_MESSAGE_TICKS: u64 = 50;
+ /// Public so the runtime can pin its `[cluster] prepare_retransmit_interval`
+ /// config default against this built-in.
+ pub const PREPARE_TICKS: u64 = 25;
+ /// Public so the runtime can pin its `[cluster] commit_broadcast_interval`
+ /// config default against this built-in.
+ pub const COMMIT_MESSAGE_TICKS: u64 = 50;
/// Public so the runtime can pin its config default (`[cluster]
/// heartbeat_timeout`) against this built-in with a static assert.
pub const NORMAL_HEARTBEAT_TICKS: u64 = 500;
- const START_VIEW_CHANGE_MESSAGE_TICKS: u64 = 50;
- const VIEW_CHANGE_STATUS_TICKS: u64 = 500;
- const DO_VIEW_CHANGE_MESSAGE_TICKS: u64 = 50;
- const REQUEST_START_VIEW_MESSAGE_TICKS: u64 = 100;
+ /// Public so the runtime can pin its `[cluster]
+ /// view_change_retransmit_interval` config default against this built-in.
+ /// Deliberately equal to [`Self::DO_VIEW_CHANGE_MESSAGE_TICKS`]: one config
+ /// knob drives both retransmit timers.
+ pub const START_VIEW_CHANGE_MESSAGE_TICKS: u64 = 50;
+ /// Public so the runtime can pin its `[cluster] view_change_status_timeout`
+ /// config default against this built-in.
+ pub const VIEW_CHANGE_STATUS_TICKS: u64 = 500;
+ /// Public so the runtime can pin its `[cluster]
+ /// view_change_retransmit_interval` config default against this built-in.
+ /// Deliberately equal to [`Self::START_VIEW_CHANGE_MESSAGE_TICKS`]: one
+ /// config knob drives both retransmit timers.
+ pub const DO_VIEW_CHANGE_MESSAGE_TICKS: u64 = 50;
+ /// Public so the runtime can pin its `[cluster]
+ /// request_start_view_retransmit_interval` config default against this
+ /// built-in.
+ pub const REQUEST_START_VIEW_MESSAGE_TICKS: u64 = 100;
#[must_use]
pub fn new(replica_id: u128) -> Self {
Self {
- ping: Timeout::new(replica_id, Self::PING_TICKS),
prepare: Timeout::new(replica_id, Self::PREPARE_TICKS),
commit_message: Timeout::new(replica_id, Self::COMMIT_MESSAGE_TICKS),
normal_heartbeat: Timeout::new(replica_id, Self::NORMAL_HEARTBEAT_TICKS),
@@ -172,11 +185,56 @@ impl TimeoutManager {
self.normal_heartbeat = Timeout::new(self.normal_heartbeat.id, ticks);
}
+ /// Override the primary's commit-broadcast interval (`[cluster]
+ /// commit_broadcast_interval`). Replaces the timeout object, so any
+ /// countdown already in flight is discarded: call before the replica
+ /// starts ticking (i.e. before `init`).
+ pub const fn set_commit_message_ticks(&mut self, ticks: u64) {
+ self.commit_message = Timeout::new(self.commit_message.id, ticks);
+ }
+
+ /// Override the primary's prepare-retransmit interval (`[cluster]
+ /// prepare_retransmit_interval`). Replaces the timeout object, so any
+ /// countdown already in flight is discarded: call before the replica
+ /// starts ticking (i.e. before `init`).
+ pub const fn set_prepare_ticks(&mut self, ticks: u64) {
+ self.prepare = Timeout::new(self.prepare.id, ticks);
+ }
+
+ /// Override the view-change retransmit interval (`[cluster]
+ /// view_change_retransmit_interval`), in consensus ticks. Drives BOTH the
+ /// `StartViewChange` and `DoViewChange` retransmit timers - kept equal by
+ /// design so a view change retransmits both messages at one cadence.
+ /// Replaces the timeout objects, so any countdown already in flight is
+ /// discarded: call before the replica starts ticking (i.e. before `init`).
+ pub const fn set_view_change_retransmit_ticks(&mut self, ticks: u64) {
+ self.start_view_change_message = Timeout::new(self.start_view_change_message.id, ticks);
+ self.do_view_change_message = Timeout::new(self.do_view_change_message.id, ticks);
+ }
+
+ /// Override the view-change status backstop (`[cluster]
+ /// view_change_status_timeout`), in consensus ticks. A view change stalled
+ /// this long escalates to a fresh cluster-wide election. Replaces the
+ /// timeout object, so any countdown already in flight is discarded: call
+ /// before the replica starts ticking (i.e. before `init`).
+ pub const fn set_view_change_status_ticks(&mut self, ticks: u64) {
+ self.view_change_status = Timeout::new(self.view_change_status.id, ticks);
+ }
+
+ /// Override the request-start-view retransmit interval (`[cluster]
+ /// request_start_view_retransmit_interval`), in consensus ticks: how often
+ /// a recovering or view-change backup re-requests the current view's
+ /// `StartView`. Replaces the timeout object, so any countdown already in
+ /// flight is discarded: call before the replica starts ticking (i.e. before
+ /// `init`).
+ pub const fn set_request_start_view_ticks(&mut self, ticks: u64) {
+ self.request_start_view_message = Timeout::new(self.request_start_view_message.id, ticks);
+ }
+
/// Tick all timeouts
/// This is the first phase of the two-phase tick-based timeout mechanism.
/// 2nd phase is checking which timeouts have fired and calling the appropriate handlers.
pub const fn tick(&mut self) {
- self.ping.tick();
self.prepare.tick();
self.commit_message.tick();
self.normal_heartbeat.tick();
@@ -194,7 +252,6 @@ impl TimeoutManager {
#[must_use]
pub const fn get(&self, kind: TimeoutKind) -> &Timeout {
match kind {
- TimeoutKind::Ping => &self.ping,
TimeoutKind::Prepare => &self.prepare,
TimeoutKind::CommitMessage => &self.commit_message,
TimeoutKind::NormalHeartbeat => &self.normal_heartbeat,
@@ -207,7 +264,6 @@ impl TimeoutManager {
pub const fn get_mut(&mut self, kind: TimeoutKind) -> &mut Timeout {
match kind {
- TimeoutKind::Ping => &mut self.ping,
TimeoutKind::Prepare => &mut self.prepare,
TimeoutKind::CommitMessage => &mut self.commit_message,
TimeoutKind::NormalHeartbeat => &mut self.normal_heartbeat,
@@ -232,7 +288,6 @@ impl TimeoutManager {
pub fn backoff(&mut self, kind: TimeoutKind) {
let timeout = match kind {
- TimeoutKind::Ping => &mut self.ping,
TimeoutKind::Prepare => &mut self.prepare,
TimeoutKind::CommitMessage => &mut self.commit_message,
TimeoutKind::NormalHeartbeat => &mut self.normal_heartbeat,
diff --git a/core/integration/tests/server/scenarios/reconnect_after_restart_scenario.rs b/core/integration/tests/server/scenarios/reconnect_after_restart_scenario.rs
index e994e5d44..bc2da26a7 100644
--- a/core/integration/tests/server/scenarios/reconnect_after_restart_scenario.rs
+++ b/core/integration/tests/server/scenarios/reconnect_after_restart_scenario.rs
@@ -621,6 +621,15 @@ async fn poll_from_zero_until(
pub async fn run_ring_overflow_rejoin(harness: &mut TestHarness) {
const RING_OVERFLOW_OPS: u32 = 4300;
const POST_RESTART_OPS: u32 = 50;
+ // The point of this scenario is overflowing the peers' evicted ring so
+ // RangeEvicted and the commit floor engage. The harness runs the server on
+ // the shipped default ring capacity; if that default ever reaches this op
+ // count the overflow stops happening and the test silently passes without
+ // covering the floor path. Fail loud instead.
+ const _: () = assert!(
+ RING_OVERFLOW_OPS as usize > configs::ng_partition::DEFAULT_EVICTED_RING_CAPACITY,
+ "RING_OVERFLOW_OPS must exceed the default evicted ring capacity, or this scenario no longer exercises RangeEvicted",
+ );
let setup_client = harness
.root_client()
diff --git a/core/message_bus/src/cache/connection.rs b/core/message_bus/src/cache/connection.rs
index f4373f7d3..8aded4271 100644
--- a/core/message_bus/src/cache/connection.rs
+++ b/core/message_bus/src/cache/connection.rs
@@ -22,8 +22,6 @@ use rand::seq::SliceRandom;
use std::cell::{Cell, RefCell};
use std::collections::{HashMap, HashSet};
-pub const DEFAULT_MAX_CONNECTIONS_PER_REPLICA: usize = 8;
-
pub trait ShardedState {
type Entry;
type Delta;
diff --git a/core/message_bus/src/client_listener/mod.rs b/core/message_bus/src/client_listener/mod.rs
index 2503be876..8c65a6a16 100644
--- a/core/message_bus/src/client_listener/mod.rs
+++ b/core/message_bus/src/client_listener/mod.rs
@@ -130,7 +130,7 @@ pub async fn bind_nodelay_listener(
.await
.map_err(|_| IggyError::CannotBindToSocket(addr.to_string()))?;
let listener = socket
- .listen(128)
+ .listen(libc::SOMAXCONN)
.await
.map_err(|_| IggyError::CannotBindToSocket(addr.to_string()))?;
let actual = listener
diff --git a/core/metadata/src/impls/metadata.rs b/core/metadata/src/impls/metadata.rs
index 7ba69922b..40cba9857 100644
--- a/core/metadata/src/impls/metadata.rs
+++ b/core/metadata/src/impls/metadata.rs
@@ -563,6 +563,14 @@ impl<C, J, S, M> IggyMetadata<C, J, S, M> {
}
}
+ /// Size the VSR client table to `[metadata] clients_table_max`
+ /// (see [`ClientTable::set_capacity`]). Boot-only, before any client
+ /// registers; server-ng bootstrap applies it alongside
+ /// [`Self::set_checkpoint_margin`].
+ pub fn set_clients_table_max(&self, max_clients: usize) {
+ self.client_table.borrow_mut().set_capacity(max_clients);
+ }
+
/// Resolved byte value for `MaxTopicSize::ServerDefault`.
#[must_use]
pub const fn default_max_topic_size(&self) -> u64 {
diff --git a/core/partitions/src/journal.rs b/core/partitions/src/journal.rs
index feb78c24b..fb24564f4 100644
--- a/core/partitions/src/journal.rs
+++ b/core/partitions/src/journal.rs
@@ -177,6 +177,14 @@ where
evicted_ring: UnsafeCell<VecDeque<(u64, JournalBuffer)>>,
/// Running byte total of the buffers held by `evicted_ring`.
evicted_ring_bytes: Cell<u64>,
+ /// Entry-count ceiling for `evicted_ring`. Defaults to
+ /// [`EVICTED_RING_CAPACITY`]; server-ng overrides it from config at
+ /// partition build.
+ evicted_ring_capacity: Cell<usize>,
+ /// Byte ceiling for `evicted_ring`. Defaults to
+ /// [`EVICTED_RING_BYTES_MAX`]; server-ng overrides it from config at
+ /// partition build.
+ evicted_ring_bytes_max: Cell<u64>,
/// Single-replica groups have nobody to repair; retaining evicted
/// entries for them is pure memory waste.
repair_retention: Cell<bool>,
@@ -207,6 +215,8 @@ where
}),
evicted_ring: UnsafeCell::new(VecDeque::new()),
evicted_ring_bytes: Cell::new(0),
+ evicted_ring_capacity: Cell::new(EVICTED_RING_CAPACITY),
+ evicted_ring_bytes_max: Cell::new(EVICTED_RING_BYTES_MAX),
repair_retention: Cell::new(true),
}
}
@@ -286,6 +296,15 @@ impl PartitionJournal<PartitionJournalMemStorage> {
}
}
+ /// Override the evicted-ring ceilings from configuration. Called once at
+ /// partition build, before any eviction, so the caps govern the first
+ /// flush onward. Leaves `repair_retention` untouched: the single-replica
+ /// disable path stands on its own.
+ pub fn set_ring_caps(&self, capacity: usize, bytes_max: u64) {
+ self.evicted_ring_capacity.set(capacity);
+ self.evicted_ring_bytes_max.set(bytes_max);
+ }
+
/// Resident (un-evicted) entry count; diagnostics only.
pub fn resident_count(&self) -> usize {
let op_to_storage_offset = unsafe { &*self.op_to_storage_offset.get() };
@@ -466,8 +485,8 @@ impl PartitionJournal<PartitionJournalMemStorage> {
};
ring_bytes += entry.len() as u64;
ring.push_back((op, entry));
- while ring.len() > EVICTED_RING_CAPACITY
- || (ring_bytes > EVICTED_RING_BYTES_MAX && ring.len() > 1)
+ while ring.len() > self.evicted_ring_capacity.get()
+ || (ring_bytes > self.evicted_ring_bytes_max.get() && ring.len() > 1)
{
if let Some((_, dropped)) = ring.pop_front() {
ring_bytes -= dropped.len() as u64;
@@ -597,6 +616,8 @@ where
inner: UnsafeCell::new(JournalInner { storage }),
evicted_ring: UnsafeCell::new(VecDeque::new()),
evicted_ring_bytes: Cell::new(0),
+ evicted_ring_capacity: Cell::new(EVICTED_RING_CAPACITY),
+ evicted_ring_bytes_max: Cell::new(EVICTED_RING_BYTES_MAX),
repair_retention: Cell::new(true),
}
}
diff --git a/core/partitions/src/lib.rs b/core/partitions/src/lib.rs
index 0abd137d1..33e33d02e 100644
--- a/core/partitions/src/lib.rs
+++ b/core/partitions/src/lib.rs
@@ -37,6 +37,7 @@ pub use iggy_index_reader::IggyIndexReader;
pub use iggy_index_writer::IggyIndexWriter;
pub use iggy_partition::IggyPartition;
pub use iggy_partitions::IggyPartitions;
+pub use journal::{EVICTED_RING_BYTES_MAX, EVICTED_RING_CAPACITY};
pub use messages_writer::MessagesWriter;
pub use offset_storage::delete_persisted_offset;
pub use poll_plan::{AutoCommitApplied, PollPlan};
diff --git a/core/server-ng/Cargo.toml b/core/server-ng/Cargo.toml
index 6659db887..d710066d5 100644
--- a/core/server-ng/Cargo.toml
+++ b/core/server-ng/Cargo.toml
@@ -88,7 +88,10 @@ default = ["mimalloc", "iggy-web"]
disable-mimalloc = []
mimalloc = ["dep:mimalloc"]
iggy-web = ["dep:rust-embed", "dep:mime_guess"]
-vsr = ["iggy_common/vsr"]
+# forward iggy/vsr: the SDK gates its VsrSessionControl impls on its own
+# feature while the trait surface follows iggy_common/vsr; without the
+# forward a bare `-p server-ng --features vsr` build breaks in the SDK
+vsr = ["iggy_common/vsr", "iggy/vsr"]
[dependencies]
ahash = { workspace = true }
diff --git a/core/server-ng/config.toml b/core/server-ng/config.toml
index 7a185a80d..b8fc909ba 100644
--- a/core/server-ng/config.toml
+++ b/core/server-ng/config.toml
@@ -183,6 +183,20 @@ cert_file = "core/certs/iggy_cert.pem"
# Path to the TLS key file.
key_file = "core/certs/iggy_key.pem"
+# Admission caps for shard-0 HTTP awaited partition writes (produce and
+# consumer-offset writes that park a handler and pin a buffered body until the
+# write commits). These bound how much buffered memory and consensus-pump delay
+# concurrent HTTP writes can impose on shard 0.
+[http_admission]
+# Shard-0-global budget for concurrently awaited partition writes across every
+# session. Requests past it are refused with a server-busy 503. Must be > 0.
+max_in_flight_writes = 128
+
+# Per-session cap on concurrently awaited partition writes, so one credential
+# that outruns its own commits gets a too-many-in-flight 429 before it can
+# starve the shared budget. Must be > 0 and must not exceed max_in_flight_writes.
+max_in_flight_writes_per_session = 32
+
# TCP server configuration.
[tcp]
# Determines if the TCP server is active.
@@ -590,10 +604,59 @@ name = "iggy-cluster"
# Backup-side liveness window for a consensus plane's primary (duration).
# A replica that sees no primary traffic for this long starts a view change.
# Raise it on oversubscribed hosts where scheduling stalls fake primary
-# death. Must be at least "2s": the primary heartbeats every second, and a
-# window at or below one ping interval would elect on every hiccup.
+# death. Must be at least "2s" and at least 4x commit_broadcast_interval: the
+# primary signals liveness through its commit broadcast, and the window must
+# span several broadcasts so one delayed broadcast never trips an election.
heartbeat_timeout = "5s"
+# How often the primary broadcasts its commit point to every backup (duration).
+# This is the cluster's liveness signal: each broadcast resets every backup's
+# heartbeat_timeout window and carries the latest commit point forward. Must be
+# nonzero and, with heartbeat_timeout, satisfy heartbeat_timeout >= 4x this
+# value. Drives the consensus CommitMessage timer.
+commit_broadcast_interval = "500ms"
+
+# How often the primary retransmits prepares that backups have not yet acked
+# (duration). Lower values recover faster from a dropped prepare at the cost of
+# more replica traffic; must be nonzero. Drives the consensus Prepare timer.
+prepare_retransmit_interval = "250ms"
+
+# How often a replica retransmits its StartViewChange / DoViewChange while a
+# view change is in progress (duration). Lower values converge a healthy
+# election faster at the cost of more replica traffic; must be nonzero. Drives
+# both consensus view-change retransmit timers.
+view_change_retransmit_interval = "500ms"
+
+# Backstop for a stalled view change (duration): one that does not conclude
+# within this window escalates to a fresh cluster-wide election. Must be nonzero
+# and at least 4x view_change_retransmit_interval, so a few dropped view-change
+# messages retransmit rather than prematurely escalate.
+view_change_status_timeout = "5s"
+
+# How often a recovering or view-change backup re-requests the current view's
+# StartView from its primary (duration); must be nonzero. Drives the consensus
+# RequestStartView timer.
+request_start_view_retransmit_interval = "1s"
+
+# How many consecutive unanswered RequestStartView probes a recovering replica
+# tolerates before falling back to an election (integer). A full-cluster restart
+# leaves nobody settled to answer, so the replica elects on its recovered log.
+# Must be between 1 and 100.
+view_probe_attempts_max = 5
+
+# How long a stalled journal-repair stream waits before re-requesting its
+# remaining window from the serving peer (duration). Repair frames are
+# fire-and-forget over the lossy bus, so a session with no retry wedges forever
+# on a single dropped frame. Paces both the metadata and partition repair loops;
+# must be nonzero.
+repair_retry_interval = "1s"
+
+# Prepares a peer serves per repair round before the requester walks to the next
+# chunk (integer). Each frame rides the per-peer message-bus queue, so this must
+# stay strictly below message_bus.peer_queue_capacity or a full round overruns
+# the queue and drops frames. Must be > 0 and <= 1024.
+repair_chunk_max = 128
+
# Replica-to-replica authentication (PSK + BLAKE3 keyed-MAC handshake).
[cluster.auth]
# When true, every replica peer must complete the authenticated handshake or be
@@ -735,6 +798,35 @@ prepare_queue_depth = 32
# rewrites per checkpoint.
journal_slots = 1024
+# Slot count of the VSR client table: how many distinct clients (TCP/QUIC/WS
+# virtual clients and HTTP sessions together) hold live session state at once.
+# When full, the client whose last commit is oldest is evicted and its next
+# request re-registers. The HTTP session cap tracks this at half, so raising
+# it lifts both. Must be between 2 and 65536.
+clients_table_max = 8192
+
+# Per-partition consensus plane tunables. Unlike [metadata] (one shard-0
+# plane), a pipeline exists per partition, so raising this multiplies pinned
+# request-buffer memory by the partition count. Keep it modest.
+[partition]
+# Depth of a partition's prepare queue: how many uncommitted produce /
+# consumer-offset ops may be in flight at once for that partition. Submits
+# beyond it are rejected with the transient prepare-queue-full path and
+# retried by the SDK. Must be > 0 and <= 256.
+prepare_queue_depth = 32
+
+# Entries the evicted ring retains per multi-replica partition for journal
+# repair after a peer rejoins. Larger widens the window a restarting peer can be
+# served from the ring before falling back to bulk sync, at the cost of pinned
+# memory per partition. Must be > 0 and <= 65536. Single-replica partitions
+# retain nothing regardless.
+evicted_ring_capacity = 4096
+
+# Byte ceiling for the evicted ring per partition; whichever ring cap (this or
+# evicted_ring_capacity) trips first evicts. Bounds the ring memory a burst of
+# large batches can pin. Must be > 0 and <= "256 MiB".
+evicted_ring_bytes_max = "16 MiB"
+
# Message bus configuration.
# Tunables for the inter-shard / inter-replica internal bus that ships
# consensus traffic between replicas and SDK-client traffic between
diff --git a/core/server-ng/src/bootstrap.rs b/core/server-ng/src/bootstrap.rs
index b7d578a3f..c7b59df25 100644
--- a/core/server-ng/src/bootstrap.rs
+++ b/core/server-ng/src/bootstrap.rs
@@ -958,6 +958,16 @@ async fn shard_main(
Rc::clone(&bus),
config.metadata.prepare_queue_depth,
cluster_heartbeat_ticks(config),
+ commit_broadcast_ticks(config),
+ prepare_retransmit_ticks(config),
+ view_change_retransmit_ticks(config),
+ view_change_status_ticks(config),
+ request_start_view_ticks(config),
+ config.cluster.view_probe_attempts_max,
+ recovery_barrier_deadline(
+ config.cluster.heartbeat_timeout.get_duration(),
+ config.cluster.view_change_status_timeout.get_duration(),
+ ),
);
(Some(consensus), Some(journal), snapshot)
} else {
@@ -978,6 +988,10 @@ async fn shard_main(
// depth: ops already pipelined while a checkpoint runs append into that
// margin (config validation keeps journal_slots >= 4x this).
metadata.set_checkpoint_margin(config.metadata.checkpoint_margin());
+ // Size the VSR client table before listeners bind and any client registers.
+ // The table is empty here on both fresh boot and restart (its slots are not
+ // restored from snapshot), which the setter's empty-table contract requires.
+ metadata.set_clients_table_max(config.metadata.clients_table_max);
let shard_metrics = ShardMetrics::for_shard();
// Notifier install deferred until after tick handler wires below.
@@ -1674,6 +1688,10 @@ async fn build_shard_for_thread(
.map_err(ServerNgError::ShardConstruction)?;
let shard = Rc::new(built.shard);
+ // Repair pacing is shared by both planes' repair loops, so it is a
+ // per-shard tunable set once here rather than per consensus group.
+ shard.set_repair_retry_ticks(repair_retry_ticks(config));
+ shard.set_repair_chunk_max(config.cluster.repair_chunk_max as u64);
*shard_handle.borrow_mut() = Some(Rc::downgrade(&shard));
Ok((shard, sessions))
}
@@ -1690,13 +1708,127 @@ const _: () = assert!(
configs::ng_metadata::DEFAULT_METADATA_JOURNAL_SLOTS
== journal::prepare_journal::DEFAULT_SLOT_COUNT
);
-/// `[cluster] heartbeat_timeout` in consensus ticks, floored at one tick.
-/// Every consensus group (metadata and per-partition planes alike) gets the
-/// same window: the failure it guards against - a primary that stopped
-/// heartbeating - is host-level, not per-plane.
+const _: () = assert!(
+ configs::ng_partition::DEFAULT_PARTITION_PREPARE_QUEUE_DEPTH
+ == consensus::PIPELINE_PREPARE_QUEUE_MAX
+);
+const _: () = assert!(
+ configs::ng_metadata::DEFAULT_METADATA_CLIENTS_TABLE_MAX == consensus::CLIENTS_TABLE_MAX
+);
+const _: () =
+ assert!(configs::ng_cluster::DEFAULT_VIEW_PROBE_ATTEMPTS_MAX == consensus::PROBE_ATTEMPTS_MAX);
+const _: () = assert!(
+ configs::ng_partition::DEFAULT_EVICTED_RING_CAPACITY == partitions::EVICTED_RING_CAPACITY
+);
+const _: () = assert!(
+ configs::ng_partition::DEFAULT_EVICTED_RING_BYTES_MAX == partitions::EVICTED_RING_BYTES_MAX
+);
+const _: () =
+ assert!(configs::ng_cluster::DEFAULT_REPAIR_CHUNK_MAX as u64 == shard::REPAIR_CHUNK_MAX);
+/// Convert a consensus-timer interval to whole ticks, floored at one tick so a
+/// sub-tick value still fires and saturated on overflow.
+fn duration_to_ticks(interval: Duration) -> u64 {
+ let ticks = interval.as_millis() / shard::CONSENSUS_TICK_INTERVAL.as_millis();
+ u64::try_from(ticks.max(1)).unwrap_or(u64::MAX)
+}
+
+/// `[cluster] heartbeat_timeout` in consensus ticks. Every consensus group
+/// (metadata and per-partition planes alike) gets the same window: the failure
+/// it guards against - a primary that stopped heartbeating - is host-level, not
+/// per-plane.
pub(crate) fn cluster_heartbeat_ticks(config: &ServerNgConfig) -> u64 {
- let window = config.cluster.heartbeat_timeout.get_duration().as_millis();
- u64::try_from((window / shard::CONSENSUS_TICK_INTERVAL.as_millis()).max(1)).unwrap_or(u64::MAX)
+ duration_to_ticks(config.cluster.heartbeat_timeout.get_duration())
+}
+
+/// Floor for the post-restart read-recovery deadline (see
+/// [`recovery_barrier_deadline`]). At and below the 5s default heartbeat the
+/// worst-case recovery is dominated by the heartbeat-independent term - the
+/// `ViewChangeStatus` backstop plus election ceremony and suffix recommit,
+/// empirically ~7s - so the scaled value must never fall under this or a
+/// fast-heartbeat cluster would 503 legitimate reads mid-recovery. The backstop
+/// is the configurable `[cluster] view_change_status_timeout`; raising it past
+/// its 5s default is why `recovery_barrier_deadline` scales that knob in too
+/// rather than leaning on this floor to cover it.
+const RECOVERY_BARRIER_DEADLINE_FLOOR: Duration = Duration::from_secs(15);
+
+/// Safety factor applied to each scaled term of the recovery deadline: a slower
+/// heartbeat stretches election and suffix recommit proportionally, and a wider
+/// status backstop stretches the ceremony it bounds. 3x reproduces the
+/// empirically chosen 15s margin at the shared 5s default (3 x 5s = 15s) and
+/// holds that factor as either knob grows.
+const RECOVERY_BARRIER_MULTIPLIER: u32 = 3;
+
+/// How long the post-restart read path waits for the recovered WAL suffix to
+/// re-commit before failing loud (retryable 503): the largest of the fixed
+/// floor, a `[cluster] heartbeat_timeout`-scaled window, and a
+/// `[cluster] view_change_status_timeout`-scaled window. Both knobs feed it
+/// because either, raised far past its default, stretches worst-case recovery
+/// past the fixed floor; see `await_recovery_barrier` for the read-side wait.
+pub(crate) fn recovery_barrier_deadline(
+ heartbeat: Duration,
+ view_change_status: Duration,
+) -> Duration {
+ // saturating: neither timeout has a config ceiling, plain `*` panics
+ heartbeat
+ .saturating_mul(RECOVERY_BARRIER_MULTIPLIER)
+ .max(view_change_status.saturating_mul(RECOVERY_BARRIER_MULTIPLIER))
+ .max(RECOVERY_BARRIER_DEADLINE_FLOOR)
+}
+
+/// `[cluster] commit_broadcast_interval` in consensus ticks: how often the
+/// primary broadcasts its commit point, the cluster's liveness feed. Applied
+/// to every consensus group, matching `cluster_heartbeat_ticks`.
+pub(crate) fn commit_broadcast_ticks(config: &ServerNgConfig) -> u64 {
+ duration_to_ticks(config.cluster.commit_broadcast_interval.get_duration())
+}
+
+/// `[cluster] prepare_retransmit_interval` in consensus ticks: how often the
+/// primary retransmits un-acked prepares. Applied to every consensus group,
+/// matching `cluster_heartbeat_ticks`.
+pub(crate) fn prepare_retransmit_ticks(config: &ServerNgConfig) -> u64 {
+ duration_to_ticks(config.cluster.prepare_retransmit_interval.get_duration())
+}
+
+/// `[cluster] view_change_retransmit_interval` in consensus ticks: how often a
+/// replica retransmits its `StartViewChange` / `DoViewChange` during a view
+/// change. Applied to every consensus group, matching `cluster_heartbeat_ticks`.
+pub(crate) fn view_change_retransmit_ticks(config: &ServerNgConfig) -> u64 {
+ duration_to_ticks(
+ config
+ .cluster
+ .view_change_retransmit_interval
+ .get_duration(),
+ )
+}
+
+/// `[cluster] view_change_status_timeout` in consensus ticks: the stalled
+/// view-change backstop before escalating to a fresh election. Applied to every
+/// consensus group, matching `cluster_heartbeat_ticks`.
+pub(crate) fn view_change_status_ticks(config: &ServerNgConfig) -> u64 {
+ duration_to_ticks(config.cluster.view_change_status_timeout.get_duration())
+}
+
+/// `[cluster] request_start_view_retransmit_interval` in consensus ticks: how
+/// often a recovering or view-change backup re-requests the current `StartView`.
+/// Applied to every consensus group, matching `cluster_heartbeat_ticks`.
+pub(crate) fn request_start_view_ticks(config: &ServerNgConfig) -> u64 {
+ duration_to_ticks(
+ config
+ .cluster
+ .request_start_view_retransmit_interval
+ .get_duration(),
+ )
+}
+
+/// `[cluster] repair_retry_interval` in consensus ticks: how long a stalled
+/// journal-repair stream waits before re-requesting its window. Both planes'
+/// repair loops share it, so it is applied once per shard (not per consensus
+/// group). Clamped to `u32`, the width of the session idle-tick counter.
+pub(crate) fn repair_retry_ticks(config: &ServerNgConfig) -> u32 {
+ u32::try_from(duration_to_ticks(
+ config.cluster.repair_retry_interval.get_duration(),
+ ))
+ .unwrap_or(u32::MAX)
}
#[allow(clippy::too_many_arguments)]
@@ -1710,6 +1842,13 @@ fn restore_metadata_consensus(
bus: Rc<IggyMessageBus>,
prepare_queue_depth: usize,
normal_heartbeat_ticks: u64,
+ commit_message_ticks: u64,
+ prepare_ticks: u64,
+ view_change_retransmit_ticks: u64,
+ view_change_status_ticks: u64,
+ request_start_view_ticks: u64,
+ probe_attempts_max: u32,
+ recovery_deadline: Duration,
) -> VsrConsensus<Rc<IggyMessageBus>> {
let mut consensus = VsrConsensus::new(
cluster_id,
@@ -1723,6 +1862,12 @@ fn restore_metadata_consensus(
LocalPipeline::with_capacities(prepare_queue_depth, prepare_queue_depth * 2),
);
consensus.set_normal_heartbeat_ticks(normal_heartbeat_ticks);
+ consensus.set_commit_message_ticks(commit_message_ticks);
+ consensus.set_prepare_ticks(prepare_ticks);
+ consensus.set_view_change_retransmit_ticks(view_change_retransmit_ticks);
+ consensus.set_view_change_status_ticks(view_change_status_ticks);
+ consensus.set_request_start_view_ticks(request_start_view_ticks);
+ consensus.set_probe_attempts_max(probe_attempts_max);
let last_header = journal
.last_op()
@@ -1787,10 +1932,11 @@ fn restore_metadata_consensus(
// primary; via StartView adoption + the local commit walk on a rejoined
// backup), serving reads would show pre-restart state that clients already
// saw acked -- gate them on the barrier regardless of role. If the suffix
- // never committed cluster-wide, the barrier times out on the read path and
- // serving resumes (`await_recovery_barrier`).
+ // never re-commits cluster-wide, the read path fails loud with a retryable
+ // 503 once the paired deadline expires (`await_recovery_barrier`).
if commit_watermark < restored_op {
consensus.set_recovery_barrier(restored_op);
+ consensus.set_recovery_deadline(recovery_deadline);
}
// Re-pipeline the prepared-but-uncommitted suffix so the primary's
@@ -1838,15 +1984,24 @@ async fn load_partition(
let stream_id = namespace.stream_id();
let topic_id = namespace.topic_id();
let partition_id = namespace.partition_id();
+ // Request queue holds 2x the prepare depth (buffered requests drain as
+ // prepares commit); depth is the per-partition `[partition]` knob.
+ let prepare_queue_depth = config.partition.prepare_queue_depth;
let consensus = VsrConsensus::new(
cluster_id,
self_replica_id,
replica_count,
namespace.inner(),
bus,
- LocalPipeline::new(),
+ LocalPipeline::with_capacities(prepare_queue_depth, prepare_queue_depth * 2),
);
consensus.set_normal_heartbeat_ticks(cluster_heartbeat_ticks(config));
+ consensus.set_commit_message_ticks(commit_broadcast_ticks(config));
+ consensus.set_prepare_ticks(prepare_retransmit_ticks(config));
+ consensus.set_view_change_retransmit_ticks(view_change_retransmit_ticks(config));
+ consensus.set_view_change_status_ticks(view_change_status_ticks(config));
+ consensus.set_request_start_view_ticks(request_start_view_ticks(config));
+ consensus.set_probe_attempts_max(config.cluster.view_probe_attempts_max);
// A recovered partition lost its consensus state with the process: the
// partition journal is in-memory and segments carry no op numbers, so
// this replica cannot know the group's (op, commit). In a cluster it
@@ -1886,6 +2041,13 @@ async fn load_partition(
})?;
let mut partition = IggyPartition::new(stats.clone(), consensus);
+ // Recovered partitions honor the same config-surfaced ring ceilings as the
+ // fresh-create path (build_partition_fresh). Retention is already off for
+ // single-replica groups, so this only sizes the multi-replica ring.
+ partition.log.journal().inner.set_ring_caps(
+ config.partition.evicted_ring_capacity,
+ config.partition.evicted_ring_bytes_max.as_bytes_u64(),
+ );
partition.set_partition_dir(config.system.get_partition_path(
stream_id,
topic_id,
@@ -2223,6 +2385,8 @@ async fn start_tcp_runtime(
shard,
http_addr,
&config.http,
+ &config.http_admission,
+ config.metadata.clients_table_max,
&config.cluster,
Arc::clone(&config.system),
self_ports,
@@ -3083,6 +3247,264 @@ mod tests {
);
}
+ #[test]
+ fn recovery_barrier_deadline_holds_the_floor_for_small_heartbeats() {
+ // Below the 5s default the heartbeat-independent recovery term (~7s of
+ // ViewChangeStatus backstop plus ceremony) dominates, so the floor
+ // governs however small the heartbeat is; 3 x 5s lands exactly on it.
+ // A default-sized status backstop stays on the floor, not above it.
+ assert_eq!(
+ recovery_barrier_deadline(Duration::from_secs(1), Duration::from_secs(5)),
+ RECOVERY_BARRIER_DEADLINE_FLOOR
+ );
+ assert_eq!(
+ recovery_barrier_deadline(Duration::from_secs(5), Duration::from_secs(5)),
+ RECOVERY_BARRIER_DEADLINE_FLOOR
+ );
+ }
+
+ #[test]
+ fn recovery_barrier_deadline_scales_past_the_floor_for_large_heartbeats() {
+ // Once 3 x heartbeat clears the floor the scaled window governs, so a
+ // slow-heartbeat cluster is not failed 503 before its longer recovery
+ // can finish. A default-sized status backstop stays under it.
+ assert_eq!(
+ recovery_barrier_deadline(Duration::from_secs(10), Duration::from_secs(5)),
+ Duration::from_secs(30)
+ );
+ assert_eq!(
+ recovery_barrier_deadline(Duration::from_secs(15), Duration::from_secs(5)),
+ Duration::from_secs(45)
+ );
+ }
+
+ #[test]
+ fn recovery_barrier_deadline_scales_with_the_status_backstop() {
+ // A raised view-change status backstop stretches worst-case recovery
+ // even when the heartbeat stays fast, so the deadline must track it or
+ // post-restart reads 503 before a slow election settles.
+ assert_eq!(
+ recovery_barrier_deadline(Duration::from_secs(1), Duration::from_secs(10)),
+ Duration::from_secs(30)
+ );
+ }
+
+ #[test]
+ fn recovery_barrier_deadline_at_config_defaults_matches_the_floor() {
+ // Folding the status term in must not move the stock deadline: at the
+ // shared 5s defaults each scaled term lands exactly on the 15s floor,
+ // so an un-tuned cluster keeps its pre-existing recovery window.
+ let cluster = configs::ng_cluster::ClusterConfig::default();
+ assert_eq!(
+ recovery_barrier_deadline(
+ cluster.heartbeat_timeout.get_duration(),
+ cluster.view_change_status_timeout.get_duration(),
+ ),
+ RECOVERY_BARRIER_DEADLINE_FLOOR
+ );
+ }
+
+ #[test]
+ fn recovery_barrier_deadline_saturates_instead_of_panicking() {
+ // Neither timeout has a config ceiling, so both multiplies must
+ // saturate rather than abort boot on an absurd parseable value.
+ assert_eq!(
+ recovery_barrier_deadline(Duration::MAX, Duration::from_secs(5)),
+ Duration::MAX
+ );
+ assert_eq!(
+ recovery_barrier_deadline(Duration::from_secs(5), Duration::MAX),
+ Duration::MAX
+ );
+ }
+
+ #[test]
+ fn default_commit_broadcast_interval_matches_consensus_constant() {
+ // The config default lives in core/server-ng/config.toml (a string,
+ // so no static assert can pin it); keep it in lockstep with the
+ // built-in the simulator and un-configured replicas run on.
+ let config_default = configs::ng_cluster::ClusterConfig::default()
+ .commit_broadcast_interval
+ .get_duration()
+ .as_millis();
+ let built_in = u128::from(consensus::TimeoutManager::COMMIT_MESSAGE_TICKS)
+ * shard::CONSENSUS_TICK_INTERVAL.as_millis();
+ assert_eq!(
+ config_default, built_in,
+ "[cluster] commit_broadcast_interval default drifted from \
+ TimeoutManager::COMMIT_MESSAGE_TICKS"
+ );
+ }
+
+ #[test]
+ fn default_prepare_retransmit_interval_matches_consensus_constant() {
+ // The config default lives in core/server-ng/config.toml (a string,
+ // so no static assert can pin it); keep it in lockstep with the
+ // built-in the simulator and un-configured replicas run on.
+ let config_default = configs::ng_cluster::ClusterConfig::default()
+ .prepare_retransmit_interval
+ .get_duration()
+ .as_millis();
+ let built_in = u128::from(consensus::TimeoutManager::PREPARE_TICKS)
+ * shard::CONSENSUS_TICK_INTERVAL.as_millis();
+ assert_eq!(
+ config_default, built_in,
+ "[cluster] prepare_retransmit_interval default drifted from \
+ TimeoutManager::PREPARE_TICKS"
+ );
+ }
+
+ #[test]
+ fn default_partition_prepare_queue_depth_matches_consensus_constant() {
+ // The config default lives in core/server-ng/config.toml and flows
+ // through PartitionConfig::default(); keep the embedded value in
+ // lockstep with the pipeline depth LocalPipeline::new() (the simulator
+ // and tests) runs on, so a default deployment is byte-identical.
+ let config_default = configs::ng_partition::PartitionConfig::default().prepare_queue_depth;
+ assert_eq!(
+ config_default,
+ consensus::PIPELINE_PREPARE_QUEUE_MAX,
+ "[partition] prepare_queue_depth default drifted from \
+ consensus::PIPELINE_PREPARE_QUEUE_MAX"
+ );
+ }
+
+ #[test]
+ fn default_view_change_retransmit_interval_matches_consensus_constant() {
+ // The config default lives in core/server-ng/config.toml (a string, so
+ // no static assert can pin it). One knob drives both view-change
+ // retransmit timers, which are equal by design, so pin it against both.
+ let config_default = configs::ng_cluster::ClusterConfig::default()
+ .view_change_retransmit_interval
+ .get_duration()
+ .as_millis();
+ let start_view_change =
+ u128::from(consensus::TimeoutManager::START_VIEW_CHANGE_MESSAGE_TICKS)
+ * shard::CONSENSUS_TICK_INTERVAL.as_millis();
+ let do_view_change = u128::from(consensus::TimeoutManager::DO_VIEW_CHANGE_MESSAGE_TICKS)
+ * shard::CONSENSUS_TICK_INTERVAL.as_millis();
+ assert_eq!(
+ config_default, start_view_change,
+ "[cluster] view_change_retransmit_interval default drifted from \
+ TimeoutManager::START_VIEW_CHANGE_MESSAGE_TICKS"
+ );
+ assert_eq!(
+ config_default, do_view_change,
+ "[cluster] view_change_retransmit_interval default drifted from \
+ TimeoutManager::DO_VIEW_CHANGE_MESSAGE_TICKS"
+ );
+ }
+
+ #[test]
+ fn default_view_change_status_timeout_matches_consensus_constant() {
+ // The config default lives in core/server-ng/config.toml (a string, so
+ // no static assert can pin it); keep it in lockstep with the built-in
+ // the simulator and un-configured replicas run on.
+ let config_default = configs::ng_cluster::ClusterConfig::default()
+ .view_change_status_timeout
+ .get_duration()
+ .as_millis();
+ let built_in = u128::from(consensus::TimeoutManager::VIEW_CHANGE_STATUS_TICKS)
+ * shard::CONSENSUS_TICK_INTERVAL.as_millis();
+ assert_eq!(
+ config_default, built_in,
+ "[cluster] view_change_status_timeout default drifted from \
+ TimeoutManager::VIEW_CHANGE_STATUS_TICKS"
+ );
+ }
+
+ #[test]
+ fn default_request_start_view_retransmit_interval_matches_consensus_constant() {
+ // The config default lives in core/server-ng/config.toml (a string, so
+ // no static assert can pin it); keep it in lockstep with the built-in
+ // the simulator and un-configured replicas run on.
+ let config_default = configs::ng_cluster::ClusterConfig::default()
+ .request_start_view_retransmit_interval
+ .get_duration()
+ .as_millis();
+ let built_in = u128::from(consensus::TimeoutManager::REQUEST_START_VIEW_MESSAGE_TICKS)
+ * shard::CONSENSUS_TICK_INTERVAL.as_millis();
+ assert_eq!(
+ config_default, built_in,
+ "[cluster] request_start_view_retransmit_interval default drifted from \
+ TimeoutManager::REQUEST_START_VIEW_MESSAGE_TICKS"
+ );
+ }
+
+ #[test]
+ fn default_view_probe_attempts_max_matches_consensus_constant() {
+ // Belt and suspenders with the static assert above: that pins the
+ // duplicated configs-crate literal, this pins the shipped config.toml
+ // value the simulator and un-configured replicas run on.
+ let config_default = configs::ng_cluster::ClusterConfig::default().view_probe_attempts_max;
+ assert_eq!(
+ config_default,
+ consensus::PROBE_ATTEMPTS_MAX,
+ "[cluster] view_probe_attempts_max default drifted from \
+ consensus::PROBE_ATTEMPTS_MAX"
+ );
+ }
+
+ #[test]
+ fn default_repair_retry_interval_matches_partitions_constant() {
+ // The config default lives in core/server-ng/config.toml (a string, so
+ // no static assert can pin it); keep it in lockstep with the built-in
+ // the simulator and un-configured replicas run on.
+ let config_default = configs::ng_cluster::ClusterConfig::default()
+ .repair_retry_interval
+ .get_duration()
+ .as_millis();
+ let built_in =
+ u128::from(partitions::REPAIR_RETRY_TICKS) * shard::CONSENSUS_TICK_INTERVAL.as_millis();
+ assert_eq!(
+ config_default, built_in,
+ "[cluster] repair_retry_interval default drifted from \
+ partitions::REPAIR_RETRY_TICKS"
+ );
+ }
+
+ #[test]
+ fn default_repair_chunk_max_matches_shard_constant() {
+ // Belt and suspenders with the static assert above: that pins the
+ // duplicated configs-crate literal, this pins the shipped config.toml
+ // value the simulator and un-configured replicas run on.
+ let config_default = configs::ng_cluster::ClusterConfig::default().repair_chunk_max;
+ assert_eq!(
+ config_default as u64,
+ shard::REPAIR_CHUNK_MAX,
+ "[cluster] repair_chunk_max default drifted from shard::REPAIR_CHUNK_MAX"
+ );
+ }
+
+ #[test]
+ fn default_evicted_ring_capacity_matches_partitions_constant() {
+ // Belt and suspenders with the static assert above; this pins the
+ // shipped config.toml value.
+ let config_default =
+ configs::ng_partition::PartitionConfig::default().evicted_ring_capacity;
+ assert_eq!(
+ config_default,
+ partitions::EVICTED_RING_CAPACITY,
+ "[partition] evicted_ring_capacity default drifted from \
+ partitions::EVICTED_RING_CAPACITY"
+ );
+ }
+
+ #[test]
+ fn default_evicted_ring_bytes_max_matches_partitions_constant() {
+ // Belt and suspenders with the static assert above; this pins the
+ // shipped config.toml value.
+ let config_default = configs::ng_partition::PartitionConfig::default()
+ .evicted_ring_bytes_max
+ .as_bytes_u64();
+ assert_eq!(
+ config_default,
+ partitions::EVICTED_RING_BYTES_MAX,
+ "[partition] evicted_ring_bytes_max default drifted from \
+ partitions::EVICTED_RING_BYTES_MAX"
+ );
+ }
+
#[test]
fn shutdown_on_drop_armed_flips_flag() {
let flag = Arc::new(AtomicBool::new(false));
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index 0f4668cf3..053b65706 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -53,6 +53,7 @@ use axum::middleware::{Next, from_fn, from_fn_with_state};
use axum::routing::{delete, get, post, put};
use configs::http::{HttpConfig, HttpCorsConfig};
use configs::ng_cluster::{ClusterConfig, TransportPorts, http_forwarding_key_material};
+use configs::ng_http_admission::HttpAdmissionConfig;
use configs::server_ng::NgSystemConfig;
use iggy_common::IggyError;
use message_bus::client_listener;
@@ -90,10 +91,13 @@ use crate::server_error::ServerNgError;
/// Returns [`ServerNgError`] if the JWT manager cannot be built from
/// `http_config.jwt`, the `[http.cors]` config is invalid, the `[http.tls]`
/// credentials cannot be loaded, or the listener cannot bind to `addr`.
+#[allow(clippy::too_many_arguments)]
pub async fn start(
shard: &Rc<ServerNgShard>,
addr: SocketAddr,
http_config: &HttpConfig,
+ admission: &HttpAdmissionConfig,
+ clients_table_max: usize,
cluster: &ClusterConfig,
system_config: Arc<NgSystemConfig>,
self_ports: TransportPorts,
@@ -151,6 +155,9 @@ pub async fn start(
// never consulted here.
metadata_view: Arc::new(AtomicU64::new(crate::cluster_meta::METADATA_VIEW_UNKNOWN)),
},
+ max_http_sessions: crate::http::session::max_http_sessions(clients_table_max),
+ max_in_flight_writes: admission.max_in_flight_writes,
+ max_in_flight_writes_per_session: admission.max_in_flight_writes_per_session,
in_flight_writes: Cell::new(0),
forward,
}));
@@ -158,8 +165,12 @@ pub async fn start(
if http_config.tls.enabled {
let server_config = tls::load_http_tls_server_config(&http_config.tls)?;
- let (connections, pump) =
- tls::spawn_accept_pump(listener, server_config, shard.bus.token());
+ let (connections, pump) = tls::spawn_accept_pump(
+ listener,
+ server_config,
+ shard.bus.config().handshake_grace,
+ shard.bus.token(),
+ );
shard.bus.track_background(pump);
info!(address = %bound_addr, "server-ng HTTPS listener started");
let handle = compio::runtime::spawn(tls::serve(connections, router, shard.bus.token()));
diff --git a/core/server-ng/src/http/admission.rs b/core/server-ng/src/http/admission.rs
index e88f79a00..afb614bed 100644
--- a/core/server-ng/src/http/admission.rs
+++ b/core/server-ng/src/http/admission.rs
@@ -23,27 +23,6 @@ use std::cell::Cell;
use crate::http::error::PartitionWriteError;
-/// Per-session cap on concurrently awaited partition writes (produce /
-/// consumer-offset). Bounds how much of [`MAX_IN_FLIGHT_WRITES_GLOBAL`] one
-/// credential can occupy, so a session that outruns its own commits saturates
-/// itself (429, its own backpressure signal) before it can starve every other
-/// session out of the shared budget.
-const MAX_IN_FLIGHT_WRITES_PER_SESSION: u32 = 32;
-
-/// Shard-0-global budget for concurrently awaited partition writes, across all
-/// sessions. Every admitted write parks a handler for up to
-/// [`PARTITION_WRITE_REPLY_TIMEOUT`] while pinning its buffered body, and its
-/// decode/encode/HS256 CPU runs on the same single-threaded core that pumps
-/// consensus. The budget therefore bounds both starvation terms: budget x
-/// `max_request_size` bounds the worst-case buffered bytes, and budget x
-/// per-request CPU bounds how far admitted HTTP work can delay the consensus
-/// pump. `?ack=none` produces are admitted through the same caps: they install
-/// no reply slot and never await a commit, but they still park inside dispatch
-/// for the routable-wait budget while pinning their buffered body, so leaving
-/// them uncapped would bypass both terms. A session that saturates its own cap
-/// reads its own 429 before it can spill onto the shared budget.
-const MAX_IN_FLIGHT_WRITES_GLOBAL: u32 = 128;
-
/// In-flight admission token for one awaited partition write. One guard owns
/// both releases (session + global) so success, every error return, the reply
/// timeout, and handler cancellation (the client hanging up mid-await drops
@@ -64,14 +43,26 @@ impl Drop for InFlightWriteGuard<'_> {
/// shard-0 global budget, incrementing both counters only when both pass. The
/// session cap is checked first so a session that saturates itself reads as
/// its own 429 rather than as server-wide pressure.
+///
+/// Both caps come from `[http_admission]`. `max_global` bounds both starvation
+/// terms every admitted write imposes on shard 0 - budget x `max_request_size`
+/// worst-case buffered bytes, budget x per-request decode/encode/HS256 CPU on
+/// the core that also pumps consensus. `max_per_session` bounds one credential's
+/// slice of that budget, so a session that outruns its own commits reads its own
+/// 429 before it can spill onto the shared budget. `?ack=none` produces are
+/// admitted through the same caps: they install no reply slot, but still park
+/// inside dispatch while pinning their buffered body, so leaving them uncapped
+/// would bypass both terms.
pub(in crate::http) fn admit_partition_write<'a>(
session_in_flight: &'a Cell<u32>,
global_in_flight: &'a Cell<u32>,
+ max_per_session: u32,
+ max_global: u32,
) -> Result<InFlightWriteGuard<'a>, PartitionWriteError> {
- if session_in_flight.get() >= MAX_IN_FLIGHT_WRITES_PER_SESSION {
+ if session_in_flight.get() >= max_per_session {
return Err(PartitionWriteError::TooManyInFlight);
}
- if global_in_flight.get() >= MAX_IN_FLIGHT_WRITES_GLOBAL {
+ if global_in_flight.get() >= max_global {
return Err(PartitionWriteError::ServerBusy);
}
session_in_flight.set(session_in_flight.get() + 1);
@@ -86,11 +77,23 @@ pub(in crate::http) fn admit_partition_write<'a>(
mod tests {
use super::*;
+ const SESSION_CAP: u32 = 32;
+ const GLOBAL_CAP: u32 = 128;
+
+ /// Admit at the fixture caps, so each test reads the counting behavior
+ /// without restating the caps on every call.
+ fn admit<'a>(
+ session_in_flight: &'a Cell<u32>,
+ global_in_flight: &'a Cell<u32>,
+ ) -> Result<InFlightWriteGuard<'a>, PartitionWriteError> {
+ admit_partition_write(session_in_flight, global_in_flight, SESSION_CAP, GLOBAL_CAP)
+ }
+
#[test]
fn in_flight_write_guard_decrements_both_counters_on_drop() {
let session = Cell::new(0);
let global = Cell::new(0);
- let guard = admit_partition_write(&session, &global).expect("below both caps");
+ let guard = admit(&session, &global).expect("below both caps");
assert_eq!(session.get(), 1);
assert_eq!(global.get(), 1);
drop(guard);
@@ -100,27 +103,27 @@ mod tests {
#[test]
fn admission_at_session_cap_rejects_with_too_many_in_flight() {
- let session = Cell::new(MAX_IN_FLIGHT_WRITES_PER_SESSION);
+ let session = Cell::new(SESSION_CAP);
let global = Cell::new(0);
assert!(matches!(
- admit_partition_write(&session, &global),
+ admit(&session, &global),
Err(PartitionWriteError::TooManyInFlight)
));
// A refusal must not leak a partial increment on either counter.
- assert_eq!(session.get(), MAX_IN_FLIGHT_WRITES_PER_SESSION);
+ assert_eq!(session.get(), SESSION_CAP);
assert_eq!(global.get(), 0);
}
#[test]
fn admission_at_global_budget_rejects_with_server_busy() {
let session = Cell::new(0);
- let global = Cell::new(MAX_IN_FLIGHT_WRITES_GLOBAL);
+ let global = Cell::new(GLOBAL_CAP);
assert!(matches!(
- admit_partition_write(&session, &global),
+ admit(&session, &global),
Err(PartitionWriteError::ServerBusy)
));
assert_eq!(session.get(), 0);
- assert_eq!(global.get(), MAX_IN_FLIGHT_WRITES_GLOBAL);
+ assert_eq!(global.get(), GLOBAL_CAP);
}
#[test]
@@ -128,21 +131,21 @@ mod tests {
let session = Cell::new(0);
let global = Cell::new(0);
let mut guards = Vec::new();
- for _ in 0..MAX_IN_FLIGHT_WRITES_PER_SESSION {
- guards.push(admit_partition_write(&session, &global).expect("below both caps"));
+ for _ in 0..SESSION_CAP {
+ guards.push(admit(&session, &global).expect("below both caps"));
}
assert!(matches!(
- admit_partition_write(&session, &global),
+ admit(&session, &global),
Err(PartitionWriteError::TooManyInFlight)
));
let released = 3;
- guards.truncate((MAX_IN_FLIGHT_WRITES_PER_SESSION - released) as usize);
- assert_eq!(session.get(), MAX_IN_FLIGHT_WRITES_PER_SESSION - released);
+ guards.truncate((SESSION_CAP - released) as usize);
+ assert_eq!(session.get(), SESSION_CAP - released);
for _ in 0..released {
- guards.push(admit_partition_write(&session, &global).expect("released slots"));
+ guards.push(admit(&session, &global).expect("released slots"));
}
assert!(matches!(
- admit_partition_write(&session, &global),
+ admit(&session, &global),
Err(PartitionWriteError::TooManyInFlight)
));
drop(guards);
@@ -153,13 +156,12 @@ mod tests {
#[test]
fn global_budget_spans_sessions_and_reopens_after_release() {
let global = Cell::new(0);
- let session_count =
- MAX_IN_FLIGHT_WRITES_GLOBAL.div_ceil(MAX_IN_FLIGHT_WRITES_PER_SESSION) as usize;
+ let session_count = GLOBAL_CAP.div_ceil(SESSION_CAP) as usize;
let sessions: Vec<Cell<u32>> = (0..session_count).map(|_| Cell::new(0)).collect();
let mut guards = Vec::new();
'fill: for session in &sessions {
- for _ in 0..MAX_IN_FLIGHT_WRITES_PER_SESSION {
- match admit_partition_write(session, &global) {
+ for _ in 0..SESSION_CAP {
+ match admit(session, &global) {
Ok(guard) => guards.push(guard),
Err(PartitionWriteError::ServerBusy) => break 'fill,
Err(other) => {
@@ -168,17 +170,17 @@ mod tests {
}
}
}
- assert_eq!(global.get(), MAX_IN_FLIGHT_WRITES_GLOBAL);
+ assert_eq!(global.get(), GLOBAL_CAP);
// A fresh session is refused on the shared budget, not its own cap.
let fresh = Cell::new(0);
assert!(matches!(
- admit_partition_write(&fresh, &global),
+ admit(&fresh, &global),
Err(PartitionWriteError::ServerBusy)
));
drop(guards.pop());
- let readmitted = admit_partition_write(&fresh, &global).expect("budget slot released");
+ let readmitted = admit(&fresh, &global).expect("budget slot released");
assert_eq!(fresh.get(), 1);
- assert_eq!(global.get(), MAX_IN_FLIGHT_WRITES_GLOBAL);
+ assert_eq!(global.get(), GLOBAL_CAP);
drop(readmitted);
}
}
diff --git a/core/server-ng/src/http/error.rs b/core/server-ng/src/http/error.rs
index 511a1dd22..f882ef730 100644
--- a/core/server-ng/src/http/error.rs
+++ b/core/server-ng/src/http/error.rs
@@ -182,10 +182,11 @@ impl IntoResponse for AuthError {
}
// A fresh session could not be established: the Register was
// canceled with its commit outcome unknown, or the session table
- // is at `MAX_HTTP_SESSIONS` and refused the fresh registration.
- // Transient server condition -> 503, retryable by the CLIENT only
- // (a forwarder must not re-issue an unknown-outcome Register under
- // this node's session budget on the caller's behalf).
+ // is at its cap (half `[metadata] clients_table_max`) and refused
+ // the fresh registration. Transient server condition -> 503,
+ // retryable by the CLIENT only (a forwarder must not re-issue an
+ // unknown-outcome Register under this node's session budget on
+ // the caller's behalf).
Self::SessionUnavailable => service_unavailable(),
}
}
@@ -244,14 +245,14 @@ pub(in crate::http) enum PartitionWriteError {
/// The in-process reply slot could not be installed. Transient server
/// condition -> the shared 503, retryable.
Unavailable,
- /// This session is already at [`MAX_IN_FLIGHT_WRITES_PER_SESSION`]
+ /// This session is already at its `http_admission.max_in_flight_writes_per_session`
/// awaited writes. 429: the caller's own concurrency is the problem, so
/// it must drain its outstanding writes before submitting more.
TooManyInFlight,
- /// Shard 0 is already at [`MAX_IN_FLIGHT_WRITES_GLOBAL`] awaited writes
- /// across all sessions. 503 with its own code (distinct from the shared
- /// consensus-unavailable body) so an operator can tell admission shedding
- /// from a consensus outage.
+ /// Shard 0 is already at its `http_admission.max_in_flight_writes` awaited
+ /// writes across all sessions. 503 with its own code (distinct from the
+ /// shared consensus-unavailable body) so an operator can tell admission
+ /// shedding from a consensus outage.
ServerBusy,
/// No committed reply within [`PARTITION_WRITE_REPLY_TIMEOUT`], or the
/// session's reply target was torn down mid-wait. 504: the commit may
@@ -346,10 +347,10 @@ fn service_unavailable() -> Response {
)
}
-/// 429 for a session at [`MAX_IN_FLIGHT_WRITES_PER_SESSION`] awaited partition
-/// writes. Shaped like every other HTTP error (`ErrorResponse`) so clients
-/// parse one error schema; the remedy is the caller's own: let outstanding
-/// writes finish, then retry.
+/// 429 for a session at its `http_admission.max_in_flight_writes_per_session`
+/// awaited partition writes. Shaped like every other HTTP error
+/// (`ErrorResponse`) so clients parse one error schema; the remedy is the
+/// caller's own: let outstanding writes finish, then retry.
fn too_many_in_flight_response() -> Response {
with_retry_after(error_response(
StatusCode::TOO_MANY_REQUESTS,
@@ -358,8 +359,8 @@ fn too_many_in_flight_response() -> Response {
))
}
-/// 503 for shard 0 at [`MAX_IN_FLIGHT_WRITES_GLOBAL`] awaited partition writes
-/// across all sessions. A distinct `server_busy` code (unlike the shared
+/// 503 for shard 0 at its `http_admission.max_in_flight_writes` awaited partition
+/// writes across all sessions. A distinct `server_busy` code (unlike the shared
/// consensus-unavailable 503) so admission shedding is tellable from a
/// consensus outage; retry with backoff.
fn server_busy_response() -> Response {
@@ -434,6 +435,12 @@ pub(in crate::http) enum ReadError {
/// query, so the caller re-issues the read against the leader (see
/// [`primary_redirect_response`]).
RedirectToPrimary(String),
+ /// The post-restart read-recovery barrier expired with the recovered WAL
+ /// suffix still uncommitted: serving now could show state that rolls back
+ /// history a client already saw acked. Fail-closed 503 via the shared
+ /// [`service_unavailable`] body, retryable once the cluster re-commits the
+ /// suffix.
+ RecoveryIncomplete,
/// A partition read (poll / consumer-offset) got no reply from the owning
/// shard within the mesh budget. 504 like a produce timeout: the outcome is
/// unknown (the abandoned read may still be running), so the caller retries.
@@ -449,6 +456,7 @@ impl IntoResponse for ReadError {
Self::NotFound => CustomError::ResourceNotFound.into_response(),
Self::NotPrimary => not_primary_response(),
Self::RedirectToPrimary(location) => primary_redirect_response(&location),
+ Self::RecoveryIncomplete => service_unavailable(),
Self::Timeout => gateway_timeout_response(
"partition_read_timeout",
"the partition owner did not answer the read in time; retry",
@@ -634,4 +642,24 @@ mod tests {
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
assert!(!response.headers().contains_key(RETRY_AFTER));
}
+
+ #[test]
+ fn recovery_incomplete_renders_retryable_503_like_not_primary() {
+ // Barrier expiry must render as the shared retryable 503: the same
+ // status and Retry-After hint as the not-primary 503, so an SDK treats
+ // it as a connection-level retry rather than a terminal error.
+ let recovery = ReadError::RecoveryIncomplete.into_response();
+ assert_eq!(recovery.status(), StatusCode::SERVICE_UNAVAILABLE);
+ assert_eq!(
+ recovery.headers().get(RETRY_AFTER),
+ Some(&HeaderValue::from(RETRY_AFTER_SECONDS))
+ );
+
+ let not_primary = ReadError::NotPrimary.into_response();
+ assert_eq!(recovery.status(), not_primary.status());
+ assert_eq!(
+ recovery.headers().get(RETRY_AFTER),
+ not_primary.headers().get(RETRY_AFTER)
+ );
+ }
}
diff --git a/core/server-ng/src/http/handlers.rs b/core/server-ng/src/http/handlers.rs
index be9bf50f8..0719609c2 100644
--- a/core/server-ng/src/http/handlers.rs
+++ b/core/server-ng/src/http/handlers.rs
@@ -176,8 +176,13 @@ pub(in crate::http) async fn login_user(
) -> Result<Json<IdentityInfo>, CustomError> {
// Credential verification is a consensus-free STM read; hold it while a
// recovered WAL suffix (which may carry the user's create/password ops)
- // re-commits, like every other local read.
- SendWrapper::new(crate::http::reads::await_recovery_barrier(&state.shard)).await;
+ // re-commits, like every other local read. On barrier expiry, fail with a
+ // retryable 503 rather than validating against rolled-back credentials:
+ // this route's error currency is `IggyError -> CustomError`, and
+ // `TransientNotCommitted` is the variant `CustomError` renders 503.
+ SendWrapper::new(crate::http::reads::await_recovery_barrier(&state.shard))
+ .await
+ .map_err(|_| IggyError::TransientNotCommitted)?;
let user_id = verify_login_credentials(
&state.shard,
&command.username,
@@ -191,7 +196,11 @@ pub(in crate::http) async fn login_with_personal_access_token(
State(state): State<HttpState>,
Json(command): Json<LoginWithPersonalAccessToken>,
) -> Result<Json<IdentityInfo>, CustomError> {
- SendWrapper::new(crate::http::reads::await_recovery_barrier(&state.shard)).await;
+ // Same recovery-barrier wait and retryable-503-on-expiry mapping as
+ // `login_user`.
+ SendWrapper::new(crate::http::reads::await_recovery_barrier(&state.shard))
+ .await
+ .map_err(|_| IggyError::TransientNotCommitted)?;
let user_id = verify_pat_credentials(&state.shard, command.token.expose_secret())
.map_err(|error| login_error_to_iggy(&error))?;
issue_identity(&state, user_id)
diff --git a/core/server-ng/src/http/reads.rs b/core/server-ng/src/http/reads.rs
index afc2397f6..e84cb9944 100644
--- a/core/server-ng/src/http/reads.rs
+++ b/core/server-ng/src/http/reads.rs
@@ -90,7 +90,7 @@ pub(in crate::http) async fn read_local(
body: &[u8],
rule: impl FnOnce(&Permissioner, u32) -> Result<(), IggyError>,
) -> Result<Bytes, ReadError> {
- await_recovery_barrier(&state.shard).await;
+ await_recovery_barrier(&state.shard).await?;
authorize_read(state, identity, consistency, rule)?;
match build_non_replicated_response(
&state.shard,
@@ -106,25 +106,54 @@ pub(in crate::http) async fn read_local(
}
}
+/// One recovery-barrier check's outcome, factored out of [`await_recovery_barrier`]
+/// so the expiry decision is unit-testable without a runtime: the loop reads the
+/// clock and injects whether the deadline has passed.
+#[derive(Debug, PartialEq, Eq)]
+enum BarrierWait {
+ /// Barrier met, or none armed: serve the read.
+ Ready,
+ /// Barrier unmet and the deadline has passed: fail loud.
+ Expired,
+ /// Barrier unmet, deadline still ahead: keep polling.
+ Pending,
+}
+
+/// Decide the barrier outcome from the armed barrier, the locally applied commit
+/// point, and whether the deadline has passed. A met barrier wins over an
+/// expired deadline, so recovery that completes as the deadline lands still
+/// serves rather than 503-ing.
+const fn barrier_state(barrier: u64, commit_min: u64, expired: bool) -> BarrierWait {
+ if barrier == 0 || commit_min >= barrier {
+ BarrierWait::Ready
+ } else if expired {
+ BarrierWait::Expired
+ } else {
+ BarrierWait::Pending
+ }
+}
+
/// Hold a local read while the recovered WAL suffix re-commits.
///
/// Recovery re-pipelines prepared-but-uncommitted ops that clients saw
/// committed before the restart; JWT-authenticated HTTP reads skip consensus
/// entirely, so without this wait they can observe state that rolls back
/// committed history in the first few hundred milliseconds after a restart.
-/// No-op (`recovery_barrier() == 0`) outside that window. Bounded: the
-/// suffix needs a backup quorum to re-commit, so serve anyway after the
-/// deadline rather than wedging reads on a partitioned cluster.
-pub(in crate::http) async fn await_recovery_barrier(shard: &Rc<ServerNgShard>) {
- // Must outlast a full post-restart convergence: the peers' election
- // (several heartbeat timeouts in the worst case) plus the new primary
- // re-committing the journaled suffix. A shorter deadline expires mid
- // view-change and serves pre-restart state that clients saw acked.
- const DEADLINE: std::time::Duration = std::time::Duration::from_secs(15);
+/// `Ok(())` immediately when no suffix is pending (`recovery_barrier() == 0`).
+///
+/// Bounded by the barrier's paired deadline (scaled from the configured
+/// cluster timeouts; see `recovery_barrier_deadline`). If the suffix has not
+/// re-committed by then the read fails loud with a retryable 503
+/// ([`ReadError::RecoveryIncomplete`]) instead of silently serving pre-restart
+/// state a client already saw acked; the caller retries against a converged
+/// cluster.
+pub(in crate::http) async fn await_recovery_barrier(
+ shard: &Rc<ServerNgShard>,
+) -> Result<(), ReadError> {
const POLL: std::time::Duration = std::time::Duration::from_millis(10);
let Some(consensus) = shard.plane.metadata().consensus.as_ref() else {
- return;
+ return Ok(());
};
let barrier = consensus.recovery_barrier();
// Gate on commit_MIN (locally applied), not commit_max (known committed):
@@ -132,20 +161,24 @@ pub(in crate::http) async fn await_recovery_barrier(shard: &Rc<ServerNgShard>) {
// journal applying ops, and this task interleaves with that walk at its
// await points -- a commit_max gate would serve state from before the
// suffix applied (e.g. a pre-restart password change not yet visible).
- if barrier == 0 || consensus.commit_min() >= barrier {
- return;
+ if barrier_state(barrier, consensus.commit_min(), false) == BarrierWait::Ready {
+ return Ok(());
}
- let deadline = std::time::Instant::now() + DEADLINE;
- while consensus.commit_min() < barrier {
- if std::time::Instant::now() >= deadline {
- tracing::warn!(
- barrier,
- commit_min = consensus.commit_min(),
- "recovered suffix still unapplied past deadline; serving read anyway"
- );
- return;
+ let deadline = std::time::Instant::now() + consensus.recovery_deadline();
+ loop {
+ let expired = std::time::Instant::now() >= deadline;
+ match barrier_state(barrier, consensus.commit_min(), expired) {
+ BarrierWait::Ready => return Ok(()),
+ BarrierWait::Expired => {
+ tracing::warn!(
+ barrier,
+ commit_min = consensus.commit_min(),
+ "recovered suffix still unapplied past deadline; failing read with retryable 503"
+ );
+ return Err(ReadError::RecoveryIncomplete);
+ }
+ BarrierWait::Pending => compio::time::sleep(POLL).await,
}
- compio::time::sleep(POLL).await;
}
}
@@ -228,3 +261,38 @@ pub(in crate::http) fn authorize_data_plane(
.users()
.authorize(|permissioner| rule(permissioner, user_id, stream_id, topic_id))
}
+
+#[cfg(test)]
+mod tests {
+ use super::{BarrierWait, barrier_state};
+
+ #[test]
+ fn barrier_state_ready_when_no_barrier_armed() {
+ assert_eq!(barrier_state(0, 0, false), BarrierWait::Ready);
+ assert_eq!(barrier_state(0, 0, true), BarrierWait::Ready);
+ }
+
+ #[test]
+ fn barrier_state_ready_when_commit_reached_barrier() {
+ assert_eq!(barrier_state(5, 5, false), BarrierWait::Ready);
+ assert_eq!(barrier_state(5, 6, false), BarrierWait::Ready);
+ }
+
+ #[test]
+ fn barrier_state_pending_while_unmet_before_deadline() {
+ assert_eq!(barrier_state(5, 3, false), BarrierWait::Pending);
+ }
+
+ #[test]
+ fn barrier_state_expires_when_unmet_past_deadline() {
+ // Red before the fail-loud change: an expired barrier used to serve the
+ // read (a bare `()`), now an unmet barrier past its deadline is a
+ // distinct terminal outcome the wait maps to a retryable 503.
+ assert_eq!(barrier_state(5, 3, true), BarrierWait::Expired);
+ }
+
+ #[test]
+ fn barrier_state_met_wins_over_expired_deadline() {
+ assert_eq!(barrier_state(5, 5, true), BarrierWait::Ready);
+ }
+}
diff --git a/core/server-ng/src/http/session.rs b/core/server-ng/src/http/session.rs
index a7f697993..1b3bc4263 100644
--- a/core/server-ng/src/http/session.rs
+++ b/core/server-ng/src/http/session.rs
@@ -29,21 +29,38 @@ use futures::channel::oneshot;
use message_bus::InstanceToken;
use tokio::sync::Mutex;
-/// Hard cap on live per-credential sessions. A leak-guard, not a tuning knob:
-/// reaching it means this many distinct live tokens are in flight at once. New
-/// sessions past the cap are refused with a transient 503 rather than evicting
-/// a live one; the client retries. Expired entries are dropped first, so the
-/// cap only bites on live oversubscription.
+/// HTTP's slice of the shared VSR client table: half the configured
+/// `[metadata] clients_table_max`. A leak-guard, not a tuning knob: reaching
+/// the cap means that many distinct live tokens are in flight at once. New
+/// sessions past it are refused with a transient 503 rather than evicting a
+/// live one; the client retries. Expired entries are dropped first, so the cap
+/// only bites on live oversubscription.
///
/// Bounded by the shared VSR client table: HTTP sessions and the TCP/QUIC/WS
-/// virtual clients all Register into the one [`CLIENTS_TABLE_MAX`]-slot table,
-/// which LRU-evicts the oldest client when full. Capping HTTP at half that
-/// bound keeps this plane from crowding the others out and keeps the combined
-/// steady state under the shared bound, so a live idle HTTP session is not
-/// routinely evicted consensus-side. The residual eviction race (both planes
-/// busy) degrades gracefully: an evicted session's next control write is
-/// classified as an eviction and re-registers (see [`HttpInner::forget_session`]).
-pub(in crate::http) const MAX_HTTP_SESSIONS: usize = CLIENTS_TABLE_MAX / 2;
+/// virtual clients all Register into the one client table, which LRU-evicts the
+/// oldest client when full. Capping HTTP at half that bound keeps this plane
+/// from crowding the others out and keeps the combined steady state under the
+/// shared bound, so a live idle HTTP session is not routinely evicted
+/// consensus-side. The residual eviction race (both planes busy) degrades
+/// gracefully: an evicted session's next control write is classified as an
+/// eviction and re-registers (see [`HttpInner::forget_session`]).
+///
+/// The half rule lives here so a change to the ratio flows to both the runtime
+/// value (threaded through `HttpInner` at boot) and the pinned default.
+pub(in crate::http) const fn max_http_sessions(clients_table_max: usize) -> usize {
+ clients_table_max / 2
+}
+
+/// HTTP session cap at the shipped-default client table ([`CLIENTS_TABLE_MAX`]).
+/// The runtime value ([`max_http_sessions`] of the configured
+/// `clients_table_max`) equals this on a default deployment; the tests pin it.
+pub(in crate::http) const DEFAULT_MAX_HTTP_SESSIONS: usize = max_http_sessions(CLIENTS_TABLE_MAX);
+
+// HTTP must never claim the whole shared VSR client table, or a login storm
+// could evict every TCP/QUIC/WS virtual client. Compile-time pin of the
+// headroom the half-cap guarantees (config validation floors the table at 2, so
+// the runtime cap keeps the same headroom).
+const _: () = assert!(DEFAULT_MAX_HTTP_SESSIONS < CLIENTS_TABLE_MAX);
/// First per-session request id the write path hands out. VSR request numbers
/// are 1-based and strictly increasing within a session.
@@ -88,7 +105,7 @@ pub(in crate::http) struct HttpSession {
/// token.
pub(in crate::http) registry_token: Cell<Option<InstanceToken>>,
/// Awaited partition writes currently in flight on this session, gated by
- /// [`MAX_IN_FLIGHT_WRITES_PER_SESSION`]. Only [`InFlightWriteGuard`]
+ /// `HttpInner::max_in_flight_writes_per_session`. Only [`InFlightWriteGuard`]
/// touches it, so every admission is paired with exactly one release.
pub(in crate::http) in_flight_writes: Cell<u32>,
}
@@ -346,15 +363,21 @@ mod tests {
assert_eq!(table.len(), 8, "no live session is evicted to make room");
}
+ // Pins the specific half ratio (not just headroom, which the module-level
+ // compile assert covers): narrowing the split would silently starve HTTP.
#[test]
- fn http_session_cap_leaves_headroom_below_the_shared_client_table_bound() {
- const {
- assert!(
- MAX_HTTP_SESSIONS < CLIENTS_TABLE_MAX,
- "HTTP must not claim the whole shared VSR client table"
- );
- }
- assert_eq!(MAX_HTTP_SESSIONS, CLIENTS_TABLE_MAX / 2);
+ fn http_session_cap_is_half_the_shared_client_table_bound() {
+ assert_eq!(DEFAULT_MAX_HTTP_SESSIONS, CLIENTS_TABLE_MAX / 2);
+ }
+
+ // Zero behavior change at defaults: the cap computed at boot from the
+ // shipped `[metadata] clients_table_max` equals the historical compile-time
+ // default, so surfacing the table size as config leaves a default
+ // deployment untouched.
+ #[test]
+ fn runtime_cap_at_default_config_equals_pinned_default() {
+ let configured = configs::ng_metadata::MetadataConfig::default().clients_table_max;
+ assert_eq!(max_http_sessions(configured), DEFAULT_MAX_HTTP_SESSIONS);
}
// Eviction recovery: forgetting the evicted session drops exactly its
diff --git a/core/server-ng/src/http/state.rs b/core/server-ng/src/http/state.rs
index f596c3ad0..dfd2e6226 100644
--- a/core/server-ng/src/http/state.rs
+++ b/core/server-ng/src/http/state.rs
@@ -43,8 +43,8 @@ use crate::http::error::{AuthError, ReadError, primary_redirect_location};
use crate::http::forward::ForwardState;
use crate::http::jwt::JwtManager;
use crate::http::session::{
- BarrierEntry, FIRST_REQUEST_ID, HttpSession, MAX_HTTP_SESSIONS, RegistrationBarrier,
- forget_if_same, live_entry, sweep_expired,
+ BarrierEntry, FIRST_REQUEST_ID, HttpSession, RegistrationBarrier, forget_if_same, live_entry,
+ sweep_expired,
};
/// Response header carrying the current VSR view number. Stamped by
@@ -82,9 +82,22 @@ pub(in crate::http) struct HttpInner {
/// requests for one credential from each running its own `Register`.
pub(in crate::http) registrations: RegistrationBarrier,
pub(in crate::http) roster: ClusterRoster,
+ /// Cap on live per-credential sessions: half the configured `[metadata]
+ /// clients_table_max`, so HTTP sessions cannot crowd the TCP/QUIC/WS virtual
+ /// clients out of the shared VSR client table. Read by `resolve_session`
+ /// when admitting a fresh session.
+ pub(in crate::http) max_http_sessions: usize,
+ /// Shard-0-global cap on concurrently awaited partition writes, from
+ /// `[http_admission] max_in_flight_writes`. Passed to `admit_partition_write`
+ /// alongside the per-session cap.
+ pub(in crate::http) max_in_flight_writes: u32,
+ /// Per-session cap on concurrently awaited partition writes, from
+ /// `[http_admission] max_in_flight_writes_per_session`. Uniform across
+ /// sessions, so it lives here rather than on each `HttpSession`.
+ pub(in crate::http) max_in_flight_writes_per_session: u32,
/// Awaited partition writes currently in flight across all sessions, gated
- /// by [`MAX_IN_FLIGHT_WRITES_GLOBAL`]. Only [`InFlightWriteGuard`] touches
- /// it, so every admission is paired with exactly one release.
+ /// by `max_in_flight_writes`. Only [`InFlightWriteGuard`] touches it, so
+ /// every admission is paired with exactly one release.
pub(in crate::http) in_flight_writes: Cell<u32>,
/// Follower-to-primary forwarding context: outbound client, scheme, body
/// bound, and its own in-flight budget (see `http::forward`).
@@ -170,7 +183,7 @@ impl HttpInner {
let (admitted, torn) = {
let mut table = self.sessions.borrow_mut();
let torn = sweep_expired(&mut table, now);
- if table.len() >= MAX_HTTP_SESSIONS {
+ if table.len() >= self.max_http_sessions {
// Still full after dropping expired entries: too many
// genuinely live sessions. Refuse rather than evict a
// live one (its `fresh` client id is orphaned on the
diff --git a/core/server-ng/src/http/submit.rs b/core/server-ng/src/http/submit.rs
index 1f7cc2971..1cacf207a 100644
--- a/core/server-ng/src/http/submit.rs
+++ b/core/server-ng/src/http/submit.rs
@@ -323,7 +323,12 @@ pub(in crate::http) async fn partition_write_replicated(
// actually unbounded - the slot install, the dispatch, and the parked
// reply await that pins this request's buffers for up to the reply
// timeout. Held across every exit below; released by `Drop`.
- let _in_flight = admit_partition_write(&session.in_flight_writes, &state.in_flight_writes)?;
+ let _in_flight = admit_partition_write(
+ &session.in_flight_writes,
+ &state.in_flight_writes,
+ state.max_in_flight_writes_per_session,
+ state.max_in_flight_writes,
+ )?;
ensure_in_process_reply_target(state, session);
let request_id = session.next_data_request_id();
let message = build_request_message(
@@ -386,7 +391,12 @@ pub(in crate::http) async fn produce_unacked(
session: &HttpSession,
body: &[u8],
) -> Result<(), PartitionWriteError> {
- let _in_flight = admit_partition_write(&session.in_flight_writes, &state.in_flight_writes)?;
+ let _in_flight = admit_partition_write(
+ &session.in_flight_writes,
+ &state.in_flight_writes,
+ state.max_in_flight_writes_per_session,
+ state.max_in_flight_writes,
+ )?;
let request_id = session.next_data_request_id();
let message = build_request_message(
Operation::SendMessages,
diff --git a/core/server-ng/src/http/tls.rs b/core/server-ng/src/http/tls.rs
index dd974c897..830e79d22 100644
--- a/core/server-ng/src/http/tls.rs
+++ b/core/server-ng/src/http/tls.rs
@@ -65,11 +65,6 @@ use crate::server_error::ServerNgError;
const ALPN_H2: &[u8] = b"h2";
const ALPN_HTTP11: &[u8] = b"http/1.1";
-/// Handshake wall-clock bound for the accept pump. The HTTP listener has no
-/// `MessageBusConfig` to source it from, so it mirrors the binary
-/// transports' `DEFAULT_HANDSHAKE_GRACE` directly.
-const HTTP_TLS_HANDSHAKE_GRACE: Duration = Duration::from_secs(10);
-
/// Depth of the handshaken-connection channel between the accept pump and
/// the serve loop. The serve loop drains one per accept and immediately
/// spawns a per-connection hyper task, so it rarely fills; the bound keeps
@@ -101,16 +96,26 @@ pub fn load_http_tls_server_config(
/// Bind the accept pump: build the acceptor, spawn the pump task, and hand
/// back the handshaken-connection receiver for [`serve`] plus the pump's
-/// join handle for the caller to track.
+/// join handle for the caller to track. `handshake_grace` bounds each
+/// connection's TLS handshake; the caller sources it from
+/// `MessageBusConfig::handshake_grace` so the HTTPS listener shares the same
+/// slowloris budget as the binary transports.
pub fn spawn_accept_pump(
listener: TcpListener,
config: Arc<rustls::ServerConfig>,
+ handshake_grace: Duration,
shutdown: ShutdownToken,
) -> (Receiver<Handshaken>, JoinHandle<()>) {
let (connections_tx, connections_rx) =
async_channel::bounded::<Handshaken>(ACCEPT_CHANNEL_DEPTH);
let acceptor = TlsAcceptor::from(config);
- let pump = compio::runtime::spawn(accept_pump(listener, acceptor, connections_tx, shutdown));
+ let pump = compio::runtime::spawn(accept_pump(
+ listener,
+ acceptor,
+ connections_tx,
+ handshake_grace,
+ shutdown,
+ ));
(connections_rx, pump)
}
@@ -205,6 +210,7 @@ async fn accept_pump(
listener: TcpListener,
acceptor: TlsAcceptor,
connections: Sender<Handshaken>,
+ handshake_grace: Duration,
shutdown: ShutdownToken,
) {
loop {
@@ -214,7 +220,9 @@ async fn accept_pump(
break;
}
result = listener.accept().fuse() => match result {
- Ok((stream, peer)) => spawn_handshake(&acceptor, &connections, stream, peer),
+ Ok((stream, peer)) => {
+ spawn_handshake(&acceptor, &connections, handshake_grace, stream, peer);
+ }
Err(error) => error!(%error, "server-ng HTTPS accept failed"),
},
}
@@ -228,13 +236,14 @@ async fn accept_pump(
fn spawn_handshake(
acceptor: &TlsAcceptor,
connections: &Sender<Handshaken>,
+ handshake_grace: Duration,
stream: TcpStream,
peer: SocketAddr,
) {
let acceptor = acceptor.clone();
let connections = connections.clone();
compio::runtime::spawn(async move {
- match compio::time::timeout(HTTP_TLS_HANDSHAKE_GRACE, acceptor.accept(stream)).await {
+ match compio::time::timeout(handshake_grace, acceptor.accept(stream)).await {
Ok(Ok(tls)) => {
// Drop on send error: the channel is closed only at
// shutdown, when the serve loop is already tearing down.
@@ -242,7 +251,7 @@ fn spawn_handshake(
}
Ok(Err(error)) => debug!(%peer, %error, "server-ng HTTPS handshake failed"),
Err(_elapsed) => {
- debug!(%peer, grace = ?HTTP_TLS_HANDSHAKE_GRACE, "server-ng HTTPS handshake timed out");
+ debug!(%peer, grace = ?handshake_grace, "server-ng HTTPS handshake timed out");
}
}
})
diff --git a/core/server-ng/src/partition_helpers.rs b/core/server-ng/src/partition_helpers.rs
index e7c33ecd9..e76da4408 100644
--- a/core/server-ng/src/partition_helpers.rs
+++ b/core/server-ng/src/partition_helpers.rs
@@ -487,15 +487,25 @@ pub async fn build_partition_fresh(
source
})?;
+ // Request queue holds 2x the prepare depth (buffered requests drain as
+ // prepares commit); depth is the per-partition `[partition]` knob.
+ let prepare_queue_depth = config.partition.prepare_queue_depth;
let consensus = VsrConsensus::new(
cluster_id,
self_replica_id,
replica_count,
namespace.inner(),
bus,
- LocalPipeline::new(),
+ LocalPipeline::with_capacities(prepare_queue_depth, prepare_queue_depth * 2),
);
consensus.set_normal_heartbeat_ticks(crate::bootstrap::cluster_heartbeat_ticks(config));
+ consensus.set_commit_message_ticks(crate::bootstrap::commit_broadcast_ticks(config));
+ consensus.set_prepare_ticks(crate::bootstrap::prepare_retransmit_ticks(config));
+ consensus
+ .set_view_change_retransmit_ticks(crate::bootstrap::view_change_retransmit_ticks(config));
+ consensus.set_view_change_status_ticks(crate::bootstrap::view_change_status_ticks(config));
+ consensus.set_request_start_view_ticks(crate::bootstrap::request_start_view_ticks(config));
+ consensus.set_probe_attempts_max(config.cluster.view_probe_attempts_max);
// A partition directory that already holds segment bytes is a RESTART
// materialization, not a fresh create: this replica's group state died
// with the process, so claiming view-0 primaryship would heartbeat
@@ -513,6 +523,14 @@ pub async fn build_partition_fresh(
}
let mut partition = IggyPartition::new(stats, consensus);
+ // Surface the evicted-ring ceilings from config onto the fresh journal.
+ // IggyPartition::new has already disabled retention for single-replica
+ // groups (nobody to serve), so this only sizes the multi-replica ring; the
+ // caps are inert while retention is off.
+ partition.log.journal().inner.set_ring_caps(
+ config.partition.evicted_ring_capacity,
+ config.partition.evicted_ring_bytes_max.as_bytes_u64(),
+ );
partition.set_partition_dir(config.system.get_partition_path(
stream_id,
topic_id,
diff --git a/core/shard/src/lib.rs b/core/shard/src/lib.rs
index 60803f5b5..a89be1ff0 100644
--- a/core/shard/src/lib.rs
+++ b/core/shard/src/lib.rs
@@ -56,7 +56,7 @@ use partitions::{IggyPartition, IggyPartitions, PollFragments, PollingArgs, Poll
use server_common::sharding::{IggyNamespace, PartitionLocation, ShardId};
use server_common::{MESSAGE_ALIGN, Message, MessageBag, iobuf::Frozen};
use shards_table::ShardsTable;
-use std::cell::RefCell;
+use std::cell::{Cell, RefCell};
use std::collections::{HashMap, VecDeque};
use std::future::Future;
use std::rc::Rc;
@@ -708,12 +708,16 @@ impl ShardFrame {
}
}
-/// Prepares served per `RequestPrepares` round. The per-peer bus queues
-/// are bounded (`peer_queue_capacity`, 256 by default) and overrun frames
-/// drop silently, so an unbounded burst loses its own tail; the receiver
-/// pulls the window chunk by chunk instead (each walked `RepairDone`
-/// immediately requests the next chunk while progress holds).
-const REPAIR_CHUNK_MAX: u64 = 128;
+/// Prepares served per `RequestPrepares` round.
+///
+/// The per-peer bus queues are bounded (`peer_queue_capacity`, 256 by default)
+/// and overrun frames drop silently, so an unbounded burst loses its own tail;
+/// the receiver pulls the window chunk by chunk instead (each walked
+/// `RepairDone` immediately requests the next chunk while progress holds).
+///
+/// Runtime default; server-ng overrides the live ceiling per shard from
+/// `[cluster] repair_chunk_max` at bootstrap.
+pub const REPAIR_CHUNK_MAX: u64 = 128;
/// One in-flight metadata journal-repair stream (shard 0 only).
#[derive(Debug, Clone, Copy)]
@@ -824,6 +828,16 @@ where
/// `ReconcileOp::InsertOwned` lands. Bounded per namespace; overflow
/// drops the frame (at-least-once: client/primary retries recover).
pending_partition_frames: RefCell<HashMap<IggyNamespace, Vec<Message<GenericHeader>>>>,
+
+ /// Live ceiling on prepares served per `RequestPrepares` round. Defaults
+ /// to [`REPAIR_CHUNK_MAX`]; server-ng overrides it from
+ /// `[cluster] repair_chunk_max` at bootstrap.
+ repair_chunk_max: Cell<u64>,
+
+ /// Live stalled-repair retry threshold in consensus ticks. Defaults to
+ /// [`partitions::REPAIR_RETRY_TICKS`]; server-ng overrides it from
+ /// `[cluster] repair_retry_interval` at bootstrap.
+ repair_retry_ticks: Cell<u32>,
}
impl<B, MJ, S, M, T> IggyShard<B, MJ, S, M, T>
@@ -916,9 +930,25 @@ where
reconcile_queue: RefCell::new(VecDeque::new()),
pending_partition_frames: RefCell::new(HashMap::new()),
metadata_repair: RefCell::new(None),
+ repair_chunk_max: Cell::new(REPAIR_CHUNK_MAX),
+ repair_retry_ticks: Cell::new(partitions::REPAIR_RETRY_TICKS),
})
}
+ /// Override the stalled-repair retry threshold (consensus ticks) from
+ /// configuration. Called once per shard at bootstrap; the simulator and
+ /// tests keep the compile-time [`partitions::REPAIR_RETRY_TICKS`] default.
+ pub fn set_repair_retry_ticks(&self, ticks: u32) {
+ self.repair_retry_ticks.set(ticks);
+ }
+
+ /// Override the per-round repair-serving chunk ceiling from configuration.
+ /// Called once per shard at bootstrap; the simulator and tests keep the
+ /// compile-time [`REPAIR_CHUNK_MAX`] default.
+ pub fn set_repair_chunk_max(&self, chunk: u64) {
+ self.repair_chunk_max.set(chunk);
+ }
+
/// Hand a metadata consensus submit (login/logout) to shard 0.
///
/// Sends a [`LifecycleFrame::MetadataSubmit`] into shard 0's inbox. The
@@ -1123,6 +1153,8 @@ where
reconcile_queue: RefCell::new(VecDeque::new()),
pending_partition_frames: RefCell::new(HashMap::new()),
metadata_repair: RefCell::new(None),
+ repair_chunk_max: Cell::new(REPAIR_CHUNK_MAX),
+ repair_retry_ticks: Cell::new(partitions::REPAIR_RETRY_TICKS),
}
}
@@ -2036,6 +2068,9 @@ where
{
let header = *msg.header();
let target = header.replica;
+ // Snapshot the config-overridable chunk ceiling once; both plane
+ // branches below serve the same per-round window.
+ let repair_chunk_max = self.repair_chunk_max.get();
let planes = self.plane.inner();
if let Some(ref consensus) = planes.0.consensus
&& consensus.namespace() == header.namespace
@@ -2097,7 +2132,7 @@ where
)
.await;
}
- let chunk_end = to_op.min(from_op.saturating_add(REPAIR_CHUNK_MAX - 1));
+ let chunk_end = to_op.min(from_op.saturating_add(repair_chunk_max - 1));
let mut served_through = from_op.saturating_sub(1);
for op in from_op..=chunk_end {
#[allow(clippy::cast_possible_truncation)]
@@ -2157,7 +2192,7 @@ where
.await;
from_op = retained_from;
}
- let chunk_end = to_op.min(from_op.saturating_add(REPAIR_CHUNK_MAX - 1));
+ let chunk_end = to_op.min(from_op.saturating_add(repair_chunk_max - 1));
let mut served_through = from_op.saturating_sub(1);
for op in from_op..=chunk_end {
let Some(entry) = partition.log.journal().inner.repair_entry(op) else {
@@ -2541,6 +2576,7 @@ where
>,
{
let partitions = self.plane.partitions();
+ let repair_retry_ticks = self.repair_retry_ticks.get();
// Fan out over every group (each partition's heartbeat/retransmit timer
// must advance), so the keyed single-namespace lookup the control-frame
// handlers use does not apply here. The namespaces are snapshotted into
@@ -2579,7 +2615,7 @@ where
return None;
}
session.idle_ticks += 1;
- if session.idle_ticks < partitions::REPAIR_RETRY_TICKS {
+ if session.idle_ticks < repair_retry_ticks {
return None;
}
session.idle_ticks = 0;
@@ -2693,6 +2729,7 @@ where
// Stall retry, mirroring `tick_partitions`: a lost repair frame must
// not wedge the session forever.
+ let repair_retry_ticks = self.repair_retry_ticks.get();
let stalled = {
let mut session = self.metadata_repair.borrow_mut();
session.as_mut().and_then(|session| {
@@ -2700,7 +2737,7 @@ where
return None;
}
session.idle_ticks += 1;
- if session.idle_ticks < partitions::REPAIR_RETRY_TICKS {
+ if session.idle_ticks < repair_retry_ticks {
return None;
}
session.idle_ticks = 0;
diff --git a/justfile b/justfile
index ba8096a96..c8bf1ba4f 100644
--- a/justfile
+++ b/justfile
@@ -62,7 +62,7 @@ nextest: build
#!/usr/bin/env bash
set -euo pipefail
trap "{{reap_test_containers}}" EXIT
- cargo nextest run
+ cargo nextest run --retries 2
# Like `nextest` but with the `vsr` feature; builds vsr first so the
# harness-spawned iggy-server-ng carries the vsr wire format. `--no-fail-fast`
@@ -73,7 +73,7 @@ nextest-vsr: build-vsr
#!/usr/bin/env bash
set -euo pipefail
trap "{{reap_test_containers}}" EXIT
- cargo nextest run --features vsr --no-fail-fast
+ cargo nextest run --features vsr --no-fail-fast --retries 2
nextests TEST: build
#!/usr/bin/env bash