You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "hubcio (via GitHub)" <gi...@apache.org> on 2026/10/13 11:36:55 UTC
[PR] feat(server): route partition requests to their primaries [iggy]
hubcio opened a new pull request, #4506:
URL: https://github.com/apache/iggy/pull/4506
Backups refused requests without directing clients to a partition's
primary. HTTP tried writes on every node, and partition lookup
required send permission.
Add command 105 for partition routes with send or poll permission,
NotPrimary (89), and replica IDs in cluster metadata. Routes carry
incarnation, frontier and view. HTTP forwards to the partition primary
in one hop; stale routes cannot chain relays.
Run reads, polls and metadata commit waits outside the connection
drain. Serialize metadata writes per connection and serve QUIC streams
concurrently to prevent stalls.
The Rust SDK shares session state, correlates TCP/WebSocket replies by
request ID and bounds QUIC requests by deadlines. Add a driver with
cached routes, one link per replica and one mutation at a time per
partition history and for metadata. Clients do not use the driver yet.
Derive owner generations from metadata ops to fence recreated groups.
IGGY-OWNERSHIP-HISTORY-5 rejects older data before modifying it. Fence
old partition control frames with incarnation in reserved bytes,
preserving header sizes and field offsets.
Close the owner handoff, retirement and offset-transfer gaps found in
#4456. Relay partition completions to the metadata primary and
transfer emptied-table markers with offsets artifact v8. Exhausted
partition repair windows escalate to state transfer. Rotations
preserve the session ceiling and yield capacity; transfers reset
repair budgets.
Prove retained StartView history before committing or serving repair.
Preserve unresolved proof across failed truncation and view changes,
and retain acknowledged tails on commit-only updates. Metadata
recovery protects pipeline-owned prepares; internal completions no
longer become external reply attempts.
Carry the transport disconnect fix from #4388 and retry relayed reads
after primary failure. Simulator checks cover replacement histories
and restarts. Keep unresolved requests through same-session re-login
and transient refusals, bind the refusing endpoint and preserve
committed-offset flags across restart.
After offset file loss at boot, including all files or a whole kind
directory, deleting the donor's last key can emit an emptied-table
marker that drops those offsets on lagging receivers. That key may
have been written after boot. Deleted groups remain only while
stranded offsets are durable, avoiding permanent catalog retention for
unreadable or unremovable files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): route partition requests to their primaries [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4260687487
##########
core/server/src/http/forward.rs:
##########
@@ -332,92 +537,61 @@ fn record_relayed_floor(state: &HttpInner, user_id: u32, response: &Response) {
state.metadata_watermarks.record(user_id, applied);
}
-async fn forward_partition_or_pass(state: HttpState, request: Request, next: Next) -> Response {
- if !state.forward.active || request.headers().contains_key(FORWARDED_HEADER) {
- return next.run(request).await;
- }
- // A read that does not move consumer progress is answered by whichever
- // replica the caller reached, which is the point of reading from one. Only
- // an automatic commit needs a replica that can originate the offset
- // operation, so only that poll pays for the buffering and the credential
- // resolution below.
- if request.method() == Method::GET && !wants_auto_commit(request.uri().query()) {
- return next.run(request).await;
- }
+async fn attach_partition_request(state: HttpState, request: Request, next: Next) -> Response {
+ let received = Instant::now();
+ let hop = match partition_hop(request.headers()) {
+ Ok(hop) => hop,
+ Err(error) => return CustomError::from(error).into_response(),
+ };
+ // Verify-only, like the metadata relay: an unauthenticated caller gets no
+ // buffered body and no relay. The handler authenticates again.
let bearer = match bearer_token(request.headers()) {
Ok(bearer) => bearer,
Err(error) => return CustomError::from(error).into_response(),
};
if let Err(rejection) = resolve_credential(&state, bearer).await {
return rejection.into_response();
}
-
- let (parts, request_body) = request.into_parts();
+ let (mut parts, request_body) = request.into_parts();
let Ok(body) = to_bytes(request_body, state.forward.body_limit).await else {
Review Comment:
nit: every partition request decodes the credential and buffers the whole body here before the handler, which authenticates again. For an unauthenticated caller an oversized body becomes 413 instead of 401 and is buffered first. A cheap bearer-presence check before buffering, and skipping this when the request is served locally, would help.
##########
core/server/src/http/forward.rs:
##########
@@ -42,24 +42,32 @@
//! - The primary re-authenticates and re-authorizes the forwarded request
//! through its ordinary extractor stack; the forward marker header is a loop
//! guard only and never a trust input.
-//! - A forwarded request is retried only when it provably never entered the
-//! primary's pipeline: a connect-phase failure, a 503 whose body carries the
-//! `TransientNotAccepted` code, or a 307 from a stale target. Every other
-//! outcome - including a 503 carrying `TransientNotCommitted`, whose op may
-//! still commit - is relayed as-is, because re-issuing it under a fresh
+//! - The route header, like the marker, carries only values a caller can send
+//! itself, so the hop target authenticates and authorizes the request again.
+//! - A forwarded write is retried only when it provably never entered the
+//! primary's pipeline: a connect-phase failure, and for a metadata relay a
+//! 503 whose body carries the `TransientNotAccepted` or `NotPrimary` code or
+//! a 307 from a stale target, for a partition relay a 503 carrying
+//! `NotPrimary`. Every
Review Comment:
nit: broken wrap, "`NotPrimary`. Every" sits on its own line.
##########
core/server/src/http/forward.rs:
##########
@@ -494,18 +693,15 @@ async fn attempt(
return AttemptOutcome::Relay(bad_gateway());
}
};
- let request = builder
- .headers(forwarded_headers(request_headers))
- .body(body.clone())
- .build();
+ let request = builder.headers(headers).body(body.clone()).build();
let attempt = async {
let response = match state.forward.client.execute(request).await {
Ok(response) => response,
- Err(error) => return classify_transport_error(&error),
+ Err(error) => return classify_transport_error(&error, kind),
};
- classify_forwarded_reply(response, method, retry_redirect).await
+ classify_forwarded_reply(response, method, kind).await
};
- match compio::time::timeout(FORWARD_ATTEMPT_TIMEOUT, attempt).await {
+ match compio::time::timeout(timeout, attempt).await {
Review Comment:
nit: an elapsed attempt answers 504 "outcome unknown" for every relay kind, including `PartitionEffect::Read`, which changes nothing. A timed-out read relay can answer a retryable 503 instead.
##########
core/server/src/http/handlers.rs:
##########
@@ -1256,25 +1253,27 @@ pub(in crate::http) async fn delete_segments(
/// [`poll_wire_request`]) since HTTP has no persistent connection and cannot observe
/// group's partition assignment (that can be subject to change).
///
-/// A non-replicated read served in band: the same resolution the TCP dispatch
-/// runs ([`resolve_poll_request`]), then a mesh read on the owning shard and a
-/// re-encode of the stored batches into the legacy wire body, decoded here by
-/// the SDK's own [`PolledMessages::from_bytes`] so the JSON is field-identical
-/// to a TCP poll. `auto_commit` rides [`resolve_poll_request`]'s args and is
-/// honored by the owning shard's poll plan; no extra HTTP work.
+/// A non-replicated read served by the partition primary: the same resolution
+/// the TCP dispatch runs ([`resolve_poll_request`]), then a mesh read on the
+/// owning shard and a re-encode of the stored batches into the legacy wire
+/// body, decoded here by the SDK's own [`PolledMessages::from_bytes`] so the
+/// JSON is field-identical to a TCP poll. `auto_commit` rides
+/// [`resolve_poll_request`]'s args and is honored by the owning shard's poll
+/// plan. Another node relays the poll to the primary ([`route_partition`]).
pub(in crate::http) async fn poll_messages(
State(state): State<HttpState>,
identity: Identity,
Path((stream_id, topic_id)): Path<(String, String)>,
Query(query): Query<PollMessages>,
Query(consistency): Query<ConsistencyQuery>,
-) -> Result<Json<PolledMessages>, ReadError> {
+ Extension(request): Extension<Arc<PartitionRequest>>,
+) -> Result<Response, ReadError> {
let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
let topic_id = Identifier::from_str_value(&topic_id).map_err(ReadError::Rejected)?;
SendWrapper::new(gate_local_read(
Review Comment:
nit: with `?consistency=linearizable`, a node that is neither the metadata nor the partition primary first 307s to the metadata primary, which then relays to the partition primary again: two hops where the partition relay alone would do.
##########
core/sdk/src/tcp/tcp_client.rs:
##########
@@ -1707,115 +1589,65 @@ impl TcpClient {
Err(error) => return (preencoded.header, Err(error), outcome_so_far),
}
};
- let mut frame_complete = false;
- let outcome = async {
- loop {
- let header_bytes = bytemuck::bytes_of(&request_header);
- frame_complete = false;
- stream.write(header_bytes).await?;
- if !payload.is_empty() {
- stream.write(&payload).await?;
+ let mut link_failed = false;
+ let mut replay = false;
+ let outcome = loop {
+ let frame = Frame {
+ header: request_header,
+ body: payload.clone(),
+ };
+ trace!("Sent a TCP request with code: {code}, waiting for a response...");
+ // The deadline guards against server-side reply loss (e.g. a
+ // stalled replication quorum that never commits the op), which
+ // would hold the link lock forever and wedge every later
+ // request on this client.
+ let reply = match link.send(frame, read_deadline, &mut false).await {
+ Ok(reply) => reply,
+ Err(error) => {
+ link_failed = true;
+ break Err(outcome_so_far.observe_link_error(error, replay));
}
- stream.flush().await?;
- trace!("Sent a TCP request with code: {code}, waiting for a response...");
-
- let mut response_header = [0u8; iggy_binary_protocol::HEADER_SIZE];
- // `stream.read` delegates to `read_exact`; on success it
- // always returns the requested length, so no short-read
- // guard is needed here.
- //
- // Deadline guards against server-side reply loss (e.g. a
- // stalled replication quorum that never commits the op):
- // the connection is lockstep, so an unanswered read would
- // hold the stream lock forever and wedge every later
- // request on this client. On expiry drop the stream --
- // a late reply would desync framing for the next request.
- //
- // One deadline spans BOTH the header and body reads: a
- // reply that delivers a header then stalls must not get a
- // fresh full timeout for the body.
- let header_read =
- tokio::time::timeout_at(read_deadline, stream.read(&mut response_header))
- .await;
- let Ok(header_read) = header_read else {
- error!(
- "Timed out after {RESPONSE_READ_TIMEOUT:?} waiting for VSR response header for TCP request with code: {code}",
- );
- return Err(IggyError::Disconnected);
- };
- header_read.map_err(|error| {
- error!(
- "Failed to read VSR response header for TCP request with code: {code}: {error}",
- );
- IggyError::Disconnected
- })?;
-
- let response_size = crate::vsr::response_size(&response_header)?;
- let body_size = response_size - iggy_binary_protocol::HEADER_SIZE;
- let body = if body_size > 0 {
- let mut body = BytesMut::with_capacity(body_size);
- let body_read = tokio::time::timeout_at(
- read_deadline,
- stream.read_buf(&mut body, body_size),
- )
- .await;
- let Ok(body_read) = body_read else {
- error!(
- "Timed out after {RESPONSE_READ_TIMEOUT:?} waiting for VSR response body for TCP request with code: {code}",
- );
- return Err(IggyError::Disconnected);
- };
- body_read.map_err(|error| {
- error!(
- "Failed to read VSR response body for TCP request with code: {code}: {error}",
- );
- IggyError::Disconnected
- })?;
- body.freeze()
- } else {
- Bytes::new()
- };
-
- frame_complete = true;
- crate::vsr::observe_metadata_reply(&metadata_watermark, &response_header);
- match crate::vsr::decode_response_split(&response_header, body)
- .map_err(|error| outcome_so_far.observe(error))
+ };
+ replay = true;
+ session_owner.observe_metadata_reply(&reply.header);
+ match crate::vsr::decode_response_split(&reply.header, reply.body)
+ .map_err(|error| outcome_so_far.observe(error))
+ {
+ Err(error) if !retry_transient => break Err(error),
+ // `TransientNotCommitted`: the op's outcome is unknown
+ // (e.g. a view change canceled it in flight) -- ONLY a
+ // same-session replay of the same request id is safe
+ // (the client-table serves it from cache if it did
+ // commit). Replay on this connection for the whole
+ // request budget; never hand it to the failover path,
+ // which re-issues under a fresh session and could
+ // double-apply a committed write.
+ Err(IggyError::TransientNotCommitted)
+ if code != iggy_binary_protocol::codes::POLL_MESSAGES_ON_PRIMARY_CODE
+ && tokio::time::Instant::now() < read_deadline =>
{
- Err(error) if !retry_transient => return Err(error),
- // `TransientNotCommitted`: the op's outcome is unknown
- // (e.g. a view change canceled it in flight) -- ONLY a
- // same-session replay of the same request id is safe
- // (the client-table serves it from cache if it did
- // commit). Replay on this connection for the whole
- // request budget; never hand it to the failover path,
- // which re-issues under a fresh session and could
- // double-apply a committed write.
- Err(IggyError::TransientNotCommitted)
- if code != iggy_binary_protocol::codes::POLL_MESSAGES_ON_PRIMARY_CODE
- && tokio::time::Instant::now() < read_deadline =>
- {
- let remaining = read_deadline
- .saturating_duration_since(tokio::time::Instant::now());
- tokio::time::sleep(NOT_READY_RETRY_INTERVAL.min(remaining)).await;
- }
- // `TransientNotAccepted`: the server never admitted the
- // request, so it is re-issuable anywhere. Replay here
- // briefly, then hand it back to the caller for a
- // leader recheck / failover.
- Err(IggyError::TransientNotAccepted)
- if tokio::time::Instant::now() < transient_deadline =>
- {
- let remaining = transient_deadline
- .saturating_duration_since(tokio::time::Instant::now());
- tokio::time::sleep(NOT_READY_RETRY_INTERVAL.min(remaining)).await;
- }
- other => return other,
+ let remaining =
+ read_deadline.saturating_duration_since(tokio::time::Instant::now());
+ tokio::time::sleep(NOT_READY_RETRY_INTERVAL.min(remaining)).await;
Review Comment:
nit: the transient replay sleeps here while still holding the link guard (up to 2s on 58/89, up to 30s on `TransientNotCommitted`), so every other call on the client, PING included, waits. Releasing the guard before the pause and keeping the retained header would fix it (same in WS/QUIC).
##########
core/server/src/dispatch/partition.rs:
##########
@@ -690,7 +688,7 @@ pub const fn fence_poll_context(
}
if requested.owner_generation != local.owner_generation {
return Err(IggyError::ConsumerGroupPartitionNotOwned(
- group_id,
+ client_id as u32,
Review Comment:
nit: `client_id as u32` truncates the u128 VSR client id, while `iggy_partition.rs:7283` puts the group id in the same slot, so one error reports two different identities (clients only see the code). Pre-existing.
##########
core/sdk/src/tcp/tcp_client.rs:
##########
@@ -1674,28 +1559,25 @@ impl TcpClient {
Option<iggy_binary_protocol::consensus::RequestHeader>,
Result<Bytes, IggyError>,
) {
- let stream = self.stream.clone();
- let consensus_session = self.consensus_session.clone();
- let metadata_watermark = Arc::clone(&self.poll_router.metadata_watermark);
+ let link = self.link.clone();
+ let session_owner = Arc::clone(&self.session_owner);
let mut outcome_so_far = std::mem::take(retry_outcome);
- // SAFETY: we run code holding the `stream` lock in a task so we can't be cancelled while holding the lock.
+ // A task finishes the exchange when its caller goes away. An unbound
+ // request that left before its reply would close the link.
let joined = tokio::spawn(async move {
- let mut stream_guard = stream.lock().await;
- let Some(stream) = stream_guard.as_mut() else {
+ let mut link_guard = link.lock().await;
Review Comment:
**warning**: the link mutex is held across the whole exchange (same in `websocket_client.rs:1361`, `quic_client.rs:1194`), so the new 256-slot `MuxLink` runs lockstep and only adds two task hand-offs, a semaphore, a map and a oneshot per request: producer about -4..-5%, 2x client context switches. A direct lockstep path (write, flush, read on the caller task) recovers master; or lock only to clone the link if pipelining is the goal.
##########
core/sdk/src/links/mux.rs:
##########
@@ -0,0 +1,815 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! One TCP or WebSocket connection that carries many requests at once.
+//!
+//! A writer task and a reader task own the two halves of the connection, and
+//! each reply goes to the request with its id. So a caller that goes away
+//! never leaves a frame half written or half read on the connection.
+
+use crate::links::{Frame, Reply};
+use crate::vsr::{decode_eviction, peek_command, reply_request_id, response_size};
+use bytes::{BufMut, BytesMut};
+use futures_util::stream::{SplitSink, SplitStream};
+use futures_util::{SinkExt, StreamExt};
+use iggy_binary_protocol::consensus::{Command, HEADER_SIZE};
+use iggy_common::IggyError;
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::future::Future;
+use std::io;
+use std::net::SocketAddr;
+use std::sync::{Arc, Mutex as StdMutex, MutexGuard, PoisonError};
+use std::time::Duration;
+use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, BufReader, BufWriter};
+use tokio::sync::{Mutex, Semaphore, mpsc, oneshot, watch};
+use tokio::task::JoinHandle;
+use tokio::time::{Instant, timeout, timeout_at};
+use tokio_tungstenite::WebSocketStream;
+use tokio_tungstenite::tungstenite::Message;
+use tokio_tungstenite::tungstenite::protocol::CloseFrame;
+use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode;
+use tracing::{debug, error, trace};
+
+/// Frames queued for the writer task before a sender waits for room.
+const LINK_WRITE_QUEUE: usize = 32;
+/// Requests that wait for a reply on one link at once. It matches
+/// `message_bus.client_queue_capacity`, the most requests the server runs at
+/// once for one connection.
+const MAX_PENDING_PER_LINK: usize = 256;
+/// A peer that stops reading cannot hold up a shutdown for longer than this.
+const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(2);
+const WEBSOCKET_CLOSE_REASON: &str = "Client requested shutdown";
+
+type Waiter = oneshot::Sender<Result<Reply, IggyError>>;
+
+/// A connection that carries many requests at once. Dropping the link stops
+/// its tasks, which closes the connection.
+#[derive(Debug)]
+pub(crate) struct MuxLink {
+ shared: Arc<MuxShared>,
+ queue: mpsc::Sender<Frame>,
+ tasks: LinkTasks,
+}
+
+#[derive(Debug)]
+struct MuxShared {
+ peer: SocketAddr,
+ pending: StdMutex<Pending>,
+ permits: Semaphore,
+ /// Before bind, request ids repeat, so one such exchange runs at a time.
+ unbound: Mutex<()>,
+ /// Turns true once, when the link closes.
+ closed: watch::Sender<bool>,
+}
+
+#[derive(Debug)]
+struct Pending {
+ /// Cleared when the link closes. A closed link never opens again.
+ open: bool,
+ waiters: HashMap<u64, Waiter>,
+}
+
+#[derive(Debug)]
+struct LinkTasks {
+ writer: JoinHandle<()>,
+ reader: JoinHandle<()>,
+}
+
+impl Drop for LinkTasks {
+ fn drop(&mut self) {
+ self.writer.abort();
+ self.reader.abort();
+ }
+}
+
+/// A request that waits for its reply. Dropping it ends the wait.
+struct PendingEntry<'a> {
+ shared: &'a MuxShared,
+ request: u64,
+ /// Set for a repeated id that reached the queue: its late reply would
+ /// answer the next request with the same id.
+ close_if_abandoned: bool,
+}
+
+impl MuxLink {
+ /// A link over the two halves of a TCP connection, plain or TLS.
+ pub(crate) fn tcp<R, W>(reader: R, writer: W, peer: SocketAddr) -> Self
+ where
+ R: AsyncRead + Unpin + Send + 'static,
+ W: AsyncWrite + Unpin + Send + 'static,
+ {
+ Self::start(
+ peer,
+ |frames, shared| write_tcp(BufWriter::new(writer), frames, shared),
+ |shared| read_tcp(BufReader::new(reader), shared),
+ )
+ }
+
+ /// A link over a WebSocket connection, plain or TLS.
+ pub(crate) fn websocket<S>(stream: WebSocketStream<S>, peer: SocketAddr) -> Self
+ where
+ S: AsyncRead + AsyncWrite + Unpin + Send + 'static,
+ {
+ let (sink, stream) = stream.split();
+ Self::start(
+ peer,
+ |frames, shared| write_websocket(sink, frames, shared),
+ |shared| read_websocket(stream, shared),
+ )
+ }
+
+ /// Sends `frame` and waits for its reply until `deadline`. `sent` turns
+ /// true when the frame reaches the queue, so a caller that goes away
+ /// before the reply still knows that the frame left.
+ ///
+ /// `NotConnected` means the frame never reached the queue, so the request
+ /// can go again. `Disconnected` or an eviction reason means it did, and no
+ /// reply came.
+ pub(crate) async fn send(
+ &self,
+ frame: Frame,
+ deadline: Instant,
+ sent: &mut bool,
+ ) -> Result<Reply, IggyError> {
+ let request = frame.header.request;
+ let unbound = frame.header.session == 0;
+ let _unbound_turn = if unbound {
+ let turn = timeout_at(deadline, self.shared.unbound.lock()).await;
+ Some(turn.map_err(|_| IggyError::NotConnected)?)
+ } else {
+ None
+ };
+ let _permit = timeout_at(deadline, self.shared.permits.acquire())
+ .await
+ .map_err(|_| IggyError::NotConnected)?
+ .map_err(|_| IggyError::NotConnected)?;
+ let (waiter, reply) = oneshot::channel();
+ let mut entry = self.shared.wait_for(request, waiter)?;
+ if !matches!(
+ timeout_at(deadline, self.queue.send(frame)).await,
+ Ok(Ok(()))
+ ) {
+ return Err(IggyError::NotConnected);
+ }
+ *sent = true;
+ entry.close_if_abandoned = unbound;
+ match timeout_at(deadline, reply).await {
+ Ok(Ok(result)) => result,
+ Ok(Err(_)) => Err(IggyError::Disconnected),
Review Comment:
nit: every transport failure now surfaces as `Disconnected`, so the `TcpError` / `QuicError` / `WebSocketSendError` / `WebSocketReceiveError` arms in the clients (e.g. `websocket_client.rs:366-378`) are dead. In-tree reconnect still works, but downstream code matching those variants silently stops matching; worth dropping the dead arms and noting the error contract change.
##########
core/sdk/src/links/mux.rs:
##########
@@ -0,0 +1,815 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! One TCP or WebSocket connection that carries many requests at once.
+//!
+//! A writer task and a reader task own the two halves of the connection, and
+//! each reply goes to the request with its id. So a caller that goes away
+//! never leaves a frame half written or half read on the connection.
+
+use crate::links::{Frame, Reply};
+use crate::vsr::{decode_eviction, peek_command, reply_request_id, response_size};
+use bytes::{BufMut, BytesMut};
+use futures_util::stream::{SplitSink, SplitStream};
+use futures_util::{SinkExt, StreamExt};
+use iggy_binary_protocol::consensus::{Command, HEADER_SIZE};
+use iggy_common::IggyError;
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::future::Future;
+use std::io;
+use std::net::SocketAddr;
+use std::sync::{Arc, Mutex as StdMutex, MutexGuard, PoisonError};
+use std::time::Duration;
+use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, BufReader, BufWriter};
+use tokio::sync::{Mutex, Semaphore, mpsc, oneshot, watch};
+use tokio::task::JoinHandle;
+use tokio::time::{Instant, timeout, timeout_at};
+use tokio_tungstenite::WebSocketStream;
+use tokio_tungstenite::tungstenite::Message;
+use tokio_tungstenite::tungstenite::protocol::CloseFrame;
+use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode;
+use tracing::{debug, error, trace};
+
+/// Frames queued for the writer task before a sender waits for room.
+const LINK_WRITE_QUEUE: usize = 32;
+/// Requests that wait for a reply on one link at once. It matches
+/// `message_bus.client_queue_capacity`, the most requests the server runs at
+/// once for one connection.
+const MAX_PENDING_PER_LINK: usize = 256;
+/// A peer that stops reading cannot hold up a shutdown for longer than this.
+const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(2);
+const WEBSOCKET_CLOSE_REASON: &str = "Client requested shutdown";
+
+type Waiter = oneshot::Sender<Result<Reply, IggyError>>;
+
+/// A connection that carries many requests at once. Dropping the link stops
+/// its tasks, which closes the connection.
+#[derive(Debug)]
+pub(crate) struct MuxLink {
+ shared: Arc<MuxShared>,
+ queue: mpsc::Sender<Frame>,
+ tasks: LinkTasks,
+}
+
+#[derive(Debug)]
+struct MuxShared {
+ peer: SocketAddr,
+ pending: StdMutex<Pending>,
+ permits: Semaphore,
+ /// Before bind, request ids repeat, so one such exchange runs at a time.
+ unbound: Mutex<()>,
+ /// Turns true once, when the link closes.
+ closed: watch::Sender<bool>,
+}
+
+#[derive(Debug)]
+struct Pending {
+ /// Cleared when the link closes. A closed link never opens again.
+ open: bool,
+ waiters: HashMap<u64, Waiter>,
+}
+
+#[derive(Debug)]
+struct LinkTasks {
+ writer: JoinHandle<()>,
+ reader: JoinHandle<()>,
+}
+
+impl Drop for LinkTasks {
+ fn drop(&mut self) {
+ self.writer.abort();
+ self.reader.abort();
+ }
+}
+
+/// A request that waits for its reply. Dropping it ends the wait.
+struct PendingEntry<'a> {
+ shared: &'a MuxShared,
+ request: u64,
+ /// Set for a repeated id that reached the queue: its late reply would
+ /// answer the next request with the same id.
+ close_if_abandoned: bool,
+}
+
+impl MuxLink {
+ /// A link over the two halves of a TCP connection, plain or TLS.
+ pub(crate) fn tcp<R, W>(reader: R, writer: W, peer: SocketAddr) -> Self
+ where
+ R: AsyncRead + Unpin + Send + 'static,
+ W: AsyncWrite + Unpin + Send + 'static,
+ {
+ Self::start(
+ peer,
+ |frames, shared| write_tcp(BufWriter::new(writer), frames, shared),
+ |shared| read_tcp(BufReader::new(reader), shared),
+ )
+ }
+
+ /// A link over a WebSocket connection, plain or TLS.
+ pub(crate) fn websocket<S>(stream: WebSocketStream<S>, peer: SocketAddr) -> Self
+ where
+ S: AsyncRead + AsyncWrite + Unpin + Send + 'static,
+ {
+ let (sink, stream) = stream.split();
+ Self::start(
+ peer,
+ |frames, shared| write_websocket(sink, frames, shared),
+ |shared| read_websocket(stream, shared),
+ )
+ }
+
+ /// Sends `frame` and waits for its reply until `deadline`. `sent` turns
+ /// true when the frame reaches the queue, so a caller that goes away
+ /// before the reply still knows that the frame left.
+ ///
+ /// `NotConnected` means the frame never reached the queue, so the request
+ /// can go again. `Disconnected` or an eviction reason means it did, and no
+ /// reply came.
+ pub(crate) async fn send(
+ &self,
+ frame: Frame,
+ deadline: Instant,
+ sent: &mut bool,
+ ) -> Result<Reply, IggyError> {
+ let request = frame.header.request;
+ let unbound = frame.header.session == 0;
+ let _unbound_turn = if unbound {
+ let turn = timeout_at(deadline, self.shared.unbound.lock()).await;
+ Some(turn.map_err(|_| IggyError::NotConnected)?)
+ } else {
+ None
+ };
+ let _permit = timeout_at(deadline, self.shared.permits.acquire())
+ .await
+ .map_err(|_| IggyError::NotConnected)?
+ .map_err(|_| IggyError::NotConnected)?;
+ let (waiter, reply) = oneshot::channel();
+ let mut entry = self.shared.wait_for(request, waiter)?;
+ if !matches!(
+ timeout_at(deadline, self.queue.send(frame)).await,
+ Ok(Ok(()))
+ ) {
+ return Err(IggyError::NotConnected);
+ }
+ *sent = true;
+ entry.close_if_abandoned = unbound;
+ match timeout_at(deadline, reply).await {
+ Ok(Ok(result)) => result,
+ Ok(Err(_)) => Err(IggyError::Disconnected),
+ Err(_) => {
+ error!(
+ "No reply to request {request} from {} before its deadline.",
+ self.shared.peer
+ );
+ Err(IggyError::Disconnected)
+ }
+ }
+ }
+
+ /// Fails every waiting request. The writer task then ends the connection
+ /// the way its protocol expects.
+ pub(crate) fn close(&self) {
+ self.shared.close(&IggyError::Disconnected);
+ }
+
+ /// Returns when the link closes, for any reason.
+ pub(crate) async fn closed(&self) {
+ let mut closed = self.shared.closed.subscribe();
+ // The sender lives as long as the link.
+ let _ = closed.wait_for(|closed| *closed).await;
+ }
+
+ /// Ends the connection the way its protocol expects: a FIN for TCP, a
+ /// Close message for WebSocket. The link counts as closed first, so the
+ /// reader does not report the end of the connection as an error.
+ pub(crate) async fn shutdown(self) {
+ let Self {
+ shared,
+ queue,
+ mut tasks,
+ } = self;
+ shared.close(&IggyError::Disconnected);
+ drop(queue);
+ let _ = timeout(SHUTDOWN_TIMEOUT, &mut tasks.writer).await;
+ }
+
+ fn start<Writer, Reader>(
+ peer: SocketAddr,
+ writer: impl FnOnce(mpsc::Receiver<Frame>, Arc<MuxShared>) -> Writer,
+ reader: impl FnOnce(Arc<MuxShared>) -> Reader,
+ ) -> Self
+ where
+ Writer: Future<Output = ()> + Send + 'static,
+ Reader: Future<Output = ()> + Send + 'static,
+ {
+ let shared = Arc::new(MuxShared {
+ peer,
+ pending: StdMutex::new(Pending {
+ open: true,
+ waiters: HashMap::new(),
+ }),
+ permits: Semaphore::new(MAX_PENDING_PER_LINK),
+ unbound: Mutex::new(()),
+ closed: watch::Sender::new(false),
+ });
+ let (queue, frames) = mpsc::channel(LINK_WRITE_QUEUE);
+ let tasks = LinkTasks {
+ writer: tokio::spawn(writer(frames, Arc::clone(&shared))),
+ reader: tokio::spawn(reader(Arc::clone(&shared))),
+ };
+ Self {
+ shared,
+ queue,
+ tasks,
+ }
+ }
+}
+
+impl MuxShared {
+ fn wait_for(&self, request: u64, waiter: Waiter) -> Result<PendingEntry<'_>, IggyError> {
+ let mut pending = self.lock_pending();
+ if !pending.open {
+ return Err(IggyError::NotConnected);
+ }
+ match pending.waiters.entry(request) {
+ // Two requests with one id could not tell their replies apart.
+ Entry::Occupied(_) => {
+ error!(
+ "Request {request} already waits for a reply from {}.",
+ self.peer
+ );
+ Err(IggyError::NotConnected)
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(waiter);
+ Ok(PendingEntry {
+ shared: self,
+ request,
+ close_if_abandoned: false,
+ })
+ }
+ }
+ }
+
+ /// Cuts every whole frame off the front of `buffer` and dispatches it.
+ fn deliver_buffered(&self, buffer: &mut BytesMut) -> Result<(), IggyError> {
+ while let Some(&header) = buffer.first_chunk::<HEADER_SIZE>() {
+ let Ok(size) = response_size(&header) else {
+ error!(
+ "Received a frame shorter than its header from {}.",
+ self.peer
+ );
+ return Err(IggyError::Disconnected);
+ };
+ if buffer.len() < size {
+ break;
+ }
+ let body = buffer.split_to(size).freeze().slice(HEADER_SIZE..);
+ self.dispatch(Reply { header, body })?;
+ }
+ Ok(())
+ }
+
+ /// Gives a reply to the request that waits for it. An error is the reason
+ /// to close the link: an eviction ends the session, so it ends every
+ /// request.
+ fn dispatch(&self, reply: Reply) -> Result<(), IggyError> {
+ match peek_command(&reply.header) {
+ Command::Reply => {
+ let request = reply_request_id(&reply.header);
+ let waiter = self.lock_pending().waiters.remove(&request);
+ match waiter {
+ Some(waiter) => {
+ let _ = waiter.send(Ok(reply));
+ }
+ None => trace!(
+ "Dropped a reply to request {request} from {}, which nothing waits for.",
+ self.peer
+ ),
+ }
+ Ok(())
+ }
+ Command::Eviction => Err(decode_eviction(&reply.header)),
+ _ => {
+ error!(
+ "Received a frame that is neither a reply nor an eviction from {}.",
+ self.peer
+ );
+ Err(IggyError::Disconnected)
+ }
+ }
+ }
+
+ /// Fails every waiting request with `reason`. A request that comes later
+ /// fails with `NotConnected`.
+ fn close(&self, reason: &IggyError) {
+ let waiters = {
+ let mut pending = self.lock_pending();
+ pending.open = false;
+ std::mem::take(&mut pending.waiters)
+ };
+ self.permits.close();
+ self.closed.send_replace(true);
+ for waiter in waiters.into_values() {
+ let _ = waiter.send(Err(reason.clone()));
+ }
+ }
+
+ /// The next frame to write, or `None` once the link closed or every
+ /// sender went away.
+ async fn next_frame(&self, frames: &mut mpsc::Receiver<Frame>) -> Option<Frame> {
+ let mut closed = self.closed.subscribe();
+ tokio::select! {
+ biased;
+ _ = closed.wait_for(|closed| *closed) => None,
+ frame = frames.recv() => frame,
+ }
+ }
+
+ fn is_open(&self) -> bool {
+ self.lock_pending().open
+ }
+
+ fn lock_pending(&self) -> MutexGuard<'_, Pending> {
+ self.pending.lock().unwrap_or_else(PoisonError::into_inner)
+ }
+}
+
+impl Drop for PendingEntry<'_> {
+ fn drop(&mut self) {
+ let abandoned = self
+ .shared
+ .lock_pending()
+ .waiters
+ .remove(&self.request)
+ .is_some();
+ if abandoned && self.close_if_abandoned {
+ debug!(
+ "Closing the link to {}: request {} left before its reply.",
+ self.shared.peer, self.request
+ );
+ self.shared.close(&IggyError::Disconnected);
+ }
+ }
+}
+
+async fn write_tcp<W: AsyncWrite + Unpin>(
+ mut writer: W,
+ mut frames: mpsc::Receiver<Frame>,
+ shared: Arc<MuxShared>,
+) {
+ while let Some(frame) = shared.next_frame(&mut frames).await {
+ let written = async {
+ writer.write_all(bytemuck::bytes_of(&frame.header)).await?;
Review Comment:
nit: header and body go out as two `write_all` calls through the `BufWriter`; a body of 8 KiB or more becomes two segments, and with `nodelay=false` (default) an 8-64 KiB body can hit a Nagle / delayed-ACK stall. One gather write avoids it.
##########
core/sdk/src/session_owner/lanes.rs:
##########
@@ -0,0 +1,605 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A lane admits one mutation at a time, in arrival order. There is one lane
+//! for metadata and one for each partition history. A mutation whose outcome
+//! is unknown stays retained in its lane, and the lane admits nothing else
+//! until a resolve or an abandon frees it. So a retained operation is always
+//! the newest operation of its lane, and its replay can never age out.
+
+use super::SessionOwner;
+use crate::driver::PartitionKey;
+use crate::links::Frame;
+use crate::vsr::encode_request_header;
+use bytes::Bytes;
+use iggy_binary_protocol::consensus::{Operation, operation_for_code};
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
+use iggy_common::IggyError;
+use std::collections::HashMap;
+use std::sync::Arc;
+use tokio::sync::{Mutex, OwnedMutexGuard};
+use tokio::time::{Instant, timeout_at};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum LaneKey {
+ Metadata,
+ /// The incarnation names one history of the partition for every spelling
+ /// of its stream and topic.
+ Partition {
+ incarnation: u64,
+ partition_id: u32,
+ },
+}
+
+impl LaneKey {
+ /// The lane of `code`. A write to a partition takes the lane of the
+ /// incarnation that its context carries. Reads, Register and Logout take
+ /// no lane.
Review Comment:
nit: polls with auto-commit are NonReplicated and take no lane, but they move the stored offset of their partition history, so "one mutation at a time per partition history" doesn't hold for them (client-side twin of the duplicate-delivery race). Give them the partition lane or document the exemption.
##########
core/sdk/src/links/replicas.rs:
##########
@@ -0,0 +1,1571 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! One link per replica of the roster, and the sweep that signs in.
+//!
+//! `connect()` sweeps the known endpoints. The first link that answers
+//! becomes the bootstrap link, and the session signs in there. A read of the
+//! roster then gives one slot per replica, keyed by its replica id. Each slot
+//! runs one task that dials its replica, binds the session there and redials
+//! after a loss. A request never dials: it takes a ready link or fails with
+//! `NotConnected`.
+
+use super::connect::{
+ ConnectCoordinator, is_same_address, is_same_spelling, node_address, transport_port,
+};
+use super::{
+ Dialer, Link, OPERATION_TIMEOUT, backoff, event_channel, exchange, lost_link,
+ pause_before_retry,
+};
+use crate::SDK_VERSION;
+use crate::session_owner::{SessionEpoch, SessionOwner};
+use async_broadcast::{Receiver, Sender};
+use futures::future::join_all;
+use iggy_binary_protocol::codec::{WireDecode, WireEncode};
+use iggy_binary_protocol::codes::{BIND_SESSION_CODE, GET_CLUSTER_METADATA_CODE, PING_CODE};
+use iggy_binary_protocol::requests::system::{
+ BindSessionRequest, GetClusterMetadataRequest, PingRequest,
+};
+use iggy_binary_protocol::responses::system::ClusterMetadataResponse;
+use iggy_binary_protocol::responses::users::LoginRegisterResponse;
+use iggy_common::{
+ ClientState, ClusterMetadata, ClusterNodeRole, Credentials, DiagnosticEvent, IggyError,
+ QuicClientReconnectionConfig, TcpClientReconnectionConfig, WebSocketClientReconnectionConfig,
+ decode_session_binding, rust_sdk_version_info, sign_in_request,
+};
+use std::collections::{BTreeMap, BTreeSet};
+use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
+use std::time::Duration;
+use tokio::sync::Semaphore;
+use tokio::task::AbortHandle;
+use tokio::time::{Instant, sleep};
+use tracing::{info, warn};
+
+/// Dials that run at once, for one client.
+const MAX_CONCURRENT_DIALS: usize = 4;
+
+/// The longest pause before a slot or a sweep tries again.
+const RECONNECT_BACKOFF_MAX: Duration = Duration::from_secs(10);
+
+/// The reconnection configuration, the same for every transport.
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct ReconnectPolicy {
+ /// Without it, a lost link stays down, and a sweep makes one pass.
+ pub(super) enabled: bool,
+ /// The attempts of a slot after its first, and the passes of a sweep
+ /// after its first. `None` means no limit.
+ pub(super) max_retries: Option<u32>,
+ /// The base of the backoff.
+ pub(super) interval: Duration,
+ /// A link that stayed up this long redials at once after a loss.
+ pub(super) reestablish_after: Duration,
+}
+
+impl ReconnectPolicy {
+ fn allows(&self, retry: u32) -> bool {
+ self.enabled && self.max_retries.is_none_or(|max| retry <= max)
+ }
+
+ fn pause(&self, retry: u32) -> Duration {
+ backoff(retry, self.interval, RECONNECT_BACKOFF_MAX, rand::random())
+ }
+}
+
+macro_rules! reconnect_policy_from {
+ ($($config:ty),+) => {$(
+ impl From<&$config> for ReconnectPolicy {
+ fn from(config: &$config) -> Self {
+ Self {
+ enabled: config.enabled,
+ max_retries: config.max_retries,
+ interval: config.interval.get_duration(),
+ reestablish_after: config.reestablish_after.get_duration(),
+ }
+ }
+ }
+ )+};
+}
+
+reconnect_policy_from!(
+ TcpClientReconnectionConfig,
+ QuicClientReconnectionConfig,
+ WebSocketClientReconnectionConfig
+);
+
+/// Where a request goes.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum Target {
+ /// The metadata primary of the last roster, else the bootstrap link.
+ MetadataPrimary,
+ Replica(u8),
+}
+
+/// The links of one client. Dropping it aborts every task and drops every
+/// link, without a runtime.
+#[derive(Debug)]
+pub(crate) struct ReplicaLinks {
+ shared: Arc<LinksShared>,
+}
+
+/// The part that the tasks hold. No task holds [`ReplicaLinks`], so its
+/// `Drop` always runs.
+#[derive(Debug)]
+struct LinksShared {
+ dialer: Dialer,
+ configured_endpoint: String,
+ policy: ReconnectPolicy,
+ session: Arc<SessionOwner>,
+ /// Never held across an await. The epoch of the session is read before
+ /// this lock, never under it.
+ table: Mutex<Table>,
+ dial_permits: Semaphore,
+ coordinator: ConnectCoordinator,
+ events: Sender<DiagnosticEvent>,
+ /// Never read. Each subscriber is a clone, so it replays the backlog.
+ subscribers: Receiver<DiagnosticEvent>,
+}
+
+#[derive(Debug)]
+struct Table {
+ state: ClientState,
+ /// One slot per replica of the last roster. A replica id is a `u8`, so
+ /// the table holds at most 256 slots.
+ slots: BTreeMap<u8, Slot>,
+ bootstrap: Option<Bootstrap>,
+ metadata_primary: Option<u8>,
+ /// The endpoints of the last roster. A sweep dials them after the
+ /// configured endpoint.
+ roster: Vec<String>,
+}
+
+#[derive(Debug)]
+struct Slot {
+ endpoint: String,
+ link: Option<ReadyLink>,
+ /// `None` after the task stopped. Only the task of this handle changes
+ /// the slot.
+ task: Option<AbortHandle>,
+}
+
+/// The link that the sweep signed in on, until a slot takes it over or the
+/// metadata primary slot is ready.
+#[derive(Debug)]
+struct Bootstrap {
+ endpoint: String,
+ link: ReadyLink,
+ /// `None` while the sweep still uses the link.
+ watcher: Option<AbortHandle>,
+}
+
+#[derive(Debug)]
+struct ReadyLink {
+ link: Arc<Link>,
+ /// The fence of the session that the link carries, `None` before a
+ /// sign-in.
+ bound_fence: Option<u64>,
+}
+
+/// How one endpoint of a sweep ended.
+enum Attempt {
+ /// Another endpoint can still work.
+ EndpointFailed,
+ /// No retry can fix it, but the endpoints behind this one can still work.
+ ConfigFault(IggyError),
+ /// The sign-in lost its link. Another endpoint can still work.
+ SignInLost(IggyError),
+ /// The sweep ends with this error.
+ Failed(IggyError),
+}
+
+#[derive(Debug)]
+struct SignInFailure {
+ error: IggyError,
+ link_lost: bool,
+}
+
+impl From<IggyError> for SignInFailure {
+ fn from(error: IggyError) -> Self {
+ Self {
+ link_lost: lost_link(&error),
+ error,
+ }
+ }
+}
+
+impl ReplicaLinks {
+ pub(crate) fn new(
+ dialer: Dialer,
+ configured_endpoint: String,
+ policy: ReconnectPolicy,
+ session: Arc<SessionOwner>,
+ ) -> Self {
+ let (events, subscribers) = event_channel();
+ let table = Table {
+ state: ClientState::Disconnected,
+ slots: BTreeMap::new(),
+ bootstrap: None,
+ metadata_primary: None,
+ roster: Vec::new(),
+ };
+ Self {
+ shared: Arc::new(LinksShared {
+ dialer,
+ configured_endpoint,
+ policy,
+ session,
+ table: Mutex::new(table),
+ dial_permits: Semaphore::new(MAX_CONCURRENT_DIALS),
+ coordinator: ConnectCoordinator::new(),
+ events,
+ subscribers,
+ }),
+ }
+ }
+
+ /// Sweeps the known endpoints until one of them signs in. Concurrent
+ /// callers share one result. On a connected client, every slot without a
+ /// link dials again at once.
+ pub(crate) async fn connect(&self) -> Result<(), IggyError> {
+ let shared = &self.shared;
+ shared
+ .coordinator
+ .run(|abandoned, _| shared.sweep(abandoned))
+ .await
+ }
+
+ /// Aborts every task and closes every link, which fails each pending
+ /// request. The roster stays for the next sweep.
+ pub(crate) fn disconnect(&self) {
+ self.shared.disconnect();
+ }
+
+ /// Ends every link the way its protocol expects. It is final.
+ pub(crate) async fn shutdown(&self) {
+ let links = {
+ let mut table = self.shared.table();
+ if table.state == ClientState::Shutdown {
+ return;
+ }
+ table.state = ClientState::Shutdown;
+ table.clear()
+ };
+ join_all(links.into_iter().map(Link::shutdown)).await;
+ self.shared.dialer.shutdown().await;
+ self.shared.publish(DiagnosticEvent::Shutdown);
+ }
+
+ /// Pings every link that serves the session, all at once. A failed ping
+ /// only logs, because a link that lost its connection redials on its own.
+ pub(crate) async fn keepalive(&self) -> Result<(), IggyError> {
+ let links: Vec<Arc<Link>> = {
+ let epoch = self.shared.session.epoch();
+ let table = self.shared.table();
+ if table.state == ClientState::Shutdown {
+ return Err(IggyError::ClientShutdown);
+ }
+ table
+ .links()
+ .filter(|ready| serves(ready, epoch))
+ .map(|ready| Arc::clone(&ready.link))
+ .collect()
+ };
+ let deadline = Instant::now() + OPERATION_TIMEOUT;
+ let pings = links.iter().map(|link| {
+ exchange(
+ &self.shared.session,
+ link,
+ PING_CODE,
+ PingRequest.to_bytes(),
+ deadline,
+ )
+ });
+ for result in join_all(pings).await {
+ if let Err(error) = result {
+ warn!("A keepalive ping failed: {error}");
+ }
+ }
+ Ok(())
+ }
+
+ /// The link for `target`. A request never dials, so a target without a
+ /// ready link fails with `NotConnected`.
+ pub(crate) fn link(&self, target: Target) -> Result<Arc<Link>, IggyError> {
+ let epoch = self.shared.session.epoch();
+ let table = self.shared.table();
+ let slot = |id: u8| table.slots.get(&id).and_then(|slot| slot.link.as_ref());
+ let ready = match target {
+ Target::Replica(id) => slot(id),
+ Target::MetadataPrimary => table
+ .metadata_primary
+ .and_then(slot)
+ .filter(|ready| serves(ready, epoch))
+ .or(table.bootstrap.as_ref().map(|bootstrap| &bootstrap.link)),
+ };
+ ready
+ .filter(|ready| serves(ready, epoch))
+ .map(|ready| Arc::clone(&ready.link))
+ .ok_or(IggyError::NotConnected)
+ }
+
+ /// The replicas with a link that serves the session, in id order.
+ pub(crate) fn ready_replicas(&self) -> Vec<u8> {
+ let epoch = self.shared.session.epoch();
+ self.shared
+ .table()
+ .slots
+ .iter()
+ .filter(|(_, slot)| slot.link.as_ref().is_some_and(|ready| serves(ready, epoch)))
+ .map(|(id, _)| *id)
+ .collect()
+ }
+
+ pub(crate) fn metadata_primary(&self) -> Option<u8> {
+ self.shared.table().metadata_primary
+ }
+
+ pub(crate) fn session(&self) -> &Arc<SessionOwner> {
+ &self.shared.session
+ }
+
+ /// Records that `link` carries the session of `fence`. A sign-in calls it
+ /// before it binds the session, so the slots keep this link when they
+ /// close the links of the old session.
+ pub(crate) fn mark_bound(&self, link: &Arc<Link>, fence: u64) {
+ self.shared.mark_bound(link, fence);
+ }
+
+ /// Reads the roster on `link` and rebuilds the slots from it.
+ pub(crate) async fn refresh_roster(&self, link: &Arc<Link>) -> Result<(), IggyError> {
+ self.shared.refresh_roster(link).await
+ }
+
+ pub(crate) fn subscribe(&self) -> Receiver<DiagnosticEvent> {
+ self.shared.subscribers.clone()
+ }
+
+ pub(crate) fn publish(&self, event: DiagnosticEvent) {
+ self.shared.publish(event);
+ }
+
+ pub(crate) fn state(&self) -> ClientState {
+ self.shared.table().state
+ }
+
+ /// Sets the state, except after a shutdown, which is final.
+ pub(crate) fn set_state(&self, state: ClientState) {
+ let mut table = self.shared.table();
+ if table.state != ClientState::Shutdown {
+ table.state = state;
+ }
+ }
+
+ /// The endpoint of the metadata primary, else of the bootstrap link, else
+ /// the configured one.
+ pub(crate) fn connection_address(&self) -> String {
+ let table = self.shared.table();
+ table
+ .metadata_primary
+ .and_then(|id| table.slots.get(&id))
+ .map(|slot| &slot.endpoint)
+ .or(table
+ .bootstrap
+ .as_ref()
+ .map(|bootstrap| &bootstrap.endpoint))
+ .unwrap_or(&self.shared.configured_endpoint)
+ .clone()
+ }
+}
+
+impl Drop for ReplicaLinks {
+ fn drop(&mut self) {
+ // An abort needs no runtime, and each dropped link stops its own
+ // tasks.
+ drop(self.shared.table().clear());
+ }
+}
+
+impl LinksShared {
+ fn table(&self) -> MutexGuard<'_, Table> {
+ self.table.lock().unwrap_or_else(PoisonError::into_inner)
+ }
+
+ /// The channel runs in overflow mode, so a publish never waits.
+ fn publish(&self, event: DiagnosticEvent) {
+ let _ = self.events.try_broadcast(event);
+ }
+
+ fn disconnect(&self) {
+ let (links, previous) = {
+ let mut table = self.table();
+ if table.state == ClientState::Shutdown {
+ return;
+ }
+ let links = table.clear();
+ (
+ links,
+ std::mem::replace(&mut table.state, ClientState::Disconnected),
+ )
+ };
+ links.iter().for_each(|link| link.close());
+ if previous != ClientState::Disconnected {
+ self.publish(DiagnosticEvent::Disconnected);
+ }
+ }
+
+ async fn sweep(self: &Arc<Self>, abandoned: bool) -> Result<(), IggyError> {
+ if abandoned {
+ // The last sweep lost its caller halfway.
+ self.disconnect();
+ }
+ let (candidates, stopped) = {
+ let mut table = self.table();
+ match table.state {
+ ClientState::Shutdown => return Err(IggyError::ClientShutdown),
+ ClientState::Connected
+ | ClientState::Authenticating
+ | ClientState::Authenticated => {
+ self.restart_down_slots(&mut table);
+ return Ok(());
+ }
+ ClientState::Disconnected | ClientState::Connecting => {}
+ }
+ table.state = ClientState::Connecting;
+ (self.candidates(&table.roster), table.clear())
+ };
+ stopped.iter().for_each(|link| link.close());
+
+ let mut pass = 0;
+ loop {
+ let mut config_fault = None;
+ let mut sign_in_failure = None;
+ for endpoint in &candidates {
+ match self.try_endpoint(endpoint).await {
+ Ok(()) => return Ok(()),
+ Err(Attempt::EndpointFailed) => {}
+ Err(Attempt::ConfigFault(error)) => config_fault = Some(error),
+ Err(Attempt::SignInLost(error)) => sign_in_failure = Some(error),
+ Err(Attempt::Failed(error)) => return Err(error),
+ }
+ }
+ pass += 1;
+ let failure = match config_fault {
+ Some(error) => Some(error),
+ None if !self.policy.allows(pass) => {
+ Some(sign_in_failure.unwrap_or(IggyError::CannotEstablishConnection))
+ }
+ None => None,
+ };
+ if let Some(error) = failure {
+ self.fail_sweep();
+ return Err(error);
+ }
+ let pause = self.policy.pause(pass);
+ info!("No endpoint signed in. Pass {pass} starts in {pause:?}.");
+ sleep(pause).await;
+ }
+ }
+
+ /// The configured endpoint, then the endpoints of the last roster, each
+ /// spelling once. Spellings need no name lookup, which can stall a sweep.
+ fn candidates(&self, roster: &[String]) -> Vec<String> {
+ let mut candidates = vec![self.configured_endpoint.clone()];
+ for endpoint in roster {
+ if !candidates
+ .iter()
+ .any(|known| is_same_spelling(known, endpoint))
+ {
+ candidates.push(endpoint.clone());
+ }
+ }
+ candidates
+ }
+
+ async fn try_endpoint(self: &Arc<Self>, endpoint: &str) -> Result<(), Attempt> {
+ sweeping(self.table().state).map_err(Attempt::Failed)?;
+ info!("Connecting to {endpoint}...");
+ let dialed = match self.dial_permits.acquire().await {
+ Ok(_permit) => self.dialer.dial(endpoint).await,
+ Err(_) => Err(IggyError::CannotEstablishConnection),
+ };
+ let link = match dialed {
+ Ok(link) => Arc::new(link),
+ Err(IggyError::CannotEstablishConnection) => return Err(Attempt::EndpointFailed),
+ Err(error) => return Err(Attempt::ConfigFault(error)),
+ };
+ if let Err(error) = self.install_bootstrap(endpoint, &link) {
+ link.close();
+ return Err(Attempt::Failed(error));
+ }
+ self.publish(DiagnosticEvent::Connected);
+ info!("Connected to {endpoint}.");
+
+ if let Err(failure) = self.resume_or_sign_in(&link).await {
+ if failure.link_lost {
+ warn!(
+ "The sign-in on {endpoint} did not complete: {}",
+ failure.error
+ );
+ self.lose_bootstrap(&link);
+ return Err(Attempt::SignInLost(failure.error));
+ }
+ // The link stands, and only the session is missing. Every
+ // endpoint gives the same answer to the same credentials.
+ self.watch_bootstrap(&link);
+ return Err(Attempt::Failed(failure.error));
+ }
+ if self.session.epoch().bound
+ && let Err(error) = self.refresh_roster(&link).await
+ {
+ warn!("Failed to read the roster from {endpoint}: {error}");
+ }
+ self.watch_bootstrap(&link);
+ Ok(())
+ }
+
+ fn install_bootstrap(&self, endpoint: &str, link: &Arc<Link>) -> Result<(), IggyError> {
+ let mut table = self.table();
+ sweeping(table.state)?;
+ table.bootstrap = Some(Bootstrap {
+ endpoint: endpoint.to_owned(),
+ link: ReadyLink {
+ link: Arc::clone(link),
+ bound_fence: None,
+ },
+ watcher: None,
+ });
+ table.state = ClientState::Connected;
+ Ok(())
+ }
+
+ /// Drops a bootstrap link that its sign-in lost. The sweep goes on.
+ fn lose_bootstrap(&self, link: &Arc<Link>) {
+ link.close();
+ let mut table = self.table();
+ if !table.is_bootstrap(link) {
+ return;
+ }
+ table.bootstrap = None;
+ table.state = ClientState::Connecting;
+ drop(table);
+ self.publish(DiagnosticEvent::Disconnected);
+ }
+
+ fn fail_sweep(&self) {
+ let mut table = self.table();
+ if table.state == ClientState::Connecting {
+ table.state = ClientState::Disconnected;
+ drop(table);
+ self.publish(DiagnosticEvent::Disconnected);
+ }
+ }
+
+ /// Watches a bootstrap link that the sweep is done with, until it closes.
+ fn watch_bootstrap(self: &Arc<Self>, link: &Arc<Link>) {
+ let mut table = self.table();
+ let Some(bootstrap) = table
+ .bootstrap
+ .as_mut()
+ .filter(|bootstrap| Arc::ptr_eq(&bootstrap.link.link, link))
+ else {
+ return;
+ };
+ let shared = Arc::clone(self);
+ let link = Arc::clone(link);
+ let watcher = tokio::spawn(async move {
+ shared.hold(&link).await;
+ shared.bootstrap_down(&link);
+ });
+ bootstrap.watcher = Some(watcher.abort_handle());
+ }
+
+ fn bootstrap_down(&self, link: &Arc<Link>) {
+ let mut table = self.table();
+ if !table.is_bootstrap(link) {
+ return;
+ }
+ table.bootstrap = None;
+ let last = table.ready_count() == 0;
+ table.settle();
+ drop(table);
+ if last {
+ self.publish(DiagnosticEvent::Disconnected);
+ }
+ }
+
+ /// Resumes a bound session on the bootstrap link, else signs in there.
+ async fn resume_or_sign_in(&self, link: &Arc<Link>) -> Result<(), SignInFailure> {
+ let epoch = self.session.epoch();
+ if epoch.bound {
+ match self.bind_session(link).await {
+ Ok(()) => {
+ self.mark_bound(link, epoch.fence);
+ self.set_state_on(link, ClientState::Authenticated);
+ self.publish(DiagnosticEvent::SignedIn);
+ info!("Resumed the session.");
+ return Ok(());
+ }
+ Err(error) if session_lost(&error) => {
+ info!("The session ended ({error}). Signing in again.");
+ self.session.reset_if_fence(epoch.fence)?;
+ }
+ // A replica that cannot bind inside its read budget is behind
+ // the session, and another one can be further ahead.
+ Err(error) => {
+ return Err(SignInFailure {
+ link_lost: lost_link(&error)
+ || matches!(error, IggyError::TransientNotAccepted),
+ error,
+ });
+ }
+ }
+ }
+ self.sign_in(link).await
+ }
+
+ /// Signs in on `link` with the known credentials, if any. A failure that
+ /// leaves the link up leaves the client connected without a session. A
+ /// rejected credential is forgotten, because it stays rejected, and each
+ /// replay costs the server a password hash.
+ async fn sign_in(&self, link: &Arc<Link>) -> Result<(), SignInFailure> {
+ let Some(credentials) = self.session.sign_in_credentials() else {
+ info!("No credentials to sign in with.");
+ return Ok(());
+ };
+ self.set_state_on(link, ClientState::Authenticating);
+ let error = match self.register(link, &credentials).await {
+ Ok(user_id) => {
+ self.set_state_on(link, ClientState::Authenticated);
+ self.session.remember(credentials, user_id);
+ self.publish(DiagnosticEvent::SignedIn);
+ info!("Signed in.");
+ return Ok(());
+ }
+ Err(error) => error,
+ };
+ let failure = SignInFailure::from(error);
+ if !failure.link_lost {
+ self.set_state_on(link, ClientState::Connected);
+ }
+ if matches!(
+ failure.error,
+ IggyError::InvalidCredentials
+ | IggyError::InvalidUsername
+ | IggyError::InvalidPassword
+ | IggyError::Unauthenticated
+ ) {
+ self.session.forget();
+ }
+ Err(failure)
+ }
+
+ /// Registers a new session on `link` and binds it. A Register that the
+ /// server did not commit or did not admit goes again with the same client
+ /// id, so it registers one session.
+ async fn register(
+ &self,
+ link: &Arc<Link>,
+ credentials: &Credentials,
+ ) -> Result<u32, IggyError> {
+ let fence = self.session.fence();
+ let (code, body) = sign_in_request(
+ credentials,
+ rust_sdk_version_info(SDK_VERSION)?,
+ self.session.bind_secret()?,
+ )?;
+ let deadline = Instant::now() + OPERATION_TIMEOUT;
+ let mut retry = 0_u32;
+ let reply = loop {
+ let error = match exchange(&self.session, link, code, body.clone(), deadline).await {
+ Ok(reply) => break reply,
+ Err(error) => error,
+ };
+ if !matches!(
+ error,
+ IggyError::TransientNotCommitted | IggyError::TransientNotAccepted
+ ) || !pause_before_retry(retry, deadline, rand::random()).await
+ {
+ return Err(error);
+ }
+ retry = retry.saturating_add(1);
+ };
+ let registered =
+ LoginRegisterResponse::decode_from(&reply).map_err(|_| IggyError::InvalidFormat)?;
+ self.mark_bound(link, fence);
+ self.session.bind_at(fence, registered.session)?;
+ Ok(registered.user_id)
+ }
+
+ /// Binds the bound session on `link`. The reply must name the same
+ /// session.
+ async fn bind_session(&self, link: &Link) -> Result<(), IggyError> {
+ let identity = self.session.identity()?;
+ let request = BindSessionRequest {
+ version_info: rust_sdk_version_info(SDK_VERSION)?,
+ identity,
+ bind_secret: self.session.bind_secret()?,
+ };
+ let reply = exchange(
+ &self.session,
+ link,
+ BIND_SESSION_CODE,
+ request.to_bytes(),
+ Instant::now() + OPERATION_TIMEOUT,
+ )
+ .await?;
+ decode_session_binding(&identity, &reply).map(drop)
+ }
+
+ async fn refresh_roster(self: &Arc<Self>, link: &Arc<Link>) -> Result<(), IggyError> {
+ let reply = exchange(
+ &self.session,
+ link,
+ GET_CLUSTER_METADATA_CODE,
+ GetClusterMetadataRequest.to_bytes(),
+ Instant::now() + OPERATION_TIMEOUT,
+ )
+ .await?;
+ let roster =
+ ClusterMetadataResponse::decode_from(&reply).map_err(|_| IggyError::InvalidFormat)?;
+ self.apply_roster(&ClusterMetadata::try_from(roster)?, link)
+ .await
+ }
+
+ /// Rebuilds the slots from the roster that `via` carried. A removed
+ /// replica or a moved endpoint retires its link, so the requests that
+ /// hold it finish first.
+ async fn apply_roster(
+ self: &Arc<Self>,
+ roster: &ClusterMetadata,
+ via: &Arc<Link>,
+ ) -> Result<(), IggyError> {
+ let protocol = self.dialer.protocol();
+ let mut ids = BTreeSet::new();
+ let mut wanted = BTreeMap::new();
+ for node in &roster.nodes {
+ if !ids.insert(node.replica_id) {
+ warn!(
+ "The roster names replica {} twice, so the links stay as they are.",
+ node.replica_id
+ );
+ return Err(IggyError::InvalidFormat);
+ }
+ // A node without a port for this transport gets no slot, so the
+ // client never dials `ip:0`.
+ let port = transport_port(node, protocol);
+ if port != 0 {
+ wanted.insert(node.replica_id, node_address(node, port));
+ }
+ }
+ let primary = roster
+ .nodes
+ .iter()
+ .find(|node| node.role == ClusterNodeRole::Leader)
+ .map(|node| node.replica_id)
+ .filter(|id| wanted.contains_key(id));
+ let adopt = self.bootstrap_replica(&wanted).await;
+
+ let mut retired = Vec::new();
+ let mut table = self.table();
+ if table.find(via).is_none() {
+ // A disconnect ran during the read.
+ return Err(IggyError::Disconnected);
+ }
+ table.slots.retain(|id, slot| {
+ let kept = wanted.get(id) == Some(&slot.endpoint);
+ if !kept {
+ if let Some(task) = slot.task.take() {
+ task.abort();
+ }
+ retired.extend(slot.link.take());
+ }
+ kept
+ });
+ for (id, endpoint) in &wanted {
+ if table.slots.contains_key(id) {
+ continue;
+ }
+ let adopted = match &adopt {
+ Some((link, replica)) if replica == id && table.is_bootstrap(link) => {
+ table.retire_bootstrap()
+ }
+ _ => None,
+ };
+ let task = self.spawn_slot(
+ *id,
+ endpoint.clone(),
+ adopted.as_ref().map(|ready| Arc::clone(&ready.link)),
+ );
+ table.slots.insert(
+ *id,
+ Slot {
+ endpoint: endpoint.clone(),
+ link: adopted,
+ task: Some(task),
+ },
+ );
+ }
+ table.metadata_primary = primary;
+ table.roster = wanted.into_values().collect();
+ if primary.is_some_and(|id| table.slots.get(&id).is_some_and(|slot| slot.link.is_some())) {
+ retired.extend(table.retire_bootstrap());
+ }
+ drop(table);
+ drop(retired);
+ Ok(())
+ }
+
+ /// The replica that the bootstrap link reaches. With one replica in the
+ /// roster, that replica: a single node advertises its bind address, and
+ /// that can be `0.0.0.0`.
+ async fn bootstrap_replica(&self, wanted: &BTreeMap<u8, String>) -> Option<(Arc<Link>, u8)> {
+ let (link, endpoint) = {
+ let table = self.table();
+ let bootstrap = table.bootstrap.as_ref()?;
+ (Arc::clone(&bootstrap.link.link), bootstrap.endpoint.clone())
+ };
+ if wanted.len() == 1 {
+ return wanted.keys().next().map(|id| (link, *id));
+ }
+ for (id, replica) in wanted {
+ if is_same_address(&endpoint, replica).await {
+ return Some((link, *id));
+ }
+ }
+ None
+ }
+
+ fn spawn_slot(
+ self: &Arc<Self>,
+ id: u8,
+ endpoint: String,
+ adopted: Option<Arc<Link>>,
+ ) -> AbortHandle {
+ tokio::spawn(Arc::clone(self).run_slot(id, endpoint, adopted)).abort_handle()
+ }
+
+ /// Starts every slot without a link again, at once and without its
+ /// backoff.
+ fn restart_down_slots(self: &Arc<Self>, table: &mut Table) {
+ for (id, slot) in table
+ .slots
+ .iter_mut()
+ .filter(|(_, slot)| slot.link.is_none())
+ {
+ if let Some(task) = slot.task.take() {
+ task.abort();
+ }
+ slot.task = Some(self.spawn_slot(*id, slot.endpoint.clone(), None));
+ }
+ }
+
+ /// Keeps one link to replica `id` up. Each retry waits its backoff, and a
+ /// link that stayed up for `reestablish_after` redials at once.
+ async fn run_slot(self: Arc<Self>, id: u8, endpoint: String, mut adopted: Option<Arc<Link>>) {
+ let mut signs_in_at = None;
+ let mut retry = 0_u32;
+ loop {
+ let link = match adopted.take() {
+ Some(link) => link,
+ None => {
+ if retry > 0 {
+ if !self.policy.allows(retry) {
+ break;
+ }
+ sleep(self.policy.pause(retry)).await;
+ }
+ match self.open(id, &endpoint, &mut signs_in_at).await {
+ Some(link) => link,
+ None => {
+ retry = retry.saturating_add(1);
+ continue;
+ }
+ }
+ }
+ };
+ let up_since = Instant::now();
+ self.hold(&link).await;
+ self.slot_down(id, &link);
+ if !self.policy.enabled {
+ break;
+ }
+ retry = if up_since.elapsed() >= self.policy.reestablish_after {
Review Comment:
nit: after a long-lived link drops, `retry` resets to 0 and the slot redials immediately with no jitter, so a server restart makes every client redial in the same instant. Drawing jitter for the first redial too (`backoff(0, ..)`) avoids the herd. Driver unused today.
##########
core/sdk/src/driver.rs:
##########
@@ -0,0 +1,1309 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The operation driver sends every request of a binary client. One deadline
+//! bounds the lane wait, the lookups, the attempts and the pauses of a call.
+//! A frame that can have taken effect is never frozen again, so a retry never
+//! applies a request twice.
+
+// Unused until the clients send through the driver.
+#![allow(dead_code)]
+
+use crate::links::{
+ Frame, Link, OPERATION_TIMEOUT, ReplicaLinks, Target, lost_link, pause_before_retry, read_reply,
+};
+use crate::session_owner::{LaneKey, LaneTicket, RetainedOperation, SessionOwner};
+use bytes::Bytes;
+use iggy_binary_protocol::codes::{
+ BIND_SESSION_CODE, GET_PARTITION_ROUTE_CODE, LOGIN_REGISTER_CODE, LOGIN_REGISTER_WITH_PAT_CODE,
+ SEND_MESSAGES_CODE,
+};
+use iggy_binary_protocol::consensus::{Operation, operation_for_code};
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
+use iggy_binary_protocol::requests::messages::GetPartitionRouteRequest;
+use iggy_binary_protocol::responses::messages::PartitionRouteResponse;
+use iggy_binary_protocol::responses::users::LoginRegisterResponse;
+use iggy_binary_protocol::{WireDecode, WireEncode};
+use iggy_common::{ClientState, IggyError, RequestTarget};
+use route_cache::RouteCache;
+use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
+use tokio::time::{Instant, timeout_at};
+use tracing::debug;
+
+mod poll_router;
+mod route_cache;
+
+pub(crate) use poll_router::{
+ PollRouter, PollTransport, ROSTER_READ_TIMEOUT, is_poll_routing_code,
+};
+pub(crate) use route_cache::PartitionKey;
+
+/// Sends the requests of one client over its replica links.
+pub(crate) struct OperationDriver {
+ session: Arc<SessionOwner>,
+ links: ReplicaLinks,
+ routes: Mutex<RouteCache>,
+}
+
+impl OperationDriver {
+ /// The driver shares the session owner of `links`.
+ pub(crate) fn new(links: ReplicaLinks) -> Self {
+ Self {
+ session: Arc::clone(links.session()),
+ links,
+ routes: Mutex::default(),
+ }
+ }
+
+ /// Sends one request and returns the body of its reply.
+ pub(crate) async fn send(
+ &self,
+ code: u32,
+ payload: Bytes,
+ target: RequestTarget<'_>,
+ ) -> Result<Bytes, IggyError> {
+ let deadline = Instant::now() + OPERATION_TIMEOUT;
+ self.drive(Call::new(code, payload, target, deadline)).await
+ }
+
+ /// Replays the retained operation `op` until it has an outcome or
+ /// `deadline` passes. Its success frees the lane.
+ pub(crate) async fn resolve(&self, op: u64, deadline: Instant) -> Result<(), IggyError> {
+ let (ticket, retained) = self.session.resume(op, deadline).await?;
+ self.drive(Call::resumed(ticket, retained, deadline))
+ .await
+ .map(|_| ())
+ }
+
+ /// Frees the lane of the retained operation `op`. Its outcome stays
+ /// unknown.
+ pub(crate) fn abandon(&self, op: u64) -> Result<(), IggyError> {
+ self.session.abandon(op)
+ }
+
+ async fn drive(&self, mut call: Call) -> Result<Bytes, IggyError> {
+ loop {
+ let next = match self.attempt(&mut call).await {
+ Ok(body) => {
+ call.complete();
+ return Ok(body);
+ }
+ Err(next) => next,
+ };
+ call.settle_sent();
+ match next {
+ Next::Again if Instant::now() < call.deadline => {}
+ Next::Again => return Err(call.expire()),
+ Next::Pause => {
+ if !call.pause().await {
+ return Err(call.expire());
+ }
+ }
+ Next::Fail(error) => return Err(call.end(error)),
+ }
+ }
+ }
+
+ /// One attempt of `call`, from the choice of its replica to its reply.
+ async fn attempt(&self, call: &mut Call) -> Result<Bytes, Next> {
+ if let Some(error) = self.stopped() {
+ return Err(Next::Fail(error));
+ }
+ if call.frame.is_some() && call.fence() != Some(self.session.fence()) {
+ // A replaced session can never apply the frame. An uncertain call
+ // ends with its uncertainty, any other call freezes again.
+ if call.uncertain {
+ return Err(Next::Fail(IggyError::Unauthenticated));
+ }
+ call.frame = None;
+ }
+ if call.frame.is_none() {
+ self.wait_for_session(call).await?;
+ }
+ let route = self.route_for(call).await?;
+ let context = self.context(call, route.as_ref());
+ // The lane comes first, so a blocked lane ends the call at once, also
+ // while no link is ready.
+ let ticket = self.ticket(call, &context).await?;
+ let hop = match self.hop(call, route).await {
+ Ok(hop) => hop,
+ Err(next) => {
+ call.ticket = Some(ticket);
+ return Err(next);
+ }
+ };
+ let deadline = call.deadline;
+ let (frame, ticket) = call.frame(ticket, context)?;
+ let result = hop.link.send(frame, deadline, ticket.sent_flag()).await;
+ let sent = *ticket.sent_flag();
+ let result = result.and_then(|reply| read_reply(&self.session, reply));
+ self.settle(call, &hop, result, sent).await
+ }
+
+ /// The error of a call on a client that nothing brings back.
+ fn stopped(&self) -> Option<IggyError> {
+ match self.links.state() {
+ ClientState::Shutdown => Some(IggyError::ClientShutdown),
+ ClientState::Disconnected => Some(IggyError::NotConnected),
+ _ => None,
+ }
+ }
+
+ /// Waits for a bound session while the links sign in again. Without
+ /// credentials nothing signs in, so the call goes on unbound.
+ async fn wait_for_session(&self, call: &Call) -> Result<(), Next> {
+ if is_sign_in(call.code)
+ || self.session.epoch().bound
+ || self.session.sign_in_credentials().is_none()
+ {
+ return Ok(());
+ }
+ let mut epoch = self.session.subscribe_epoch();
+ match timeout_at(call.deadline, epoch.wait_for(|epoch| epoch.bound)).await {
+ Ok(Ok(_)) => Ok(()),
+ Ok(Err(_)) | Err(_) => Err(Next::Fail(IggyError::Unauthenticated)),
+ }
+ }
+
+ /// The route of a partition call, `None` for any other call. A partition
+ /// call looks up its route unless the cached one is usable.
+ async fn route_for(&self, call: &mut Call) -> Result<Option<PartitionRouteResponse>, Next> {
+ let Some(partition) = &call.partition else {
+ return Ok(None);
+ };
+ match self.route(&partition.key, call.deadline).await {
+ Ok(route) => Ok(Some(route)),
+ Err(error) if retries_lookup(&error) => {
+ call.refusal = error;
+ Err(Next::Pause)
+ }
+ Err(error) => Err(Next::Fail(error)),
+ }
+ }
+
+ /// The context that a new frame of `call` carries. Every attempt and
+ /// every replay keeps the context of the first frame.
+ fn context(&self, call: &Call, route: Option<&PartitionRouteResponse>) -> PartitionContext {
+ match (&call.partition, route) {
+ (Some(partition), Some(route)) => partition.context.unwrap_or_else(|| route.context()),
+ _ if operation_for_code(call.code) == Operation::NonReplicated => self.read_context(),
+ _ => PartitionContext::default(),
+ }
+ }
+
+ /// A read without a partition context waits until its replica applied
+ /// the metadata that the session saw.
+ fn read_context(&self) -> PartitionContext {
+ PartitionContext {
+ metadata_op: self.session.metadata_watermark(),
+ ..PartitionContext::default()
+ }
+ }
+
+ /// The lane ticket of `call`. The call enters its lane on its first
+ /// attempt and keeps the ticket until it ends.
+ async fn ticket(
+ &self,
+ call: &mut Call,
+ context: &PartitionContext,
+ ) -> Result<LaneTicket, Next> {
+ match call.ticket.take() {
+ Some(ticket) => Ok(ticket),
+ None => {
+ let key = LaneKey::of(call.code, context, call.partition_id());
+ self.session
+ .enter_lane(key, call.deadline)
+ .await
+ .map_err(Next::Fail)
+ }
+ }
+ }
+
+ /// The link of the next attempt: to the primary of the route, else to
+ /// the metadata primary.
+ async fn hop(
+ &self,
+ call: &mut Call,
+ route: Option<PartitionRouteResponse>,
+ ) -> Result<Hop, Next> {
+ let target = route.map_or(Target::MetadataPrimary, |route| {
+ Target::Replica(route.primary_replica)
+ });
+ let error = match self.links.link(target) {
+ Ok(link) => return Ok(Hop { link, route }),
+ Err(error) => error,
+ };
+ call.refusal = error;
+ match (&call.partition, route) {
+ (Some(partition), Some(route)) => {
+ self.routes()
+ .failed(&route, partition.key.partition_id, route.primary_replica);
+ }
+ _ => self.find_metadata_primary(call.deadline).await,
+ }
+ Err(Next::Pause)
+ }
+
+ /// Reads the outcome of one attempt. `sent` tells that this or an
+ /// earlier attempt can have taken effect.
+ async fn settle(
+ &self,
+ call: &mut Call,
+ hop: &Hop,
+ result: Result<Bytes, IggyError>,
+ sent: bool,
+ ) -> Result<Bytes, Next> {
+ let error = match result {
+ Ok(body) => return self.signed_in(call, hop, body),
+ Err(error) => error,
+ };
+ let repeated = std::mem::replace(
+ &mut call.not_primary,
+ matches!(error, IggyError::NotPrimary),
+ );
+ match error {
+ IggyError::TransientNotCommitted => {
+ call.uncertain = true;
+ Err(Next::Pause)
+ }
+ IggyError::TransientNotAccepted => {
+ call.refusal = error;
+ self.refresh(call, hop, Failure::Refused).await;
+ Err(Next::Pause)
+ }
+ IggyError::NotPrimary => {
+ // Neither was admitted, so the deadline reports both as 58.
+ call.refusal = IggyError::TransientNotAccepted;
+ self.refresh(call, hop, Failure::NotPrimary).await;
+ Err(if repeated { Next::Pause } else { Next::Again })
+ }
+ error if lost_link(&error) => {
+ call.uncertain |= sent;
+ call.refusal = error;
+ self.refresh(call, hop, Failure::LostLink).await;
+ Err(Next::Pause)
+ }
+ // The refusal spent the request id, so the call starts a new
+ // operation in the same lane.
+ IggyError::LifecycleBusy if !call.uncertain => {
+ call.refusal = error;
+ call.frame = None;
+ Err(Next::Pause)
+ }
+ IggyError::HistoryUnavailable => {
+ self.invalidate(call, hop);
+ if call.restarts_on_new_history() {
+ call.restart();
+ return Err(Next::Again);
+ }
+ Err(Next::Fail(error))
+ }
+ IggyError::ConsumerGroupPartitionNotOwned(..) | IggyError::PartitionNotFound(..) => {
+ self.invalidate(call, hop);
+ Err(Next::Fail(error))
+ }
+ error => Err(Next::Fail(error)),
Review Comment:
**warning** (unused driver, must be fixed before it is wired): once a call is uncertain, a committed business rejection (e.g. replayed CreateStream answered `StreamNameAlreadyExists`) falls here and ends as `OperationUncertain`; `resolve()` rebuilds with `uncertain: true` and hits the same arm forever, so the lane stays blocked. A fenced pre-consensus rejection is byte-identical on the wire, so the server needs to mark committed replies before the driver can tell them apart.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4268027118
##########
core/consensus/src/plane_helpers.rs:
##########
@@ -903,6 +903,62 @@ where
.await
}
+/// Find a held entry below a view's suffix that the new history does not prove.
+///
+/// `canonical` is ordered by descending op. Entries above the local commit floor
+/// survive only if their parent chain reaches its lowest header, or a prepare's
+/// commit stamp proves them through the uninterrupted local chain. This runs once
+/// per installed log, before repair starts filling any resulting gap.
+#[must_use]
+pub fn first_unproven_op(
+ commit_floor: u64,
+ canonical: &[PrepareHeader],
+ op_head: u64,
+ last_op: Option<u64>,
+ header_at: impl Fn(u64) -> Option<PrepareHeader>,
+) -> Option<u64> {
+ let last_op = last_op?;
+ let proven = locally_proven_commit(commit_floor, &header_at);
+ let (mut op, mut parent) = canonical.last().map_or((op_head, None), |lowest| {
Review Comment:
**critical** (second trigger): with an empty `canonical` this starts at op_head and stops on the first step. shard/lib.rs:10656 `respond_start_view` ships `local_dvc_suffix()` without refreshing it, despite its comment at :10652. On a Normal primary the cache tag goes stale after every append (the refresh at :7993 is gated on status != Normal), so impls.rs:2538-2546 returns an empty suffix.
The stale old primary that receives it reconciles with `canonical = []`, so `first_unproven_op` stops at op_head on the first step and truncates every held op above its locally proven commit, acked ones included.
Neither caller (:5076-5089, :5122-5132) refreshes, unlike the view-change paths (:4692/:4734). Fix: refresh the metadata/partition suffix before building this StartView, plus the gap fix.
##########
core/shard/src/lib.rs:
##########
@@ -4511,17 +4609,38 @@ where
}
}
+ fn partition_incarnation_matches(
+ &self,
+ partition: &IggyPartition<B, SB>,
+ partition_incarnation: u64,
+ frame: &'static str,
+ ) -> bool {
+ if partition_incarnation == partition.created_revision() {
Review Comment:
nit: frames with incarnation 0, i.e. every SVC/DVC/SV/Commit/RequestPrepares/repair/state-transfer frame from a pre-PR replica, are dropped here, so a mixed-version partition view change cannot complete. Fine if rolling upgrades aren't supported (the boot format gate already refuses old data dirs), but please state that explicitly in the upgrade notes.
##########
core/simulator/src/workload/oracle.rs:
##########
@@ -449,14 +500,15 @@ pub fn assert_converged(sim: &Simulator, workload: &mut Workload) -> Convergence
"committed metadata prefixes agree across every live replica"
);
- let Some(leader) = metadata_leader(sim, &live) else {
+ let Some(leader) = sim.metadata_leader().map(usize::from) else {
panic!("no metadata leader live at quiesce (seed={seed:#x})");
};
// Settlement treats "no live replica hosts this group" as settled, so losing
// every instance of a LIVE partition reads as converged. The metadata read
// separates that from a group whose stream the workload deleted.
let namespaces_checked = assert_live_namespaces_have_a_primary(sim, workload, leader, seed);
Review Comment:
nit (pre-existing, not from this PR; the crash campaigns surfaced them and they're worth their own issues):
1. impls.rs:4180 `RebuildPipeline` starts at `merged.commit_max + 1` even when the parked-view repair walk already applied that op, so the metadata pipeline head sits below commit until the next view (invariants.rs:291 AppliedHead; seeds lifecycle 1191/383, strict 536; base seed 5187 too). Fix: start from `max(commit_min, merged.commit_max) + 1`.
2. shard/lib.rs:4527: a partition created while 2/3 replicas are down rebuilds with elections forbidden wherever there is no retained state, so it never elects and its Delete* never completes (#4476 family). Seed 24 `--plane uniform --shell --clients 2 --journal-slots 96 --faults light --crash-prob 0.003 --restart-prob 0.03 --crash-primary --ticks 30000 --heal-before-quiesce`.
3. Metadata-plane forks under crash-primary swarm exist on base too (base fails 56/93, 13 "committed different history"), so the plane_helpers metadata twin can't be attributed by sim alone.
##########
core/server/src/http/forward.rs:
##########
@@ -332,92 +536,61 @@ fn record_relayed_floor(state: &HttpInner, user_id: u32, response: &Response) {
state.metadata_watermarks.record(user_id, applied);
}
-async fn forward_partition_or_pass(state: HttpState, request: Request, next: Next) -> Response {
- if !state.forward.active || request.headers().contains_key(FORWARDED_HEADER) {
- return next.run(request).await;
- }
- // A read that does not move consumer progress is answered by whichever
- // replica the caller reached, which is the point of reading from one. Only
- // an automatic commit needs a replica that can originate the offset
- // operation, so only that poll pays for the buffering and the credential
- // resolution below.
- if request.method() == Method::GET && !wants_auto_commit(request.uri().query()) {
- return next.run(request).await;
- }
+async fn attach_partition_request(state: HttpState, request: Request, next: Next) -> Response {
+ let received = Instant::now();
+ let hop = match partition_hop(request.headers()) {
+ Ok(hop) => hop,
+ Err(error) => return CustomError::from(error).into_response(),
+ };
+ // Verify-only, like the metadata relay: an unauthenticated caller gets no
+ // buffered body and no relay. The handler authenticates again.
let bearer = match bearer_token(request.headers()) {
Ok(bearer) => bearer,
Err(error) => return CustomError::from(error).into_response(),
};
if let Err(rejection) = resolve_credential(&state, bearer).await {
return rejection.into_response();
}
-
- let (parts, request_body) = request.into_parts();
+ let (mut parts, request_body) = request.into_parts();
let Ok(body) = to_bytes(request_body, state.forward.body_limit).await else {
return error_response(
StatusCode::PAYLOAD_TOO_LARGE,
"payload_too_large",
"request body exceeds http.max_request_size",
);
};
- let method = parts.method.clone();
- let request_headers = parts.headers.clone();
- let path_and_query = parts
- .uri
- .path_and_query()
- .map_or("/", |path_and_query| path_and_query.as_str())
- .to_owned();
- let local = next
- .run(Request::from_parts(parts, Body::from(body.clone())))
- .await;
- if let AttemptOutcome::Relay(response) = classify_local_partition_reply(local).await {
- return response;
- }
-
- let Some(guard) = ForwardGuard::admit(&state.forward.in_flight) else {
- return with_retry_after(error_response(
- StatusCode::SERVICE_UNAVAILABLE,
- "forward_busy",
- "node is at its forward budget; retry with backoff",
- ));
- };
- let self_id = state
- .shard
- .plane
- .metadata()
- .consensus
- .as_ref()
- .map(consensus::VsrConsensus::replica);
- let deadline = Instant::now() + FORWARD_RETRY_DEADLINE;
- let mut skip_node = self_id;
- loop {
- for socket in partition_http_sockets(&state.roster, skip_node) {
- if Instant::now() >= deadline {
- break;
- }
- let url = format!("{}://{socket}{path_and_query}", state.forward.scheme);
- match attempt(&state, &method, &request_headers, &body, &url, false).await {
- AttemptOutcome::Relay(response) => {
- return if method == Method::GET && response.status().is_success() {
- retain_forward_guard(response, guard, FORWARD_ATTEMPT_TIMEOUT)
- } else {
- response
- };
- }
- AttemptOutcome::Retry => {}
- }
- }
- let remaining = deadline.saturating_duration_since(Instant::now());
- if remaining <= FORWARD_RETRY_INTERVAL {
- break;
+ let deadline = received
+ + match hop {
+ PartitionHop::Target(route) => route.budget.min(PARTITION_REQUEST_DEADLINE),
+ PartitionHop::Contacted | PartitionHop::Unrouted => PARTITION_REQUEST_DEADLINE,
+ };
+ // The contacted node chose the partition from metadata at this op. Before
+ // that op is applied here, the topic or the partition may not exist yet,
+ // and the handler would answer 404 for a request it can serve.
+ if let PartitionHop::Target(route) = hop {
+ let frontier = state.shard.plane.metadata().applied_frontier();
+ let budget = compio::time::sleep(deadline.saturating_duration_since(Instant::now()));
+ if hold_for_frontier(frontier, route.context.metadata_op, budget)
Review Comment:
nit: `iggy-forwarded` + `iggy-partition-route` come from the request, so any authenticated client can make this node hold up to the deadline on an arbitrary `metadata_op`, outside the `forward.in_flight` cap. Bounded, but a cheap slow-request lever. Admit Target holds against the in-flight cap.
##########
core/simulator/src/workload/shadow.rs:
##########
@@ -94,7 +315,327 @@ impl Shadow {
consumer_offsets: HashMap::new(),
id_permutation,
next_index: 1,
+ fixtures,
+ fixture_deletes_open: true,
+ deleted_namespaces: Vec::new(),
+ captured_owners: BTreeMap::new(),
+ }
+ }
+
+ pub(crate) fn observe_owner(
+ &mut self,
+ namespace: IggyNamespace,
+ owner: ConsumerGroupOwner,
+ context: PartitionContext,
+ ) {
+ self.captured_owners.retain(|&(ns, _), observed| {
+ ns != namespace || observed.captured.context.incarnation == context.incarnation
+ });
+ for (&(ns, _), observed) in &mut self.captured_owners {
+ if ns == namespace {
+ observed.current = owner;
+ }
+ }
+ if !owner.is_unassigned() {
+ self.captured_owners
+ .entry((namespace, owner.client_id))
+ .or_insert(ObservedOwner {
+ captured: CapturedOwner {
+ namespace,
+ owner,
+ context,
+ },
+ current: owner,
+ });
+ }
+ }
+
+ pub(crate) fn pick_stale_owner(&self, prng: &mut Xoshiro256PlusPlus) -> Option<CapturedOwner> {
+ let stale = || {
+ self.captured_owners.values().filter(|observed| {
+ observed.current.generation > observed.captured.owner.generation
+ && self.namespaces_live.contains(&observed.captured.namespace)
+ })
+ };
+ let count = stale().count();
+ if count == 0 {
+ return None;
+ }
+ stale()
+ .nth(prng.random_range(0..count))
+ .map(|observed| observed.captured)
+ }
+
+ /// Start a toggle of a fixture topic picked uniformly among those that can
+ /// take one. A delete starts a replacement.
+ ///
+ /// After a toggle of unknown fate, the next toggle of its topic is a probe: the
+ /// same half again, whose reply tells what the first one did.
+ ///
+ /// Topic slabs are reused last-freed-first, so a delete starts only while every
+ /// fixture topic is live: with one topic away at a time, each recreated topic
+ /// lands on its own ids.
+ pub fn begin_fixture_toggle(
+ &mut self,
+ prng: &mut Xoshiro256PlusPlus,
+ ) -> Option<(usize, Toggle)> {
+ let deletes_open = self.fixture_deletes_open
+ && self
+ .fixtures
+ .iter()
+ .all(|fixture| fixture.state == FixtureState::Live);
+ let candidates: Vec<usize> = self
+ .fixtures
+ .iter()
+ .enumerate()
+ .filter(|(_, fixture)| fixture.next_toggle(deletes_open).is_some())
+ .map(|(index, _)| index)
+ .collect();
+ if candidates.is_empty() {
+ return None;
+ }
+ let index = candidates[prng.random_range(0..candidates.len())];
+ let fixture = &mut self.fixtures[index];
+ let (state, toggle) = fixture.next_toggle(deletes_open)?;
+ if fixture.state == FixtureState::Live {
+ fixture.replacements.started += 1;
+ }
+ fixture.state = state;
+ Some((index, toggle))
+ }
+
+ /// Whether the toggle in flight on fixture topic `index` is a probe.
+ #[must_use]
+ pub fn fixture_probing(&self, index: usize) -> bool {
+ matches!(
+ self.fixtures[index].state,
+ FixtureState::Deleting { probe: true } | FixtureState::Creating { probe: true }
+ )
+ }
+
+ /// The seeded stream and topic names of fixture topic `index`.
+ #[must_use]
+ pub fn fixture_names(&self, index: usize) -> (String, String) {
+ let fixture = &self.fixtures[index];
+ (fixture.stream.clone(), fixture.topic.clone())
+ }
+
+ /// Move fixture topic `index` by what its toggle's reply proved.
+ ///
+ /// # Panics
+ /// If the topic has no toggle in flight: every toggle finishes exactly once.
+ pub fn finish_fixture_toggle(&mut self, index: usize, fate: ToggleFate) {
+ let previous = self.fixtures[index].state;
+ let Some(next) = previous.after(fate) else {
+ panic!(
+ "fixture topic {} finished a toggle with {fate:?} while {previous:?}",
+ self.fixtures[index].topic,
+ );
+ };
+ self.fixtures[index].state = next;
+ let namespaces = &self.fixtures[index].namespaces;
+ match next {
+ // A refused delete replaced nothing. Counted as a settle instead, it
+ // would leave a request sampled during it free to see a replacement.
+ FixtureState::Live => self.fixtures[index].replacements.started -= 1,
+ FixtureState::Deleted if matches!(previous, FixtureState::Deleting { .. }) => {
+ self.namespaces_live
+ .retain(|namespace| !namespaces.contains(namespace));
+ self.sends_committed
+ .retain(|namespace, _| !namespaces.contains(namespace));
+ self.consumer_offsets
+ .retain(|(namespace, _, _), _| !namespaces.contains(namespace));
+ self.captured_owners
+ .retain(|(namespace, _), _| !namespaces.contains(namespace));
+ self.deleted_namespaces.extend(namespaces);
+ if let Some(group) = &mut self.fixtures[index].group {
+ group.exists = false;
+ group.joined = false;
+ if let ToggleFate::Reached { op } = fate {
+ group.deletes_reached.push(op);
+ }
+ }
+ }
+ FixtureState::Settling { .. } => self.namespaces_live.extend(namespaces),
+ _ => {}
+ }
+ }
+
+ /// Settle every fixture topic that `hosted(op, namespaces)` reports as applied
+ /// at its create op and hosted at the recreated incarnation.
+ pub fn settle_fixtures(&mut self, mut hosted: impl FnMut(u64, &[IggyNamespace]) -> bool) {
+ for fixture in &mut self.fixtures {
+ if let FixtureState::Settling { op } = fixture.state
+ && hosted(op, &fixture.namespaces)
+ {
+ fixture.state = FixtureState::Live;
+ fixture.replacements.settled += 1;
+ }
+ }
+ }
+
+ /// Take the namespaces of the fixture topics whose delete was reached since
+ /// the last call. Every replica hosts the deleted incarnation until the topic
+ /// is recreated.
+ pub fn take_deleted_namespaces(&mut self) -> Vec<IggyNamespace> {
+ std::mem::take(&mut self.deleted_namespaces)
+ }
+
+ /// Start no more fixture deletes, so the toggles left recreate every topic.
+ pub const fn close_fixture_deletes(&mut self) {
+ self.fixture_deletes_open = false;
+ }
+
+ /// Whether every fixture topic is live with no toggle in flight, and its
+ /// fixture group, if the run has one, was joined since the topic was last
+ /// replaced.
+ #[must_use]
+ pub fn fixtures_restored(&self) -> bool {
+ self.fixtures.iter().all(|fixture| {
+ fixture.state == FixtureState::Live
+ && fixture.group.as_ref().is_none_or(|group| group.joined)
+ })
+ }
+
+ /// Replacement counts of the fixture topic holding `namespace`, zero for any
+ /// other namespace.
+ #[must_use]
+ pub fn replacements(&self, namespace: IggyNamespace) -> Replacements {
+ self.fixtures
+ .iter()
+ .find(|fixture| fixture.namespaces.contains(&namespace))
+ .map_or_else(Replacements::default, |fixture| fixture.replacements)
+ }
+
+ /// A pending delete can still refuse, so retain committed effects until its
+ /// success clears them. After a successful delete, late replies belong to the
+ /// old history until the replacement settles.
+ #[must_use]
+ pub fn effect_is_current(&self, namespace: IggyNamespace, sampled: Replacements) -> bool {
+ self.fixtures
+ .iter()
+ .find(|fixture| fixture.namespaces.contains(&namespace))
+ .is_none_or(|fixture| {
Review Comment:
nit: the strict offset oracle gives false positives across fixture replacement. During `Settling` the namespaces are live again (:456), so a StoreConsumerOffset on the recreated incarnation commits and is acked, but this returns false and the effect becomes `Effect::None`. At quiesce, oracle.rs:900 panics `left: {(1,2): 0} right: {}`.
12/600 seeds of `--plane lifecycle --shell --clients 1 --ack-quorum-ratio 1.0 --faults swarm --crash-prob 0.002 --restart-prob 0.02 --crash-primary --ticks 25000 --heal-before-quiesce`: 132, 146, 168, ... The server is correct.
Fix: treat `Settling` as current when `sampled.started == fixture.replacements.started`, or gate the strict compare on every fixture being `Live`.
##########
core/consensus/src/impls.rs:
##########
@@ -3666,6 +3696,9 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
// it: the new primary's numbers are at least as fresh as any probe
// answer.
self.finish_view_probe();
+ let needs_reconciliation = self.view_log_unproven.get()
Review Comment:
nit: if a metadata backup's truncate fails (I/O error) it keeps `view_log_unproven`, but the same-view early return at :3647-3653 never re-runs the reconcile, so commit stays fenced (metadata.rs:4227) until the next view change. Retry the reconcile here when the proof is still unsettled.
##########
core/message_bus/src/transports/quic.rs:
##########
@@ -326,12 +316,205 @@ fn reply_route(frame: &BusMessage) -> ReplyRoute {
if !is_reply {
return ReplyRoute::Unkeyed;
}
+ read_u64(bytes, offset_of!(ReplyHeader, request)).map_or(ReplyRoute::Unkeyed, ReplyRoute::Keyed)
+}
+
+/// The request id of a client frame, read as raw bytes like [`reply_route`],
+/// so a frame with an undeclared operation still has one.
+fn request_id(request: &Message<GenericHeader>) -> Option<u64> {
+ read_u64(request.as_slice(), offset_of!(RequestHeader, request))
+}
+
+fn read_u64(bytes: &[u8], offset: usize) -> Option<u64> {
bytes
- .get(..HEADER_SIZE)
- .and_then(|header| bytemuck::checked::try_from_bytes::<ReplyHeader>(header).ok())
- .map_or(ReplyRoute::Unkeyed, |header| {
- ReplyRoute::Keyed(header.request)
- })
+ .get(offset..offset + size_of::<u64>())
+ .and_then(|field| field.try_into().ok())
+ .map(u64::from_le_bytes)
+}
+
+/// Hands the outbound frames of one connection to the streams that wait
+/// for them.
+///
+/// A stream registers its waiter before it forwards its request, so a reply
+/// never comes before the stream it answers.
+struct ReplyRouter {
+ /// Waiting streams by request id, oldest first.
+ waiters: HashMap<u64, VecDeque<oneshot::Sender<BusMessage>>>,
+ /// Frames that came while no stream waited for them, oldest first.
+ unclaimed: VecDeque<BusMessage>,
+ /// When `unclaimed` holds this many frames, it drops its oldest one.
+ unclaimed_max: usize,
+}
+
+impl ReplyRouter {
+ fn new(unclaimed_max: usize) -> Self {
+ Self {
+ waiters: HashMap::new(),
+ unclaimed: VecDeque::new(),
+ unclaimed_max,
+ }
+ }
+
+ /// Register a stream that waits for the reply to `request_id`.
+ ///
+ /// The stream first takes the oldest unclaimed frame that has no request
+ /// id or has its own. The SDK replays an explicit transient with the same
+ /// request id on a new stream, so a late reply of the first attempt
+ /// answers the replay. The replies for other ids ahead of that frame are
+ /// dropped, because their streams already ended.
+ fn register(&mut self, request_id: u64) -> oneshot::Receiver<BusMessage> {
+ let (waiter, reply) = oneshot::channel();
+ while let Some(frame) = self.unclaimed.pop_front() {
+ if !matches!(reply_route(&frame), ReplyRoute::Keyed(id) if id != request_id) {
Review Comment:
nit: an unclaimed unkeyed Eviction is handed to whichever stream registers next, even one opened after the eviction. It is harmless today because Eviction is session-terminal. Dropping unclaimed Evictions older than the registering stream would make that explicit.
##########
core/consensus/src/plane_helpers.rs:
##########
@@ -903,6 +903,62 @@ where
.await
}
+/// Find a held entry below a view's suffix that the new history does not prove.
+///
+/// `canonical` is ordered by descending op. Entries above the local commit floor
+/// survive only if their parent chain reaches its lowest header, or a prepare's
+/// commit stamp proves them through the uninterrupted local chain. This runs once
+/// per installed log, before repair starts filling any resulting gap.
+#[must_use]
+pub fn first_unproven_op(
+ commit_floor: u64,
+ canonical: &[PrepareHeader],
+ op_head: u64,
+ last_op: Option<u64>,
+ header_at: impl Fn(u64) -> Option<PrepareHeader>,
+) -> Option<u64> {
+ let last_op = last_op?;
+ let proven = locally_proven_commit(commit_floor, &header_at);
+ let (mut op, mut parent) = canonical.last().map_or((op_head, None), |lowest| {
+ (lowest.op.saturating_sub(1), Some(lowest.parent))
+ });
+ while op > proven {
Review Comment:
**critical**: committed ops are lost. Both this walk and `locally_proven_commit` (:946) stop at the first op this replica doesn't hold, so a repair gap counts as divergence. Every held entry in `(proven, gap]` is returned and `truncate_uncommitted_from` (iggy_partition.rs:8727) cuts it, including acked entries that link to the canonical head and ops at or below the merged commit_max.
If a later view-change quorum excludes the remaining holder, the committed op is gone and its slot is re-prepared with different bytes.
Repro: `workload-fuzz --seed 7016 --ticks 60000 --clients 4 --replicas 3 --plane partition --faults swarm --crash-prob 0.005 --restart-prob 0.02 --crash-primary --require-faults` panics at state_checker.rs:230: "replica 2 and witnesses {0: 6} disagree on committed partition op 93". With only the call at shard/lib.rs:11646 disabled, the seed passes.
Fix: treat MISSING as unknown. Keep held entries above the first gap but fence them (no commit, PrepareOk, DVC present-bit or serving), let repair replace a held entry that doesn't match (`apply_repaired_prepare` skips held ops at iggy_partition.rs:11600), and re-prove once gaps fill. "Never cut <= commit_max" alone is unsafe: a dead-branch op next to a gap would be applied. The metadata path (shard/lib.rs:6486) has the same construction.
##########
core/simulator/src/workload/oracle.rs:
##########
@@ -449,14 +500,15 @@ pub fn assert_converged(sim: &Simulator, workload: &mut Workload) -> Convergence
"committed metadata prefixes agree across every live replica"
);
- let Some(leader) = metadata_leader(sim, &live) else {
+ let Some(leader) = sim.metadata_leader().map(usize::from) else {
panic!("no metadata leader live at quiesce (seed={seed:#x})");
};
// Settlement treats "no live replica hosts this group" as settled, so losing
// every instance of a LIVE partition reads as converged. The metadata read
// separates that from a group whose stream the workload deleted.
let namespaces_checked = assert_live_namespaces_have_a_primary(sim, workload, leader, seed);
+ assert_hosted_incarnations_named(sim, workload, &live, seed);
// Safety direction: no live replica may have COMMITTED more of a group than
Review Comment:
nit: convergence compares prepare identity chains and offset tables, never applied segment bytes or messages read back across replicas. Divergence introduced at apply, rebuild or state-transfer install goes unseen. Separately, partition state transfer is refused in the sim ("no on-disk directory"), so crash campaigns report false wedges (seed 1078 `--plane mixed`).
Add a per-namespace read-back digest at quiesce. Until then the PR body's "Simulator checks cover replacement histories" should be softened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): route partition requests to their primaries [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4260669737
##########
core/server/src/dispatch/mod.rs:
##########
@@ -557,25 +688,48 @@ async fn handle_client_request<B, MJ, S, SB>(
// (`submit_register_local_or_forward`). Every other non-replicated
// code MUST go through Register first, which binds the acting user
// the per-op authz gates resolve.
- handle_non_replicated_request(
- shard,
- sessions,
- server_config,
- transport_client_id,
- request,
- ConnectionContext {
- bound,
- user_id,
- address: client_address,
- metadata_watermark,
- },
- )
- .await;
+ let context = ConnectionContext {
+ bound,
+ user_id,
+ address: client_address,
+ metadata_watermark,
+ };
+ // `ClientDrain` says which frames stay on the drain.
+ if matches!(non_replicated_code(header), PING_CODE | BIND_SESSION_CODE) {
+ handle_non_replicated_request(
+ shard,
+ sessions,
+ server_config,
+ transport_client_id,
+ request,
+ context,
+ )
+ .await;
+ return;
+ }
+ let in_flight = drain.admit(Admission::Cap).await;
+ let task_shard = Rc::clone(shard);
+ let task_sessions = Rc::clone(sessions);
+ let task_config = Arc::clone(server_config);
+ shard.bus.spawn(async move {
Review Comment:
**warning**: every non-PING read is spawned per frame, and every in-tree SDK is lockstep, so each poll pays a new drain + task. Measured vs master: standalone poll -14..-17%, consumer under concurrent produce -24..-48%. It also lets two pipelined `Next`/auto-commit disk-tier polls on one connection plan the same offset and deliver twice (and a promoted poll can overwrite a later StoreConsumerOffset).
Suggest: take `InFlight` first, run the read inline and promote it to a task only when another frame arrives (PING still overtakes a held read); a one-slot poll lane keyed by the resolved (namespace, consumer), released after the reply, that offset store/delete also wait on.
##########
core/metadata/src/impls/metadata.rs:
##########
@@ -3107,20 +3099,22 @@ where
.as_ref()
.expect("submit_request_in_process: consensus only exists on shard 0");
- // Not-primary is transient: the same request replayed against the
- // current primary commits fine. Reply with the explicit transient
- // frame (relayed to the socket by the home shard) so the client
- // replays immediately rather than waiting out its read-timeout.
- // `TransientNotAccepted` specifically: the request never entered the
- // pipeline here, so the client may re-issue it ANYWHERE -- including
- // under a fresh session after failing over to the current leader --
- // without double-apply risk. (`TransientNotCommitted` conversely means
- // the outcome is unknown and only a same-session replay is safe.)
+ // Reply before admission so the client can retry without a read timeout.
+ // This refusal covers only this attempt; earlier copies may still commit.
+ // Retrying with their session and request id preserves deduplication.
//
// No catch-up gate: a non-register op admitted mid-commit-window
// simply pipelines behind the in-flight batch (the wire path has
// always done this); the register-specific invariant is guarded in
// `submit_register_in_process` / `register_preflight`.
+ if consensus.is_normal_backup() {
+ return Ok(build_result_rejection_reply(
Review Comment:
**critical**: backups now answer `NotPrimary` (89) here and in `iggy_partition.rs:6755` where they answered 58. No foreign SDK knows 89, and their retry / roster-walk predicates only match 58 (node `client.socket.ts:59`, go `tcp_core.go:746`, java `AsyncTcpConnection`, C# `VsrConnection.cs:150`), so in a cluster these requests fail terminally. Single-node e2e won't catch it.
Please add 89 everywhere and treat it exactly like 58, keeping the rule that a refusal after an uncertain attempt doesn't prove non-commit.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -11670,12 +11919,11 @@ where
fn request_history_available(&self, header: &RoutedRequestHeader) -> bool {
header.operation == Operation::RetireSession
|| header.operation.is_partition_lifecycle()
- || (header.partition_incarnation == self.created_revision && !self.history_deleted())
+ || (header.partition_incarnation == self.created_revision() && !self.history_deleted())
}
- fn prepare_history_available(&self, header: &PrepareHeader) -> bool {
- header.operation == Operation::RetireSession
- || (header.partition_incarnation == self.created_revision && !self.history_deleted())
+ const fn prepare_history_available(&self, header: &PrepareHeader) -> bool {
Review Comment:
**warning**: the `RetireSession` exemption was dropped here but kept in `request_history_available` above, so the primary prepares a retirement that every backup with the delete fence applied discards (`on_replicate`, `apply_repaired_prepare`). No quorum, the op sits at the pipeline head and the liveness sweep never completes; if a lagging backup acks, the third replica can't repair it.
Suggest `incarnation == created_revision() && (op == RetireSession || !history_deleted())`, fix the stale doc above, and let `SessionRetired` (`shard/src/poll.rs:253`) answer true once the fence applied, carrying `created_revision` so a recreated incarnation never answers for the old one.
##########
core/binary_protocol/src/responses/system/get_cluster_metadata.rs:
##########
@@ -60,6 +64,7 @@ impl WireEncode for ClusterNodeResponse {
#[allow(clippy::cast_possible_truncation)]
fn encode(&self, buf: &mut BytesMut) {
+ buf.put_u8(self.replica_id);
Review Comment:
**critical**: the leading `replica_id` byte shifts every node for old decoders (node `cluster.utils.ts:56`, go `cluster_node.go:91`, java `BytesDeserializer.java:560`, C# `BinaryMapper.cs:1428`), breaking GetClusterMetadata and the poll/offset routing replies that embed a node. Nodes have no length prefix, so there is no compatible server-side form.
This also turns bdd-go/java/node/csharp red (they depend on rust-server and call GetClusterMetadata). Please update the decoders, plus `replicaId: 0` in node `tcp.cluster.e2e.ts` expectedMeta.
##########
core/server/src/dispatch/mod.rs:
##########
@@ -659,6 +819,7 @@ async fn handle_client_request<B, MJ, S, SB>(
// stamps the consensus client / session / group over the header,
// and a pre-consensus deny still has to echo what the client sent.
let header = *header;
+ let in_flight = drain.admit(Admission::MetadataLane).await;
Review Comment:
**warning**: the metadata write commits in a spawned task, and nothing holds back the frames behind it on the same connection. A pipelined CreateTopic then SendMessages resolves the namespace from the local STM before the topic commits and gets a permanent `TopicIdNotFound`; a write then GetTopic misses the write (the read's watermark is captured before this one is recorded at :886). Reachable today via the Java async client.
Suggest parking every later non-PING frame (before it takes a slot) while the metadata lane is held.
##########
core/server/src/dispatch/host.rs:
##########
@@ -19,27 +19,28 @@
//! mode build a shard on.
//!
//! One [`ServerHost`] per shard owns everything the request plane shares
-//! across its transports: the per-client FIFO queues and their drain slots,
-//! the [`SessionManager`], and the bus connection-lost hook that logs a
-//! dropped connection out. All connections installed on the shard dispatch
-//! through the same instance, which preserves ordering and disconnect
-//! cleanup across transports: the destination shard installs it on delegated
-//! TCP/WS/TCP-TLS/WSS connections, and shard 0 also on its local QUIC
-//! connections.
+//! across its transports: the per-client FIFO queues and their drains, the
+//! [`SessionManager`], and the bus connection-lost hook that logs a dropped
+//! connection out. All connections installed on the shard dispatch through
+//! the same instance, which keeps disconnect cleanup equal across transports:
+//! the destination shard installs it on delegated TCP/WS/TCP-TLS/WSS
+//! connections, and shard 0 also on its local QUIC connections. A drain runs
+//! the frames that later frames depend on in arrival order, and every other
Review Comment:
nit: the commit wait of a metadata write is not in the set of frames the drain orders (see :822 in `mod.rs`), so "the frames that later frames depend on" overstates the contract. Please state the actual ordering guarantee here and in `mod.rs:256-262`.
##########
core/server/src/http/handlers.rs:
##########
@@ -1314,12 +1313,28 @@ pub(in crate::http) async fn poll_messages(
.streams()
.poll_metadata(namespace, None, HTTP_READ_CLIENT_ID)
.ok_or(ReadError::Rejected(IggyError::TransientNotAccepted))?;
- let context = metadata.context(state.shard.plane.metadata().applied_frontier().get());
- if let Some(requested) = query.strategy.context {
- // TCP parity: a group poll already got the re-sync sentinel at resolve
- // above, so this poll has no group and names group 0, as TCP does.
- fence_poll_context(requested, context, 0, partition_id).map_err(ReadError::Rejected)?;
+ let local = metadata.context(state.shard.plane.metadata().applied_frontier().get());
+ let requested = request
+ .route()
+ .map(|route| route.context)
+ .or(query.strategy.context)
+ .unwrap_or(local);
+ let effect = if query.auto_commit {
+ PartitionEffect::Write
+ } else {
+ PartitionEffect::Read
+ };
+ if let Some(response) = SendWrapper::new(route_partition(
Review Comment:
**warning**: every HTTP partition request (here and :1437, :1558, :1657, :1723) first does a cross-shard `PartitionRead::Primary` probe via `route_partition`, even on the primary and on the hop target; plain GETs used to run locally. HTTP produce benched about -10%.
Suggest running locally first with a primary-gated read, and relaying only on a refusal that proves non-admission with deadline left. A missing reply or spent deadline must be the final answer (never re-run), and the session-gate timeout should get its own error rather than look like "not primary".
##########
core/server/src/http/handlers.rs:
##########
@@ -1314,12 +1313,28 @@ pub(in crate::http) async fn poll_messages(
.streams()
.poll_metadata(namespace, None, HTTP_READ_CLIENT_ID)
.ok_or(ReadError::Rejected(IggyError::TransientNotAccepted))?;
- let context = metadata.context(state.shard.plane.metadata().applied_frontier().get());
- if let Some(requested) = query.strategy.context {
- // TCP parity: a group poll already got the re-sync sentinel at resolve
- // above, so this poll has no group and names group 0, as TCP does.
- fence_poll_context(requested, context, 0, partition_id).map_err(ReadError::Rejected)?;
+ let local = metadata.context(state.shard.plane.metadata().applied_frontier().get());
+ let requested = request
+ .route()
+ .map(|route| route.context)
+ .or(query.strategy.context)
+ .unwrap_or(local);
+ let effect = if query.auto_commit {
+ PartitionEffect::Write
+ } else {
+ PartitionEffect::Read
+ };
+ if let Some(response) = SendWrapper::new(route_partition(
+ &state, &request, namespace, requested, effect,
+ ))
+ .await
+ {
+ return Ok(response);
}
+ // `poll_wire_request` refused a group poll above, so this poll has no
+ // member and names client 0, the HTTP read client.
+ fence_poll_context(requested, local, HTTP_READ_CLIENT_ID, partition_id)
+ .map_err(ReadError::Rejected)?;
let reply = SendWrapper::new(state.shard.partition_read(
namespace,
PartitionRead::Poll {
Review Comment:
nit: `PartitionRead::Poll` never refuses on a backup, and the primary check in `route_partition` is a separate cross-shard message, so a view change in between serves this on a backup (stale stored offset for `Next`; auto-commit only refused later in `complete_poll`). Small window today, but a local-first change would make it the main path. An OnPrimary-gated read variant closes it.
##########
core/server/src/http/forward.rs:
##########
@@ -241,19 +361,108 @@ pub(in crate::http) async fn forward_to_primary(
SendWrapper::new(forward_or_pass(state, request, next)).await
}
-/// Route-layer fallback for acknowledged partition writes over HTTP.
-///
-/// HTTP has no persistent leader-aware connection to retarget. Execute on the
-/// contacted node first, then retry across the configured HTTP nodes within
-/// the deadline after a typed `TransientNotAccepted` denial. That
-/// denial proves the write never entered a partition pipeline. Every ambiguous
-/// outcome is returned without replay.
-pub(in crate::http) async fn forward_partition_write(
+/// Route layer of the partition routes: buffer the request once and attach it
+/// as a [`PartitionRequest`], so its handler can relay it to the partition
+/// primary with [`route_partition`]. No attempt runs here.
+pub(in crate::http) async fn forward_partition_request(
State(state): State<HttpState>,
request: Request,
next: Next,
) -> Response {
- SendWrapper::new(forward_partition_or_pass(state, request, next)).await
+ SendWrapper::new(attach_partition_request(state, request, next)).await
+}
+
+/// Decide where a partition request runs once its partition and context are
+/// fixed. `None` serves it on this node. `Some` is the final answer: the reply
+/// of the partition primary, or a refusal that proves nothing was admitted.
+///
+/// Until the deadline, every hop waits for a view with a primary, because a
+/// new partition starts on each replica at its own pace. A hop target never
+/// relays: it refuses with `NotPrimary` when another replica is the primary,
+/// so a stale view never chains hops. The contacted node starts another turn
+/// only after such a proven non-admission, a failed connect, or a read that
+/// failed after its send.
+pub(in crate::http) async fn route_partition(
+ state: &HttpInner,
+ request: &PartitionRequest,
+ namespace: IggyNamespace,
+ context: PartitionContext,
+ effect: PartitionEffect,
+) -> Option<Response> {
+ if request.hop == PartitionHop::Unrouted {
+ return None;
+ }
+ let Ok(partition_id) = u32::try_from(namespace.partition_id()) else {
+ return Some(CustomError::from(IggyError::InvalidCommand).into_response());
+ };
+ let mut guard = None;
+ loop {
+ if let Some(primary) = partition_primary(state, namespace).await {
Review Comment:
nit: after the hop target answers `NotPrimary`, the retry re-resolves the primary only from this node's own view, so a lagging node keeps relaying to the same stale primary until the 10s deadline. Carrying the refusing node's view/primary as a hint (like `RouteCache::not_primary`) would fix it; fine as a follow-up.
##########
core/binary_protocol/src/responses/messages/partition_route.rs:
##########
@@ -0,0 +1,157 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::WireError;
+use crate::codec::{WireDecode, WireEncode, read_u8, read_u32_le, read_u64_le};
+use crate::primitives::partition_history::PartitionContext;
+use bytes::{BufMut, BytesMut};
+
+/// One replica's route to a partition, the reply to `GetPartitionRoute` (105).
+///
+/// The incarnation and the partition count come from metadata at
+/// `metadata_op` or later. The view and its primary come from this replica's
+/// copy of the partition group, so another replica can answer a higher view.
+///
+/// Wire format, exactly 25 bytes, little endian:
+/// ```text
+/// [incarnation:8][metadata_op:8][view:4][partitions_count:4][primary_replica:1]
+/// ```
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PartitionRouteResponse {
Review Comment:
**critical**: 105 keeps its code but the reply grows from the 24-byte `PartitionContext` to this 25-byte route. Node, Go, Java and C# still decode the old shape: Go rejects every explicit-partition send (`contracts/partitions.go:36` wants 24), Node/C# check only `<` and misread `metadata_op` as `view|partitions_count<<32`, so sends spin on `TransientNotAccepted` until the deadline.
Please update all four SDKs (rename to GetPartitionRoute, decode exactly 25 bytes into {incarnation, owner 0, metadata_op}, use `!=` not `<`, regenerate fixtures).
##########
core/server/src/dispatch/mod.rs:
##########
@@ -240,13 +249,130 @@ impl Drop for ActiveDrainSlot {
}
}
+/// The state that the drain task of one connection shares with the requests
+/// it spawned.
+///
+/// The drain takes the frames of the connection in arrival order and runs the
+/// prologue of each. It finishes a frame that later frames depend on before it
+/// takes the next one: a login, a logout, a `BIND_SESSION`, the rewrite of a
+/// metadata write, or a partition admission. A `PING` finishes there too,
Review Comment:
nit: "A `PING` finishes there too, because it never waits" holds only when the PING is at the queue head. A PING behind a frame parked in `admit`, `wait_for_partition_routable` or an attached relay waits too, and `touch_connection` never runs for it, so a long metadata stall can evict a pipelining client that keeps pinging. Answer or touch PING on enqueue, or narrow the doc.
##########
core/server/src/dispatch/mod.rs:
##########
@@ -240,13 +249,130 @@ impl Drop for ActiveDrainSlot {
}
}
+/// The state that the drain task of one connection shares with the requests
+/// it spawned.
+///
+/// The drain takes the frames of the connection in arrival order and runs the
+/// prologue of each. It finishes a frame that later frames depend on before it
+/// takes the next one: a login, a logout, a `BIND_SESSION`, the rewrite of a
+/// metadata write, or a partition admission. A `PING` finishes there too,
+/// because it never waits. Every other read, every poll and every wait for a
+/// metadata commit runs in a task of its own, so a held frame does not stall
+/// the frames behind it.
+///
+/// `capacity` caps those tasks at `message_bus.client_queue_capacity`, the
+/// outbound queue size of a connection, so their replies never overflow that
+/// queue. A metadata write, a segment delete, a login and a logout also take
+/// the metadata lane, which holds one of them at a time. The metadata client
+/// table keeps one pending write per client and refuses a lower request id
+/// after a higher one committed, so a second write in flight would fail with
+/// `TransientNotAccepted` or `RequestTooOld`. The lane also runs the rewrite of
+/// a write after the previous write replied, because the rewrite reads state
+/// that the previous write can change.
+///
+/// At the cap or the lane the drain waits, and the frames behind it queue up
+/// to [`MAX_QUEUED_CLIENT_REQUESTS`]. While tasks run, the drain outlives its
+/// queue, so the cap and the lane still count them when the next frame
+/// arrives.
+pub(in crate::dispatch) struct ClientDrain {
+ capacity: usize,
+ in_flight: Cell<usize>,
+ /// Whether a frame holds the metadata lane.
+ metadata_write: Cell<bool>,
+ /// The drain task, while it waits for a frame or for room to admit one.
+ parked: Cell<Option<Waker>>,
+}
+
+/// What a frame waits for before [`ClientDrain::admit`] counts it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum Admission {
+ /// Room under the in-flight cap.
+ Cap,
+ /// Room under the cap and a free metadata lane.
+ MetadataLane,
+}
+
+impl ClientDrain {
+ pub(in crate::dispatch) const fn new(server_config: &ServerConfig) -> Self {
+ Self {
+ capacity: server_config.message_bus.client_queue_capacity,
Review Comment:
**warning**: spawned reads are capped by `client_queue_capacity` (256), so one pipelining connection can hold 256 concurrent polls and take a large share of the shard's `poll_completion_capacity` (1024, counts ops not bytes). A small dedicated read cap (around 8) would bound it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): route partition requests to their primaries [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4506:
URL: https://github.com/apache/iggy/pull/4506#issuecomment-6146510377
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/4506?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 87.77%. Comparing base ([`a89b26a`](https://app.codecov.io/gh/apache/iggy/commit/a89b26a4a441bf09cfa1930f7b6648f859cb57e9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`94aa5de`](https://app.codecov.io/gh/apache/iggy/commit/94aa5de40ff5d5a3e007f6a5f2eaf605c97cd748?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4506 +/- ##
============================================
- Coverage 88.25% 87.77% -0.48%
Complexity 1726 1726
============================================
Files 1316 1313 -3
Lines 266111 264553 -1558
Branches 224746 223399 -1347
============================================
- Hits 234863 232220 -2643
- Misses 25731 26889 +1158
+ Partials 5517 5444 -73
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/4506/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/4506/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `89.94% <ø> (-0.04%)` | :arrow_down: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/4506/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.12% <ø> (-0.06%)` | :arrow_down: |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/4506/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `65.01% <ø> (-0.92%)` | :arrow_down: |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/4506/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `90.43% <ø> (ø)` | |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/4506/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.84% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/4506/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `92.32% <ø> (-2.82%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/4506/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `60.68% <ø> (-9.72%)` | :arrow_down: |
| [C++ SDK](https://app.codecov.io/gh/apache/iggy/pull/4506/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.43% <ø> (ø)` | |
| [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/4506?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core/binary\_protocol/src/codes.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fcodes.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL2NvZGVzLnJz) | `100.00% <ø> (ø)` | |
| [core/binary\_protocol/src/consensus/command.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fconsensus%2Fcommand.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL2NvbnNlbnN1cy9jb21tYW5kLnJz) | `100.00% <ø> (ø)` | |
| [core/binary\_protocol/src/consensus/header.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fconsensus%2Fheader.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL2NvbnNlbnN1cy9oZWFkZXIucnM=) | `83.57% <ø> (ø)` | |
| [core/binary\_protocol/src/consensus/request.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fconsensus%2Frequest.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL2NvbnNlbnN1cy9yZXF1ZXN0LnJz) | `95.65% <ø> (ø)` | |
| [core/binary\_protocol/src/dispatch.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fdispatch.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL2Rpc3BhdGNoLnJz) | `89.28% <ø> (-0.08%)` | :arrow_down: |
| [core/binary\_protocol/src/primitives/identifier.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fprimitives%2Fidentifier.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL3ByaW1pdGl2ZXMvaWRlbnRpZmllci5ycw==) | `95.96% <ø> (ø)` | |
| [...inary\_protocol/src/primitives/partition\_history.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fprimitives%2Fpartition_history.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL3ByaW1pdGl2ZXMvcGFydGl0aW9uX2hpc3RvcnkucnM=) | `100.00% <ø> (ø)` | |
| [...equests/partitions/retire\_consumer\_group\_owners.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Frequests%2Fpartitions%2Fretire_consumer_group_owners.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL3JlcXVlc3RzL3BhcnRpdGlvbnMvcmV0aXJlX2NvbnN1bWVyX2dyb3VwX293bmVycy5ycw==) | `92.30% <ø> (ø)` | |
| [...ry\_protocol/src/responses/messages/poll\_routing.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fresponses%2Fmessages%2Fpoll_routing.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL3Jlc3BvbnNlcy9tZXNzYWdlcy9wb2xsX3JvdXRpbmcucnM=) | `100.00% <ø> (ø)` | |
| [...tocol/src/responses/system/get\_cluster\_metadata.rs](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fresponses%2Fsystem%2Fget_cluster_metadata.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL3Jlc3BvbnNlcy9zeXN0ZW0vZ2V0X2NsdXN0ZXJfbWV0YWRhdGEucnM=) | `100.00% <ø> (ø)` | |
| ... and [85 more](https://app.codecov.io/gh/apache/iggy/pull/4506?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [105 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/4506/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4263203905
##########
core/simulator/src/workload/oracle.rs:
##########
@@ -666,17 +767,161 @@ fn assert_committed_metadata_agrees(sim: &Simulator, live: &[usize], seed: u64)
compared
}
-/// The live replica whose metadata consensus is the current primary, i.e. the
-/// authoritative holder of the committed metadata log.
-fn metadata_leader(sim: &Simulator, live: &[usize]) -> Option<usize> {
- live.iter().copied().find(|&replica_idx| {
- sim.replicas[replica_idx].shards[0]
- .plane
- .metadata()
- .consensus
- .as_ref()
- .is_some_and(|consensus| consensus.is_primary() && consensus.status() == Status::Normal)
- })
+/// Require the partition primary to hold each owner that leader metadata settled,
+/// and equally committed replicas to agree. Returns how many owners it compared.
+///
+/// Metadata settles an owner only after a partition installed it, so a settled
+/// owner the primary lacks was lost or never installed. Metadata names no owner
+/// for an unassigned partition, which the partition installs as an owner with no
+/// client.
+fn assert_settled_owners_installed(
+ sim: &Simulator,
+ workload: &Workload,
+ leader: usize,
+ seed: u64,
+) -> usize {
+ let streams = sim.replicas[leader].shards[0]
+ .plane
+ .metadata()
+ .mux_stm
+ .streams();
+ let mut compared = 0;
+ for &ns in &workload.options.namespaces {
+ let Some(primary) = sim.primary_index(ns) else {
+ continue;
+ };
+ let settled: Vec<(u64, Option<ConsumerGroupOwner>)> = streams.read(|inner| {
+ let Some(topic) = inner
+ .items
+ .get(ns.stream_id())
+ .and_then(|stream| stream.topics.get(ns.topic_id()))
+ else {
+ return Vec::new();
+ };
+ topic
+ .consumer_groups
+ .iter()
+ .filter_map(|(&group_id, group)| {
+ let assignment = group.assignments.get(&ns.partition_id())?;
+ assignment
+ .pending
+ .is_none()
+ .then_some((group_id, assignment.owner))
+ })
+ .collect()
+ });
+ for (group_id, owner) in settled {
+ let installed = sim.installed_group_owner(usize::from(primary), ns, group_id);
+ let agrees = owner.map_or_else(
+ || installed.is_none_or(ConsumerGroupOwner::is_unassigned),
+ |owner| installed == Some(owner),
+ );
+ assert!(
+ agrees,
+ "leader {leader} settled owner {owner:?} of consumer group {group_id} on \
+ ns {ns:?}, but partition primary {primary} installed {installed:?} \
+ (seed={seed:#x})",
+ );
+ let primary_state = sim.partition_consensus_state(usize::from(primary), ns);
+ for replica in 0..sim.replica_count {
+ let index = usize::from(replica);
+ if sim.is_crashed(replica)
+ || sim.partition_incarnation(index, ns)
+ != sim.partition_incarnation(usize::from(primary), ns)
+ || sim
+ .partition_consensus_state(index, ns)
+ .map(|state| state.commit_min)
+ != primary_state.as_ref().map(|state| state.commit_min)
+ {
+ continue;
+ }
+ assert_eq!(
+ sim.installed_group_owner(index, ns, group_id),
+ installed,
+ "replica {replica} and primary {primary} installed different owners at the same committed prefix \
+ (ns={ns:?}, group={group_id}, seed={seed:#x})",
+ );
+ }
+ compared += 1;
+ }
+ }
+ compared
+}
+
+fn assert_partition_offsets_agree(sim: &Simulator, workload: &Workload, live: &[usize], seed: u64) {
+ for &namespace in &workload.options.namespaces {
+ let mut committed = BTreeMap::new();
+ for &replica in live {
+ let Some(partition) = sim.replicas[replica]
+ .partition_shard(namespace)
+ .plane
+ .partitions()
+ .get_by_ns(&namespace)
+ else {
+ continue;
+ };
+ let mut offsets = BTreeMap::new();
+ for kind in ConsumerKind::ALL {
+ for (consumer, offset) in partition.retained_consumer_offsets(kind) {
+ offsets.insert((kind.as_code(), consumer), offset);
+ }
+ }
+ let prefix = (
+ partition.created_revision(),
+ partition.consensus().commit_min(),
+ );
+ if let Some((other, expected)) = committed.get(&prefix) {
+ assert_eq!(
+ &offsets, expected,
+ "replicas {replica} and {other} have different consumer offsets at committed prefix {prefix:?} \
+ (ns={namespace:?}, seed={seed:#x})"
+ );
+ }
+ if workload.strict_outcome_oracle()
+ && workload.options.ack_quorum_ratio >= 1.0
Review Comment:
nit: gated on `ack_quorum_ratio >= 1.0` while the default is 0.5, so the shadow offset compare is off in every default campaign. 16 strict seeds (clients=1, ack=1.0) pass with no false positive.
Add one strict cell to CI.
##########
core/simulator/src/workload/state_checker.rs:
##########
@@ -265,6 +381,53 @@ impl StateChecker {
}
}
+/// Membership is metadata state; the partition's ordered install barrier is
+/// what fences a departed member's later offset writes.
+fn check_partition_owner(
+ owners: &mut BTreeMap<u64, ConsumerGroupOwner>,
+ header: &PrepareHeader,
+ body: &[u8],
+ seed: u64,
+) {
+ if header.operation == Operation::InstallConsumerGroupOwner {
+ let install = InstallConsumerGroupOwnerRequest::decode_from(body)
+ .expect("a committed owner installation decodes");
+ owners.insert(install.group_id, install.owner);
+ return;
+ }
+ if is_auto_commit_client(header.client) {
Review Comment:
nit: auto-commit stores skip the generation check entirely, and `owners` ignores `RetireConsumerGroupOwners`, so a stale store after retirement would pass.
Drop retired groups when that op applies and check the generation for auto-commit stores too.
##########
core/sdk/src/vsr.rs:
##########
@@ -125,26 +126,22 @@ pub(crate) fn encode_request_header(
}
_ => {
let operation = operation_for_code(code);
- // NonReplicated ops (ping, reads) bypass server-side dedup --
- // `ClientTable` only tracks request_ids for replicated ops, and
- // the table accepts any id above the watermark with no
- // contiguity requirement (client_table.rs: "There is no
- // `RequestGap`"), so consuming the counter would not break the
- // next metadata op. Read the current id without advancing
- // because the server ignores it for NonReplicated and burning
- // ids for requests the table never sees buys nothing.
+ // NonReplicated ops (ping, reads) bypass server-side dedup, and
+ // `ClientTable` admits any id above its watermark
+ // (client_table.rs: "There is no `RequestGap`"), so a read can
+ // spend an id. A bound session spends one per read, so a reply
+ // names exactly one request on a link that matches by id.
//
- // They are also sessionless on the server (routed by transport
- // id; protected codes are auth-gated server-side), so send with
- // session 0 when unregistered -- preserving the legacy "ping
- // works without auth" contract. Replicated ops still require a
- // bound session for dedup and fail fast here.
+ // Unbound, they are sessionless on the server (routed by
+ // transport id; protected codes are auth-gated server-side), so
+ // they carry session 0, which keeps the legacy "ping works
+ // without auth" contract. Replicated ops still require a bound
+ // session for dedup and fail fast here.
if operation == Operation::NonReplicated {
- (
- operation,
- session.current_request_id(),
- session.session().unwrap_or(0),
- )
+ match session.session() {
+ Some(session_id) => (operation, session.next_request_id()?, session_id),
Review Comment:
nit: bound reads now spend a shared request id, here and in the Go/Java/Node session counters; master did not. The 128-id committed window (`client_table.rs:493-512`) is documented in terms of writes, so polls and pings dilute it: a replayed write behind a busy poller can get `RequestTooOld`. That fails safe (unknown outcome), but it isn't what the doc says.
Either use a separate counter for reads or update the window doc.
##########
foreign/go/client/tcp/tcp_poll_routing.go:
##########
@@ -200,33 +198,90 @@ func (c *IggyTcpClient) sendRouted(caller context.Context, code command.Code, ke
c.polls.mu.Unlock()
if heartbeatDue {
if _, _, err := c.sendPollRequest(ctx, uint32(command.PingCode), nil); err != nil {
Review Comment:
nit: the 5 s heartbeat PING runs synchronously in the caller's send/poll path (+1 RTT on that request). There are also `string(query)` copies at :191 and :421, and the router mutex is taken 3x per op.
Move the heartbeat to a background ticker and reuse the key.
##########
foreign/go/client/tcp/tcp_poll_routing.go:
##########
@@ -200,33 +198,90 @@ func (c *IggyTcpClient) sendRouted(caller context.Context, code command.Code, ke
c.polls.mu.Unlock()
if heartbeatDue {
if _, _, err := c.sendPollRequest(ctx, uint32(command.PingCode), nil); err != nil {
- if !isReconnectable(err) {
- return nil, err
- }
- if _, err := c.do(ctx, &command.Ping{}); err != nil {
- return nil, err
- }
+ return nil, err
}
}
-
+ stamped, refreshed, repeatedPrimary := false, false, false
+ lifecycleRetryInterval := replayInterval
+ var requestParent *consumerSession
+ exact := captured != nil
for {
if err := ctx.Err(); err != nil {
return nil, err
}
- route, err := c.consumerRoute(ctx, key, routePayload, routeCode)
- var response []byte
+ if requestParent != nil && !c.matchesPollParent(*requestParent) {
+ return nil, ierror.ErrUnauthenticated
+ }
+ state := pollExchangeState{replay: isPartitionMutation(code)}
+ route, err := c.consumerRoute(ctx, query)
if err == nil {
+ if requestParent != nil && (requestParent.client != route.parent.client || requestParent.session != route.parent.session) {
+ return nil, ierror.ErrUnauthenticated
+ }
if captured == nil {
value := route.context
captured = &value
}
route.context = *captured
- response, err = c.pollOnRoute(ctx, code, key, payload, route)
+ if !exact {
+ route.context.MetadataOp = max(route.context.MetadataOp, route.parent.watermark)
+ }
+ if !stamped {
+ snapshot := c.pollSession.Load()
+ if snapshot == nil || snapshot.parent.client != route.parent.client || snapshot.parent.session != route.parent.session {
+ return nil, ierror.ErrUnauthenticated
+ }
+ if err := vsr.StampRequestHeader(snapshot.session, uint32(code), frame); err != nil {
Review Comment:
nit: the id is stamped from the shared counter before the per-endpoint gate at :337 (master stamped inside the gate). Two goroutines on one partition can send N+1 before N. The slice window still runs N, but with the counter shared across all connections the 128-id margin shrinks with fanout, and the residual is an explicit `RequestTooOld`. The LifecycleBusy restamp (:274) has the same shape.
Stamp after the gate is acquired.
##########
foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/PollRouter.java:
##########
@@ -89,33 +95,105 @@ final class PollRouter {
}
CompletableFuture<ByteBuf> poll(ByteBuf payload, Optional<PartitionContext> context) {
+ return partitionRequest(CommandCode.Messages.POLL.getValue(), payload, context);
+ }
+
+ CompletableFuture<ByteBuf> partitionRequest(int command, ByteBuf payload, Optional<PartitionContext> context) {
byte[] request = takeBytes(payload);
- return send(new Poll(
- CommandCode.Messages.GET_POLL_ROUTING,
- request,
- CommandCode.Messages.POLL_ON_PRIMARY.getValue(),
- request,
- request.length - POLL_PARAMETERS_BYTES,
- pollTimeout,
- context));
- }
-
- /**
- * Sends a store or delete offset request to the partition primary. The route is
- * looked up by the consumer, stream, topic and partition in the first
- * {@code targetLength} bytes, as in Rust's {@code write_offset}.
- */
- CompletableFuture<ByteBuf> writeOffset(
- int command, ByteBuf payload, int targetLength, Optional<PartitionContext> context) {
+ try {
+ byte[] query = partitionQuery(command, request);
+ return send(new Poll(query, command, request, pollTimeout, context));
+ } catch (RuntimeException error) {
+ return CompletableFuture.failedFuture(error);
+ }
+ }
+
+ CompletableFuture<ByteBuf> sendMessages(ByteBuf payload, ByteBuf query) {
byte[] request = takeBytes(payload);
- return send(new Poll(
- CommandCode.ConsumerOffset.GET_ROUTING,
- Arrays.copyOf(request, targetLength),
- command,
- request,
- targetLength,
- pollTimeout,
- context));
+ byte[] target = takeBytes(query);
+ return send(new Poll(target, CommandCode.Messages.SEND.getValue(), request, pollTimeout, Optional.empty()));
+ }
+
+ CompletableFuture<Long> partitionCount(StreamId stream, TopicId topic) {
+ ByteBuf query = BytesSerializer.toBytes(stream);
+ ByteBuf topicBytes = BytesSerializer.toBytes(topic);
+ try {
+ query.writeBytes(topicBytes);
+ query.writeIntLE(0);
+ } finally {
+ topicBytes.release();
+ }
+ byte[] target = takeBytes(query);
+ String topicKey = ByteBufUtil.hexDump(target, 0, target.length - Integer.BYTES);
+ synchronized (this) {
+ Long count = partitionCounts.get(topicKey);
+ if (count != null) {
+ return CompletableFuture.completedFuture(count);
+ }
+ }
+ Poll lookup = new Poll(
+ target, CommandCode.Messages.GET_PARTITION_ROUTE.getValue(), target, pollTimeout, Optional.empty());
+ return send(lookup).thenApply(response -> {
+ response.release();
+ return lookup.partitionsCount;
+ });
+ }
+
+ synchronized void clear() {
+ routes.clear();
+ partitionCounts.clear();
+ }
+
+ synchronized void observeTopology(ClusterMetadata metadata) {
+ replicas.clear();
+ rosterSize = metadata.nodes().size();
+ for (var node : metadata.nodes()) {
+ if (node.endpoints().tcp() != 0) {
+ replicas.put(
+ node.replicaId(),
+ new ConnectionInfo(node.ip(), node.endpoints().tcp()));
+ }
+ }
+ }
+
+ private static byte[] partitionQuery(int command, byte[] request) {
+ ByteBuf body = Unpooled.wrappedBuffer(request);
+ try {
+ int consumerKind = body.readUnsignedByte();
+ skipIdentifier(body);
+ int streamStart = body.readerIndex();
+ skipIdentifier(body);
+ skipIdentifier(body);
+ int partitionStart = body.readerIndex();
+ boolean explicit = body.readUnsignedByte() != 0;
+ long partition = body.readUnsignedIntLE();
+ if (!explicit) {
+ if (command == CommandCode.ConsumerOffset.STORE.getValue()
+ || command == CommandCode.ConsumerOffset.DELETE.getValue()
+ || (command == CommandCode.Messages.POLL.getValue() && consumerKind == 2)) {
+ throw new IggyInvalidArgumentException("A group poll or offset mutation requires a partition id");
+ }
+ partition = 0;
+ body.setByte(partitionStart, 1);
+ body.setIntLE(partitionStart + 1, 0);
+ }
+ ByteBuf query = Unpooled.buffer(partitionStart - streamStart + Integer.BYTES);
+ query.writeBytes(request, streamStart, partitionStart - streamStart).writeIntLE((int) partition);
+ return takeBytes(query);
+ } catch (IndexOutOfBoundsException error) {
+ throw new IggyInvalidArgumentException("Truncated partition target");
+ } finally {
+ body.release();
+ }
+ }
+
+ private static void skipIdentifier(ByteBuf body) {
+ int kind = body.readUnsignedByte();
+ int length = body.readUnsignedByte();
+ if ((kind == 1 && length != Integer.BYTES) || (kind != 1 && kind != 2) || length == 0) {
+ throw new IggyInvalidArgumentException("Invalid partition target identifier");
+ }
+ body.skipBytes(length);
}
private static byte[] takeBytes(ByteBuf payload) {
Review Comment:
nit: every routed send now heap-copies the whole batch (single-node too) plus a hexDump key and a scheduled timeout. In a cluster, the Slot tail chain allows one send in flight per primary. Master pipelined on the main connection via `retainedDuplicate`, though that clustered path never worked.
Keep the `ByteBuf` (`retainedDuplicate`) and pipeline SendMessages on the data connection. Request identity doesn't need the serialization, since each Poll owns its failover state.
##########
foreign/csharp/Iggy_SDK/IggyClient/Implementations/TcpMessageStream.PollRouting.cs:
##########
@@ -201,105 +120,141 @@ private static IggyInvalidStatusCodeException TruncatedSend()
return VsrError.Exception(VsrError.INVALID_COMMAND, "The SendMessages body is truncated.");
}
- private static ContextQuery? TryBuildContextQuery(int code, ReadOnlySpan<byte> body)
+ private static SendTarget PartitionTarget(int code, ReadOnlySpan<byte> body)
{
- if (code is CommandCodes.STORE_CONSUMER_OFFSET_CODE or CommandCodes.DELETE_CONSUMER_OFFSET_CODE)
+ if (code == CommandCodes.SEND_MESSAGES_CODE)
{
- var suffixSize = code == CommandCodes.STORE_CONSUMER_OFFSET_CODE
- ? StoreOffsetSuffixSize : DeleteOffsetSuffixSize;
- if (body.Length <= suffixSize)
- {
- return null;
- }
- var query = body[..^suffixSize].ToArray();
- return new ContextQuery(CommandCodes.GET_CONSUMER_OFFSET_ROUTING_CODE, query, ConsumerSessionSize,
- Convert.ToBase64String(query));
+ return ParseSendTarget(body);
}
- if (code != CommandCodes.POLL_MESSAGES_CODE || body.Length <= PollOptionsSize)
+
+ if (body.IsEmpty)
{
- return null;
+ throw TruncatedSend();
+ }
+ var group = body[0] == ConsumerGroupWireKind;
+ var position = 1;
+ _ = ReadIdentifier(body, ref position);
+ var identifiersStart = position;
+ var streamId = ReadIdentifier(body, ref position);
+ var topicId = ReadIdentifier(body, ref position);
+ var identifiersLength = position - identifiersStart;
+ if (position + 1 + sizeof(uint) > body.Length)
+ {
+ throw TruncatedSend();
}
- return new ContextQuery(CommandCodes.GET_POLL_ROUTING_CODE, body.ToArray(), ConsumerSessionSize,
- Convert.ToBase64String(body[..^PollOptionsSize]));
+ var present = body[position++] != 0;
+ if (!present && (code is CommandCodes.STORE_CONSUMER_OFFSET_CODE or CommandCodes.DELETE_CONSUMER_OFFSET_CODE
+ || code == CommandCodes.POLL_MESSAGES_CODE && group))
+ {
+ throw VsrError.Exception(VsrError.INVALID_COMMAND, "This partition request requires an explicit partition id.");
+ }
+ var partitionId = present ? BinaryPrimitives.ReadUInt32LittleEndian(body[position..]) : 0;
+ var query = new byte[identifiersLength + sizeof(uint)];
+ body.Slice(identifiersStart, identifiersLength).CopyTo(query);
+ BinaryPrimitives.WriteUInt32LittleEndian(query.AsSpan(identifiersLength), partitionId);
+ return new SendTarget(new PartitionRouteKey(streamId, topicId, partitionId), query);
+ }
+
+ private static byte[] RouteQuery(PartitionRouteKey key)
+ {
+ var query = new byte[IdentifierPrefixSize * 2 + key.StreamId.Length + key.TopicId.Length + sizeof(uint)];
+ query.AsSpan().WriteBytesFromStreamAndTopicIdentifiers(key.StreamId, key.TopicId);
+ BinaryPrimitives.WriteUInt32LittleEndian(query.AsSpan(query.Length - sizeof(uint)), key.PartitionId);
+ return query;
}
- /// <summary>
- /// Sends a poll or an offset write to the partition primary, over a data connection that leaves the
- /// membership connection where it is. A caller context, or else the first context a route reports, is
- /// retained across every retry, so a retry never moves the request onto a newer incarnation or owner.
- /// </summary>
- /// <param name="code">The poll or offset write command code.</param>
- /// <param name="key">The cached route of the request.</param>
- /// <param name="routing">The body of the routing request.</param>
- /// <param name="payload">The body of the request.</param>
- /// <param name="context">The caller context, or null to take the context the route reports.</param>
- /// <param name="token">The cancellation token.</param>
- private async Task<IMemoryOwner<byte>> SendRoutedAsync(int code, PollRouteKey key, ReadOnlyMemory<byte> routing,
+ private async Task<IMemoryOwner<byte>> SendRoutedAsync(int code, SendTarget target,
ReadOnlyMemory<byte> payload, PartitionContext? context, CancellationToken token)
{
using var cancellation = CancellationTokenSource.CreateLinkedTokenSource(token);
cancellation.CancelAfter(VsrRequestTimeoutMs);
var deadline = Environment.TickCount64 + VsrRequestTimeoutMs;
+ // A new route can change the destination, but not the request's identity or captured position.
var captured = context;
+ SessionFrame? frame = null;
+ ulong? generation = null;
+ var refreshHistory = code == CommandCodes.SEND_MESSAGES_CODE && context is null;
+ var refusedPrimary = false;
+ var lifecycleInterval = LifecycleRetryIntervalMs;
try
{
while (true)
{
try
{
- if (Volatile.Read(ref _clusterNodeCount) == 0)
+ var route = await GetPartitionRouteAsync(target.Key, target.Query, deadline, cancellation.Token);
+ captured ??= route.Route.Context;
+ if (generation is not null && generation != route.Generation)
{
- using var metadata = await SendPollControlAsync(CommandCodes.GET_CLUSTER_METADATA_CODE,
- ReadOnlyMemory<byte>.Empty, deadline, cancellation.Token);
- if (metadata.Memory.Length == 0)
- {
- throw new MalformedResponseException("Poll routing requires a nonempty cluster roster.");
- }
-
- RememberRoster(BinaryMapper.MapClusterMetadata(metadata.Memory.Span));
+ throw new VsrRequestOutcomeUnknownException(new NotConnectedException());
Review Comment:
nit: a session generation change throws `VsrRequestOutcomeUnknownException` even when nothing has been sent yet, whereas Node answers NotAccepted (`client.socket.ts:768`). Only report an unknown outcome once a frame actually went out.
##########
foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/PollRouter.java:
##########
@@ -182,102 +258,177 @@ private void attempt(Poll poll) {
}
return;
}
- synchronized (this) {
- routes.remove(poll.key);
- }
- if (isNotAccepted(error)) {
- if (!poll.retryRefusal()) {
- return;
- }
- coordinator
- .get()
- .eventLoop()
- .schedule(() -> attempt(poll), RETRY_INTERVAL_MILLIS, TimeUnit.MILLISECONDS);
- } else {
- poll.result.completeExceptionally(unwrap(error));
- }
+ failedAttempt(poll, error);
});
}
+ private void failedAttempt(Poll poll, Throwable error) {
+ if (poll.retryingRefusal && unwrap(error) instanceof IggyTimeoutException) {
+ poll.result.completeExceptionally(notAccepted());
+ return;
+ }
+ IggyServerException refusal = AsyncTcpConnection.findServerError(error);
+ boolean historyRefresh = poll.canRefreshHistory(refusal);
+ boolean notPrimary = refusal != null && refusal.getRawErrorCode() == AsyncTcpConnection.NOT_PRIMARY;
+ invalidateRoute(poll, error, historyRefresh, notPrimary);
+ if (poll.isLifecycleRefusal(refusal)) {
+ retryLifecycle(poll, refusal);
+ return;
+ }
+ if (!poll.terminalRoutingRefusal && (isNotAccepted(error) || notPrimary || historyRefresh)) {
+ retry(poll, historyRefresh, notPrimary);
+ } else {
+ poll.result.completeExceptionally(unwrap(error));
+ }
+ }
+
+ private synchronized void invalidateRoute(Poll poll, Throwable error, boolean historyRefresh, boolean notPrimary) {
+ if (historyRefresh) {
+ routes.keySet().removeIf(key -> key.startsWith(poll.topicKey()));
+ partitionCounts.remove(poll.topicKey());
+ } else if (isNotAccepted(error) || notPrimary || poll.dataCommand != CommandCode.Messages.SEND.getValue()) {
+ routes.remove(poll.key);
+ }
+ }
+
+ private void retryLifecycle(Poll poll, IggyServerException refusal) {
+ long delay = poll.lifecycleRetryMillis;
+ if (poll.deadline - System.nanoTime() <= TimeUnit.MILLISECONDS.toNanos(delay)) {
+ poll.result.completeExceptionally(refusal);
+ return;
+ }
+ poll.lifecycleRefusal = refusal;
+ poll.state.renewRequestId();
+ poll.lifecycleRetryMillis = Math.min(delay * 2, AsyncTcpConnection.LIFECYCLE_RETRY_MAX_INTERVAL_MS);
+ coordinator.get().eventLoop().schedule(() -> attempt(poll), delay, TimeUnit.MILLISECONDS);
+ }
+
+ private void retry(Poll poll, boolean historyRefresh, boolean notPrimary) {
+ if (historyRefresh) {
+ poll.refreshedHistory = true;
+ poll.context.set(null);
+ poll.state = new AsyncTcpConnection.TransientFailoverState();
+ }
+ if (!poll.retryRefusal()) {
+ return;
+ }
+ long delay = (notPrimary && !poll.notPrimarySeen) || historyRefresh ? 0 : RETRY_INTERVAL_MILLIS;
+ poll.notPrimarySeen |= notPrimary;
+ coordinator.get().eventLoop().schedule(() -> attempt(poll), delay, TimeUnit.MILLISECONDS);
+ }
+
private CompletableFuture<Route> route(Poll poll) {
AsyncTcpConnection parent = coordinator.get();
if (!parent.isAuthenticated()) {
return CompletableFuture.failedFuture(new IggyNotConnectedException("Not authenticated, call login first"));
}
synchronized (this) {
- observeMetadata(parent.metadataWatermark());
Route cached = routes.get(poll.key);
if (cached != null && routeIsCurrent(cached)) {
return CompletableFuture.completedFuture(cached);
}
}
- return parent.send(poll.routingCommand, Unpooled.wrappedBuffer(poll.routingBody))
- .thenApply(response -> decodeRoute(parent, poll, response))
- .exceptionallyCompose(error -> {
- if (!connectionFailed(error) || poll.result.isDone()) {
- return CompletableFuture.failedFuture(unwrap(error));
+ CompletableFuture<Void> roster = rosterSize == 0
+ ? refreshRoster(parent, poll.deadline).exceptionallyCompose(error -> {
+ poll.terminalRoutingRefusal = true;
+ return CompletableFuture.failedFuture(unwrap(error));
+ })
+ : CompletableFuture.completedFuture(null);
+ return roster.thenCompose(ignored -> parent.sendRouted(
+ CommandCode.Messages.GET_PARTITION_ROUTE.getValue(),
+ Unpooled.wrappedBuffer(poll.routingBody),
+ 0,
+ poll.deadline,
+ new AsyncTcpConnection.TransientFailoverState(new PartitionContext(
+ BigInteger.ZERO, BigInteger.ZERO, BigInteger.valueOf(parent.metadataWatermark())))))
+ .thenCompose(response -> {
+ PartitionRoute decoded;
+ try {
+ decoded = PartitionRoute.decode(response);
+ } finally {
+ response.release();
+ }
+ CompletableFuture<Void> known;
+ synchronized (this) {
+ known = replicas.containsKey(decoded.primaryReplica()) || connectData == null
+ ? CompletableFuture.completedFuture(null)
+ : refreshRoster(parent, poll.deadline);
}
- return coordinator
- .get()
- .send(CommandCode.System.PING, Unpooled.EMPTY_BUFFER)
- .thenApply(response -> {
- response.release();
- throw notAccepted();
- });
+ return known.thenApply(ignored -> cacheRoute(parent, poll, decoded));
});
}
- private Route decodeRoute(AsyncTcpConnection parent, Poll poll, ByteBuf response) {
- try {
- if (response.readableBytes() < ATTACHMENT_BYTES + PartitionContext.ENCODED_SIZE) {
- throw new IggyClientException("Truncated primary poll session attachment");
- }
- Attachment attachment = new Attachment(
- response.readLongLE(), response.readLongLE(), response.readLongLE(), response.readLongLE());
- long generation = parent.sessionGeneration();
- byte[] bindSecret = parent.bindSecret(attachment.clientLow, attachment.clientHigh, attachment.session);
- if (generation != parent.sessionGeneration()) {
- throw new IggyClientException("Parent session changed while decoding a poll route");
- }
- var context = BytesDeserializer.readPartitionContext(response);
- var node = BytesDeserializer.readClusterNode(response);
- if (response.isReadable() || node.endpoints().tcp() == 0) {
- throw new IggyClientException("Invalid TCP primary poll routing response");
- }
- synchronized (this) {
- observeMetadata(parent.metadataWatermark());
- Route route = new Route(
- new ConnectionInfo(node.ip(), node.endpoints().tcp()),
- attachment.withWatermark(metadataWatermark),
- parent,
- generation,
- bindSecret,
- context);
- if (routes.size() >= MAX_ROUTES) {
- routes.clear();
- }
- if (!poll.result.isDone() && coordinator.get() == parent) {
- routes.put(poll.key, route);
- }
- return route;
- }
- } catch (IndexOutOfBoundsException | IggyMalformedResponseException error) {
- throw new IggyClientException("Invalid TCP primary poll routing response", error);
- } finally {
- response.release();
+ private CompletableFuture<Void> refreshRoster(AsyncTcpConnection parent, long deadline) {
+ return parent.sendRouted(
+ CommandCode.System.GET_CLUSTER_METADATA.getValue(),
+ Unpooled.EMPTY_BUFFER,
+ 0,
+ deadline,
+ new AsyncTcpConnection.TransientFailoverState())
+ .thenAccept(response -> {
+ try {
+ observeTopology(BytesDeserializer.readClusterMetadata(response));
+ } finally {
+ response.release();
+ }
+ });
+ }
+
+ private synchronized Route cacheRoute(AsyncTcpConnection parent, Poll poll, PartitionRoute decoded) {
+ ConnectionInfo endpoint = replicas.get(decoded.primaryReplica());
+ if (decoded.partitionsCount() == 0) {
+ throw new IggyMalformedResponseException("Partition route contains no partitions");
}
+ if (rosterSize == 0 || (endpoint == null && connectData != null)) {
+ poll.terminalRoutingRefusal = true;
+ throw notAccepted();
+ }
+ long generation = parent.sessionGeneration();
+ long[] identity = parent.sessionIdentity();
+ Attachment attachment = new Attachment(
+ identity[0], identity[1], identity[2], decoded.metadataOp().longValue());
+ byte[] secret = parent.bindSecret(identity[0], identity[1], identity[2]);
+ Route route = new Route(
+ endpoint,
+ attachment,
+ parent,
+ generation,
+ secret,
+ decoded.context(),
+ decoded.partitionsCount(),
+ rosterSize <= 1 || connectData == null);
+ if (routes.size() >= MAX_ROUTES) {
+ clear();
+ }
+ if (coordinator.get() != parent || parent.sessionGeneration() != generation) {
+ throw notAccepted();
+ }
+ routes.put(poll.key, route);
+ partitionCounts.put(poll.topicKey(), decoded.partitionsCount());
+ return route;
}
private synchronized boolean routeIsCurrent(Route route) {
AsyncTcpConnection parent = coordinator.get();
- observeMetadata(parent.metadataWatermark());
- return route.parent == parent
- && route.generation == parent.sessionGeneration()
- && Long.compareUnsigned(route.attachment.watermark, metadataWatermark) >= 0;
+ return route.parent == parent && route.generation == parent.sessionGeneration();
Review Comment:
nit: unlike Go (`tcp_poll_routing.go:431`), the cached route is not aged by the metadata watermark (Node `client.socket.ts` is the same). After a partition is recreated, one `HistoryUnavailable` surfaces on poll/offset calls before re-routing.
Align with Go.
##########
foreign/csharp/Iggy_SDK/Contracts/ClusterNode.cs:
##########
@@ -27,6 +27,9 @@ public class ClusterNode
/// </summary>
public required string Name { get; set; }
+ /// <summary>Configured replica identity, independent of its position in the cluster roster.</summary>
+ public byte ReplicaId { get; set; }
Review Comment:
nit: `ReplicaId` silently defaults to 0 when the field is absent, which names the wrong replica (`node.rs:26-28` warns about exactly this). The Java `ClusterNode.replicaId` primitive int has the same problem.
Make it `required` here, and boxed or failing on missing in Java.
##########
core/cli/src/commands/binary_cluster/get_cluster_metadata.rs:
##########
@@ -101,7 +103,8 @@ impl CliCommand for GetClusterMetadataCmd {
GetClusterMetadataOutput::List => {
cluster_metadata.nodes.iter().for_each(|node| {
event!(target: PRINT_TARGET, Level::INFO,
- "{}|{}|{}|{}|{}|{}|{}|{}",
+ "{}|{}|{}|{}|{}|{}|{}|{}|{}",
+ node.replica_id,
Review Comment:
nit: putting `replica_id` first shifts every `|`-separated field of the list output and breaks scripts that parse it. Append it as the last column instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4270237258
##########
core/shard/src/lib.rs:
##########
@@ -6559,11 +6718,17 @@ where
}
if consensus.is_normal()
&& !consensus.is_transferring()
- && consensus.commit_min() < consensus.commit_max()
+ && (consensus.commit_min() < consensus.commit_max() || consensus.view_log_unproven())
&& self.metadata_repair.borrow().is_none()
{
- let to_op = consensus.commit_max();
- let from_op = consensus.commit_min() + 1;
+ let to_op = if consensus.view_log_unproven() {
+ consensus.sequencer().current_sequence()
Review Comment:
**critical**: server panic. While unproven, `to_op` is the sequencer and `from_op` is `view_log_repair_op` (the adopted head). A backup's `on_replicate` (metadata.rs:1705, `set_sequence(header.op)`) rewinds the sequencer below that head when a retransmit fills a journal gap. That builds `RequestPrepares [38,37]`, and the `expect` at consensus_message.rs:463 ("repair range must be non-empty and 1-based") aborts the process.
Repro: metadata swarm seed 7002, and `--plane lifecycle --shell --clients 3 --faults swarm --crash-prob 0.003 --restart-prob 0.02 --crash-primary --ticks 40000 --heal-before-quiesce` seed 20037. 32/200 metadata seeds at head, 0 on the previous head and base; a clamp mutant gives 0.
Fix: `to_op = max(seq, pending.op_head, from_op)`, and never send an empty range. The partition twin (:9288) is lower risk because partition backups only move the sequencer forward.
##########
core/server/src/http/forward.rs:
##########
@@ -332,92 +534,69 @@ fn record_relayed_floor(state: &HttpInner, user_id: u32, response: &Response) {
state.metadata_watermarks.record(user_id, applied);
}
-async fn forward_partition_or_pass(state: HttpState, request: Request, next: Next) -> Response {
- if !state.forward.active || request.headers().contains_key(FORWARDED_HEADER) {
- return next.run(request).await;
- }
- // A read that does not move consumer progress is answered by whichever
- // replica the caller reached, which is the point of reading from one. Only
- // an automatic commit needs a replica that can originate the offset
- // operation, so only that poll pays for the buffering and the credential
- // resolution below.
- if request.method() == Method::GET && !wants_auto_commit(request.uri().query()) {
- return next.run(request).await;
- }
+async fn attach_partition_request(state: HttpState, request: Request, next: Next) -> Response {
+ let received = Instant::now();
+ let hop = match partition_hop(request.headers()) {
+ Ok(hop) => hop,
+ Err(error) => return CustomError::from(error).into_response(),
+ };
+ // Verify-only, like the metadata relay: an unauthenticated caller gets no
+ // buffered body and no relay. The handler authenticates again.
let bearer = match bearer_token(request.headers()) {
Ok(bearer) => bearer,
Err(error) => return CustomError::from(error).into_response(),
};
if let Err(rejection) = resolve_credential(&state, bearer).await {
return rejection.into_response();
}
-
- let (parts, request_body) = request.into_parts();
+ let _guard = if matches!(hop, PartitionHop::Target(_)) {
+ let Some(guard) = ForwardGuard::admit(&state.forward.in_flight) else {
Review Comment:
warning: the Target guard is admitted before the untimed `to_bytes` body read, and Target comes from client-supplied headers. 128 authenticated slow-body connections pin `MAX_IN_FLIGHT_FORWARDS`, and this node's own outbound relays then get `forward_busy` 503, which is passed to clients unretried. No test pins the bound.
Admit after the body is read (scope the guard to the hold), or give Target holds their own cap.
##########
core/shard/src/lib.rs:
##########
@@ -5469,6 +5639,8 @@ where
return;
}
partition.apply_repaired_prepare(msg).await;
+ let pending = partition.consensus().pending_view_log();
+ reconcile_partition_view_divergence(self.id, partition, pending.as_ref()).await;
Review Comment:
warning: every repaired frame, held duplicates included, clones `pending_view_log` and reruns the full reconcile (O(op_head - gap) walk plus a linear canonical find), so a gap of G costs O(G^2) CPU on the shard pump. Reconcile once per RepairDone, or only when the frame filled the missing op.
##########
core/consensus/src/impls.rs:
##########
@@ -3666,6 +3704,9 @@ impl<B: MessageBus, P: Pipeline<Entry = PipelineEntry>> VsrConsensus<B, P> {
// it: the new primary's numbers are at least as fresh as any probe
// answer.
self.finish_view_probe();
+ let needs_reconciliation = self.view_log_unproven.get()
Review Comment:
nit: the same-view fix (skip the early return while unproven) and the metadata StartView suffix refresh (shard/lib.rs:5092) are unpinned: mutants survive consensus (221), shard (183) and sim (242) tests. The new metadata unproven-repair arming (shard/lib.rs:6724) is also untested, which is how the empty-range panic got in. Add a metadata twin of partition_io.rs:4174 and a same-view Commit-while-unproven test.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -11540,8 +11847,12 @@ where
self.consensus().set_commit_floor(floor);
Review Comment:
warning (suspect, conf L): `complete_repair` can raise the commit floor while the view log is unproven. That can jump commit_min over durable dead-branch entries, which then count as proven (impls.rs:4023-4027) and are served by repair. Refuse the floor jump while unproven, or re-prove first.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8542,11 +8764,45 @@ where
}
async fn truncate_resident_from(&mut self, from_op: u64) -> Result<usize, IggyError> {
+ self.remove_resident_range(from_op..=u64::MAX).await
+ }
+
+ /// Replace a proven mismatch while retaining the canonical tail across gaps.
+ ///
+ /// # Errors
+ /// Returns an error if the WAL or resident indexes cannot be rewritten.
+ pub async fn remove_divergent_prepare(&mut self, op: u64) -> Result<(), IggyError> {
Review Comment:
nit: removing a divergent prepare clears `certified_log_view` durably (partition_journal.rs:565), so a crash before the refill reboots as materialization-missing and needs a full state transfer (iggy_partition.rs:1724-1734). Correct, but expensive; worth a comment or keeping the certificate for the unchanged prefix.
##########
core/simulator/src/bin/workload-fuzz.rs:
##########
@@ -625,6 +653,23 @@ fn run_quiesce_phase(
args.min_metadata_ops_compared, convergence.ops_compared,
));
}
+ if sim.is_shell()
+ && workload.options.weights.draws_fixture_groups()
+ && convergence.owners_compared < args.min_owners_compared
+ {
+ vacuous(format_args!(
+ "--min-owners-compared {}: the run joined fixture groups, but the owner oracle \
+ compared only {} settled owner(s) (seed={seed:#x})",
+ args.min_owners_compared, convergence.owners_compared,
+ ));
+ }
+ if sim.is_shell() && args.plane == Plane::Lifecycle && workload.owner_fence_refusals() == 0 {
Review Comment:
nit: the vacuity gate exits 3 on many lifecycle `--shell` swarm seeds (7/24 and 18/30 in two runs; e.g. seeds 1, 2, 5, 8, 18, 21, 23), all with `stale_owner_stores=0`. The workload never forces a stale-owner store, so a CI cell on a random seed would flake. Deterministically schedule one stale store per run.
##########
core/server/README.md:
##########
@@ -39,6 +39,22 @@ cargo run --bin iggy-server --release -- --replica-id 0
Command-line arguments include `--replica-id`, `--fresh` (`-f`), `--with-default-root-credentials`, and `--list-config-env-vars`. Other settings come from configuration.
+## Upgrading
+
+Deploy matching server and SDK releases together. Rolling upgrades and mixed
+server versions are unsupported. Partition control frames from older servers
+carry incarnation zero and are rejected, so a mixed-version cluster cannot
+complete partition view changes. The `IGGY-OWNERSHIP-HISTORY-5` storage gate also
+rejects older data directories without modifying them. Existing data requires a
+separately validated offline migration or restore procedure before this release
+can use it.
+
+Cluster metadata includes `replica_id`. Java's `ClusterNode` record adds a leading
+`replicaId` component, changing constructor calls and record patterns. C# callers
+must initialize the required `ReplicaId` property. The CLI's cluster metadata list
+appends a replica ID column. Go, Java, Node.js and C# clients use command 105's new
Review Comment:
nit: the upgrade list misses the Swift `ClusterNode(replicaID:)` initializer (Resources.swift:586), the C++ FFI `ClusterNode.replica_id` (foreign/cpp/src/lib.rs:338) and the new pub field on Rust `iggy_common` `ClusterNode`/`ClusterNodeResponse`. The PR body (squash message) still lacks an SDK paragraph and still says "Simulator checks cover replacement histories".
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8125,6 +8360,7 @@ where
if self.fatal.is_some()
|| self.materialization_missing
|| self.commit_waits_for_checkpoint()
+ || self.consensus.view_log_unproven()
Review Comment:
**critical**: this gate covers only `commit_journal`. The `resume_io` walk at :5909-5914 (`select_persistable_commits` / `collect_committable_from_journal` -> `select_commit` (:10075) -> `finish_pending_commit` never checks `view_log_unproven` / `view_log_entry_proven`. The walk is capped at `persistence.head()`, which still covers dead-branch ops made durable in the old view, so a fenced backup applies them.
Seed 7009 (7016 flags): r1 restarts, proof `Missing{60}`, then commits its discarded v7 op 59 while unproven (backtrace through `finish_pending_commit`). Seed 7056 does the same with op 75.
Committed divergence is worse than before this fix: partition 7000-7299 17/300 (previous head 6/300), mixed 1000-1149 9/150 (previously 0). With `select_commit` refusing while unproven, both drop to 0.
Fix: gate per op on `view_log_entry_proven` in `select_commit` and `collect_committable_from_journal`.
##########
core/shard/src/lib.rs:
##########
@@ -8991,32 +9262,32 @@ where
let commit_to_op = consensus.commit_max();
let commit_lag = consensus.commit_min() < commit_to_op;
let head = consensus.sequencer().current_sequence();
- if !commit_lag && head <= commit_to_op {
+ if !commit_lag && head <= commit_to_op && !consensus.view_log_unproven() {
return false;
}
let missing_suffix = partition_missing_suffix_through(partition);
// Fetch the adopted suffix even while committed operations lag. Later
// live prepares can advance the head while an adopted body is missing.
- let Some(fetch_to_op) =
+ let Some(fetch_to_op) = consensus.view_log_repair_op().map(|_| head).or_else(|| {
partition_repair_fetch_to_op(consensus.commit_min(), commit_to_op, missing_suffix)
- else {
+ }) else {
return false;
};
// A resident window may only be waiting for persistence or a bounded commit walk.
- if partition
- .log
- .journal()
- .inner
- .repaired_window_shape(consensus.commit_min(), fetch_to_op)
- .complete
+ if !consensus.view_log_unproven()
+ && partition
+ .log
+ .journal()
+ .inner
+ .repaired_window_shape(consensus.commit_min(), fetch_to_op)
+ .complete
{
return false;
}
let nonce = iggy_common::random_id::get_uuid();
- let from_op = consensus.commit_min() + 1;
- let cluster = consensus.cluster();
- let self_id = consensus.replica();
- let namespace = consensus.group();
+ let from_op = consensus
+ .view_log_repair_op()
+ .unwrap_or_else(|| consensus.commit_min() + 1);
Review Comment:
**critical** (liveness): while fenced, repair starts at `view_log_repair_op`, the HIGHEST missing op, because the proof walks top-down, and requests up to head. Each round trip fills one hole and re-serves up to 128 bodies already held. A unit probe filled a 2046-op gap in 2076 rounds (258k bodies, 2.2 s of pump CPU, about quadratic); a ring-sized gap is about 65k round trips before falling back to state transfer. Base repairs upward from commit_min+1 in 128-op chunks.
Sim: 7016 flags, seeds 20000-20399, "did not drain" 17/400 at head vs 0/400 on base. On seed 20007, with r0 down, op 300 was served 125 times but never ingested, so the two live replicas stayed wedged for 50k ticks: the fenced backup withholds PrepareOk and there is no quorum.
Fix: request the chunk that ends at the hole (`from = max(commit_min+1, missing-127)`, `to = missing`), so the parent chain proves 128 ops per round trip. The same applies to the resume path at :5966.
##########
core/metadata/src/impls/metadata.rs:
##########
@@ -4231,6 +4224,9 @@ where
#[allow(clippy::future_not_send)]
pub async fn commit_journal(&self) {
let consensus = self.consensus.as_ref().unwrap();
+ if consensus.view_log_unproven() {
Review Comment:
warning (suspect): commit is fenced here, but the metadata superblock (:3719, via shard tick) persists the new `log_view` even while `view_log_unproven`, but the fence itself is not durable (impls.rs init false). After a crash, entries that were unproven at or below the fence are reported under the new log_view. The merge refuses on a same-log_view conflict, so this looks like a stall rather than loss. The partition plane is safe (certify_log_view requires the WAL head checksum). Consider deferring the persist until proven.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio merged PR #4506:
URL: https://github.com/apache/iggy/pull/4506
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4268040971
##########
core/server/src/dispatch/mod.rs:
##########
@@ -256,25 +440,50 @@ async fn drain_client_requests<B, MJ, S, SB>(
SB: SuperblockStore + 'static,
{
loop {
- let Some(message) = pop_next_client_request(&queues, client_id) else {
+ let metadata_frontier = shard.plane.metadata().applied_frontier().get();
+ let next = pop_next_client_request(&queues, client_id, |queued| {
+ if is_ping(&queued.message) {
+ return true;
+ }
+ if drain.metadata_write.get() {
+ return false;
+ }
+ queued
+ .progress_key(metadata_frontier, |message| progress_key(&shard, message))
+ .is_none_or(|key| {
+ drain.in_flight.get() < drain.capacity
+ && !drain.progress.borrow().contains(&key)
Review Comment:
nit: lane removal is barely pinned. Ignoring `drain.progress` here, or never calling `admit_progress` (:458-461), survives all 102 dispatch tests, and pipelined_frames_vsr.rs:182 still passes 3/3; only :156 catches it. No test covers Store -> Next(auto_commit=false) or Store -> GetConsumerOffset ordering. Add a drain-level unit test where a held keyed store keeps a queued Next poll queued until its guard drops.
Minor, PR6-able: the lane is exclusive, so read-only keyed frames also serialize among themselves.
##########
core/server/src/dispatch/mod.rs:
##########
@@ -557,25 +814,51 @@ async fn handle_client_request<B, MJ, S, SB>(
// (`submit_register_local_or_forward`). Every other non-replicated
// code MUST go through Register first, which binds the acting user
// the per-op authz gates resolve.
- handle_non_replicated_request(
- shard,
- sessions,
- server_config,
- transport_client_id,
- request,
- ConnectionContext {
- bound,
- user_id,
- address: client_address,
- metadata_watermark,
- },
- )
- .await;
+ let context = ConnectionContext {
+ bound,
+ user_id,
+ address: client_address,
+ metadata_watermark,
+ };
+ // `ClientDrain` says which frames stay on the drain.
+ if matches!(non_replicated_code(header), PING_CODE | BIND_SESSION_CODE) {
+ handle_non_replicated_request(
+ shard,
+ sessions,
+ server_config,
+ transport_client_id,
+ request,
+ context,
+ )
+ .await;
+ return;
+ }
+ let in_flight = match progress {
+ Some(progress) => progress,
+ None => drain.admit(Admission::Cap).await,
+ };
+ let task_shard = Rc::clone(shard);
+ let task_sessions = Rc::clone(sessions);
+ let task_config = Arc::clone(server_config);
+ shard.bus.spawn(async move {
Review Comment:
**warning** (re the PR6 deferral): the held-poll PING test only refutes running the whole poll inline. The proposed design is different: run the read future inline on the drain, select it against the frame-arrival wake (:178-181), and on wake move the same future, with its `InFlight` and lane guard, into `shard.bus.spawn`.
PING is then answered before the held owner reply by construction. Cap, lane, cancellation and unknown-outcome behaviour are unchanged (same future, same guard drop). Next/auto-commit order is stricter while inline and identical after promotion. Lockstep connections spawn nothing, which is master's cost.
Master ran reads inline and fails this PING test too, so this is a new contract the promotion meets. It's about 40 lines here. As it stands the PR ships -17% poll / -48% consumer for every lockstep SDK through the edge builds.
##########
foreign/go/client/tcp/tcp_core.go:
##########
@@ -860,6 +857,9 @@ func (c *IggyTcpClient) attempt(
if captured, ok := ctx.Value(capturedPartitionContext{}).(iggcon.PartitionContext); ok {
Review Comment:
nit: two `ctx.Value` chain walks per stamp (:857, :860), plus a `context.WithValue` that boxes `PartitionContext` once per routed op (tcp_poll_routing.go:244, :406). Do one lookup with if/else.
##########
core/simulator/src/bin/workload-fuzz.rs:
##########
@@ -586,13 +608,19 @@ fn run_quiesce_phase(
};
println!(
"quiesced and converged (leader-relative; entity oracle: {entity_oracle}; \
- evictions={}; ops_compared={} partition_ops_compared={} replicas_compared={} \
- namespaces_checked={})",
+ evictions={}; ops_compared={} partition_ops_compared={} \
+ deleted_partition_ops_compared={} replicas_compared={} namespaces_checked={} \
+ owners_compared={} offsets_compared_with_shadow={} stale_owner_stores={} owner_fence_refusals={})",
workload.evictions(),
convergence.ops_compared,
convergence.partition_ops_compared,
+ workload.deleted_partition_ops_compared(),
convergence.replicas_compared,
convergence.namespaces_checked,
+ convergence.owners_compared,
+ convergence.offsets_compared_with_shadow,
+ workload.stale_owner_stores(),
+ workload.owner_fence_refusals(),
Review Comment:
nit: `stale_owner_stores` / `owner_fence_refusals` are only printed. Unlike `min_owners_compared` (:656-659) there is no vacuity floor, and 4/12 lifecycle seeds submit zero stale stores. Exit non-zero on lifecycle `--shell` runs with zero refusals.
##########
.github/workflows/pre-merge.yml:
##########
@@ -237,11 +237,17 @@ jobs:
uses: actions/github-script@v9
with:
script: |
- const jobs = await github.rest.actions.listJobsForWorkflowRun({
- owner: context.repo.owner,
- repo: context.repo.repo,
- run_id: context.runId
- });
+ let jobs;
+ try {
+ jobs = await github.rest.actions.listJobsForWorkflowRun({
+ owner: context.repo.owner,
+ repo: context.repo.repo,
+ run_id: context.runId
+ });
+ } catch (error) {
+ core.warning(`Job timings unavailable: ${error.message}`);
Review Comment:
nit: this CI fix is unrelated to partition routing and will ride inside the `feat(server)!` squash. Better as its own `ci` PR.
##########
foreign/csharp/Iggy_SDK/IggyClient/Implementations/TcpMessageStream.PollRouting.cs:
##########
@@ -201,105 +120,142 @@ private static IggyInvalidStatusCodeException TruncatedSend()
return VsrError.Exception(VsrError.INVALID_COMMAND, "The SendMessages body is truncated.");
}
- private static ContextQuery? TryBuildContextQuery(int code, ReadOnlySpan<byte> body)
+ private static SendTarget PartitionTarget(int code, ReadOnlySpan<byte> body)
{
- if (code is CommandCodes.STORE_CONSUMER_OFFSET_CODE or CommandCodes.DELETE_CONSUMER_OFFSET_CODE)
+ if (code == CommandCodes.SEND_MESSAGES_CODE)
{
- var suffixSize = code == CommandCodes.STORE_CONSUMER_OFFSET_CODE
- ? StoreOffsetSuffixSize : DeleteOffsetSuffixSize;
- if (body.Length <= suffixSize)
- {
- return null;
- }
- var query = body[..^suffixSize].ToArray();
- return new ContextQuery(CommandCodes.GET_CONSUMER_OFFSET_ROUTING_CODE, query, ConsumerSessionSize,
- Convert.ToBase64String(query));
+ return ParseSendTarget(body);
}
- if (code != CommandCodes.POLL_MESSAGES_CODE || body.Length <= PollOptionsSize)
+
+ if (body.IsEmpty)
{
- return null;
+ throw TruncatedSend();
}
- return new ContextQuery(CommandCodes.GET_POLL_ROUTING_CODE, body.ToArray(), ConsumerSessionSize,
- Convert.ToBase64String(body[..^PollOptionsSize]));
+ var group = body[0] == ConsumerGroupWireKind;
+ var position = 1;
+ _ = ReadIdentifier(body, ref position);
+ var identifiersStart = position;
+ var streamId = ReadIdentifier(body, ref position);
+ var topicId = ReadIdentifier(body, ref position);
+ var identifiersLength = position - identifiersStart;
+ if (position + 1 + sizeof(uint) > body.Length)
+ {
+ throw TruncatedSend();
+ }
+ var present = body[position++] != 0;
+ if (!present && (code is CommandCodes.STORE_CONSUMER_OFFSET_CODE or CommandCodes.DELETE_CONSUMER_OFFSET_CODE
+ || code == CommandCodes.POLL_MESSAGES_CODE && group))
+ {
+ throw VsrError.Exception(VsrError.INVALID_COMMAND, "This partition request requires an explicit partition id.");
+ }
+ var partitionId = present ? BinaryPrimitives.ReadUInt32LittleEndian(body[position..]) : 0;
+ var query = new byte[identifiersLength + sizeof(uint)];
+ body.Slice(identifiersStart, identifiersLength).CopyTo(query);
+ BinaryPrimitives.WriteUInt32LittleEndian(query.AsSpan(identifiersLength), partitionId);
+ return new SendTarget(new PartitionRouteKey(streamId, topicId, partitionId), query);
+ }
+
+ private static byte[] RouteQuery(PartitionRouteKey key)
+ {
+ var query = new byte[IdentifierPrefixSize * 2 + key.StreamId.Length + key.TopicId.Length + sizeof(uint)];
+ query.AsSpan().WriteBytesFromStreamAndTopicIdentifiers(key.StreamId, key.TopicId);
+ BinaryPrimitives.WriteUInt32LittleEndian(query.AsSpan(query.Length - sizeof(uint)), key.PartitionId);
+ return query;
}
- /// <summary>
- /// Sends a poll or an offset write to the partition primary, over a data connection that leaves the
- /// membership connection where it is. A caller context, or else the first context a route reports, is
- /// retained across every retry, so a retry never moves the request onto a newer incarnation or owner.
- /// </summary>
- /// <param name="code">The poll or offset write command code.</param>
- /// <param name="key">The cached route of the request.</param>
- /// <param name="routing">The body of the routing request.</param>
- /// <param name="payload">The body of the request.</param>
- /// <param name="context">The caller context, or null to take the context the route reports.</param>
- /// <param name="token">The cancellation token.</param>
- private async Task<IMemoryOwner<byte>> SendRoutedAsync(int code, PollRouteKey key, ReadOnlyMemory<byte> routing,
+ private async Task<IMemoryOwner<byte>> SendRoutedAsync(int code, SendTarget target,
ReadOnlyMemory<byte> payload, PartitionContext? context, CancellationToken token)
{
using var cancellation = CancellationTokenSource.CreateLinkedTokenSource(token);
cancellation.CancelAfter(VsrRequestTimeoutMs);
var deadline = Environment.TickCount64 + VsrRequestTimeoutMs;
+ // A new route can change the destination, but not the request's identity or captured position.
var captured = context;
+ SessionFrame? frame = null;
+ ulong? generation = null;
+ var refreshHistory = code == CommandCodes.SEND_MESSAGES_CODE && context is null;
+ var refusedPrimary = false;
+ var lifecycleInterval = LifecycleRetryIntervalMs;
try
{
while (true)
{
try
{
- if (Volatile.Read(ref _clusterNodeCount) == 0)
+ var route = await GetPartitionRouteAsync(target.Key, target.Query, deadline, cancellation.Token);
+ captured ??= route.Route.Context;
+ if (generation is not null && generation != route.Generation)
{
- using var metadata = await SendPollControlAsync(CommandCodes.GET_CLUSTER_METADATA_CODE,
- ReadOnlyMemory<byte>.Empty, deadline, cancellation.Token);
- if (metadata.Memory.Length == 0)
- {
- throw new MalformedResponseException("Poll routing requires a nonempty cluster roster.");
- }
-
- RememberRoster(BinaryMapper.MapClusterMetadata(metadata.Memory.Span));
+ throw PollNotAccepted();
}
-
+ generation ??= route.Generation;
+ frame ??= _consensusSession.Resolve(VsrOperations.ForCode(code));
if (Volatile.Read(ref _clusterNodeCount) <= 1)
{
- captured ??= await CapturePartitionContextAsync(code, payload, cancellation.Token);
- return await SendWithResponseAsync(code, payload, token: cancellation.Token,
- capturedContext: captured);
+ var attempt = await SendVsrAttemptAsync(code, payload, Environment.TickCount64, deadline,
+ false, cancellation.Token, captured.Value, frame,
+ retryTransient: VsrOperations.ForCode(code) != VsrOperation.NonReplicated);
+ if (attempt.Error is not null)
+ {
+ if (attempt.RequestStarted && !VsrOperations.IsReplaySafeRead(code, false, payload.Span)
+ && !IsDefinitiveVerdict(attempt.Error))
+ {
+ throw new VsrRequestOutcomeUnknownException(attempt.Error);
+ }
+ ExceptionDispatchInfo.Throw(attempt.Error);
+ }
+ return attempt.Response!;
}
- var route = await GetPollRouteAsync(key, routing, deadline, cancellation.Token);
- captured ??= route.Context;
- return await SendPrimaryOnceAsync(key,
- code == CommandCodes.POLL_MESSAGES_CODE ? CommandCodes.POLL_MESSAGES_ON_PRIMARY_CODE : code,
- payload, route, captured.Value, deadline, cancellation.Token);
+ return await SendPrimaryOnceAsync(target.Key, code, payload, route, captured.Value,
+ frame.Value, deadline, cancellation.Token);
}
- catch (IggyInvalidStatusCodeException error) when (error.StatusCode == VsrError.TRANSIENT_NOT_ACCEPTED)
+ catch (IggyInvalidStatusCodeException error) when (error.StatusCode is
+ VsrError.TRANSIENT_NOT_ACCEPTED or VsrError.NOT_PRIMARY)
{
- RemovePollRoute(key);
- if (Environment.TickCount64 + PollRoutingRetryIntervalMs >= deadline)
+ RemovePartitionRoute(target.Key);
+ if ((generation is not null && generation != _consensusSession.Generation)
Review Comment:
nit: the new test only fails if both :189 and :216 are reverted. Reverting :216 alone still passes 117/117. Add a case that reaches :189 (LifecycleBusy/HistoryUnavailable, then relogin) so each guard is pinned.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -377,6 +381,74 @@ async fn relay_partition_reply<B, MJ, S, SB>(
});
}
+/// The offset key shared by stored-offset reads and progress writes on a connection.
+/// Failed resolution is left to the normal handler's authorization and error path.
+pub(super) fn progress_key<B, MJ, S, SB>(
+ shard: &Rc<ShellShard<B, MJ, S, SB>>,
+ message: &Message<GenericHeader>,
+) -> Option<ConsumerProgress>
+where
+ B: ShellBus,
+ MJ: JournalHandle + 'static,
+ MJ::Target: Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>,
+ S: 'static,
+ SB: SuperblockStore + 'static,
+{
+ let request = message.try_as_typed::<RequestHeader>().ok()?;
+ let wire = match request.header().operation {
+ Operation::NonReplicated
+ if request.header().reserved[..4] == GET_CONSUMER_OFFSET_CODE.to_le_bytes() =>
+ {
+ GetConsumerOffsetRequest::decode_from(request.body()).ok()?
+ }
+ Operation::NonReplicated
+ if request.header().reserved[..4] == POLL_MESSAGES_CODE.to_le_bytes()
+ || request.header().reserved[..4]
+ == POLL_MESSAGES_ON_PRIMARY_CODE.to_le_bytes() =>
+ {
+ let poll = PollMessagesRequest::decode_from(request.body()).ok()?;
Review Comment:
nit: polls that are neither `Next` nor auto-commit fully decode `PollMessagesRequest` (WireName String allocs) just to return `None`, then the handler decodes again. Return early on the strategy/flag bytes, or carry the decoded request into the handler.
##########
foreign/java/java-sdk/src/main/java/org/apache/iggy/cluster/ClusterNode.java:
##########
@@ -19,14 +19,26 @@
package org.apache.iggy.cluster;
+import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonSetter;
+import com.fasterxml.jackson.annotation.Nulls;
+
/**
* A single node in the cluster roster.
*
+ * @param replicaId the configured replica id, independent of roster order
* @param name the node name
* @param ip the node IP address or hostname
* @param endpoints the per-transport ports of the node
* @param role the node role
* @param status the node status
*/
public record ClusterNode(
- String name, String ip, TransportEndpoints endpoints, ClusterNodeRole role, ClusterNodeStatus status) {}
+ @JsonProperty(value = "replica_id", required = true) @JsonSetter(nulls = Nulls.FAIL)
+ int replicaId,
Review Comment:
nit (PR description): the leading `replicaId` record component breaks constructor calls and record patterns. Together with C#'s `required ReplicaId` and the CLI list's extra column, these public breaks, and the Go/Java/Node/C# routing migration as a whole, aren't mentioned in the PR body, which becomes the squash message and release notes. Please add an SDK paragraph listing the breaking API changes.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -1035,21 +1077,31 @@ where
if kind != ConsumerKind::ConsumerGroup {
return owner_generation == 0;
}
- self.consumer_group_owner(u64::from(consumer_id))
- .is_some_and(|owner| {
- owner.generation == owner_generation
- && (!owner.is_unassigned()
- || (operation == Operation::DeleteConsumerOffset
- && is_auto_commit_client(client_id)))
- && (is_auto_commit_client(client_id)
- || (owner.client_id == client_id && owner.session == session))
- })
+ let group_id = u64::from(consumer_id);
+ let Some(owner) = self.consumer_group_owner(group_id) else {
+ // Group ids are never reused, so no owner follows a retired group.
+ // A replica can still hold its offset when the primary that retired
+ // it had none, and a delete admitted under the fence can commit
+ // after the catalog, so this cleanup passes at any generation.
+ return operation == Operation::DeleteConsumerOffset
+ && is_auto_commit_client(client_id)
+ && self.consumer_group_retired(group_id);
+ };
+ owner.generation == owner_generation
Review Comment:
nit: the auto-commit generation fence is unpinned. Mutating to `is_auto_commit_client(client_id) || owner.generation == owner_generation` survives 580 partitions tests, 236 sim tests and 12 lifecycle fuzz seeds, because the workload never emits a stale auto-commit (state_checker.rs:458 never fires end to end).
Add a partition unit test: auto-commit admitted at generation G, install G+1 commits before it applies, expect refusal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4270145159
##########
core/server/src/dispatch/mod.rs:
##########
@@ -256,25 +440,50 @@ async fn drain_client_requests<B, MJ, S, SB>(
SB: SuperblockStore + 'static,
{
loop {
- let Some(message) = pop_next_client_request(&queues, client_id) else {
+ let metadata_frontier = shard.plane.metadata().applied_frontier().get();
+ let next = pop_next_client_request(&queues, client_id, |queued| {
+ if is_ping(&queued.message) {
+ return true;
+ }
+ if drain.metadata_write.get() {
+ return false;
+ }
+ queued
+ .progress_key(metadata_frontier, |message| progress_key(&shard, message))
+ .is_none_or(|key| {
+ drain.in_flight.get() < drain.capacity
+ && !drain.progress.borrow().contains(&key)
Review Comment:
PR4 now tests a held keyed Store followed by queued Next(auto=false) and GetConsumerOffset: both remain behind its progress guard. Optional shared read-only lanes are deferred to PR6 and require explicit Store/Delete ordering regressions before changing that serialization. This split is recorded in B7 and the PR6 plan.
##########
foreign/go/client/tcp/tcp_core.go:
##########
@@ -860,6 +857,9 @@ func (c *IggyTcpClient) attempt(
if captured, ok := ctx.Value(capturedPartitionContext{}).(iggcon.PartitionContext); ok {
Review Comment:
PR4 removes the duplicate context lookup. The remaining context.WithValue boxing/allocation work is deferred to PR6, recorded with Go RTT, copies, key reuse and locking measurements in B7. Those measurements follow PR5's managed link/heartbeat lifecycle gates; they do not replace the lifecycle proof.
##########
foreign/csharp/Iggy_SDK/IggyClient/Implementations/TcpMessageStream.PollRouting.cs:
##########
@@ -201,105 +120,142 @@ private static IggyInvalidStatusCodeException TruncatedSend()
return VsrError.Exception(VsrError.INVALID_COMMAND, "The SendMessages body is truncated.");
}
- private static ContextQuery? TryBuildContextQuery(int code, ReadOnlySpan<byte> body)
+ private static SendTarget PartitionTarget(int code, ReadOnlySpan<byte> body)
{
- if (code is CommandCodes.STORE_CONSUMER_OFFSET_CODE or CommandCodes.DELETE_CONSUMER_OFFSET_CODE)
+ if (code == CommandCodes.SEND_MESSAGES_CODE)
{
- var suffixSize = code == CommandCodes.STORE_CONSUMER_OFFSET_CODE
- ? StoreOffsetSuffixSize : DeleteOffsetSuffixSize;
- if (body.Length <= suffixSize)
- {
- return null;
- }
- var query = body[..^suffixSize].ToArray();
- return new ContextQuery(CommandCodes.GET_CONSUMER_OFFSET_ROUTING_CODE, query, ConsumerSessionSize,
- Convert.ToBase64String(query));
+ return ParseSendTarget(body);
}
- if (code != CommandCodes.POLL_MESSAGES_CODE || body.Length <= PollOptionsSize)
+
+ if (body.IsEmpty)
{
- return null;
+ throw TruncatedSend();
}
- return new ContextQuery(CommandCodes.GET_POLL_ROUTING_CODE, body.ToArray(), ConsumerSessionSize,
- Convert.ToBase64String(body[..^PollOptionsSize]));
+ var group = body[0] == ConsumerGroupWireKind;
+ var position = 1;
+ _ = ReadIdentifier(body, ref position);
+ var identifiersStart = position;
+ var streamId = ReadIdentifier(body, ref position);
+ var topicId = ReadIdentifier(body, ref position);
+ var identifiersLength = position - identifiersStart;
+ if (position + 1 + sizeof(uint) > body.Length)
+ {
+ throw TruncatedSend();
+ }
+ var present = body[position++] != 0;
+ if (!present && (code is CommandCodes.STORE_CONSUMER_OFFSET_CODE or CommandCodes.DELETE_CONSUMER_OFFSET_CODE
+ || code == CommandCodes.POLL_MESSAGES_CODE && group))
+ {
+ throw VsrError.Exception(VsrError.INVALID_COMMAND, "This partition request requires an explicit partition id.");
+ }
+ var partitionId = present ? BinaryPrimitives.ReadUInt32LittleEndian(body[position..]) : 0;
+ var query = new byte[identifiersLength + sizeof(uint)];
+ body.Slice(identifiersStart, identifiersLength).CopyTo(query);
+ BinaryPrimitives.WriteUInt32LittleEndian(query.AsSpan(identifiersLength), partitionId);
+ return new SendTarget(new PartitionRouteKey(streamId, topicId, partitionId), query);
+ }
+
+ private static byte[] RouteQuery(PartitionRouteKey key)
+ {
+ var query = new byte[IdentifierPrefixSize * 2 + key.StreamId.Length + key.TopicId.Length + sizeof(uint)];
+ query.AsSpan().WriteBytesFromStreamAndTopicIdentifiers(key.StreamId, key.TopicId);
+ BinaryPrimitives.WriteUInt32LittleEndian(query.AsSpan(query.Length - sizeof(uint)), key.PartitionId);
+ return query;
}
- /// <summary>
- /// Sends a poll or an offset write to the partition primary, over a data connection that leaves the
- /// membership connection where it is. A caller context, or else the first context a route reports, is
- /// retained across every retry, so a retry never moves the request onto a newer incarnation or owner.
- /// </summary>
- /// <param name="code">The poll or offset write command code.</param>
- /// <param name="key">The cached route of the request.</param>
- /// <param name="routing">The body of the routing request.</param>
- /// <param name="payload">The body of the request.</param>
- /// <param name="context">The caller context, or null to take the context the route reports.</param>
- /// <param name="token">The cancellation token.</param>
- private async Task<IMemoryOwner<byte>> SendRoutedAsync(int code, PollRouteKey key, ReadOnlyMemory<byte> routing,
+ private async Task<IMemoryOwner<byte>> SendRoutedAsync(int code, SendTarget target,
ReadOnlyMemory<byte> payload, PartitionContext? context, CancellationToken token)
{
using var cancellation = CancellationTokenSource.CreateLinkedTokenSource(token);
cancellation.CancelAfter(VsrRequestTimeoutMs);
var deadline = Environment.TickCount64 + VsrRequestTimeoutMs;
+ // A new route can change the destination, but not the request's identity or captured position.
var captured = context;
+ SessionFrame? frame = null;
+ ulong? generation = null;
+ var refreshHistory = code == CommandCodes.SEND_MESSAGES_CODE && context is null;
+ var refusedPrimary = false;
+ var lifecycleInterval = LifecycleRetryIntervalMs;
try
{
while (true)
{
try
{
- if (Volatile.Read(ref _clusterNodeCount) == 0)
+ var route = await GetPartitionRouteAsync(target.Key, target.Query, deadline, cancellation.Token);
+ captured ??= route.Route.Context;
+ if (generation is not null && generation != route.Generation)
{
- using var metadata = await SendPollControlAsync(CommandCodes.GET_CLUSTER_METADATA_CODE,
- ReadOnlyMemory<byte>.Empty, deadline, cancellation.Token);
- if (metadata.Memory.Length == 0)
- {
- throw new MalformedResponseException("Poll routing requires a nonempty cluster roster.");
- }
-
- RememberRoster(BinaryMapper.MapClusterMetadata(metadata.Memory.Span));
+ throw PollNotAccepted();
}
-
+ generation ??= route.Generation;
+ frame ??= _consensusSession.Resolve(VsrOperations.ForCode(code));
if (Volatile.Read(ref _clusterNodeCount) <= 1)
{
- captured ??= await CapturePartitionContextAsync(code, payload, cancellation.Token);
- return await SendWithResponseAsync(code, payload, token: cancellation.Token,
- capturedContext: captured);
+ var attempt = await SendVsrAttemptAsync(code, payload, Environment.TickCount64, deadline,
+ false, cancellation.Token, captured.Value, frame,
+ retryTransient: VsrOperations.ForCode(code) != VsrOperation.NonReplicated);
+ if (attempt.Error is not null)
+ {
+ if (attempt.RequestStarted && !VsrOperations.IsReplaySafeRead(code, false, payload.Span)
+ && !IsDefinitiveVerdict(attempt.Error))
+ {
+ throw new VsrRequestOutcomeUnknownException(attempt.Error);
+ }
+ ExceptionDispatchInfo.Throw(attempt.Error);
+ }
+ return attempt.Response!;
}
- var route = await GetPollRouteAsync(key, routing, deadline, cancellation.Token);
- captured ??= route.Context;
- return await SendPrimaryOnceAsync(key,
- code == CommandCodes.POLL_MESSAGES_CODE ? CommandCodes.POLL_MESSAGES_ON_PRIMARY_CODE : code,
- payload, route, captured.Value, deadline, cancellation.Token);
+ return await SendPrimaryOnceAsync(target.Key, code, payload, route, captured.Value,
+ frame.Value, deadline, cancellation.Token);
}
- catch (IggyInvalidStatusCodeException error) when (error.StatusCode == VsrError.TRANSIENT_NOT_ACCEPTED)
+ catch (IggyInvalidStatusCodeException error) when (error.StatusCode is
+ VsrError.TRANSIENT_NOT_ACCEPTED or VsrError.NOT_PRIMARY)
{
- RemovePollRoute(key);
- if (Environment.TickCount64 + PollRoutingRetryIntervalMs >= deadline)
+ RemovePartitionRoute(target.Key);
+ if ((generation is not null && generation != _consensusSession.Generation)
Review Comment:
Deferred to PR5 native route-activation acceptance. B7 and the PR5 plan require LifecycleBusy or HistoryUnavailable followed by relogin so the post-refresh generation guard is exercised independently. Removing either guard alone must fail its own regression while the other remains enabled. The existing dual-guard mutation test is not claimed as independent proof.
##########
core/shard/src/lib.rs:
##########
@@ -5197,7 +5345,18 @@ where
let Some(partition) = planes.1.0.get_mut_by_ns(&namespace) else {
return;
};
- if !partition.consensus().is_normal() {
+ if !self.partition_incarnation_matches(
+ partition,
+ header.partition_incarnation,
+ Command::RequestPrepares,
+ ) {
+ return;
+ }
+ if !matches!(
Review Comment:
The new ViewChange donor exposure is fixed in b048d0263: requests carry the requester view; a ViewChange donor serves its uncommitted suffix only to that view's primary, while ordinary backups and other views receive at most its locally applied prefix. RepairDone/RangeEvicted must also match the selected peer and nonce. The regression captures real repair frames and preserves primary-elect missing-body serving.
The broader pre-existing requester-side trust problem is deferred to PR6 as a correctness/release gate in B5 and the plan. Repair bodies still lack sender/nonce attribution. Closure requires canonical or committed evidence for unanchored history and end-to-end five-/seven-replica commit, failover and readback tests, including stale normal donors, blank canonical heads and delayed bodies after peer rotation. The donor regression and terminator check do not claim that broader proof.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4270143419
##########
core/simulator/src/workload/oracle.rs:
##########
@@ -449,14 +500,15 @@ pub fn assert_converged(sim: &Simulator, workload: &mut Workload) -> Convergence
"committed metadata prefixes agree across every live replica"
);
- let Some(leader) = metadata_leader(sim, &live) else {
+ let Some(leader) = sim.metadata_leader().map(usize::from) else {
panic!("no metadata leader live at quiesce (seed={seed:#x})");
};
// Settlement treats "no live replica hosts this group" as settled, so losing
// every instance of a LIVE partition reads as converged. The metadata read
// separates that from a group whose stream the workload deleted.
let namespaces_checked = assert_live_namespaces_have_a_primary(sim, workload, leader, seed);
Review Comment:
Deferred to PR6 as recovery release blockers. B5 now retains the RebuildPipeline floor fix (`max(commit_min, merged.commit_max) + 1`), quorum-loss creation/delete wedge (#4476 family, including seed 24), and the baseline metadata fork campaigns. Closure requires separate baseline/candidate reproductions with full settings and acknowledged-history/recovery proof. The passing PR4 seed 7016 does not close these findings; the baseline attribution remains your evidence, not a campaign I reran.
##########
core/simulator/src/workload/oracle.rs:
##########
@@ -449,14 +500,15 @@ pub fn assert_converged(sim: &Simulator, workload: &mut Workload) -> Convergence
"committed metadata prefixes agree across every live replica"
);
- let Some(leader) = metadata_leader(sim, &live) else {
+ let Some(leader) = sim.metadata_leader().map(usize::from) else {
panic!("no metadata leader live at quiesce (seed={seed:#x})");
};
// Settlement treats "no live replica hosts this group" as settled, so losing
// every instance of a LIVE partition reads as converged. The metadata read
// separates that from a group whose stream the workload deleted.
let namespaces_checked = assert_live_namespaces_have_a_primary(sim, workload, leader, seed);
+ assert_hosted_incarnations_named(sim, workload, &live, seed);
// Safety direction: no live replica may have COMMITTED more of a group than
Review Comment:
Deferred to PR6. The plan and consolidated simulator acceptance now require per-namespace applied-data/readback digests through apply, rebuild and transfer installation, plus a negative control where prepare identities agree but applied bytes differ. Simulator partition transfer must be implemented or its unsupported-directory runs explicitly classified, including seed 1078 with its full campaign settings. Such a classified run does not count as passing recovery evidence. The PR description limits PR4's convergence claim to prepare identities and offset tables.
##########
core/message_bus/src/transports/quic.rs:
##########
@@ -326,12 +316,205 @@ fn reply_route(frame: &BusMessage) -> ReplyRoute {
if !is_reply {
return ReplyRoute::Unkeyed;
}
+ read_u64(bytes, offset_of!(ReplyHeader, request)).map_or(ReplyRoute::Unkeyed, ReplyRoute::Keyed)
+}
+
+/// The request id of a client frame, read as raw bytes like [`reply_route`],
+/// so a frame with an undeclared operation still has one.
+fn request_id(request: &Message<GenericHeader>) -> Option<u64> {
+ read_u64(request.as_slice(), offset_of!(RequestHeader, request))
+}
+
+fn read_u64(bytes: &[u8], offset: usize) -> Option<u64> {
bytes
- .get(..HEADER_SIZE)
- .and_then(|header| bytemuck::checked::try_from_bytes::<ReplyHeader>(header).ok())
- .map_or(ReplyRoute::Unkeyed, |header| {
- ReplyRoute::Keyed(header.request)
- })
+ .get(offset..offset + size_of::<u64>())
+ .and_then(|field| field.try_into().ok())
+ .map(u64::from_le_bytes)
+}
+
+/// Hands the outbound frames of one connection to the streams that wait
+/// for them.
+///
+/// A stream registers its waiter before it forwards its request, so a reply
+/// never comes before the stream it answers.
+struct ReplyRouter {
+ /// Waiting streams by request id, oldest first.
+ waiters: HashMap<u64, VecDeque<oneshot::Sender<BusMessage>>>,
+ /// Frames that came while no stream waited for them, oldest first.
+ unclaimed: VecDeque<BusMessage>,
+ /// When `unclaimed` holds this many frames, it drops its oldest one.
+ unclaimed_max: usize,
+}
+
+impl ReplyRouter {
+ fn new(unclaimed_max: usize) -> Self {
+ Self {
+ waiters: HashMap::new(),
+ unclaimed: VecDeque::new(),
+ unclaimed_max,
+ }
+ }
+
+ /// Register a stream that waits for the reply to `request_id`.
+ ///
+ /// The stream first takes the oldest unclaimed frame that has no request
+ /// id or has its own. The SDK replays an explicit transient with the same
+ /// request id on a new stream, so a late reply of the first attempt
+ /// answers the replay. The replies for other ids ahead of that frame are
+ /// dropped, because their streams already ended.
+ fn register(&mut self, request_id: u64) -> oneshot::Receiver<BusMessage> {
+ let (waiter, reply) = oneshot::channel();
+ while let Some(frame) = self.unclaimed.pop_front() {
+ if !matches!(reply_route(&frame), ReplyRoute::Keyed(id) if id != request_id) {
Review Comment:
Deferred to PR6 with the QUIC reply-tracking work. An unclaimed Eviction that predates a new stream must not be assigned to that stream. Acceptance requires late-event isolation together with session-terminal Eviction behavior, bounded tracking, late acknowledgements and draining within close_grace. The B7 and PR6 plan entries retain these conditions; PR4 does not change Eviction into a nonterminal event.
##########
core/server/src/dispatch/mod.rs:
##########
@@ -557,25 +814,51 @@ async fn handle_client_request<B, MJ, S, SB>(
// (`submit_register_local_or_forward`). Every other non-replicated
// code MUST go through Register first, which binds the acting user
// the per-op authz gates resolve.
- handle_non_replicated_request(
- shard,
- sessions,
- server_config,
- transport_client_id,
- request,
- ConnectionContext {
- bound,
- user_id,
- address: client_address,
- metadata_watermark,
- },
- )
- .await;
+ let context = ConnectionContext {
+ bound,
+ user_id,
+ address: client_address,
+ metadata_watermark,
+ };
+ // `ClientDrain` says which frames stay on the drain.
+ if matches!(non_replicated_code(header), PING_CODE | BIND_SESSION_CODE) {
+ handle_non_replicated_request(
+ shard,
+ sessions,
+ server_config,
+ transport_client_id,
+ request,
+ context,
+ )
+ .await;
+ return;
+ }
+ let in_flight = match progress {
+ Some(progress) => progress,
+ None => drain.admit(Admission::Cap).await,
+ };
+ let task_shard = Rc::clone(shard);
+ let task_sessions = Rc::clone(sessions);
+ let task_config = Arc::clone(server_config);
+ shard.bus.spawn(async move {
Review Comment:
Deferred to PR6. The recorded proposal is to poll inline, select against frame arrival, then promote the same pending future, InFlight permit and progress-lane guard when another frame arrives. The existing PING negative control rejects blanket inline serialization; it does not disprove this proposal. B7 requires lockstep and pipelined benchmarks plus preserved cancellation, bounded admission, unknown outcomes and Next/auto-commit ordering. The reported performance figures remain attributed measurements, not results I reproduced.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -377,6 +381,74 @@ async fn relay_partition_reply<B, MJ, S, SB>(
});
}
+/// The offset key shared by stored-offset reads and progress writes on a connection.
+/// Failed resolution is left to the normal handler's authorization and error path.
+pub(super) fn progress_key<B, MJ, S, SB>(
+ shard: &Rc<ShellShard<B, MJ, S, SB>>,
+ message: &Message<GenericHeader>,
+) -> Option<ConsumerProgress>
+where
+ B: ShellBus,
+ MJ: JournalHandle + 'static,
+ MJ::Target: Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>,
+ S: 'static,
+ SB: SuperblockStore + 'static,
+{
+ let request = message.try_as_typed::<RequestHeader>().ok()?;
+ let wire = match request.header().operation {
+ Operation::NonReplicated
+ if request.header().reserved[..4] == GET_CONSUMER_OFFSET_CODE.to_le_bytes() =>
+ {
+ GetConsumerOffsetRequest::decode_from(request.body()).ok()?
+ }
+ Operation::NonReplicated
+ if request.header().reserved[..4] == POLL_MESSAGES_CODE.to_le_bytes()
+ || request.header().reserved[..4]
+ == POLL_MESSAGES_ON_PRIMARY_CODE.to_le_bytes() =>
+ {
+ let poll = PollMessagesRequest::decode_from(request.body()).ok()?;
Review Comment:
Deferred to PR6. B7 now explicitly requires removing the duplicate full PollMessages decode/allocation for non-Next, non-auto-commit polls, using the flag/strategy fields or passing the decoded request onward. Closure requires allocation/throughput measurements and unchanged Next/auto-commit progress ordering.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4261451684
##########
core/sdk/src/links/mux.rs:
##########
@@ -0,0 +1,815 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! One TCP or WebSocket connection that carries many requests at once.
+//!
+//! A writer task and a reader task own the two halves of the connection, and
+//! each reply goes to the request with its id. So a caller that goes away
+//! never leaves a frame half written or half read on the connection.
+
+use crate::links::{Frame, Reply};
+use crate::vsr::{decode_eviction, peek_command, reply_request_id, response_size};
+use bytes::{BufMut, BytesMut};
+use futures_util::stream::{SplitSink, SplitStream};
+use futures_util::{SinkExt, StreamExt};
+use iggy_binary_protocol::consensus::{Command, HEADER_SIZE};
+use iggy_common::IggyError;
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::future::Future;
+use std::io;
+use std::net::SocketAddr;
+use std::sync::{Arc, Mutex as StdMutex, MutexGuard, PoisonError};
+use std::time::Duration;
+use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, BufReader, BufWriter};
+use tokio::sync::{Mutex, Semaphore, mpsc, oneshot, watch};
+use tokio::task::JoinHandle;
+use tokio::time::{Instant, timeout, timeout_at};
+use tokio_tungstenite::WebSocketStream;
+use tokio_tungstenite::tungstenite::Message;
+use tokio_tungstenite::tungstenite::protocol::CloseFrame;
+use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode;
+use tracing::{debug, error, trace};
+
+/// Frames queued for the writer task before a sender waits for room.
+const LINK_WRITE_QUEUE: usize = 32;
+/// Requests that wait for a reply on one link at once. It matches
+/// `message_bus.client_queue_capacity`, the most requests the server runs at
+/// once for one connection.
+const MAX_PENDING_PER_LINK: usize = 256;
+/// A peer that stops reading cannot hold up a shutdown for longer than this.
+const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(2);
+const WEBSOCKET_CLOSE_REASON: &str = "Client requested shutdown";
+
+type Waiter = oneshot::Sender<Result<Reply, IggyError>>;
+
+/// A connection that carries many requests at once. Dropping the link stops
+/// its tasks, which closes the connection.
+#[derive(Debug)]
+pub(crate) struct MuxLink {
+ shared: Arc<MuxShared>,
+ queue: mpsc::Sender<Frame>,
+ tasks: LinkTasks,
+}
+
+#[derive(Debug)]
+struct MuxShared {
+ peer: SocketAddr,
+ pending: StdMutex<Pending>,
+ permits: Semaphore,
+ /// Before bind, request ids repeat, so one such exchange runs at a time.
+ unbound: Mutex<()>,
+ /// Turns true once, when the link closes.
+ closed: watch::Sender<bool>,
+}
+
+#[derive(Debug)]
+struct Pending {
+ /// Cleared when the link closes. A closed link never opens again.
+ open: bool,
+ waiters: HashMap<u64, Waiter>,
+}
+
+#[derive(Debug)]
+struct LinkTasks {
+ writer: JoinHandle<()>,
+ reader: JoinHandle<()>,
+}
+
+impl Drop for LinkTasks {
+ fn drop(&mut self) {
+ self.writer.abort();
+ self.reader.abort();
+ }
+}
+
+/// A request that waits for its reply. Dropping it ends the wait.
+struct PendingEntry<'a> {
+ shared: &'a MuxShared,
+ request: u64,
+ /// Set for a repeated id that reached the queue: its late reply would
+ /// answer the next request with the same id.
+ close_if_abandoned: bool,
+}
+
+impl MuxLink {
+ /// A link over the two halves of a TCP connection, plain or TLS.
+ pub(crate) fn tcp<R, W>(reader: R, writer: W, peer: SocketAddr) -> Self
+ where
+ R: AsyncRead + Unpin + Send + 'static,
+ W: AsyncWrite + Unpin + Send + 'static,
+ {
+ Self::start(
+ peer,
+ |frames, shared| write_tcp(BufWriter::new(writer), frames, shared),
+ |shared| read_tcp(BufReader::new(reader), shared),
+ )
+ }
+
+ /// A link over a WebSocket connection, plain or TLS.
+ pub(crate) fn websocket<S>(stream: WebSocketStream<S>, peer: SocketAddr) -> Self
+ where
+ S: AsyncRead + AsyncWrite + Unpin + Send + 'static,
+ {
+ let (sink, stream) = stream.split();
+ Self::start(
+ peer,
+ |frames, shared| write_websocket(sink, frames, shared),
+ |shared| read_websocket(stream, shared),
+ )
+ }
+
+ /// Sends `frame` and waits for its reply until `deadline`. `sent` turns
+ /// true when the frame reaches the queue, so a caller that goes away
+ /// before the reply still knows that the frame left.
+ ///
+ /// `NotConnected` means the frame never reached the queue, so the request
+ /// can go again. `Disconnected` or an eviction reason means it did, and no
+ /// reply came.
+ pub(crate) async fn send(
+ &self,
+ frame: Frame,
+ deadline: Instant,
+ sent: &mut bool,
+ ) -> Result<Reply, IggyError> {
+ let request = frame.header.request;
+ let unbound = frame.header.session == 0;
+ let _unbound_turn = if unbound {
+ let turn = timeout_at(deadline, self.shared.unbound.lock()).await;
+ Some(turn.map_err(|_| IggyError::NotConnected)?)
+ } else {
+ None
+ };
+ let _permit = timeout_at(deadline, self.shared.permits.acquire())
+ .await
+ .map_err(|_| IggyError::NotConnected)?
+ .map_err(|_| IggyError::NotConnected)?;
+ let (waiter, reply) = oneshot::channel();
+ let mut entry = self.shared.wait_for(request, waiter)?;
+ if !matches!(
+ timeout_at(deadline, self.queue.send(frame)).await,
+ Ok(Ok(()))
+ ) {
+ return Err(IggyError::NotConnected);
+ }
+ *sent = true;
+ entry.close_if_abandoned = unbound;
+ match timeout_at(deadline, reply).await {
+ Ok(Ok(result)) => result,
+ Ok(Err(_)) => Err(IggyError::Disconnected),
Review Comment:
Deferred to multi-leader PR5 with the transport/driver error-contract audit. Separate exchange failures normalized to Disconnected from establishment paths that still use transport-specific errors, then remove only unreachable arms and document downstream matching changes.
##########
core/sdk/src/links/replicas.rs:
##########
@@ -0,0 +1,1571 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! One link per replica of the roster, and the sweep that signs in.
+//!
+//! `connect()` sweeps the known endpoints. The first link that answers
+//! becomes the bootstrap link, and the session signs in there. A read of the
+//! roster then gives one slot per replica, keyed by its replica id. Each slot
+//! runs one task that dials its replica, binds the session there and redials
+//! after a loss. A request never dials: it takes a ready link or fails with
+//! `NotConnected`.
+
+use super::connect::{
+ ConnectCoordinator, is_same_address, is_same_spelling, node_address, transport_port,
+};
+use super::{
+ Dialer, Link, OPERATION_TIMEOUT, backoff, event_channel, exchange, lost_link,
+ pause_before_retry,
+};
+use crate::SDK_VERSION;
+use crate::session_owner::{SessionEpoch, SessionOwner};
+use async_broadcast::{Receiver, Sender};
+use futures::future::join_all;
+use iggy_binary_protocol::codec::{WireDecode, WireEncode};
+use iggy_binary_protocol::codes::{BIND_SESSION_CODE, GET_CLUSTER_METADATA_CODE, PING_CODE};
+use iggy_binary_protocol::requests::system::{
+ BindSessionRequest, GetClusterMetadataRequest, PingRequest,
+};
+use iggy_binary_protocol::responses::system::ClusterMetadataResponse;
+use iggy_binary_protocol::responses::users::LoginRegisterResponse;
+use iggy_common::{
+ ClientState, ClusterMetadata, ClusterNodeRole, Credentials, DiagnosticEvent, IggyError,
+ QuicClientReconnectionConfig, TcpClientReconnectionConfig, WebSocketClientReconnectionConfig,
+ decode_session_binding, rust_sdk_version_info, sign_in_request,
+};
+use std::collections::{BTreeMap, BTreeSet};
+use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
+use std::time::Duration;
+use tokio::sync::Semaphore;
+use tokio::task::AbortHandle;
+use tokio::time::{Instant, sleep};
+use tracing::{info, warn};
+
+/// Dials that run at once, for one client.
+const MAX_CONCURRENT_DIALS: usize = 4;
+
+/// The longest pause before a slot or a sweep tries again.
+const RECONNECT_BACKOFF_MAX: Duration = Duration::from_secs(10);
+
+/// The reconnection configuration, the same for every transport.
+#[derive(Debug, Clone, Copy)]
+pub(crate) struct ReconnectPolicy {
+ /// Without it, a lost link stays down, and a sweep makes one pass.
+ pub(super) enabled: bool,
+ /// The attempts of a slot after its first, and the passes of a sweep
+ /// after its first. `None` means no limit.
+ pub(super) max_retries: Option<u32>,
+ /// The base of the backoff.
+ pub(super) interval: Duration,
+ /// A link that stayed up this long redials at once after a loss.
+ pub(super) reestablish_after: Duration,
+}
+
+impl ReconnectPolicy {
+ fn allows(&self, retry: u32) -> bool {
+ self.enabled && self.max_retries.is_none_or(|max| retry <= max)
+ }
+
+ fn pause(&self, retry: u32) -> Duration {
+ backoff(retry, self.interval, RECONNECT_BACKOFF_MAX, rand::random())
+ }
+}
+
+macro_rules! reconnect_policy_from {
+ ($($config:ty),+) => {$(
+ impl From<&$config> for ReconnectPolicy {
+ fn from(config: &$config) -> Self {
+ Self {
+ enabled: config.enabled,
+ max_retries: config.max_retries,
+ interval: config.interval.get_duration(),
+ reestablish_after: config.reestablish_after.get_duration(),
+ }
+ }
+ }
+ )+};
+}
+
+reconnect_policy_from!(
+ TcpClientReconnectionConfig,
+ QuicClientReconnectionConfig,
+ WebSocketClientReconnectionConfig
+);
+
+/// Where a request goes.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(crate) enum Target {
+ /// The metadata primary of the last roster, else the bootstrap link.
+ MetadataPrimary,
+ Replica(u8),
+}
+
+/// The links of one client. Dropping it aborts every task and drops every
+/// link, without a runtime.
+#[derive(Debug)]
+pub(crate) struct ReplicaLinks {
+ shared: Arc<LinksShared>,
+}
+
+/// The part that the tasks hold. No task holds [`ReplicaLinks`], so its
+/// `Drop` always runs.
+#[derive(Debug)]
+struct LinksShared {
+ dialer: Dialer,
+ configured_endpoint: String,
+ policy: ReconnectPolicy,
+ session: Arc<SessionOwner>,
+ /// Never held across an await. The epoch of the session is read before
+ /// this lock, never under it.
+ table: Mutex<Table>,
+ dial_permits: Semaphore,
+ coordinator: ConnectCoordinator,
+ events: Sender<DiagnosticEvent>,
+ /// Never read. Each subscriber is a clone, so it replays the backlog.
+ subscribers: Receiver<DiagnosticEvent>,
+}
+
+#[derive(Debug)]
+struct Table {
+ state: ClientState,
+ /// One slot per replica of the last roster. A replica id is a `u8`, so
+ /// the table holds at most 256 slots.
+ slots: BTreeMap<u8, Slot>,
+ bootstrap: Option<Bootstrap>,
+ metadata_primary: Option<u8>,
+ /// The endpoints of the last roster. A sweep dials them after the
+ /// configured endpoint.
+ roster: Vec<String>,
+}
+
+#[derive(Debug)]
+struct Slot {
+ endpoint: String,
+ link: Option<ReadyLink>,
+ /// `None` after the task stopped. Only the task of this handle changes
+ /// the slot.
+ task: Option<AbortHandle>,
+}
+
+/// The link that the sweep signed in on, until a slot takes it over or the
+/// metadata primary slot is ready.
+#[derive(Debug)]
+struct Bootstrap {
+ endpoint: String,
+ link: ReadyLink,
+ /// `None` while the sweep still uses the link.
+ watcher: Option<AbortHandle>,
+}
+
+#[derive(Debug)]
+struct ReadyLink {
+ link: Arc<Link>,
+ /// The fence of the session that the link carries, `None` before a
+ /// sign-in.
+ bound_fence: Option<u64>,
+}
+
+/// How one endpoint of a sweep ended.
+enum Attempt {
+ /// Another endpoint can still work.
+ EndpointFailed,
+ /// No retry can fix it, but the endpoints behind this one can still work.
+ ConfigFault(IggyError),
+ /// The sign-in lost its link. Another endpoint can still work.
+ SignInLost(IggyError),
+ /// The sweep ends with this error.
+ Failed(IggyError),
+}
+
+#[derive(Debug)]
+struct SignInFailure {
+ error: IggyError,
+ link_lost: bool,
+}
+
+impl From<IggyError> for SignInFailure {
+ fn from(error: IggyError) -> Self {
+ Self {
+ link_lost: lost_link(&error),
+ error,
+ }
+ }
+}
+
+impl ReplicaLinks {
+ pub(crate) fn new(
+ dialer: Dialer,
+ configured_endpoint: String,
+ policy: ReconnectPolicy,
+ session: Arc<SessionOwner>,
+ ) -> Self {
+ let (events, subscribers) = event_channel();
+ let table = Table {
+ state: ClientState::Disconnected,
+ slots: BTreeMap::new(),
+ bootstrap: None,
+ metadata_primary: None,
+ roster: Vec::new(),
+ };
+ Self {
+ shared: Arc::new(LinksShared {
+ dialer,
+ configured_endpoint,
+ policy,
+ session,
+ table: Mutex::new(table),
+ dial_permits: Semaphore::new(MAX_CONCURRENT_DIALS),
+ coordinator: ConnectCoordinator::new(),
+ events,
+ subscribers,
+ }),
+ }
+ }
+
+ /// Sweeps the known endpoints until one of them signs in. Concurrent
+ /// callers share one result. On a connected client, every slot without a
+ /// link dials again at once.
+ pub(crate) async fn connect(&self) -> Result<(), IggyError> {
+ let shared = &self.shared;
+ shared
+ .coordinator
+ .run(|abandoned, _| shared.sweep(abandoned))
+ .await
+ }
+
+ /// Aborts every task and closes every link, which fails each pending
+ /// request. The roster stays for the next sweep.
+ pub(crate) fn disconnect(&self) {
+ self.shared.disconnect();
+ }
+
+ /// Ends every link the way its protocol expects. It is final.
+ pub(crate) async fn shutdown(&self) {
+ let links = {
+ let mut table = self.shared.table();
+ if table.state == ClientState::Shutdown {
+ return;
+ }
+ table.state = ClientState::Shutdown;
+ table.clear()
+ };
+ join_all(links.into_iter().map(Link::shutdown)).await;
+ self.shared.dialer.shutdown().await;
+ self.shared.publish(DiagnosticEvent::Shutdown);
+ }
+
+ /// Pings every link that serves the session, all at once. A failed ping
+ /// only logs, because a link that lost its connection redials on its own.
+ pub(crate) async fn keepalive(&self) -> Result<(), IggyError> {
+ let links: Vec<Arc<Link>> = {
+ let epoch = self.shared.session.epoch();
+ let table = self.shared.table();
+ if table.state == ClientState::Shutdown {
+ return Err(IggyError::ClientShutdown);
+ }
+ table
+ .links()
+ .filter(|ready| serves(ready, epoch))
+ .map(|ready| Arc::clone(&ready.link))
+ .collect()
+ };
+ let deadline = Instant::now() + OPERATION_TIMEOUT;
+ let pings = links.iter().map(|link| {
+ exchange(
+ &self.shared.session,
+ link,
+ PING_CODE,
+ PingRequest.to_bytes(),
+ deadline,
+ )
+ });
+ for result in join_all(pings).await {
+ if let Err(error) = result {
+ warn!("A keepalive ping failed: {error}");
+ }
+ }
+ Ok(())
+ }
+
+ /// The link for `target`. A request never dials, so a target without a
+ /// ready link fails with `NotConnected`.
+ pub(crate) fn link(&self, target: Target) -> Result<Arc<Link>, IggyError> {
+ let epoch = self.shared.session.epoch();
+ let table = self.shared.table();
+ let slot = |id: u8| table.slots.get(&id).and_then(|slot| slot.link.as_ref());
+ let ready = match target {
+ Target::Replica(id) => slot(id),
+ Target::MetadataPrimary => table
+ .metadata_primary
+ .and_then(slot)
+ .filter(|ready| serves(ready, epoch))
+ .or(table.bootstrap.as_ref().map(|bootstrap| &bootstrap.link)),
+ };
+ ready
+ .filter(|ready| serves(ready, epoch))
+ .map(|ready| Arc::clone(&ready.link))
+ .ok_or(IggyError::NotConnected)
+ }
+
+ /// The replicas with a link that serves the session, in id order.
+ pub(crate) fn ready_replicas(&self) -> Vec<u8> {
+ let epoch = self.shared.session.epoch();
+ self.shared
+ .table()
+ .slots
+ .iter()
+ .filter(|(_, slot)| slot.link.as_ref().is_some_and(|ready| serves(ready, epoch)))
+ .map(|(id, _)| *id)
+ .collect()
+ }
+
+ pub(crate) fn metadata_primary(&self) -> Option<u8> {
+ self.shared.table().metadata_primary
+ }
+
+ pub(crate) fn session(&self) -> &Arc<SessionOwner> {
+ &self.shared.session
+ }
+
+ /// Records that `link` carries the session of `fence`. A sign-in calls it
+ /// before it binds the session, so the slots keep this link when they
+ /// close the links of the old session.
+ pub(crate) fn mark_bound(&self, link: &Arc<Link>, fence: u64) {
+ self.shared.mark_bound(link, fence);
+ }
+
+ /// Reads the roster on `link` and rebuilds the slots from it.
+ pub(crate) async fn refresh_roster(&self, link: &Arc<Link>) -> Result<(), IggyError> {
+ self.shared.refresh_roster(link).await
+ }
+
+ pub(crate) fn subscribe(&self) -> Receiver<DiagnosticEvent> {
+ self.shared.subscribers.clone()
+ }
+
+ pub(crate) fn publish(&self, event: DiagnosticEvent) {
+ self.shared.publish(event);
+ }
+
+ pub(crate) fn state(&self) -> ClientState {
+ self.shared.table().state
+ }
+
+ /// Sets the state, except after a shutdown, which is final.
+ pub(crate) fn set_state(&self, state: ClientState) {
+ let mut table = self.shared.table();
+ if table.state != ClientState::Shutdown {
+ table.state = state;
+ }
+ }
+
+ /// The endpoint of the metadata primary, else of the bootstrap link, else
+ /// the configured one.
+ pub(crate) fn connection_address(&self) -> String {
+ let table = self.shared.table();
+ table
+ .metadata_primary
+ .and_then(|id| table.slots.get(&id))
+ .map(|slot| &slot.endpoint)
+ .or(table
+ .bootstrap
+ .as_ref()
+ .map(|bootstrap| &bootstrap.endpoint))
+ .unwrap_or(&self.shared.configured_endpoint)
+ .clone()
+ }
+}
+
+impl Drop for ReplicaLinks {
+ fn drop(&mut self) {
+ // An abort needs no runtime, and each dropped link stops its own
+ // tasks.
+ drop(self.shared.table().clear());
+ }
+}
+
+impl LinksShared {
+ fn table(&self) -> MutexGuard<'_, Table> {
+ self.table.lock().unwrap_or_else(PoisonError::into_inner)
+ }
+
+ /// The channel runs in overflow mode, so a publish never waits.
+ fn publish(&self, event: DiagnosticEvent) {
+ let _ = self.events.try_broadcast(event);
+ }
+
+ fn disconnect(&self) {
+ let (links, previous) = {
+ let mut table = self.table();
+ if table.state == ClientState::Shutdown {
+ return;
+ }
+ let links = table.clear();
+ (
+ links,
+ std::mem::replace(&mut table.state, ClientState::Disconnected),
+ )
+ };
+ links.iter().for_each(|link| link.close());
+ if previous != ClientState::Disconnected {
+ self.publish(DiagnosticEvent::Disconnected);
+ }
+ }
+
+ async fn sweep(self: &Arc<Self>, abandoned: bool) -> Result<(), IggyError> {
+ if abandoned {
+ // The last sweep lost its caller halfway.
+ self.disconnect();
+ }
+ let (candidates, stopped) = {
+ let mut table = self.table();
+ match table.state {
+ ClientState::Shutdown => return Err(IggyError::ClientShutdown),
+ ClientState::Connected
+ | ClientState::Authenticating
+ | ClientState::Authenticated => {
+ self.restart_down_slots(&mut table);
+ return Ok(());
+ }
+ ClientState::Disconnected | ClientState::Connecting => {}
+ }
+ table.state = ClientState::Connecting;
+ (self.candidates(&table.roster), table.clear())
+ };
+ stopped.iter().for_each(|link| link.close());
+
+ let mut pass = 0;
+ loop {
+ let mut config_fault = None;
+ let mut sign_in_failure = None;
+ for endpoint in &candidates {
+ match self.try_endpoint(endpoint).await {
+ Ok(()) => return Ok(()),
+ Err(Attempt::EndpointFailed) => {}
+ Err(Attempt::ConfigFault(error)) => config_fault = Some(error),
+ Err(Attempt::SignInLost(error)) => sign_in_failure = Some(error),
+ Err(Attempt::Failed(error)) => return Err(error),
+ }
+ }
+ pass += 1;
+ let failure = match config_fault {
+ Some(error) => Some(error),
+ None if !self.policy.allows(pass) => {
+ Some(sign_in_failure.unwrap_or(IggyError::CannotEstablishConnection))
+ }
+ None => None,
+ };
+ if let Some(error) = failure {
+ self.fail_sweep();
+ return Err(error);
+ }
+ let pause = self.policy.pause(pass);
+ info!("No endpoint signed in. Pass {pass} starts in {pause:?}.");
+ sleep(pause).await;
+ }
+ }
+
+ /// The configured endpoint, then the endpoints of the last roster, each
+ /// spelling once. Spellings need no name lookup, which can stall a sweep.
+ fn candidates(&self, roster: &[String]) -> Vec<String> {
+ let mut candidates = vec![self.configured_endpoint.clone()];
+ for endpoint in roster {
+ if !candidates
+ .iter()
+ .any(|known| is_same_spelling(known, endpoint))
+ {
+ candidates.push(endpoint.clone());
+ }
+ }
+ candidates
+ }
+
+ async fn try_endpoint(self: &Arc<Self>, endpoint: &str) -> Result<(), Attempt> {
+ sweeping(self.table().state).map_err(Attempt::Failed)?;
+ info!("Connecting to {endpoint}...");
+ let dialed = match self.dial_permits.acquire().await {
+ Ok(_permit) => self.dialer.dial(endpoint).await,
+ Err(_) => Err(IggyError::CannotEstablishConnection),
+ };
+ let link = match dialed {
+ Ok(link) => Arc::new(link),
+ Err(IggyError::CannotEstablishConnection) => return Err(Attempt::EndpointFailed),
+ Err(error) => return Err(Attempt::ConfigFault(error)),
+ };
+ if let Err(error) = self.install_bootstrap(endpoint, &link) {
+ link.close();
+ return Err(Attempt::Failed(error));
+ }
+ self.publish(DiagnosticEvent::Connected);
+ info!("Connected to {endpoint}.");
+
+ if let Err(failure) = self.resume_or_sign_in(&link).await {
+ if failure.link_lost {
+ warn!(
+ "The sign-in on {endpoint} did not complete: {}",
+ failure.error
+ );
+ self.lose_bootstrap(&link);
+ return Err(Attempt::SignInLost(failure.error));
+ }
+ // The link stands, and only the session is missing. Every
+ // endpoint gives the same answer to the same credentials.
+ self.watch_bootstrap(&link);
+ return Err(Attempt::Failed(failure.error));
+ }
+ if self.session.epoch().bound
+ && let Err(error) = self.refresh_roster(&link).await
+ {
+ warn!("Failed to read the roster from {endpoint}: {error}");
+ }
+ self.watch_bootstrap(&link);
+ Ok(())
+ }
+
+ fn install_bootstrap(&self, endpoint: &str, link: &Arc<Link>) -> Result<(), IggyError> {
+ let mut table = self.table();
+ sweeping(table.state)?;
+ table.bootstrap = Some(Bootstrap {
+ endpoint: endpoint.to_owned(),
+ link: ReadyLink {
+ link: Arc::clone(link),
+ bound_fence: None,
+ },
+ watcher: None,
+ });
+ table.state = ClientState::Connected;
+ Ok(())
+ }
+
+ /// Drops a bootstrap link that its sign-in lost. The sweep goes on.
+ fn lose_bootstrap(&self, link: &Arc<Link>) {
+ link.close();
+ let mut table = self.table();
+ if !table.is_bootstrap(link) {
+ return;
+ }
+ table.bootstrap = None;
+ table.state = ClientState::Connecting;
+ drop(table);
+ self.publish(DiagnosticEvent::Disconnected);
+ }
+
+ fn fail_sweep(&self) {
+ let mut table = self.table();
+ if table.state == ClientState::Connecting {
+ table.state = ClientState::Disconnected;
+ drop(table);
+ self.publish(DiagnosticEvent::Disconnected);
+ }
+ }
+
+ /// Watches a bootstrap link that the sweep is done with, until it closes.
+ fn watch_bootstrap(self: &Arc<Self>, link: &Arc<Link>) {
+ let mut table = self.table();
+ let Some(bootstrap) = table
+ .bootstrap
+ .as_mut()
+ .filter(|bootstrap| Arc::ptr_eq(&bootstrap.link.link, link))
+ else {
+ return;
+ };
+ let shared = Arc::clone(self);
+ let link = Arc::clone(link);
+ let watcher = tokio::spawn(async move {
+ shared.hold(&link).await;
+ shared.bootstrap_down(&link);
+ });
+ bootstrap.watcher = Some(watcher.abort_handle());
+ }
+
+ fn bootstrap_down(&self, link: &Arc<Link>) {
+ let mut table = self.table();
+ if !table.is_bootstrap(link) {
+ return;
+ }
+ table.bootstrap = None;
+ let last = table.ready_count() == 0;
+ table.settle();
+ drop(table);
+ if last {
+ self.publish(DiagnosticEvent::Disconnected);
+ }
+ }
+
+ /// Resumes a bound session on the bootstrap link, else signs in there.
+ async fn resume_or_sign_in(&self, link: &Arc<Link>) -> Result<(), SignInFailure> {
+ let epoch = self.session.epoch();
+ if epoch.bound {
+ match self.bind_session(link).await {
+ Ok(()) => {
+ self.mark_bound(link, epoch.fence);
+ self.set_state_on(link, ClientState::Authenticated);
+ self.publish(DiagnosticEvent::SignedIn);
+ info!("Resumed the session.");
+ return Ok(());
+ }
+ Err(error) if session_lost(&error) => {
+ info!("The session ended ({error}). Signing in again.");
+ self.session.reset_if_fence(epoch.fence)?;
+ }
+ // A replica that cannot bind inside its read budget is behind
+ // the session, and another one can be further ahead.
+ Err(error) => {
+ return Err(SignInFailure {
+ link_lost: lost_link(&error)
+ || matches!(error, IggyError::TransientNotAccepted),
+ error,
+ });
+ }
+ }
+ }
+ self.sign_in(link).await
+ }
+
+ /// Signs in on `link` with the known credentials, if any. A failure that
+ /// leaves the link up leaves the client connected without a session. A
+ /// rejected credential is forgotten, because it stays rejected, and each
+ /// replay costs the server a password hash.
+ async fn sign_in(&self, link: &Arc<Link>) -> Result<(), SignInFailure> {
+ let Some(credentials) = self.session.sign_in_credentials() else {
+ info!("No credentials to sign in with.");
+ return Ok(());
+ };
+ self.set_state_on(link, ClientState::Authenticating);
+ let error = match self.register(link, &credentials).await {
+ Ok(user_id) => {
+ self.set_state_on(link, ClientState::Authenticated);
+ self.session.remember(credentials, user_id);
+ self.publish(DiagnosticEvent::SignedIn);
+ info!("Signed in.");
+ return Ok(());
+ }
+ Err(error) => error,
+ };
+ let failure = SignInFailure::from(error);
+ if !failure.link_lost {
+ self.set_state_on(link, ClientState::Connected);
+ }
+ if matches!(
+ failure.error,
+ IggyError::InvalidCredentials
+ | IggyError::InvalidUsername
+ | IggyError::InvalidPassword
+ | IggyError::Unauthenticated
+ ) {
+ self.session.forget();
+ }
+ Err(failure)
+ }
+
+ /// Registers a new session on `link` and binds it. A Register that the
+ /// server did not commit or did not admit goes again with the same client
+ /// id, so it registers one session.
+ async fn register(
+ &self,
+ link: &Arc<Link>,
+ credentials: &Credentials,
+ ) -> Result<u32, IggyError> {
+ let fence = self.session.fence();
+ let (code, body) = sign_in_request(
+ credentials,
+ rust_sdk_version_info(SDK_VERSION)?,
+ self.session.bind_secret()?,
+ )?;
+ let deadline = Instant::now() + OPERATION_TIMEOUT;
+ let mut retry = 0_u32;
+ let reply = loop {
+ let error = match exchange(&self.session, link, code, body.clone(), deadline).await {
+ Ok(reply) => break reply,
+ Err(error) => error,
+ };
+ if !matches!(
+ error,
+ IggyError::TransientNotCommitted | IggyError::TransientNotAccepted
+ ) || !pause_before_retry(retry, deadline, rand::random()).await
+ {
+ return Err(error);
+ }
+ retry = retry.saturating_add(1);
+ };
+ let registered =
+ LoginRegisterResponse::decode_from(&reply).map_err(|_| IggyError::InvalidFormat)?;
+ self.mark_bound(link, fence);
+ self.session.bind_at(fence, registered.session)?;
+ Ok(registered.user_id)
+ }
+
+ /// Binds the bound session on `link`. The reply must name the same
+ /// session.
+ async fn bind_session(&self, link: &Link) -> Result<(), IggyError> {
+ let identity = self.session.identity()?;
+ let request = BindSessionRequest {
+ version_info: rust_sdk_version_info(SDK_VERSION)?,
+ identity,
+ bind_secret: self.session.bind_secret()?,
+ };
+ let reply = exchange(
+ &self.session,
+ link,
+ BIND_SESSION_CODE,
+ request.to_bytes(),
+ Instant::now() + OPERATION_TIMEOUT,
+ )
+ .await?;
+ decode_session_binding(&identity, &reply).map(drop)
+ }
+
+ async fn refresh_roster(self: &Arc<Self>, link: &Arc<Link>) -> Result<(), IggyError> {
+ let reply = exchange(
+ &self.session,
+ link,
+ GET_CLUSTER_METADATA_CODE,
+ GetClusterMetadataRequest.to_bytes(),
+ Instant::now() + OPERATION_TIMEOUT,
+ )
+ .await?;
+ let roster =
+ ClusterMetadataResponse::decode_from(&reply).map_err(|_| IggyError::InvalidFormat)?;
+ self.apply_roster(&ClusterMetadata::try_from(roster)?, link)
+ .await
+ }
+
+ /// Rebuilds the slots from the roster that `via` carried. A removed
+ /// replica or a moved endpoint retires its link, so the requests that
+ /// hold it finish first.
+ async fn apply_roster(
+ self: &Arc<Self>,
+ roster: &ClusterMetadata,
+ via: &Arc<Link>,
+ ) -> Result<(), IggyError> {
+ let protocol = self.dialer.protocol();
+ let mut ids = BTreeSet::new();
+ let mut wanted = BTreeMap::new();
+ for node in &roster.nodes {
+ if !ids.insert(node.replica_id) {
+ warn!(
+ "The roster names replica {} twice, so the links stay as they are.",
+ node.replica_id
+ );
+ return Err(IggyError::InvalidFormat);
+ }
+ // A node without a port for this transport gets no slot, so the
+ // client never dials `ip:0`.
+ let port = transport_port(node, protocol);
+ if port != 0 {
+ wanted.insert(node.replica_id, node_address(node, port));
+ }
+ }
+ let primary = roster
+ .nodes
+ .iter()
+ .find(|node| node.role == ClusterNodeRole::Leader)
+ .map(|node| node.replica_id)
+ .filter(|id| wanted.contains_key(id));
+ let adopt = self.bootstrap_replica(&wanted).await;
+
+ let mut retired = Vec::new();
+ let mut table = self.table();
+ if table.find(via).is_none() {
+ // A disconnect ran during the read.
+ return Err(IggyError::Disconnected);
+ }
+ table.slots.retain(|id, slot| {
+ let kept = wanted.get(id) == Some(&slot.endpoint);
+ if !kept {
+ if let Some(task) = slot.task.take() {
+ task.abort();
+ }
+ retired.extend(slot.link.take());
+ }
+ kept
+ });
+ for (id, endpoint) in &wanted {
+ if table.slots.contains_key(id) {
+ continue;
+ }
+ let adopted = match &adopt {
+ Some((link, replica)) if replica == id && table.is_bootstrap(link) => {
+ table.retire_bootstrap()
+ }
+ _ => None,
+ };
+ let task = self.spawn_slot(
+ *id,
+ endpoint.clone(),
+ adopted.as_ref().map(|ready| Arc::clone(&ready.link)),
+ );
+ table.slots.insert(
+ *id,
+ Slot {
+ endpoint: endpoint.clone(),
+ link: adopted,
+ task: Some(task),
+ },
+ );
+ }
+ table.metadata_primary = primary;
+ table.roster = wanted.into_values().collect();
+ if primary.is_some_and(|id| table.slots.get(&id).is_some_and(|slot| slot.link.is_some())) {
+ retired.extend(table.retire_bootstrap());
+ }
+ drop(table);
+ drop(retired);
+ Ok(())
+ }
+
+ /// The replica that the bootstrap link reaches. With one replica in the
+ /// roster, that replica: a single node advertises its bind address, and
+ /// that can be `0.0.0.0`.
+ async fn bootstrap_replica(&self, wanted: &BTreeMap<u8, String>) -> Option<(Arc<Link>, u8)> {
+ let (link, endpoint) = {
+ let table = self.table();
+ let bootstrap = table.bootstrap.as_ref()?;
+ (Arc::clone(&bootstrap.link.link), bootstrap.endpoint.clone())
+ };
+ if wanted.len() == 1 {
+ return wanted.keys().next().map(|id| (link, *id));
+ }
+ for (id, replica) in wanted {
+ if is_same_address(&endpoint, replica).await {
+ return Some((link, *id));
+ }
+ }
+ None
+ }
+
+ fn spawn_slot(
+ self: &Arc<Self>,
+ id: u8,
+ endpoint: String,
+ adopted: Option<Arc<Link>>,
+ ) -> AbortHandle {
+ tokio::spawn(Arc::clone(self).run_slot(id, endpoint, adopted)).abort_handle()
+ }
+
+ /// Starts every slot without a link again, at once and without its
+ /// backoff.
+ fn restart_down_slots(self: &Arc<Self>, table: &mut Table) {
+ for (id, slot) in table
+ .slots
+ .iter_mut()
+ .filter(|(_, slot)| slot.link.is_none())
+ {
+ if let Some(task) = slot.task.take() {
+ task.abort();
+ }
+ slot.task = Some(self.spawn_slot(*id, slot.endpoint.clone(), None));
+ }
+ }
+
+ /// Keeps one link to replica `id` up. Each retry waits its backoff, and a
+ /// link that stayed up for `reestablish_after` redials at once.
+ async fn run_slot(self: Arc<Self>, id: u8, endpoint: String, mut adopted: Option<Arc<Link>>) {
+ let mut signs_in_at = None;
+ let mut retry = 0_u32;
+ loop {
+ let link = match adopted.take() {
+ Some(link) => link,
+ None => {
+ if retry > 0 {
+ if !self.policy.allows(retry) {
+ break;
+ }
+ sleep(self.policy.pause(retry)).await;
+ }
+ match self.open(id, &endpoint, &mut signs_in_at).await {
+ Some(link) => link,
+ None => {
+ retry = retry.saturating_add(1);
+ continue;
+ }
+ }
+ }
+ };
+ let up_since = Instant::now();
+ self.hold(&link).await;
+ self.slot_down(id, &link);
+ if !self.policy.enabled {
+ break;
+ }
+ retry = if up_since.elapsed() >= self.policy.reestablish_after {
Review Comment:
Deferred to multi-leader PR5 connection/failover work. Apply jitter to the first redial after a live link drops and test simultaneous reconnects while preserving the operation deadline and bounded replica-link lifecycle.
##########
core/sdk/src/driver.rs:
##########
@@ -0,0 +1,1309 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The operation driver sends every request of a binary client. One deadline
+//! bounds the lane wait, the lookups, the attempts and the pauses of a call.
+//! A frame that can have taken effect is never frozen again, so a retry never
+//! applies a request twice.
+
+// Unused until the clients send through the driver.
+#![allow(dead_code)]
+
+use crate::links::{
+ Frame, Link, OPERATION_TIMEOUT, ReplicaLinks, Target, lost_link, pause_before_retry, read_reply,
+};
+use crate::session_owner::{LaneKey, LaneTicket, RetainedOperation, SessionOwner};
+use bytes::Bytes;
+use iggy_binary_protocol::codes::{
+ BIND_SESSION_CODE, GET_PARTITION_ROUTE_CODE, LOGIN_REGISTER_CODE, LOGIN_REGISTER_WITH_PAT_CODE,
+ SEND_MESSAGES_CODE,
+};
+use iggy_binary_protocol::consensus::{Operation, operation_for_code};
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
+use iggy_binary_protocol::requests::messages::GetPartitionRouteRequest;
+use iggy_binary_protocol::responses::messages::PartitionRouteResponse;
+use iggy_binary_protocol::responses::users::LoginRegisterResponse;
+use iggy_binary_protocol::{WireDecode, WireEncode};
+use iggy_common::{ClientState, IggyError, RequestTarget};
+use route_cache::RouteCache;
+use std::sync::{Arc, Mutex, MutexGuard, PoisonError};
+use tokio::time::{Instant, timeout_at};
+use tracing::debug;
+
+mod poll_router;
+mod route_cache;
+
+pub(crate) use poll_router::{
+ PollRouter, PollTransport, ROSTER_READ_TIMEOUT, is_poll_routing_code,
+};
+pub(crate) use route_cache::PartitionKey;
+
+/// Sends the requests of one client over its replica links.
+pub(crate) struct OperationDriver {
+ session: Arc<SessionOwner>,
+ links: ReplicaLinks,
+ routes: Mutex<RouteCache>,
+}
+
+impl OperationDriver {
+ /// The driver shares the session owner of `links`.
+ pub(crate) fn new(links: ReplicaLinks) -> Self {
+ Self {
+ session: Arc::clone(links.session()),
+ links,
+ routes: Mutex::default(),
+ }
+ }
+
+ /// Sends one request and returns the body of its reply.
+ pub(crate) async fn send(
+ &self,
+ code: u32,
+ payload: Bytes,
+ target: RequestTarget<'_>,
+ ) -> Result<Bytes, IggyError> {
+ let deadline = Instant::now() + OPERATION_TIMEOUT;
+ self.drive(Call::new(code, payload, target, deadline)).await
+ }
+
+ /// Replays the retained operation `op` until it has an outcome or
+ /// `deadline` passes. Its success frees the lane.
+ pub(crate) async fn resolve(&self, op: u64, deadline: Instant) -> Result<(), IggyError> {
+ let (ticket, retained) = self.session.resume(op, deadline).await?;
+ self.drive(Call::resumed(ticket, retained, deadline))
+ .await
+ .map(|_| ())
+ }
+
+ /// Frees the lane of the retained operation `op`. Its outcome stays
+ /// unknown.
+ pub(crate) fn abandon(&self, op: u64) -> Result<(), IggyError> {
+ self.session.abandon(op)
+ }
+
+ async fn drive(&self, mut call: Call) -> Result<Bytes, IggyError> {
+ loop {
+ let next = match self.attempt(&mut call).await {
+ Ok(body) => {
+ call.complete();
+ return Ok(body);
+ }
+ Err(next) => next,
+ };
+ call.settle_sent();
+ match next {
+ Next::Again if Instant::now() < call.deadline => {}
+ Next::Again => return Err(call.expire()),
+ Next::Pause => {
+ if !call.pause().await {
+ return Err(call.expire());
+ }
+ }
+ Next::Fail(error) => return Err(call.end(error)),
+ }
+ }
+ }
+
+ /// One attempt of `call`, from the choice of its replica to its reply.
+ async fn attempt(&self, call: &mut Call) -> Result<Bytes, Next> {
+ if let Some(error) = self.stopped() {
+ return Err(Next::Fail(error));
+ }
+ if call.frame.is_some() && call.fence() != Some(self.session.fence()) {
+ // A replaced session can never apply the frame. An uncertain call
+ // ends with its uncertainty, any other call freezes again.
+ if call.uncertain {
+ return Err(Next::Fail(IggyError::Unauthenticated));
+ }
+ call.frame = None;
+ }
+ if call.frame.is_none() {
+ self.wait_for_session(call).await?;
+ }
+ let route = self.route_for(call).await?;
+ let context = self.context(call, route.as_ref());
+ // The lane comes first, so a blocked lane ends the call at once, also
+ // while no link is ready.
+ let ticket = self.ticket(call, &context).await?;
+ let hop = match self.hop(call, route).await {
+ Ok(hop) => hop,
+ Err(next) => {
+ call.ticket = Some(ticket);
+ return Err(next);
+ }
+ };
+ let deadline = call.deadline;
+ let (frame, ticket) = call.frame(ticket, context)?;
+ let result = hop.link.send(frame, deadline, ticket.sent_flag()).await;
+ let sent = *ticket.sent_flag();
+ let result = result.and_then(|reply| read_reply(&self.session, reply));
+ self.settle(call, &hop, result, sent).await
+ }
+
+ /// The error of a call on a client that nothing brings back.
+ fn stopped(&self) -> Option<IggyError> {
+ match self.links.state() {
+ ClientState::Shutdown => Some(IggyError::ClientShutdown),
+ ClientState::Disconnected => Some(IggyError::NotConnected),
+ _ => None,
+ }
+ }
+
+ /// Waits for a bound session while the links sign in again. Without
+ /// credentials nothing signs in, so the call goes on unbound.
+ async fn wait_for_session(&self, call: &Call) -> Result<(), Next> {
+ if is_sign_in(call.code)
+ || self.session.epoch().bound
+ || self.session.sign_in_credentials().is_none()
+ {
+ return Ok(());
+ }
+ let mut epoch = self.session.subscribe_epoch();
+ match timeout_at(call.deadline, epoch.wait_for(|epoch| epoch.bound)).await {
+ Ok(Ok(_)) => Ok(()),
+ Ok(Err(_)) | Err(_) => Err(Next::Fail(IggyError::Unauthenticated)),
+ }
+ }
+
+ /// The route of a partition call, `None` for any other call. A partition
+ /// call looks up its route unless the cached one is usable.
+ async fn route_for(&self, call: &mut Call) -> Result<Option<PartitionRouteResponse>, Next> {
+ let Some(partition) = &call.partition else {
+ return Ok(None);
+ };
+ match self.route(&partition.key, call.deadline).await {
+ Ok(route) => Ok(Some(route)),
+ Err(error) if retries_lookup(&error) => {
+ call.refusal = error;
+ Err(Next::Pause)
+ }
+ Err(error) => Err(Next::Fail(error)),
+ }
+ }
+
+ /// The context that a new frame of `call` carries. Every attempt and
+ /// every replay keeps the context of the first frame.
+ fn context(&self, call: &Call, route: Option<&PartitionRouteResponse>) -> PartitionContext {
+ match (&call.partition, route) {
+ (Some(partition), Some(route)) => partition.context.unwrap_or_else(|| route.context()),
+ _ if operation_for_code(call.code) == Operation::NonReplicated => self.read_context(),
+ _ => PartitionContext::default(),
+ }
+ }
+
+ /// A read without a partition context waits until its replica applied
+ /// the metadata that the session saw.
+ fn read_context(&self) -> PartitionContext {
+ PartitionContext {
+ metadata_op: self.session.metadata_watermark(),
+ ..PartitionContext::default()
+ }
+ }
+
+ /// The lane ticket of `call`. The call enters its lane on its first
+ /// attempt and keeps the ticket until it ends.
+ async fn ticket(
+ &self,
+ call: &mut Call,
+ context: &PartitionContext,
+ ) -> Result<LaneTicket, Next> {
+ match call.ticket.take() {
+ Some(ticket) => Ok(ticket),
+ None => {
+ let key = LaneKey::of(call.code, context, call.partition_id());
+ self.session
+ .enter_lane(key, call.deadline)
+ .await
+ .map_err(Next::Fail)
+ }
+ }
+ }
+
+ /// The link of the next attempt: to the primary of the route, else to
+ /// the metadata primary.
+ async fn hop(
+ &self,
+ call: &mut Call,
+ route: Option<PartitionRouteResponse>,
+ ) -> Result<Hop, Next> {
+ let target = route.map_or(Target::MetadataPrimary, |route| {
+ Target::Replica(route.primary_replica)
+ });
+ let error = match self.links.link(target) {
+ Ok(link) => return Ok(Hop { link, route }),
+ Err(error) => error,
+ };
+ call.refusal = error;
+ match (&call.partition, route) {
+ (Some(partition), Some(route)) => {
+ self.routes()
+ .failed(&route, partition.key.partition_id, route.primary_replica);
+ }
+ _ => self.find_metadata_primary(call.deadline).await,
+ }
+ Err(Next::Pause)
+ }
+
+ /// Reads the outcome of one attempt. `sent` tells that this or an
+ /// earlier attempt can have taken effect.
+ async fn settle(
+ &self,
+ call: &mut Call,
+ hop: &Hop,
+ result: Result<Bytes, IggyError>,
+ sent: bool,
+ ) -> Result<Bytes, Next> {
+ let error = match result {
+ Ok(body) => return self.signed_in(call, hop, body),
+ Err(error) => error,
+ };
+ let repeated = std::mem::replace(
+ &mut call.not_primary,
+ matches!(error, IggyError::NotPrimary),
+ );
+ match error {
+ IggyError::TransientNotCommitted => {
+ call.uncertain = true;
+ Err(Next::Pause)
+ }
+ IggyError::TransientNotAccepted => {
+ call.refusal = error;
+ self.refresh(call, hop, Failure::Refused).await;
+ Err(Next::Pause)
+ }
+ IggyError::NotPrimary => {
+ // Neither was admitted, so the deadline reports both as 58.
+ call.refusal = IggyError::TransientNotAccepted;
+ self.refresh(call, hop, Failure::NotPrimary).await;
+ Err(if repeated { Next::Pause } else { Next::Again })
+ }
+ error if lost_link(&error) => {
+ call.uncertain |= sent;
+ call.refusal = error;
+ self.refresh(call, hop, Failure::LostLink).await;
+ Err(Next::Pause)
+ }
+ // The refusal spent the request id, so the call starts a new
+ // operation in the same lane.
+ IggyError::LifecycleBusy if !call.uncertain => {
+ call.refusal = error;
+ call.frame = None;
+ Err(Next::Pause)
+ }
+ IggyError::HistoryUnavailable => {
+ self.invalidate(call, hop);
+ if call.restarts_on_new_history() {
+ call.restart();
+ return Err(Next::Again);
+ }
+ Err(Next::Fail(error))
+ }
+ IggyError::ConsumerGroupPartitionNotOwned(..) | IggyError::PartitionNotFound(..) => {
+ self.invalidate(call, hop);
+ Err(Next::Fail(error))
+ }
+ error => Err(Next::Fail(error)),
Review Comment:
Tracked as a blocking prerequisite for multi-leader PR5 driver activation. The wire must distinguish a committed business rejection from a pre-consensus denial. Before activation, test an uncertain request replayed to a committed rejection: resolve must terminate and release its lane without treating an uncommitted denial as a settled outcome.
##########
core/sdk/src/session_owner/lanes.rs:
##########
@@ -0,0 +1,605 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A lane admits one mutation at a time, in arrival order. There is one lane
+//! for metadata and one for each partition history. A mutation whose outcome
+//! is unknown stays retained in its lane, and the lane admits nothing else
+//! until a resolve or an abandon frees it. So a retained operation is always
+//! the newest operation of its lane, and its replay can never age out.
+
+use super::SessionOwner;
+use crate::driver::PartitionKey;
+use crate::links::Frame;
+use crate::vsr::encode_request_header;
+use bytes::Bytes;
+use iggy_binary_protocol::consensus::{Operation, operation_for_code};
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
+use iggy_common::IggyError;
+use std::collections::HashMap;
+use std::sync::Arc;
+use tokio::sync::{Mutex, OwnedMutexGuard};
+use tokio::time::{Instant, timeout_at};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum LaneKey {
+ Metadata,
+ /// The incarnation names one history of the partition for every spelling
+ /// of its stream and topic.
+ Partition {
+ incarnation: u64,
+ partition_id: u32,
+ },
+}
+
+impl LaneKey {
+ /// The lane of `code`. A write to a partition takes the lane of the
+ /// incarnation that its context carries. Reads, Register and Logout take
+ /// no lane.
Review Comment:
PR4 documents the auto-commit exemption in e247f1ef3 and fixes server-side consumer-progress ordering in 5a2e85e54. Client-side lane semantics for auto-commit polls are deferred to multi-leader PR5 driver activation, with poll/store/delete ordering and cancellation tests required.
##########
core/sdk/src/session_owner/lanes.rs:
##########
@@ -0,0 +1,605 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A lane admits one mutation at a time, in arrival order. There is one lane
+//! for metadata and one for each partition history. A mutation whose outcome
+//! is unknown stays retained in its lane, and the lane admits nothing else
+//! until a resolve or an abandon frees it. So a retained operation is always
+//! the newest operation of its lane, and its replay can never age out.
+
+use super::SessionOwner;
+use crate::driver::PartitionKey;
+use crate::links::Frame;
+use crate::vsr::encode_request_header;
+use bytes::Bytes;
+use iggy_binary_protocol::consensus::{Operation, operation_for_code};
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
+use iggy_common::IggyError;
+use std::collections::HashMap;
+use std::sync::Arc;
+use tokio::sync::{Mutex, OwnedMutexGuard};
+use tokio::time::{Instant, timeout_at};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum LaneKey {
+ Metadata,
+ /// The incarnation names one history of the partition for every spelling
+ /// of its stream and topic.
+ Partition {
+ incarnation: u64,
+ partition_id: u32,
+ },
+}
+
+impl LaneKey {
+ /// The lane of `code`. A write to a partition takes the lane of the
+ /// incarnation that its context carries. Reads, Register and Logout take
+ /// no lane.
+ pub(crate) fn of(code: u32, context: &PartitionContext, partition_id: u32) -> Option<Self> {
+ match operation_for_code(code) {
+ Operation::NonReplicated | Operation::Register | Operation::Logout => None,
+ operation if operation.is_partition() => Some(Self::Partition {
+ incarnation: context.incarnation,
+ partition_id,
+ }),
+ _ => Some(Self::Metadata),
+ }
+ }
+}
+
+pub(super) type Lanes = HashMap<LaneKey, Lane>;
+
+#[derive(Debug, Default)]
+pub(super) struct Lane {
+ gate: Arc<Mutex<()>>,
+ /// The tickets that wait for the gate, hold it or resume. The lane leaves
+ /// the table when none is left and nothing is retained.
+ tickets: usize,
+ retained: Option<RetainedOperation>,
+}
+
+impl Lane {
+ fn holds(&self, op: u64) -> bool {
+ self.retained
+ .as_ref()
+ .is_some_and(|retained| retained.op() == op)
+ }
+}
+
+/// A frozen mutation. Every attempt and every replay sends the same frame.
+#[derive(Debug, Clone)]
+pub(crate) struct RetainedOperation {
+ pub(crate) code: u32,
+ pub(crate) frame: Frame,
+ /// The partition of a partition write, `None` for any other request. A
+ /// replay looks up its route by this key.
+ pub(crate) partition: Option<PartitionKey>,
+}
+
+impl RetainedOperation {
+ pub(crate) fn op(&self) -> u64 {
+ self.frame.header.request
+ }
+}
+
+/// The right of one call to send in its lane. Dropping the ticket frees the
+/// lane, except that a frame which left the client without a definitive
+/// outcome stays retained.
+#[derive(Debug)]
+pub(crate) struct LaneTicket {
+ owner: Arc<SessionOwner>,
+ /// `None` for a call that takes no lane.
+ key: Option<LaneKey>,
+ guard: Option<OwnedMutexGuard<()>>,
+ operation: Option<RetainedOperation>,
+ fence: u64,
+ /// A resume replays an operation that its lane still retains.
+ resumed: bool,
+ sent: bool,
+ done: bool,
+}
+
+impl SessionOwner {
+ /// Waits for the lane of `key` in arrival order. A lane that retains an
+ /// operation fails at once with `LaneBlocked`, and a lane still busy at
+ /// `deadline` fails with `TransientNotAccepted`. Neither failure takes a
+ /// request id.
+ pub(crate) async fn enter_lane(
+ self: &Arc<Self>,
+ key: Option<LaneKey>,
+ deadline: Instant,
+ ) -> Result<LaneTicket, IggyError> {
+ let Some(key) = key else {
+ return Ok(LaneTicket::new(Arc::clone(self), None, false));
+ };
+ let (mut ticket, gate) = {
+ let mut lanes = self.lanes();
+ let lane = lanes.entry(key).or_default();
Review Comment:
Deferred to multi-leader PR6 after driver activation. Profile uncontended lane allocations and route-cache insertion at capacity before choosing a change; retain bounded cache growth and correct lane cleanup on cancellation.
##########
core/sdk/src/driver/route_cache.rs:
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The routes of one client. A route has no age limit. It goes stale after
+//! a refusal or a failure, and a stale route keeps its view as the lowest
+//! view that its next lookup accepts. So a lookup never goes back to an
+//! older primary of the same partition history.
+
+use iggy_binary_protocol::requests::messages::GetPartitionRouteRequest;
+use iggy_binary_protocol::responses::messages::PartitionRouteResponse;
+use iggy_common::wire_conversions::identifier_to_wire;
+use iggy_common::{Identifier, IggyError, PartitionTarget};
+use std::collections::HashMap;
+
+/// The most routes that one client keeps. A new route beyond it evicts the
+/// oldest lookup.
+pub(crate) const MAX_CACHED_ROUTES: usize = 4096;
+
+/// A partition as its caller spelled it. A stream or topic named by number
+/// and the same one named by name are two keys.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct PartitionKey {
+ pub(crate) stream_id: Identifier,
+ pub(crate) topic_id: Identifier,
+ pub(crate) partition_id: u32,
+}
+
+impl From<&PartitionTarget<'_>> for PartitionKey {
+ fn from(target: &PartitionTarget<'_>) -> Self {
+ Self {
+ stream_id: target.stream_id.clone(),
+ topic_id: target.topic_id.clone(),
+ partition_id: target.partition_id,
+ }
+ }
+}
+
+impl TryFrom<&PartitionKey> for GetPartitionRouteRequest {
+ type Error = IggyError;
+
+ fn try_from(key: &PartitionKey) -> Result<Self, Self::Error> {
+ Ok(Self {
+ stream_id: identifier_to_wire(&key.stream_id)?,
+ topic_id: identifier_to_wire(&key.topic_id)?,
+ partition_id: key.partition_id,
+ })
+ }
+}
+
+#[derive(Debug)]
+struct Entry {
+ route: PartitionRouteResponse,
+ /// The session fence at the lookup. A route serves only that session.
+ fence: u64,
+ stale: bool,
+ /// The lowest view that the next lookup accepts.
+ floor: u32,
+ /// The replica that the next lookup asks first. The ids after it follow.
+ ask_first: Option<u8>,
+ /// Orders the entries for eviction.
+ lookup: u64,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct RouteCache {
+ entries: HashMap<PartitionKey, Entry>,
+ lookups: u64,
+}
+
+impl RouteCache {
+ /// The route of `key`, unless it is stale, from another session, or
+ /// older than the metadata `watermark` that the session saw.
+ pub(crate) fn usable(
+ &self,
+ key: &PartitionKey,
+ fence: u64,
+ watermark: u64,
+ ) -> Option<PartitionRouteResponse> {
+ self.entries
Review Comment:
Deferred to multi-leader PR6. The global watermark remains the safety fence in PR4. Topic-specific invalidation requires trustworthy metadata-change provenance and tests for deletion/recreation and missed updates before it can replace that fence.
##########
core/shard/src/lib.rs:
##########
@@ -4511,17 +4581,38 @@ where
}
}
+ fn partition_incarnation_matches(
+ &self,
+ partition: &IggyPartition<B, SB>,
+ partition_incarnation: u64,
+ frame: &'static str,
+ ) -> bool {
+ if partition_incarnation == partition.created_revision() {
+ return true;
+ }
+ tracing::debug!(
Review Comment:
Deferred to multi-leader PR6 operational diagnostics: add rate-limited incarnation-mismatch visibility, including incarnation zero, with bounded metric cardinality and log volume.
##########
core/consensus/src/plane_helpers.rs:
##########
@@ -903,6 +903,62 @@ where
.await
}
+/// Find a held entry below a view's suffix that the new history does not prove.
+///
+/// `canonical` is ordered by descending op. Entries above the local commit floor
+/// survive only if their parent chain reaches its lowest header, or a prepare's
+/// commit stamp proves them through the uninterrupted local chain. This runs once
+/// per installed log, before repair starts filling any resulting gap.
+#[must_use]
+pub fn first_unproven_op(
+ commit_floor: u64,
+ canonical: &[PrepareHeader],
+ op_head: u64,
+ last_op: Option<u64>,
+ header_at: impl Fn(u64) -> Option<PrepareHeader>,
+) -> Option<u64> {
+ let last_op = last_op?;
+ let proven = locally_proven_commit(commit_floor, &header_at);
+ let (mut op, mut parent) = canonical.last().map_or((op_head, None), |lowest| {
+ (lowest.op.saturating_sub(1), Some(lowest.parent))
+ });
+ while op > proven {
+ match header_at(op) {
+ Some(local)
+ if local.checksum != CHECKSUM_UNSEALED && Some(local.checksum) == parent =>
+ {
+ parent = Some(local.parent);
+ op -= 1;
+ }
+ _ => break,
+ }
+ }
+ (proven.checked_add(1)?..=op.min(last_op)).find(|&held| header_at(held).is_some())
Review Comment:
Deferred to multi-leader PR6. Measure an ordered resident-index range scan during lagging-replica view installation, preserving the existing gap and repair-anchor checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4263189782
##########
core/server/src/dispatch/mod.rs:
##########
@@ -557,25 +769,51 @@ async fn handle_client_request<B, MJ, S, SB>(
// (`submit_register_local_or_forward`). Every other non-replicated
// code MUST go through Register first, which binds the acting user
// the per-op authz gates resolve.
- handle_non_replicated_request(
- shard,
- sessions,
- server_config,
- transport_client_id,
- request,
- ConnectionContext {
- bound,
- user_id,
- address: client_address,
- metadata_watermark,
- },
- )
- .await;
+ let context = ConnectionContext {
+ bound,
+ user_id,
+ address: client_address,
+ metadata_watermark,
+ };
+ // `ClientDrain` says which frames stay on the drain.
+ if matches!(non_replicated_code(header), PING_CODE | BIND_SESSION_CODE) {
+ handle_non_replicated_request(
+ shard,
+ sessions,
+ server_config,
+ transport_client_id,
+ request,
+ context,
+ )
+ .await;
+ return;
+ }
+ let in_flight = match progress {
+ Some(progress) => progress,
+ None => drain.admit(Admission::Cap).await,
+ };
+ let task_shard = Rc::clone(shard);
+ let task_sessions = Rc::clone(sessions);
+ let task_config = Arc::clone(server_config);
+ shard.bus.spawn(async move {
Review Comment:
**warning**: re the PR6 deferral: this still spawns a task for every non-PING read where master runs it inline, which measured -17% poll and -48% consumer under concurrent produce for every lockstep SDK. Merging ships that via the edge builds.
The new progress lane already makes an inline poll order-safe, and the doc at :259-266 already accepts PING waiting behind a bounded frame. An inline-only `POLL_MESSAGES*` fast path recovered 97% in the earlier measurement. Please land it here rather than in PR6.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -377,6 +379,71 @@ async fn relay_partition_reply<B, MJ, S, SB>(
});
}
+/// The offset key shared by polls and explicit progress writes on a connection.
+/// Failed resolution is left to the normal handler's authorization and error path.
+pub(super) fn progress_key<B, MJ, S, SB>(
+ shard: &Rc<ShellShard<B, MJ, S, SB>>,
+ message: &Message<GenericHeader>,
+) -> Option<ConsumerProgress>
+where
+ B: ShellBus,
+ MJ: JournalHandle + 'static,
+ MJ::Target: Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>,
+ S: 'static,
+ SB: SuperblockStore + 'static,
+{
+ let request = message.try_as_typed::<RequestHeader>().ok()?;
+ let wire = match request.header().operation {
+ Operation::NonReplicated
+ if request.header().reserved[..4] == GET_CONSUMER_OFFSET_CODE.to_le_bytes() =>
+ {
+ GetConsumerOffsetRequest::decode_from(request.body()).ok()?
+ }
+ Operation::NonReplicated
Review Comment:
nit: the lane key covers every poll and `GetConsumerOffset`, so Offset/Timestamp polls without auto_commit also serialize behind a store.
Only polls that read or commit the stored offset (`Next` or `auto_commit`) need it. Keep store<->store serialized, though: a transient-refused store retried after a later accepted one would move the offset backwards.
##########
core/server/src/dispatch/mod.rs:
##########
@@ -240,13 +252,149 @@ impl Drop for ActiveDrainSlot {
}
}
+/// The state that the drain task of one connection shares with the requests
+/// it spawned.
+///
+/// The drain takes the frames of the connection in arrival order and runs the
+/// prologue of each. Login, logout, `BIND_SESSION` and partition admission
+/// finish on the drain. Metadata commits run in tasks, but later non-PING
+/// frames wait in the queue until the metadata lane releases, before they
+/// capture session state or resolve a namespace. PING can pass that queue.
Review Comment:
nit: the doc says later frames wait "before they ... resolve a namespace", but the Partition arm drops `context.metadata_watermark` (:864-900) and resolves from the local STM. On a backup-homed or peer-shard connection, a pipelined JoinConsumerGroup -> StoreConsumerOffset can get `ConsumerGroupPartitionNotOwned` (namespace.rs:278-283).
Either hold the partition frame on the session watermark or narrow the doc.
##########
core/server/src/dispatch/mod.rs:
##########
@@ -283,13 +449,53 @@ fn pop_next_client_request(
// process exit.
let mut queues = queues.borrow_mut();
let queue = queues.get_mut(&client_id)?;
- let message = queue.pop_front();
+ let mut waiting = false;
+ let mut barrier = false;
+ let index = queue.iter().position(|message| {
Review Comment:
nit: every wake re-runs `progress_key` (body decode, `WireName` allocs, STM resolve) for each queued keyed frame, so N pipelined polls on one key cost O(N²) up to the 1024-frame queue. Together with the handler's own pass a poll now decodes its key 3 times.
Compute the key once at enqueue and store it with the frame.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -9919,6 +10175,8 @@ where
self.publish_consumer_offset_effect(*effect);
self.apply_consumer_offset_commit(effect.pending);
}
+ self.durable_consumer_offsets
+ .set_emptied_at(pending.emptied_at);
Review Comment:
nit (repeat, no reply): `emptied_at` is still written back unconditionally from the `select_commit` snapshot. A no-ack store that publishes in between gets overwritten with the stale value. That's safe (it only refuses more), but memory, disk and the offer diverge.
Write it only when this run's plan set `emptied_at`.
##########
core/server/src/http/forward.rs:
##########
@@ -42,24 +42,32 @@
//! - The primary re-authenticates and re-authorizes the forwarded request
//! through its ordinary extractor stack; the forward marker header is a loop
//! guard only and never a trust input.
-//! - A forwarded request is retried only when it provably never entered the
-//! primary's pipeline: a connect-phase failure, a 503 whose body carries the
-//! `TransientNotAccepted` code, or a 307 from a stale target. Every other
-//! outcome - including a 503 carrying `TransientNotCommitted`, whose op may
-//! still commit - is relayed as-is, because re-issuing it under a fresh
+//! - The route header, like the marker, carries only values a caller can send
+//! itself, so the hop target authenticates and authorizes the request again.
+//! - A forwarded write is retried only when it provably never entered the
+//! primary's pipeline: a connect-phase failure, and for a metadata relay a
+//! 503 whose body carries the `TransientNotAccepted` or `NotPrimary` code or
+//! a 307 from a stale target, for a partition relay a 503 carrying
+//! `NotPrimary`. Every other outcome - including a 503 carrying
+//! `TransientNotCommitted`, whose op
Review Comment:
nit (partial): the reflow still leaves a short line, "`TransientNotCommitted`, whose op". Please rewrap the paragraph.
##########
core/server/src/http/handlers.rs:
##########
@@ -1256,25 +1253,27 @@ pub(in crate::http) async fn delete_segments(
/// [`poll_wire_request`]) since HTTP has no persistent connection and cannot observe
/// group's partition assignment (that can be subject to change).
///
-/// A non-replicated read served in band: the same resolution the TCP dispatch
-/// runs ([`resolve_poll_request`]), then a mesh read on the owning shard and a
-/// re-encode of the stored batches into the legacy wire body, decoded here by
-/// the SDK's own [`PolledMessages::from_bytes`] so the JSON is field-identical
-/// to a TCP poll. `auto_commit` rides [`resolve_poll_request`]'s args and is
-/// honored by the owning shard's poll plan; no extra HTTP work.
+/// A non-replicated read served by the partition primary: the same resolution
+/// the TCP dispatch runs ([`resolve_poll_request`]), then a mesh read on the
+/// owning shard and a re-encode of the stored batches into the legacy wire
+/// body, decoded here by the SDK's own [`PolledMessages::from_bytes`] so the
+/// JSON is field-identical to a TCP poll. `auto_commit` rides
+/// [`resolve_poll_request`]'s args and is honored by the owning shard's poll
+/// plan. Another node relays the poll to the primary ([`route_partition`]).
pub(in crate::http) async fn poll_messages(
State(state): State<HttpState>,
identity: Identity,
Path((stream_id, topic_id)): Path<(String, String)>,
Query(query): Query<PollMessages>,
Query(consistency): Query<ConsistencyQuery>,
-) -> Result<Json<PolledMessages>, ReadError> {
+ Extension(request): Extension<Arc<PartitionRequest>>,
+) -> Result<Response, ReadError> {
let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
let topic_id = Identifier::from_str_value(&topic_id).map_err(ReadError::Rejected)?;
SendWrapper::new(gate_local_read(
&state,
&identity,
- consistency.consistency,
+ partition_read_consistency(&request, consistency.consistency),
Review Comment:
nit (repeat, no reply): with `?consistency=linearizable`, a node that is neither the metadata nor the partition primary first 307s to the metadata primary, which then relays to the partition primary. That is two hops where the partition relay alone would do.
On `PartitionHop::Contacted`, the relay to the partition primary should satisfy linearizable without the 307.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -345,6 +346,7 @@ async fn relay_partition_reply<B, MJ, S, SB>(
// Through the bus, not the runtime directly: the simulator supplies its
// own executor and virtual clock.
shard.bus.clone().spawn(async move {
+ let _progress = progress;
Review Comment:
nit: when `await_partition_submit` expires (30 s) the lane is released here, but the store can still commit later and overwrite a newer auto-commit for the same consumer. The outcome is already unknown to the client, so a doc line on the lane contract is enough.
##########
core/simulator/src/workload/state_checker.rs:
##########
@@ -265,6 +381,53 @@ impl StateChecker {
}
}
+/// Membership is metadata state; the partition's ordered install barrier is
+/// what fences a departed member's later offset writes.
+fn check_partition_owner(
+ owners: &mut BTreeMap<u64, ConsumerGroupOwner>,
+ header: &PrepareHeader,
+ body: &[u8],
+ seed: u64,
+) {
+ if header.operation == Operation::InstallConsumerGroupOwner {
+ let install = InstallConsumerGroupOwnerRequest::decode_from(body)
+ .expect("a committed owner installation decodes");
+ owners.insert(install.group_id, install.owner);
+ return;
+ }
+ if is_auto_commit_client(header.client) {
+ return;
+ }
+ let (request, generation) =
+ split_prepared_offset(body).expect("a committed offset has its owner token");
+ let (request, _) =
+ GetConsumerOffsetRequest::decode(request).expect("a committed offset prefix decodes");
+ if request.consumer.kind != KIND_CONSUMER_GROUP {
+ return;
+ }
+ let group_id = u64::from(
+ request
+ .consumer
+ .id
+ .as_u32()
+ .expect("a prepared group id is numeric"),
+ );
+ // The repair ring may have evicted the install before this checker started.
+ if let Some(owner) = owners.get(&group_id) {
Review Comment:
nit: this check never fires in fuzz. Two of us mutated `matches_consumer_owner` to always accept, and separately to skip only the generation compare or only the session compare. 24 fuzz seeds produced logs byte-identical to base, and all 228 sim tests still pass with `owners_compared=0`. The workload stores from a random consumer id (`ops/store_consumer_offset.rs:64`) and never from a stale owner.
Add a stale-owner store action plus a counter of owner-fence refusals that must be > 0. The PR body claim "Simulator checks cover replacement histories" should be softened until then.
##########
core/message_bus/src/transports/quic.rs:
##########
@@ -326,12 +316,205 @@ fn reply_route(frame: &BusMessage) -> ReplyRoute {
if !is_reply {
return ReplyRoute::Unkeyed;
}
+ read_u64(bytes, offset_of!(ReplyHeader, request)).map_or(ReplyRoute::Unkeyed, ReplyRoute::Keyed)
+}
+
+/// The request id of a client frame, read as raw bytes like [`reply_route`],
+/// so a frame with an undeclared operation still has one.
+fn request_id(request: &Message<GenericHeader>) -> Option<u64> {
+ read_u64(request.as_slice(), offset_of!(RequestHeader, request))
+}
+
+fn read_u64(bytes: &[u8], offset: usize) -> Option<u64> {
bytes
- .get(..HEADER_SIZE)
- .and_then(|header| bytemuck::checked::try_from_bytes::<ReplyHeader>(header).ok())
- .map_or(ReplyRoute::Unkeyed, |header| {
- ReplyRoute::Keyed(header.request)
- })
+ .get(offset..offset + size_of::<u64>())
+ .and_then(|field| field.try_into().ok())
+ .map(u64::from_le_bytes)
+}
+
+/// Hands the outbound frames of one connection to the streams that wait
+/// for them.
+///
+/// A stream registers its waiter before it forwards its request, so a reply
+/// never comes before the stream it answers.
+struct ReplyRouter {
+ /// Waiting streams by request id, oldest first.
+ waiters: HashMap<u64, VecDeque<oneshot::Sender<BusMessage>>>,
+ /// Frames that came while no stream waited for them, oldest first.
+ unclaimed: VecDeque<BusMessage>,
+ /// When `unclaimed` holds this many frames, it drops its oldest one.
+ unclaimed_max: usize,
+}
+
+impl ReplyRouter {
+ fn new(unclaimed_max: usize) -> Self {
+ Self {
+ waiters: HashMap::new(),
+ unclaimed: VecDeque::new(),
+ unclaimed_max,
+ }
+ }
+
+ /// Register a stream that waits for the reply to `request_id`.
+ ///
+ /// The stream first takes the oldest unclaimed frame that has no request
+ /// id or has its own. The SDK replays an explicit transient with the same
+ /// request id on a new stream, so a late reply of the first attempt
+ /// answers the replay. The replies for other ids ahead of that frame are
+ /// dropped, because their streams already ended.
+ fn register(&mut self, request_id: u64) -> oneshot::Receiver<BusMessage> {
+ let (waiter, reply) = oneshot::channel();
+ while let Some(frame) = self.unclaimed.pop_front() {
+ if !matches!(reply_route(&frame), ReplyRoute::Keyed(id) if id != request_id) {
+ // `reply` is alive, so the send cannot fail.
+ let _ = waiter.send(frame);
+ return reply;
+ }
+ }
+ self.waiters
+ .entry(request_id)
+ .or_default()
+ .push_back(waiter);
+ reply
+ }
+
+ /// Hand a reply to the oldest waiter of its request id, and a frame
+ /// without one to every waiter, because an eviction ends every request
+ /// of the connection. A frame no stream takes stays unclaimed.
+ fn route(&mut self, frame: BusMessage) {
+ let unclaimed = match reply_route(&frame) {
+ ReplyRoute::Keyed(request_id) => self.deliver(request_id, frame),
+ ReplyRoute::Unkeyed => self.broadcast(frame),
+ };
+ if let Some(frame) = unclaimed {
+ if self.unclaimed.len() == self.unclaimed_max {
+ self.unclaimed.pop_front();
+ }
+ self.unclaimed.push_back(frame);
+ }
+ }
+
+ /// Drop the waiters of `request_id` whose streams ended without a reply.
+ fn release(&mut self, request_id: u64) {
+ if let Some(waiters) = self.waiters.get_mut(&request_id) {
+ waiters.retain(|waiter| !waiter.is_canceled());
+ if waiters.is_empty() {
+ self.waiters.remove(&request_id);
+ }
+ }
+ }
+
+ /// Returns the frame if no waiter took it.
+ fn deliver(&mut self, request_id: u64, mut frame: BusMessage) -> Option<BusMessage> {
+ let Some(waiters) = self.waiters.get_mut(&request_id) else {
+ return Some(frame);
+ };
+ let unclaimed = loop {
+ let Some(waiter) = waiters.pop_front() else {
+ break Some(frame);
+ };
+ match waiter.send(frame) {
+ Ok(()) => break None,
+ Err(returned) => frame = returned,
+ }
+ };
+ if waiters.is_empty() {
+ self.waiters.remove(&request_id);
+ }
+ unclaimed
+ }
+
+ /// Returns the frame if no waiter took it.
+ fn broadcast(&mut self, frame: BusMessage) -> Option<BusMessage> {
+ let mut delivered = false;
+ for waiter in self.waiters.drain().flat_map(|(_, waiters)| waiters) {
+ delivered |= waiter.send(frame.clone()).is_ok();
+ }
+ (!delivered).then_some(frame)
+ }
+}
+
+/// What the stream futures of [`QuicTransportConn::run`] borrow from it.
+struct StreamContext<'a> {
+ in_tx: &'a Sender<Message<GenericHeader>>,
+ router: &'a RefCell<ReplyRouter>,
+ max_message_size: usize,
+ reply_backstop: Duration,
+ label: &'static str,
+ peer: &'a str,
+}
+
+impl StreamContext<'_> {
+ /// Serve one request stream. Breaks when the inbound queue is gone,
+ /// because the connection then has no dispatch.
+ #[allow(clippy::future_not_send)]
+ async fn serve(&self, mut send: SendStream, mut recv: RecvStream) -> ControlFlow<()> {
+ let Self {
+ in_tx,
+ router,
+ max_message_size,
+ reply_backstop,
+ label,
+ peer,
+ } = *self;
+ // The SDK `finish()`-es after its request, so the stream carries one
+ // frame.
+ let request = match framing::read_message(&mut recv, max_message_size).await {
+ Ok(request) => request,
+ Err(e) => {
+ debug!(%label, %peer, error = ?e, "quic: bidi read error");
+ return ControlFlow::Continue(());
+ }
+ };
+ let Some(request_id) = request_id(&request) else {
+ debug!(%label, %peer, "quic: frame shorter than a request header");
+ return ControlFlow::Continue(());
+ };
+ let mut reply = router.borrow_mut().register(request_id);
+ if in_tx.send(request).await.is_err() {
+ debug!(%label, %peer, "quic: inbound queue dropped");
+ return ControlFlow::Break(());
+ }
+ let replied = {
+ let mut stopped = pin!(send.stopped().fuse());
+ let mut backstop = pin!(compio::time::sleep(reply_backstop).fuse());
+ futures::select_biased! {
+ frame = reply => frame.ok(),
+ _ = stopped => {
+ debug!(%label, %peer, request_id, "quic: client stopped the stream before its reply");
+ None
+ }
+ () = backstop => {
+ warn!(%label, %peer, request_id, "quic: no reply within the backstop; resetting the stream");
+ None
+ }
+ }
+ };
+ let Some(frame) = replied else {
+ drop(reply);
+ router.borrow_mut().release(request_id);
+ if let Err(e) = send.reset(VarInt::from_u32(QUIC_NO_REPLY)) {
+ debug!(%label, %peer, error = ?e, "quic: stream reset failed");
+ }
+ return ControlFlow::Continue(());
+ };
+ for fragment in frame.into_fragments() {
+ let BufResult(result, _frozen) = send.write_all(fragment).await;
+ if let Err(e) = result {
+ debug!(%label, %peer, error = ?e, "quic: write failed");
+ return ControlFlow::Continue(());
+ }
+ }
+ // `finish()` flushes pending data + a FIN, which tells the SDK the
+ // reply is complete.
+ if let Err(e) = send.finish() {
+ debug!(%label, %peer, error = ?e, "quic: send.finish() failed");
+ }
+ // Keep the reply alive through graceful shutdown until the peer has
+ // acknowledged it. Closing the connection after FIN alone can discard it.
+ let _ = send.stopped().await;
Review Comment:
nit: `send.stopped().await` now runs on every stream in steady state, costing a per-request insert into the connection's waiter map and keeping the serve future alive for up to max_ack_delay. Only shutdown needs it.
Await `stopped()` only once shutdown has been observed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4270521778
##########
core/server/src/http/forward.rs:
##########
@@ -332,92 +534,69 @@ fn record_relayed_floor(state: &HttpInner, user_id: u32, response: &Response) {
state.metadata_watermarks.record(user_id, applied);
}
-async fn forward_partition_or_pass(state: HttpState, request: Request, next: Next) -> Response {
- if !state.forward.active || request.headers().contains_key(FORWARDED_HEADER) {
- return next.run(request).await;
- }
- // A read that does not move consumer progress is answered by whichever
- // replica the caller reached, which is the point of reading from one. Only
- // an automatic commit needs a replica that can originate the offset
- // operation, so only that poll pays for the buffering and the credential
- // resolution below.
- if request.method() == Method::GET && !wants_auto_commit(request.uri().query()) {
- return next.run(request).await;
- }
+async fn attach_partition_request(state: HttpState, request: Request, next: Next) -> Response {
+ let received = Instant::now();
+ let hop = match partition_hop(request.headers()) {
+ Ok(hop) => hop,
+ Err(error) => return CustomError::from(error).into_response(),
+ };
+ // Verify-only, like the metadata relay: an unauthenticated caller gets no
+ // buffered body and no relay. The handler authenticates again.
let bearer = match bearer_token(request.headers()) {
Ok(bearer) => bearer,
Err(error) => return CustomError::from(error).into_response(),
};
if let Err(rejection) = resolve_credential(&state, bearer).await {
return rejection.into_response();
}
-
- let (parts, request_body) = request.into_parts();
+ let _guard = if matches!(hop, PartitionHop::Target(_)) {
+ let Some(guard) = ForwardGuard::admit(&state.forward.in_flight) else {
Review Comment:
Deferred to PR6 under B7's HTTP target-hold isolation and relay-provenance gate. The docs now explicitly include the untimed body-read path: 128 authenticated slow-body uploads must not exhaust outbound forwarding capacity. Acceptance requires bounded body-read time/resources, isolated or reserved target capacity, authorized relay progress while the uploads are stalled, and cancellation/permit recovery. Moving the permit after to_bytes alone would leave body-read resource use unproven, so this remains open alongside the earlier admission-isolation finding.
##########
core/metadata/src/impls/metadata.rs:
##########
@@ -4231,6 +4224,9 @@ where
#[allow(clippy::future_not_send)]
pub async fn commit_journal(&self) {
let consensus = self.consensus.as_ref().unwrap();
+ if consensus.view_log_unproven() {
Review Comment:
Deferred to PR6 as a recovery release blocker in B5's durable metadata history certification gate. The metadata superblock can record the adopted log_view while proof remains fenced, and the fence is not recovered. Simply postponing the whole superblock write would violate the mandatory view-persistence-before-send gate. PR6 must separate the certified history view from the adopted view, bind it to verified durable WAL history, and cover checkpoint writes. Acceptance includes crash/reopen before repair, during divergence removal/refill, after proof and during checkpoint publication, plus conflicting-suffix quorum recovery without advertising unproven history under the new log_view or stalling the merge.
##########
core/simulator/src/bin/workload-fuzz.rs:
##########
@@ -625,6 +653,23 @@ fn run_quiesce_phase(
args.min_metadata_ops_compared, convergence.ops_compared,
));
}
+ if sim.is_shell()
+ && workload.options.weights.draws_fixture_groups()
+ && convergence.owners_compared < args.min_owners_compared
+ {
+ vacuous(format_args!(
+ "--min-owners-compared {}: the run joined fixture groups, but the owner oracle \
+ compared only {} settled owner(s) (seed={seed:#x})",
+ args.min_owners_compared, convergence.owners_compared,
+ ));
+ }
+ if sim.is_shell() && args.plane == Plane::Lifecycle && workload.owner_fence_refusals() == 0 {
Review Comment:
Deferred to PR6 in the consolidated simulator acceptance plan and B6. Extend the existing lifecycle workload scheduler to force a valid owner transition and at least one stale-owner store per run, then retain the random fault workload. Acceptance includes seeds 1, 2, 5, 8, 18, 21 and 23, nonzero stale-store attempts and generation refusals, successful valid-owner stores, and a production owner-check mutation caught end to end. The existing exit 3 for vacuous coverage stays; such runs will not be relabeled as passing or silently retried. The reported lifecycle seed 20037 was rerun with its supplied flags after these fixes: it converges without the reversed-range panic, then correctly exits 3 with stale_owner_stores=0 and owner_fence_refusals=0. That result is recorded as vacuous, not a passing lifecycle gate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4263738636
##########
core/server/src/dispatch/mod.rs:
##########
@@ -557,25 +769,51 @@ async fn handle_client_request<B, MJ, S, SB>(
// (`submit_register_local_or_forward`). Every other non-replicated
// code MUST go through Register first, which binds the acting user
// the per-op authz gates resolve.
- handle_non_replicated_request(
- shard,
- sessions,
- server_config,
- transport_client_id,
- request,
- ConnectionContext {
- bound,
- user_id,
- address: client_address,
- metadata_watermark,
- },
- )
- .await;
+ let context = ConnectionContext {
+ bound,
+ user_id,
+ address: client_address,
+ metadata_watermark,
+ };
+ // `ClientDrain` says which frames stay on the drain.
+ if matches!(non_replicated_code(header), PING_CODE | BIND_SESSION_CODE) {
+ handle_non_replicated_request(
+ shard,
+ sessions,
+ server_config,
+ transport_client_id,
+ request,
+ context,
+ )
+ .await;
+ return;
+ }
+ let in_flight = match progress {
+ Some(progress) => progress,
+ None => drain.admit(Admission::Cap).await,
+ };
+ let task_shard = Rc::clone(shard);
+ let task_sessions = Rc::clone(sessions);
+ let task_config = Arc::clone(server_config);
+ shard.bus.spawn(async move {
Review Comment:
Deferred to PR6's dispatch performance work. A poll still awaits its partition owner and possibly disk I/O; running the whole poll inline holds the connection drain during that wait. The held-owner regression covers both poll codes and requires PING to complete before the poll reply. The blanket-inline negative control fails this test: no PING reply arrives while the owner reply is held. PR6 must remove scheduling overhead while preserving that progress, in-flight bounds, cancellation/unknown-outcome behavior and Next/auto-commit ordering, then measure lockstep and pipelined workloads against a reproducible baseline. I have not independently reproduced the reported benchmark percentages.
##########
core/message_bus/src/transports/quic.rs:
##########
@@ -326,12 +316,205 @@ fn reply_route(frame: &BusMessage) -> ReplyRoute {
if !is_reply {
return ReplyRoute::Unkeyed;
}
+ read_u64(bytes, offset_of!(ReplyHeader, request)).map_or(ReplyRoute::Unkeyed, ReplyRoute::Keyed)
+}
+
+/// The request id of a client frame, read as raw bytes like [`reply_route`],
+/// so a frame with an undeclared operation still has one.
+fn request_id(request: &Message<GenericHeader>) -> Option<u64> {
+ read_u64(request.as_slice(), offset_of!(RequestHeader, request))
+}
+
+fn read_u64(bytes: &[u8], offset: usize) -> Option<u64> {
bytes
- .get(..HEADER_SIZE)
- .and_then(|header| bytemuck::checked::try_from_bytes::<ReplyHeader>(header).ok())
- .map_or(ReplyRoute::Unkeyed, |header| {
- ReplyRoute::Keyed(header.request)
- })
+ .get(offset..offset + size_of::<u64>())
+ .and_then(|field| field.try_into().ok())
+ .map(u64::from_le_bytes)
+}
+
+/// Hands the outbound frames of one connection to the streams that wait
+/// for them.
+///
+/// A stream registers its waiter before it forwards its request, so a reply
+/// never comes before the stream it answers.
+struct ReplyRouter {
+ /// Waiting streams by request id, oldest first.
+ waiters: HashMap<u64, VecDeque<oneshot::Sender<BusMessage>>>,
+ /// Frames that came while no stream waited for them, oldest first.
+ unclaimed: VecDeque<BusMessage>,
+ /// When `unclaimed` holds this many frames, it drops its oldest one.
+ unclaimed_max: usize,
+}
+
+impl ReplyRouter {
+ fn new(unclaimed_max: usize) -> Self {
+ Self {
+ waiters: HashMap::new(),
+ unclaimed: VecDeque::new(),
+ unclaimed_max,
+ }
+ }
+
+ /// Register a stream that waits for the reply to `request_id`.
+ ///
+ /// The stream first takes the oldest unclaimed frame that has no request
+ /// id or has its own. The SDK replays an explicit transient with the same
+ /// request id on a new stream, so a late reply of the first attempt
+ /// answers the replay. The replies for other ids ahead of that frame are
+ /// dropped, because their streams already ended.
+ fn register(&mut self, request_id: u64) -> oneshot::Receiver<BusMessage> {
+ let (waiter, reply) = oneshot::channel();
+ while let Some(frame) = self.unclaimed.pop_front() {
+ if !matches!(reply_route(&frame), ReplyRoute::Keyed(id) if id != request_id) {
+ // `reply` is alive, so the send cannot fail.
+ let _ = waiter.send(frame);
+ return reply;
+ }
+ }
+ self.waiters
+ .entry(request_id)
+ .or_default()
+ .push_back(waiter);
+ reply
+ }
+
+ /// Hand a reply to the oldest waiter of its request id, and a frame
+ /// without one to every waiter, because an eviction ends every request
+ /// of the connection. A frame no stream takes stays unclaimed.
+ fn route(&mut self, frame: BusMessage) {
+ let unclaimed = match reply_route(&frame) {
+ ReplyRoute::Keyed(request_id) => self.deliver(request_id, frame),
+ ReplyRoute::Unkeyed => self.broadcast(frame),
+ };
+ if let Some(frame) = unclaimed {
+ if self.unclaimed.len() == self.unclaimed_max {
+ self.unclaimed.pop_front();
+ }
+ self.unclaimed.push_back(frame);
+ }
+ }
+
+ /// Drop the waiters of `request_id` whose streams ended without a reply.
+ fn release(&mut self, request_id: u64) {
+ if let Some(waiters) = self.waiters.get_mut(&request_id) {
+ waiters.retain(|waiter| !waiter.is_canceled());
+ if waiters.is_empty() {
+ self.waiters.remove(&request_id);
+ }
+ }
+ }
+
+ /// Returns the frame if no waiter took it.
+ fn deliver(&mut self, request_id: u64, mut frame: BusMessage) -> Option<BusMessage> {
+ let Some(waiters) = self.waiters.get_mut(&request_id) else {
+ return Some(frame);
+ };
+ let unclaimed = loop {
+ let Some(waiter) = waiters.pop_front() else {
+ break Some(frame);
+ };
+ match waiter.send(frame) {
+ Ok(()) => break None,
+ Err(returned) => frame = returned,
+ }
+ };
+ if waiters.is_empty() {
+ self.waiters.remove(&request_id);
+ }
+ unclaimed
+ }
+
+ /// Returns the frame if no waiter took it.
+ fn broadcast(&mut self, frame: BusMessage) -> Option<BusMessage> {
+ let mut delivered = false;
+ for waiter in self.waiters.drain().flat_map(|(_, waiters)| waiters) {
+ delivered |= waiter.send(frame.clone()).is_ok();
+ }
+ (!delivered).then_some(frame)
+ }
+}
+
+/// What the stream futures of [`QuicTransportConn::run`] borrow from it.
+struct StreamContext<'a> {
+ in_tx: &'a Sender<Message<GenericHeader>>,
+ router: &'a RefCell<ReplyRouter>,
+ max_message_size: usize,
+ reply_backstop: Duration,
+ label: &'static str,
+ peer: &'a str,
+}
+
+impl StreamContext<'_> {
+ /// Serve one request stream. Breaks when the inbound queue is gone,
+ /// because the connection then has no dispatch.
+ #[allow(clippy::future_not_send)]
+ async fn serve(&self, mut send: SendStream, mut recv: RecvStream) -> ControlFlow<()> {
+ let Self {
+ in_tx,
+ router,
+ max_message_size,
+ reply_backstop,
+ label,
+ peer,
+ } = *self;
+ // The SDK `finish()`-es after its request, so the stream carries one
+ // frame.
+ let request = match framing::read_message(&mut recv, max_message_size).await {
+ Ok(request) => request,
+ Err(e) => {
+ debug!(%label, %peer, error = ?e, "quic: bidi read error");
+ return ControlFlow::Continue(());
+ }
+ };
+ let Some(request_id) = request_id(&request) else {
+ debug!(%label, %peer, "quic: frame shorter than a request header");
+ return ControlFlow::Continue(());
+ };
+ let mut reply = router.borrow_mut().register(request_id);
+ if in_tx.send(request).await.is_err() {
+ debug!(%label, %peer, "quic: inbound queue dropped");
+ return ControlFlow::Break(());
+ }
+ let replied = {
+ let mut stopped = pin!(send.stopped().fuse());
+ let mut backstop = pin!(compio::time::sleep(reply_backstop).fuse());
+ futures::select_biased! {
+ frame = reply => frame.ok(),
+ _ = stopped => {
+ debug!(%label, %peer, request_id, "quic: client stopped the stream before its reply");
+ None
+ }
+ () = backstop => {
+ warn!(%label, %peer, request_id, "quic: no reply within the backstop; resetting the stream");
+ None
+ }
+ }
+ };
+ let Some(frame) = replied else {
+ drop(reply);
+ router.borrow_mut().release(request_id);
+ if let Err(e) = send.reset(VarInt::from_u32(QUIC_NO_REPLY)) {
+ debug!(%label, %peer, error = ?e, "quic: stream reset failed");
+ }
+ return ControlFlow::Continue(());
+ };
+ for fragment in frame.into_fragments() {
+ let BufResult(result, _frozen) = send.write_all(fragment).await;
+ if let Err(e) = result {
+ debug!(%label, %peer, error = ?e, "quic: write failed");
+ return ControlFlow::Continue(());
+ }
+ }
+ // `finish()` flushes pending data + a FIN, which tells the SDK the
+ // reply is complete.
+ if let Err(e) = send.finish() {
+ debug!(%label, %peer, error = ?e, "quic: send.finish() failed");
+ }
+ // Keep the reply alive through graceful shutdown until the peer has
+ // acknowledged it. Closing the connection after FIN alone can discard it.
+ let _ = send.stopped().await;
Review Comment:
Deferred to PR6's QUIC bookkeeping work. Waiting only when shutdown has already been observed misses replies whose FIN was sent just before shutdown but is still unacknowledged when the connection closes. The optimization needs bounded tracking of those replies and proof that it preserves graceful reply draining and the close_grace deadline; the current ACK wait stays until that replacement is verified.
##########
core/server/src/http/handlers.rs:
##########
@@ -1256,25 +1253,27 @@ pub(in crate::http) async fn delete_segments(
/// [`poll_wire_request`]) since HTTP has no persistent connection and cannot observe
/// group's partition assignment (that can be subject to change).
///
-/// A non-replicated read served in band: the same resolution the TCP dispatch
-/// runs ([`resolve_poll_request`]), then a mesh read on the owning shard and a
-/// re-encode of the stored batches into the legacy wire body, decoded here by
-/// the SDK's own [`PolledMessages::from_bytes`] so the JSON is field-identical
-/// to a TCP poll. `auto_commit` rides [`resolve_poll_request`]'s args and is
-/// honored by the owning shard's poll plan; no extra HTTP work.
+/// A non-replicated read served by the partition primary: the same resolution
+/// the TCP dispatch runs ([`resolve_poll_request`]), then a mesh read on the
+/// owning shard and a re-encode of the stored batches into the legacy wire
+/// body, decoded here by the SDK's own [`PolledMessages::from_bytes`] so the
+/// JSON is field-identical to a TCP poll. `auto_commit` rides
+/// [`resolve_poll_request`]'s args and is honored by the owning shard's poll
+/// plan. Another node relays the poll to the primary ([`route_partition`]).
pub(in crate::http) async fn poll_messages(
State(state): State<HttpState>,
identity: Identity,
Path((stream_id, topic_id)): Path<(String, String)>,
Query(query): Query<PollMessages>,
Query(consistency): Query<ConsistencyQuery>,
-) -> Result<Json<PolledMessages>, ReadError> {
+ Extension(request): Extension<Arc<PartitionRequest>>,
+) -> Result<Response, ReadError> {
let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
let topic_id = Identifier::from_str_value(&topic_id).map_err(ReadError::Rejected)?;
SendWrapper::new(gate_local_read(
&state,
&identity,
- consistency.consistency,
+ partition_read_consistency(&request, consistency.consistency),
Review Comment:
Deferred to PR6's HTTP routing performance work. The contacted node currently enforces the requested metadata consistency before partition routing. A partition-primary relay proves partition read authority, but not current namespace/RBAC metadata on a lagging contacted node, so removing this redirect alone weakens the linearizable-read contract. Acceptance for a one-relay path: preserve namespace/incarnation resolution, authorization under revocation, metadata read-your-writes floors and partition view-change safety when the contacted node is neither primary.
##########
core/simulator/src/workload/state_checker.rs:
##########
@@ -265,6 +381,53 @@ impl StateChecker {
}
}
+/// Membership is metadata state; the partition's ordered install barrier is
+/// what fences a departed member's later offset writes.
+fn check_partition_owner(
+ owners: &mut BTreeMap<u64, ConsumerGroupOwner>,
+ header: &PrepareHeader,
+ body: &[u8],
+ seed: u64,
+) {
+ if header.operation == Operation::InstallConsumerGroupOwner {
+ let install = InstallConsumerGroupOwnerRequest::decode_from(body)
+ .expect("a committed owner installation decodes");
+ owners.insert(install.group_id, install.owner);
+ return;
+ }
+ if is_auto_commit_client(header.client) {
+ return;
+ }
+ let (request, generation) =
+ split_prepared_offset(body).expect("a committed offset has its owner token");
+ let (request, _) =
+ GetConsumerOffsetRequest::decode(request).expect("a committed offset prefix decodes");
+ if request.consumer.kind != KIND_CONSUMER_GROUP {
+ return;
+ }
+ let group_id = u64::from(
+ request
+ .consumer
+ .id
+ .as_u32()
+ .expect("a prepared group id is numeric"),
+ );
+ // The repair ring may have evicted the install before this checker started.
+ if let Some(owner) = owners.get(&group_id) {
Review Comment:
Added a captured stale-owner store action and a deterministic leave/rejoin scenario that retains the same client/session while replaying the old generation. It records 29 submissions and 29 owner-fence refusals, plus a strict offset-shadow comparison. Removing only the generation comparison makes the committed-log oracle fail with the same client/session and generation 3 versus installed generation 7; restoring it passes. The separate production session-only mutant proof remains deferred to PR6's session/recovery workload: a direct oracle test rejects stale sessions, but earlier authentication can reject those requests before they reach the owner check, so that unit test does not prove workload coverage. PR6 must exercise that fence independently and show the session-only mutant fails; the generation scenario does not close that gap.
##########
foreign/go/client/tcp/tcp_poll_routing.go:
##########
@@ -200,33 +198,90 @@ func (c *IggyTcpClient) sendRouted(caller context.Context, code command.Code, ke
c.polls.mu.Unlock()
if heartbeatDue {
if _, _, err := c.sendPollRequest(ctx, uint32(command.PingCode), nil); err != nil {
Review Comment:
Deferred to PR5's managed replica-link lifecycle, with RTT/allocation/lock measurements in PR6. Moving keepalive off the request path needs one session owner with cancellation on close/logout/reset, bounded reconnect and liveness when the metadata socket is unavailable but replica activity can still reach the authority. That lifecycle must replace the current owner coherently; key reuse and lock consolidation will be measured alongside it.
##########
foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/PollRouter.java:
##########
@@ -89,33 +95,105 @@ final class PollRouter {
}
CompletableFuture<ByteBuf> poll(ByteBuf payload, Optional<PartitionContext> context) {
+ return partitionRequest(CommandCode.Messages.POLL.getValue(), payload, context);
+ }
+
+ CompletableFuture<ByteBuf> partitionRequest(int command, ByteBuf payload, Optional<PartitionContext> context) {
byte[] request = takeBytes(payload);
- return send(new Poll(
- CommandCode.Messages.GET_POLL_ROUTING,
- request,
- CommandCode.Messages.POLL_ON_PRIMARY.getValue(),
- request,
- request.length - POLL_PARAMETERS_BYTES,
- pollTimeout,
- context));
- }
-
- /**
- * Sends a store or delete offset request to the partition primary. The route is
- * looked up by the consumer, stream, topic and partition in the first
- * {@code targetLength} bytes, as in Rust's {@code write_offset}.
- */
- CompletableFuture<ByteBuf> writeOffset(
- int command, ByteBuf payload, int targetLength, Optional<PartitionContext> context) {
+ try {
+ byte[] query = partitionQuery(command, request);
+ return send(new Poll(query, command, request, pollTimeout, context));
+ } catch (RuntimeException error) {
+ return CompletableFuture.failedFuture(error);
+ }
+ }
+
+ CompletableFuture<ByteBuf> sendMessages(ByteBuf payload, ByteBuf query) {
byte[] request = takeBytes(payload);
- return send(new Poll(
- CommandCode.ConsumerOffset.GET_ROUTING,
- Arrays.copyOf(request, targetLength),
- command,
- request,
- targetLength,
- pollTimeout,
- context));
+ byte[] target = takeBytes(query);
+ return send(new Poll(target, CommandCode.Messages.SEND.getValue(), request, pollTimeout, Optional.empty()));
+ }
+
+ CompletableFuture<Long> partitionCount(StreamId stream, TopicId topic) {
+ ByteBuf query = BytesSerializer.toBytes(stream);
+ ByteBuf topicBytes = BytesSerializer.toBytes(topic);
+ try {
+ query.writeBytes(topicBytes);
+ query.writeIntLE(0);
+ } finally {
+ topicBytes.release();
+ }
+ byte[] target = takeBytes(query);
+ String topicKey = ByteBufUtil.hexDump(target, 0, target.length - Integer.BYTES);
+ synchronized (this) {
+ Long count = partitionCounts.get(topicKey);
+ if (count != null) {
+ return CompletableFuture.completedFuture(count);
+ }
+ }
+ Poll lookup = new Poll(
+ target, CommandCode.Messages.GET_PARTITION_ROUTE.getValue(), target, pollTimeout, Optional.empty());
+ return send(lookup).thenApply(response -> {
+ response.release();
+ return lookup.partitionsCount;
+ });
+ }
+
+ synchronized void clear() {
+ routes.clear();
+ partitionCounts.clear();
+ }
+
+ synchronized void observeTopology(ClusterMetadata metadata) {
+ replicas.clear();
+ rosterSize = metadata.nodes().size();
+ for (var node : metadata.nodes()) {
+ if (node.endpoints().tcp() != 0) {
+ replicas.put(
+ node.replicaId(),
+ new ConnectionInfo(node.ip(), node.endpoints().tcp()));
+ }
+ }
+ }
+
+ private static byte[] partitionQuery(int command, byte[] request) {
+ ByteBuf body = Unpooled.wrappedBuffer(request);
+ try {
+ int consumerKind = body.readUnsignedByte();
+ skipIdentifier(body);
+ int streamStart = body.readerIndex();
+ skipIdentifier(body);
+ skipIdentifier(body);
+ int partitionStart = body.readerIndex();
+ boolean explicit = body.readUnsignedByte() != 0;
+ long partition = body.readUnsignedIntLE();
+ if (!explicit) {
+ if (command == CommandCode.ConsumerOffset.STORE.getValue()
+ || command == CommandCode.ConsumerOffset.DELETE.getValue()
+ || (command == CommandCode.Messages.POLL.getValue() && consumerKind == 2)) {
+ throw new IggyInvalidArgumentException("A group poll or offset mutation requires a partition id");
+ }
+ partition = 0;
+ body.setByte(partitionStart, 1);
+ body.setIntLE(partitionStart + 1, 0);
+ }
+ ByteBuf query = Unpooled.buffer(partitionStart - streamStart + Integer.BYTES);
+ query.writeBytes(request, streamStart, partitionStart - streamStart).writeIntLE((int) partition);
+ return takeBytes(query);
+ } catch (IndexOutOfBoundsException error) {
+ throw new IggyInvalidArgumentException("Truncated partition target");
+ } finally {
+ body.release();
+ }
+ }
+
+ private static void skipIdentifier(ByteBuf body) {
+ int kind = body.readUnsignedByte();
+ int length = body.readUnsignedByte();
+ if ((kind == 1 && length != Integer.BYTES) || (kind != 1 && kind != 2) || length == 0) {
+ throw new IggyInvalidArgumentException("Invalid partition target identifier");
+ }
+ body.skipBytes(length);
}
private static byte[] takeBytes(ByteBuf payload) {
Review Comment:
Deferred Java pipelining to PR5's shared lanes and reply mux. The current slot owns one exchange; removing serialization alone lets one timeout close another operation and bypasses auto-commit/explicit-offset ordering. Acceptance includes reordered replies, cancellation and late-reply isolation, fixed request identity/history and bounded retained bytes. ByteBuf ownership changes and copy/allocation measurements remain PR6 follow-ups, with releases proven on queue rejection, timeout, cancellation, retries and late replies under Netty paranoid leak detection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): route partition requests to their primaries [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4260721798
##########
core/consensus/src/plane_helpers.rs:
##########
@@ -903,6 +903,62 @@ where
.await
}
+/// Find a held entry below a view's suffix that the new history does not prove.
+///
+/// `canonical` is ordered by descending op. Entries above the local commit floor
+/// survive only if their parent chain reaches its lowest header, or a prepare's
+/// commit stamp proves them through the uninterrupted local chain. This runs once
+/// per installed log, before repair starts filling any resulting gap.
+#[must_use]
+pub fn first_unproven_op(
+ commit_floor: u64,
+ canonical: &[PrepareHeader],
+ op_head: u64,
+ last_op: Option<u64>,
+ header_at: impl Fn(u64) -> Option<PrepareHeader>,
+) -> Option<u64> {
+ let last_op = last_op?;
+ let proven = locally_proven_commit(commit_floor, &header_at);
+ let (mut op, mut parent) = canonical.last().map_or((op_head, None), |lowest| {
+ (lowest.op.saturating_sub(1), Some(lowest.parent))
+ });
+ while op > proven {
+ match header_at(op) {
+ Some(local)
+ if local.checksum != CHECKSUM_UNSEALED && Some(local.checksum) == parent =>
+ {
+ parent = Some(local.parent);
+ op -= 1;
+ }
+ _ => break,
+ }
+ }
+ (proven.checked_add(1)?..=op.min(last_op)).find(|&held| header_at(held).is_some())
Review Comment:
nit: the `find` walks op by op through the gap with a hash lookup each; on the partition path an ordered range over the resident index would be cheaper for a lagging backup. Once per view install, so low priority.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -11452,46 +11727,36 @@ where
.map(|durable| durable.saturating_add(1))
.max(self.installed_frontier);
let committed_shape = journal.repaired_window_shape(floor, session.commit_to_op);
- let connected = match (session.first_batch_offset, stand_in) {
+ // Read at check time, not collected per frame: the journal keeps
+ // each batch's own stamps, so a batch journaled before this floor
+ // settled anchors it as well as one served after.
+ let first_batch_offset = journal.first_batch_offset(floor, session.commit_to_op);
+ let connected = match (first_batch_offset, stand_in) {
(Some(first), Some(bound)) => first <= bound,
(Some(first), None) => first == 0,
- // No repaired batch arrived, so there is no offset anchor to
- // verify the floor's continuum claim against. `None` is only
- // safe when the served window itself proves it carried no
- // messages: every op in `(floor, to_op]` journaled and none
- // of them `SendMessages`. Anything less -- dropped frames, or
- // a fully evicted window -- is indistinguishable from a
- // message range below the floor that this replica does not
- // durably own, and accepting it would serve a holed log.
- // A replica with no committed message transfers too: a jump
- // would leave its counter at 0 while the group may have
- // committed messages below the floor. Such a replica then
- // moves its commit point only by walking each op, by a window
- // whose first batch starts at offset 0, or by an install, so
- // its empty offer as primary is real state.
- (None, _) => {
- self.offset_frontier() > 0
- && floor < session.commit_to_op
- && committed_shape.complete
- && !committed_shape.holds_messages
- }
+ // No batch above the floor, so nothing ties the floor to this
+ // replica's offsets. The floor stands only while this journal
+ // lacks an op up to the reported floor, and no shape of the
+ // window above it shows that the ops the jump skips carried no
+ // messages. Refused, the window goes to state transfer.
+ (None, _) => false,
Review Comment:
nit (accepted cost, just flagging): refusing control-only repair windows turns a lagging backup on an offset-commit-heavy, low-produce partition into a full state transfer. Rare, and the safety gain is real.
##########
core/message_bus/src/transports/quic.rs:
##########
@@ -326,12 +316,202 @@ fn reply_route(frame: &BusMessage) -> ReplyRoute {
if !is_reply {
return ReplyRoute::Unkeyed;
}
+ read_u64(bytes, offset_of!(ReplyHeader, request)).map_or(ReplyRoute::Unkeyed, ReplyRoute::Keyed)
+}
+
+/// The request id of a client frame, read as raw bytes like [`reply_route`],
+/// so a frame with an undeclared operation still has one.
+fn request_id(request: &Message<GenericHeader>) -> Option<u64> {
+ read_u64(request.as_slice(), offset_of!(RequestHeader, request))
+}
+
+fn read_u64(bytes: &[u8], offset: usize) -> Option<u64> {
bytes
- .get(..HEADER_SIZE)
- .and_then(|header| bytemuck::checked::try_from_bytes::<ReplyHeader>(header).ok())
- .map_or(ReplyRoute::Unkeyed, |header| {
- ReplyRoute::Keyed(header.request)
- })
+ .get(offset..offset + size_of::<u64>())
+ .and_then(|field| field.try_into().ok())
+ .map(u64::from_le_bytes)
+}
+
+/// Hands the outbound frames of one connection to the streams that wait
+/// for them.
+///
+/// A stream registers its waiter before it forwards its request, so a reply
+/// never comes before the stream it answers.
+struct ReplyRouter {
+ /// Waiting streams by request id, oldest first.
+ waiters: HashMap<u64, VecDeque<oneshot::Sender<BusMessage>>>,
+ /// Frames that came while no stream waited for them, oldest first.
+ unclaimed: VecDeque<BusMessage>,
+ /// When `unclaimed` holds this many frames, it drops its oldest one.
+ unclaimed_max: usize,
+}
+
+impl ReplyRouter {
+ fn new(unclaimed_max: usize) -> Self {
+ Self {
+ waiters: HashMap::new(),
+ unclaimed: VecDeque::new(),
+ unclaimed_max,
+ }
+ }
+
+ /// Register a stream that waits for the reply to `request_id`.
+ ///
+ /// The stream first takes the oldest unclaimed frame that has no request
+ /// id or has its own. The SDK replays an explicit transient with the same
+ /// request id on a new stream, so a late reply of the first attempt
+ /// answers the replay. The replies for other ids ahead of that frame are
+ /// dropped, because their streams already ended.
+ fn register(&mut self, request_id: u64) -> oneshot::Receiver<BusMessage> {
+ let (waiter, reply) = oneshot::channel();
Review Comment:
nit: each request allocates a waiter `VecDeque` entry, a oneshot, a SipHash map slot and arms a 60s timer. Low impact (high-RPS traffic stays on TCP), but an `AHashMap` and a connection-level deadline sweep would trim it.
##########
core/simulator/src/workload/oracle.rs:
##########
@@ -666,17 +765,91 @@ fn assert_committed_metadata_agrees(sim: &Simulator, live: &[usize], seed: u64)
compared
}
-/// The live replica whose metadata consensus is the current primary, i.e. the
-/// authoritative holder of the committed metadata log.
-fn metadata_leader(sim: &Simulator, live: &[usize]) -> Option<usize> {
- live.iter().copied().find(|&replica_idx| {
- sim.replicas[replica_idx].shards[0]
- .plane
- .metadata()
- .consensus
- .as_ref()
- .is_some_and(|consensus| consensus.is_primary() && consensus.status() == Status::Normal)
- })
+/// Require the partition primary of every workload namespace to hold each owner
Review Comment:
nit: the PR description says simulator checks cover replacement histories, but owner fencing isn't checked: with `matches_consumer_owner` forced to always accept, fuzz seeds 1-8 stay OK with identical stats and all sim tests pass. This compares only the primary's owner, and `shadow.consumer_offsets` is never compared at quiesce. Suggest comparing owners and offsets on every live replica, and re-running that mutant.
##########
core/integration/tests/server/pipelined_frames_vsr.rs:
##########
@@ -0,0 +1,76 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Frames a client pipelines on one connection, against the server (vsr). A
+//! read held for the metadata frontier runs in its own task, so a frame sent
+//! behind it on the same connection gets its reply first.
+//!
+//! The frames are hand-crafted on a raw TCP socket, because the SDK waits for
+//! each reply before it sends the next frame.
+
+use iggy::prelude::*;
+use iggy_binary_protocol::codes::{GET_ME_CODE, PING_CODE};
+use iggy_binary_protocol::consensus::ReplyHeader;
+use integration::iggy_harness;
+
+use crate::server::raw_tcp::{
+ connect, non_replicated_header, read_reply, register_root, write_frame,
+};
+
+/// The header validator requires a nonzero client id; the value is otherwise
+/// free since nothing here reconnects.
+const CLIENT_ID: u128 = 0x9_1BE;
+
+/// No commit reaches this op, so a read that waits for it is held for its
+/// whole budget and then fails retryable.
+const UNREACHED_OP: u64 = u64::MAX;
+
+const READ_REQUEST: u64 = 1;
+const PING_REQUEST: u64 = 2;
+
+#[iggy_harness]
+async fn given_a_held_read_when_a_ping_follows_on_one_tcp_connection_should_answer_the_ping_first(
Review Comment:
nit: this is the only pipelining test and it covers PING only. Nothing pins the per-connection ordering the dispatch change affects (pipelined write then partition op/read, two pipelined Next+auto-commit polls, poll then store); tests for those would guard the fixes above.
##########
core/integration/tests/cluster/partition_primary_routing.rs:
##########
@@ -262,6 +443,306 @@ async fn given_split_primaries_when_http_auto_commits_on_a_backup_should_replica
assert_replicated_offset(harness, &consumer, Some((GROUP_PAYLOADS.len() - 1) as u64)).await;
}
+/// Every partition request an HTTP caller sends to a backup runs on the
+/// partition primary: the backup chooses the partition and relays the request.
+#[iggy_harness(cluster_nodes = 3, server(
+ metadata.journal_slots = "256",
+ http.jwt.encoding_secret = "0123456789abcdef0123456789abcdef",
+ http.jwt.decoding_secret = "0123456789abcdef0123456789abcdef"
+))]
+async fn given_split_primaries_when_http_calls_a_partition_backup_should_serve_on_the_primary(
+ harness: &mut TestHarness,
+) {
+ let (partition_primary, backup, _) = seed_split_primaries(harness).await;
+ assert_ne!(partition_primary, backup);
+ let http = HttpClient::login_root_no_redirect(node_url(harness, backup)).await;
+ let messages = http.url(&format!(
+ "/streams/{STREAM_NAME}/topics/{TOPIC_NAME}/messages"
+ ));
+ // The relay copies the partition primary's applied op onto each response.
+ let mut primary_applied_op = 0;
+ for (payload, query, status) in [
+ (RELAYED_PAYLOADS[0], "", StatusCode::CREATED),
+ (RELAYED_PAYLOADS[1], "?ack=none", StatusCode::ACCEPTED),
+ ] {
+ let response = http
+ .client
+ .post(format!("{messages}{query}"))
+ .bearer_auth(&http.token)
+ .json(&send_body(payload))
+ .send()
+ .await
+ .expect("send request");
+ assert_eq!(response.status(), status, "send{query} on a backup");
+ primary_applied_op = primary_applied_op.max(applied_op(&response));
+ }
+
+ let expected: Vec<&str> = GROUP_PAYLOADS.into_iter().chain(RELAYED_PAYLOADS).collect();
+ let poll = format!(
+ "{messages}?consumer_id={RELAY_CONSUMER_ID}&partition_id={PARTITION_ID}\
+ &kind=offset&value=0&count={}&auto_commit=false",
+ expected.len() + 1
+ );
+ let polled = timeout(GROUP_POLL_BUDGET, async {
+ loop {
+ let response = http
+ .client
+ .get(&poll)
+ .bearer_auth(&http.token)
+ .send()
+ .await
+ .expect("poll request");
+ assert_eq!(response.status(), StatusCode::OK, "poll on a backup");
+ let polled: PolledMessages = response.json().await.expect("decode PolledMessages");
+ if polled.messages.len() >= expected.len() {
+ break polled;
+ }
+ sleep(PRECONDITION_POLL).await;
+ }
+ })
+ .await
+ .expect("the unacknowledged send must commit on the partition primary");
+ let payloads: Vec<&[u8]> = polled
+ .messages
+ .iter()
+ .map(|message| message.payload.as_ref())
+ .collect();
+ let expected: Vec<&[u8]> = expected.iter().map(|payload| payload.as_bytes()).collect();
+ assert_eq!(payloads, expected);
+ assert!(
+ polled.context.metadata_op >= primary_applied_op,
+ "the relayed poll context {:?} must cover the primary's applied op {primary_applied_op}",
+ polled.context
+ );
+
+ let offsets = http.url(&format!(
+ "/streams/{STREAM_NAME}/topics/{TOPIC_NAME}/consumer-offsets"
+ ));
+ let offset = format!("{offsets}?consumer_id={RELAY_CONSUMER_ID}&partition_id={PARTITION_ID}");
+ let store = StoreConsumerOffset {
+ context: None,
+ consumer: Consumer::new(Identifier::numeric(RELAY_CONSUMER_ID).unwrap()),
+ partition_id: Some(PARTITION_ID),
+ offset: 1,
+ };
+ let response = http
+ .client
+ .put(&offsets)
+ .bearer_auth(&http.token)
+ .json(&store)
+ .send()
+ .await
+ .expect("store offset request");
+ assert_eq!(
+ response.status(),
+ StatusCode::NO_CONTENT,
+ "store on a backup"
+ );
+ let response = http
+ .client
+ .get(&offset)
+ .bearer_auth(&http.token)
+ .send()
+ .await
+ .expect("get offset request");
+ assert_eq!(response.status(), StatusCode::OK, "get on a backup");
+ let info: ConsumerOffsetInfo = response.json().await.expect("decode ConsumerOffsetInfo");
+ assert_eq!(info.stored_offset, store.offset);
+ let response = http
+ .client
+ .delete(format!(
+ "{offsets}/{RELAY_CONSUMER_ID}?partition_id={PARTITION_ID}"
+ ))
+ .bearer_auth(&http.token)
+ .send()
+ .await
+ .expect("delete offset request");
+ assert_eq!(
+ response.status(),
+ StatusCode::NO_CONTENT,
+ "delete on a backup"
+ );
+ let response = http
+ .client
+ .get(&offset)
+ .bearer_auth(&http.token)
+ .send()
+ .await
+ .expect("get offset request after delete");
+ assert_eq!(response.status(), StatusCode::NOT_FOUND, "deleted offset");
+}
+
+/// A relayed request carries the partition the contacted node chose. A backup
+/// refuses it with `NotPrimary` instead of relaying it again, a malformed route
+/// is the caller's error, and a route without the relay marker is ignored.
+#[iggy_harness(cluster_nodes = 3, server(
+ metadata.journal_slots = "256",
+ http.jwt.encoding_secret = "0123456789abcdef0123456789abcdef",
+ http.jwt.decoding_secret = "0123456789abcdef0123456789abcdef"
+))]
+async fn given_split_primaries_when_a_relayed_request_reaches_a_backup_should_refuse_not_primary(
+ harness: &mut TestHarness,
+) {
+ let (partition_primary, backup, _) = seed_split_primaries(harness).await;
+ assert_ne!(partition_primary, backup);
+ let http = HttpClient::login_root_no_redirect(node_url(harness, backup)).await;
+ let messages = http.url(&format!(
+ "/streams/{STREAM_NAME}/topics/{TOPIC_NAME}/messages"
+ ));
+ let poll = format!(
+ "{messages}?consumer_id={RELAY_CONSUMER_ID}&partition_id={PARTITION_ID}\
+ &kind=offset&value=0&count=1&auto_commit=false"
+ );
+ let send = |route: &'static str| {
+ http.client
+ .post(&messages)
+ .bearer_auth(&http.token)
+ .header(PARTITION_ROUTE_HEADER, route)
+ .json(&send_body(RELAYED_PAYLOADS[0]))
+ };
+
+ for (request, name) in [
+ (
+ send(ROUTE_AT_FIRST_OP).header(FORWARDED_HEADER, "1"),
+ "send",
+ ),
+ (
+ http.client
+ .get(&poll)
+ .bearer_auth(&http.token)
+ .header(FORWARDED_HEADER, "1")
+ .header(PARTITION_ROUTE_HEADER, ROUTE_AT_FIRST_OP),
+ "poll",
+ ),
+ ] {
+ let response = request.send().await.expect("relayed request");
+ assert_eq!(
+ response.status(),
+ StatusCode::SERVICE_UNAVAILABLE,
+ "relayed {name} on a backup"
+ );
+ assert!(response.headers().contains_key(RETRY_AFTER));
+ let body: serde_json::Value = response.json().await.expect("decode the error body");
+ assert_eq!(
+ body["id"],
+ IggyError::NotPrimary.as_code(),
+ "relayed {name}"
+ );
+ }
+
+ let response = send("0,1")
+ .header(FORWARDED_HEADER, "1")
+ .send()
+ .await
+ .expect("malformed relayed send");
+ assert_eq!(response.status(), StatusCode::BAD_REQUEST);
+
+ let response = send(ROUTE_AT_FIRST_OP)
+ .send()
+ .await
+ .expect("send with a route and no relay marker");
+ assert_eq!(response.status(), StatusCode::CREATED);
+}
+
+/// A read changes no state, so a backup resends a relayed read whose partition
+/// primary died before it answered, and the caller gets the answer of the next
+/// primary. A poll with `auto_commit` may have moved its offset, so its outcome
+/// stays unknown.
+#[iggy_harness(cluster_nodes = 3, server(
+ metadata.journal_slots = "256",
+ http.jwt.encoding_secret = "0123456789abcdef0123456789abcdef",
+ http.jwt.decoding_secret = "0123456789abcdef0123456789abcdef"
+))]
+async fn given_split_primaries_when_a_relayed_read_loses_its_primary_should_resend_it(
Review Comment:
nit, flake risk: `poll(false)` asserts 200 with no 503 tolerance, but failover (5s heartbeat timeout + view changes) must fit the 10s relay deadline, so under contention it can 503. The auto-commit 502 expectation also assumes the relay was sent within 1s; if it connects after the kill, the retry gets 200. Tolerating a retryable 503 like the offset read, and killing only once the relay is observed in flight, would make it robust.
##########
core/simulator/src/workload/mod.rs:
##########
@@ -621,12 +758,34 @@ impl Workload {
// Effect-follows-actual: drive the shadow off the committed outcome, never
// the targeted one. Success mutates; a nonzero code is a committed no-op
// whose `predicted_effect` is `Effect::None`. Keeps the shadow correct
- // under at-least-once re-execution and races.
- let effect = ops::predicted_effect(&entry.input, &classified);
+ // under at-least-once re-execution and races. A partition-plane success
+ // sampled across a fixture replacement can belong to the deleted
+ // incarnation, whose state the shadow dropped with it.
+ let current = header.operation.is_metadata()
+ || self
+ .shadow
Review Comment:
nit: a send sampled Live {s,s} that replies while a delete is pending fails `unchanged_since` and is recorded as no effect; if the delete is then refused, `started` goes back to s and a committed send is missing from the shadow. Weakens detection only. Also, the workload has no poll action and the shell sim doesn't run consumer liveness (`replica.rs:407` TODO), which is why the RetireSession stall and the duplicate-delivery race aren't reachable here.
##########
core/integration/tests/cluster/partition_primary_routing.rs:
##########
@@ -262,6 +443,306 @@ async fn given_split_primaries_when_http_auto_commits_on_a_backup_should_replica
assert_replicated_offset(harness, &consumer, Some((GROUP_PAYLOADS.len() - 1) as u64)).await;
}
+/// Every partition request an HTTP caller sends to a backup runs on the
+/// partition primary: the backup chooses the partition and relays the request.
+#[iggy_harness(cluster_nodes = 3, server(
+ metadata.journal_slots = "256",
+ http.jwt.encoding_secret = "0123456789abcdef0123456789abcdef",
+ http.jwt.decoding_secret = "0123456789abcdef0123456789abcdef"
+))]
+async fn given_split_primaries_when_http_calls_a_partition_backup_should_serve_on_the_primary(
Review Comment:
nit: only the writes prove the relay here; a poll or offset GET returns the same bytes whether a backup or the primary serves it, and the `metadata_op` check also passes on a backup (shared metadata plane). A served-by signal on reads (e.g. a debug header naming the serving replica) would let this catch a backup-served read.
##########
core/message_bus/src/transports/quic.rs:
##########
@@ -375,9 +558,13 @@ impl QuicTransportConn {
}
impl TransportConn for QuicTransportConn {
- #[allow(clippy::future_not_send, clippy::too_many_lines)]
+ #[allow(clippy::future_not_send)]
async fn run(self, ctx: ActorContext) {
- let connection = self.connection;
+ let Self {
+ connection,
+ reply_backstop,
+ ..
Review Comment:
nit: `close_grace` is discarded by the `..` here, so shutdown closes immediately and drops every in-flight `serve` future whose request already went through `in_tx`. With 256 streams per connection that is up to 256 unknown outcomes per connection on a graceful restart (dedup covers retries). The module doc promises a drain bounded by `close_grace`; either drain (while `in_tx` is alive) or fix the doc.
##########
core/simulator/src/workload/shadow.rs:
##########
@@ -36,6 +37,181 @@ use server_common::sharding::IggyNamespace;
use crate::workload::effect::{ApplyResult, Effect};
use crate::workload::ids::IdPermutation;
+/// Name of the consumer group a shell run keeps on every fixture topic. Outside
+/// the `wl-` prefix, so the entity oracle leaves it out.
+pub const FIXTURE_GROUP_NAME: &str = "sim-fixture-group";
+
+/// Id of the fixture group: the first group of its topic, and every topic
+/// incarnation numbers its groups from 0.
+pub const FIXTURE_GROUP_ID: u32 = 0;
+
+/// The half of a fixture replacement one request performs.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum Toggle {
+ Delete,
+ Create { partitions_count: u32 },
+}
+
+/// A sampled join of a fixture group.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct GroupJoin {
+ /// Index of the fixture topic in the shadow.
+ pub fixture: usize,
+ /// Create the group that a replacement dropped with its topic, instead of
+ /// joining it.
+ pub create: bool,
+ /// Replacements of the topic started at sample time. A delete drops the
+ /// group with its topic, so a reply proves the current group only if no
+ /// replacement started since.
+ pub started: u64,
+}
+
+/// What the reply to a fixture toggle or a fixture group join proves.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ToggleFate {
+ /// The request reached its goal at metadata op `op`: the topic is gone or
+ /// present, the client is a member, or the group exists.
+ Reached { op: u64 },
+ /// The request did not take effect.
+ Refused,
+ /// The request may have taken effect.
+ Unknown,
+}
+
+/// Replacement counts of one fixture topic, shared by its namespaces.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct Replacements {
+ /// Deletes sampled, less those refused before they reached the server.
+ pub started: u64,
+ /// Replacements over: every live replica hosts the recreated incarnation.
+ pub settled: u64,
+}
+
+impl Replacements {
+ /// Whether a request that recorded `sampled` may find its history replaced.
+ #[must_use]
+ pub const fn may_replace(self, sampled: Self) -> bool {
+ self.started > sampled.settled
+ }
+
+ /// Whether a success of a request that recorded `sampled` belongs to the
+ /// current incarnation. A delete replies before its fences install, so the
+ /// success of an earlier request can belong to the deleted one.
+ #[must_use]
+ pub const fn unchanged_since(self, sampled: Self) -> bool {
+ sampled.started == sampled.settled && sampled.started == self.started
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum FixtureState {
+ Live,
+ /// A delete is in flight. A probe asks again after a delete of unknown fate, so
+ /// its refusal proves nothing.
+ Deleting {
+ probe: bool,
+ },
+ DeleteUnknown,
+ Deleted,
+ Creating {
+ probe: bool,
+ },
+ CreateUnknown,
+ /// The recreated topic exists since metadata op `op`. A replica may still
+ /// host the fenced incarnation, so a request can still meet it.
+ Settling {
+ op: u64,
+ },
+}
+
+impl FixtureState {
+ /// The state a reply moves an in-flight toggle to, or `None` with no toggle in
+ /// flight. A probe resolves only on `Reached`: its delete refused under the
+ /// pending one, or its create refused, says nothing about the first toggle.
+ const fn after(self, fate: ToggleFate) -> Option<Self> {
+ Some(match (self, fate) {
+ (Self::Deleting { .. }, ToggleFate::Reached { .. })
+ | (Self::Creating { probe: false }, ToggleFate::Refused) => Self::Deleted,
+ (Self::Deleting { probe: false }, ToggleFate::Refused) => Self::Live,
+ (Self::Deleting { .. }, _) => Self::DeleteUnknown,
+ (Self::Creating { .. }, ToggleFate::Reached { op }) => Self::Settling { op },
+ (Self::Creating { .. }, _) => Self::CreateUnknown,
+ _ => return None,
+ })
+ }
+}
+
+/// The fixture group of a topic, which a shell run joins and leaves.
+#[derive(Debug, Default)]
+struct FixtureGroup {
+ /// Whether the group exists in the current incarnation of its topic. A topic
+ /// delete drops it, and the next join creates it again.
+ exists: bool,
+ /// Whether a join committed on the group of the current incarnation, which
+ /// gives the group the assignments that the owner oracle compares.
+ joined: bool,
+ /// Each client whose join may have committed, with the metadata op of its
+ /// newest such join. `None` when the fate of a join is unknown, which can
+ /// still commit on any later incarnation.
Review Comment:
nit: `joined_at` is never cleared, so `may_have_joined` accepts a member that left after its successor installed, and `assert_group_writer_joined` (`workload/mod.rs:850`) passes exactly the stale-owner store this PR fences. Bounding it by the successor's install op would close that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4261449475
##########
core/server/src/dispatch/mod.rs:
##########
@@ -557,25 +688,48 @@ async fn handle_client_request<B, MJ, S, SB>(
// (`submit_register_local_or_forward`). Every other non-replicated
// code MUST go through Register first, which binds the acting user
// the per-op authz gates resolve.
- handle_non_replicated_request(
- shard,
- sessions,
- server_config,
- transport_client_id,
- request,
- ConnectionContext {
- bound,
- user_id,
- address: client_address,
- metadata_watermark,
- },
- )
- .await;
+ let context = ConnectionContext {
+ bound,
+ user_id,
+ address: client_address,
+ metadata_watermark,
+ };
+ // `ClientDrain` says which frames stay on the drain.
+ if matches!(non_replicated_code(header), PING_CODE | BIND_SESSION_CODE) {
+ handle_non_replicated_request(
+ shard,
+ sessions,
+ server_config,
+ transport_client_id,
+ request,
+ context,
+ )
+ .await;
+ return;
+ }
+ let in_flight = drain.admit(Admission::Cap).await;
+ let task_shard = Rc::clone(shard);
+ let task_sessions = Rc::clone(sessions);
+ let task_config = Arc::clone(server_config);
+ shard.bus.spawn(async move {
Review Comment:
The consumer-progress ordering bug is fixed in 5a2e85e54, with a disk-tier regression that fails when the guard is removed. Inline read promotion and the throughput regression are deferred to multi-leader PR6. Any fast path must preserve metadata barriers, per-consumer reply ordering and PING progress; rerun the standalone and concurrent-produce benchmarks before accepting it.
##########
core/server/src/dispatch/mod.rs:
##########
@@ -240,13 +249,130 @@ impl Drop for ActiveDrainSlot {
}
}
+/// The state that the drain task of one connection shares with the requests
+/// it spawned.
+///
+/// The drain takes the frames of the connection in arrival order and runs the
+/// prologue of each. It finishes a frame that later frames depend on before it
+/// takes the next one: a login, a logout, a `BIND_SESSION`, the rewrite of a
+/// metadata write, or a partition admission. A `PING` finishes there too,
+/// because it never waits. Every other read, every poll and every wait for a
+/// metadata commit runs in a task of its own, so a held frame does not stall
+/// the frames behind it.
+///
+/// `capacity` caps those tasks at `message_bus.client_queue_capacity`, the
+/// outbound queue size of a connection, so their replies never overflow that
+/// queue. A metadata write, a segment delete, a login and a logout also take
+/// the metadata lane, which holds one of them at a time. The metadata client
+/// table keeps one pending write per client and refuses a lower request id
+/// after a higher one committed, so a second write in flight would fail with
+/// `TransientNotAccepted` or `RequestTooOld`. The lane also runs the rewrite of
+/// a write after the previous write replied, because the rewrite reads state
+/// that the previous write can change.
+///
+/// At the cap or the lane the drain waits, and the frames behind it queue up
+/// to [`MAX_QUEUED_CLIENT_REQUESTS`]. While tasks run, the drain outlives its
+/// queue, so the cap and the lane still count them when the next frame
+/// arrives.
+pub(in crate::dispatch) struct ClientDrain {
+ capacity: usize,
+ in_flight: Cell<usize>,
+ /// Whether a frame holds the metadata lane.
+ metadata_write: Cell<bool>,
+ /// The drain task, while it waits for a frame or for room to admit one.
+ parked: Cell<Option<Waker>>,
+}
+
+/// What a frame waits for before [`ClientDrain::admit`] counts it.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+enum Admission {
+ /// Room under the in-flight cap.
+ Cap,
+ /// Room under the cap and a free metadata lane.
+ MetadataLane,
+}
+
+impl ClientDrain {
+ pub(in crate::dispatch) const fn new(server_config: &ServerConfig) -> Self {
+ Self {
+ capacity: server_config.message_bus.client_queue_capacity,
Review Comment:
Deferred to multi-leader PR6. PR4 retains the approved client_queue_capacity bound. A separate read cap needs multi-connection fairness and throughput measurements, plus saturation and cancellation tests, before choosing its value.
##########
core/server/src/http/handlers.rs:
##########
@@ -1314,12 +1313,28 @@ pub(in crate::http) async fn poll_messages(
.streams()
.poll_metadata(namespace, None, HTTP_READ_CLIENT_ID)
.ok_or(ReadError::Rejected(IggyError::TransientNotAccepted))?;
- let context = metadata.context(state.shard.plane.metadata().applied_frontier().get());
- if let Some(requested) = query.strategy.context {
- // TCP parity: a group poll already got the re-sync sentinel at resolve
- // above, so this poll has no group and names group 0, as TCP does.
- fence_poll_context(requested, context, 0, partition_id).map_err(ReadError::Rejected)?;
+ let local = metadata.context(state.shard.plane.metadata().applied_frontier().get());
+ let requested = request
+ .route()
+ .map(|route| route.context)
+ .or(query.strategy.context)
+ .unwrap_or(local);
+ let effect = if query.auto_commit {
+ PartitionEffect::Write
+ } else {
+ PartitionEffect::Read
+ };
+ if let Some(response) = SendWrapper::new(route_partition(
Review Comment:
Deferred to multi-leader PR6. PR4 now gates the actual HTTP read on the partition primary. Local-first dispatch changes the approved forwarding flow and must prove non-admission before rerouting, preserve the original deadline, and keep session-gate timeout distinct from a routing refusal; compare HTTP produce/read benchmarks with those guarantees intact.
##########
core/server/src/http/forward.rs:
##########
@@ -241,19 +361,108 @@ pub(in crate::http) async fn forward_to_primary(
SendWrapper::new(forward_or_pass(state, request, next)).await
}
-/// Route-layer fallback for acknowledged partition writes over HTTP.
-///
-/// HTTP has no persistent leader-aware connection to retarget. Execute on the
-/// contacted node first, then retry across the configured HTTP nodes within
-/// the deadline after a typed `TransientNotAccepted` denial. That
-/// denial proves the write never entered a partition pipeline. Every ambiguous
-/// outcome is returned without replay.
-pub(in crate::http) async fn forward_partition_write(
+/// Route layer of the partition routes: buffer the request once and attach it
+/// as a [`PartitionRequest`], so its handler can relay it to the partition
+/// primary with [`route_partition`]. No attempt runs here.
+pub(in crate::http) async fn forward_partition_request(
State(state): State<HttpState>,
request: Request,
next: Next,
) -> Response {
- SendWrapper::new(forward_partition_or_pass(state, request, next)).await
+ SendWrapper::new(attach_partition_request(state, request, next)).await
+}
+
+/// Decide where a partition request runs once its partition and context are
+/// fixed. `None` serves it on this node. `Some` is the final answer: the reply
+/// of the partition primary, or a refusal that proves nothing was admitted.
+///
+/// Until the deadline, every hop waits for a view with a primary, because a
+/// new partition starts on each replica at its own pace. A hop target never
+/// relays: it refuses with `NotPrimary` when another replica is the primary,
+/// so a stale view never chains hops. The contacted node starts another turn
+/// only after such a proven non-admission, a failed connect, or a read that
+/// failed after its send.
+pub(in crate::http) async fn route_partition(
+ state: &HttpInner,
+ request: &PartitionRequest,
+ namespace: IggyNamespace,
+ context: PartitionContext,
+ effect: PartitionEffect,
+) -> Option<Response> {
+ if request.hop == PartitionHop::Unrouted {
+ return None;
+ }
+ let Ok(partition_id) = u32::try_from(namespace.partition_id()) else {
+ return Some(CustomError::from(IggyError::InvalidCommand).into_response());
+ };
+ let mut guard = None;
+ loop {
+ if let Some(primary) = partition_primary(state, namespace).await {
Review Comment:
Deferred to multi-leader PR6: carry the refusing replica's view/primary hint and test lagging ingress views, stale hints and repeated refusals under one bounded deadline.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -690,7 +688,7 @@ pub const fn fence_poll_context(
}
if requested.owner_generation != local.owner_generation {
return Err(IggyError::ConsumerGroupPartitionNotOwned(
- group_id,
+ client_id as u32,
Review Comment:
Deferred to multi-leader PR6 diagnostics. This needs one consistent error identity contract that distinguishes the full client ID from the group ID, with tests for both reporting paths.
##########
core/sdk/src/tcp/tcp_client.rs:
##########
@@ -1674,28 +1559,25 @@ impl TcpClient {
Option<iggy_binary_protocol::consensus::RequestHeader>,
Result<Bytes, IggyError>,
) {
- let stream = self.stream.clone();
- let consensus_session = self.consensus_session.clone();
- let metadata_watermark = Arc::clone(&self.poll_router.metadata_watermark);
+ let link = self.link.clone();
+ let session_owner = Arc::clone(&self.session_owner);
let mut outcome_so_far = std::mem::take(retry_outcome);
- // SAFETY: we run code holding the `stream` lock in a task so we can't be cancelled while holding the lock.
+ // A task finishes the exchange when its caller goes away. An unbound
+ // request that left before its reply would close the link.
let joined = tokio::spawn(async move {
- let mut stream_guard = stream.lock().await;
- let Some(stream) = stream_guard.as_mut() else {
+ let mut link_guard = link.lock().await;
Review Comment:
Deferred to multi-leader PR5 with Rust driver activation. Move exchange ownership there while preserving request identity, mutation lanes, cancellation and bounded in-flight work. PR6 will compare transport throughput and task/context-switch overhead after activation.
##########
core/sdk/src/tcp/tcp_client.rs:
##########
@@ -1707,115 +1589,65 @@ impl TcpClient {
Err(error) => return (preencoded.header, Err(error), outcome_so_far),
}
};
- let mut frame_complete = false;
- let outcome = async {
- loop {
- let header_bytes = bytemuck::bytes_of(&request_header);
- frame_complete = false;
- stream.write(header_bytes).await?;
- if !payload.is_empty() {
- stream.write(&payload).await?;
+ let mut link_failed = false;
+ let mut replay = false;
+ let outcome = loop {
+ let frame = Frame {
+ header: request_header,
+ body: payload.clone(),
+ };
+ trace!("Sent a TCP request with code: {code}, waiting for a response...");
+ // The deadline guards against server-side reply loss (e.g. a
+ // stalled replication quorum that never commits the op), which
+ // would hold the link lock forever and wedge every later
+ // request on this client.
+ let reply = match link.send(frame, read_deadline, &mut false).await {
+ Ok(reply) => reply,
+ Err(error) => {
+ link_failed = true;
+ break Err(outcome_so_far.observe_link_error(error, replay));
}
- stream.flush().await?;
- trace!("Sent a TCP request with code: {code}, waiting for a response...");
-
- let mut response_header = [0u8; iggy_binary_protocol::HEADER_SIZE];
- // `stream.read` delegates to `read_exact`; on success it
- // always returns the requested length, so no short-read
- // guard is needed here.
- //
- // Deadline guards against server-side reply loss (e.g. a
- // stalled replication quorum that never commits the op):
- // the connection is lockstep, so an unanswered read would
- // hold the stream lock forever and wedge every later
- // request on this client. On expiry drop the stream --
- // a late reply would desync framing for the next request.
- //
- // One deadline spans BOTH the header and body reads: a
- // reply that delivers a header then stalls must not get a
- // fresh full timeout for the body.
- let header_read =
- tokio::time::timeout_at(read_deadline, stream.read(&mut response_header))
- .await;
- let Ok(header_read) = header_read else {
- error!(
- "Timed out after {RESPONSE_READ_TIMEOUT:?} waiting for VSR response header for TCP request with code: {code}",
- );
- return Err(IggyError::Disconnected);
- };
- header_read.map_err(|error| {
- error!(
- "Failed to read VSR response header for TCP request with code: {code}: {error}",
- );
- IggyError::Disconnected
- })?;
-
- let response_size = crate::vsr::response_size(&response_header)?;
- let body_size = response_size - iggy_binary_protocol::HEADER_SIZE;
- let body = if body_size > 0 {
- let mut body = BytesMut::with_capacity(body_size);
- let body_read = tokio::time::timeout_at(
- read_deadline,
- stream.read_buf(&mut body, body_size),
- )
- .await;
- let Ok(body_read) = body_read else {
- error!(
- "Timed out after {RESPONSE_READ_TIMEOUT:?} waiting for VSR response body for TCP request with code: {code}",
- );
- return Err(IggyError::Disconnected);
- };
- body_read.map_err(|error| {
- error!(
- "Failed to read VSR response body for TCP request with code: {code}: {error}",
- );
- IggyError::Disconnected
- })?;
- body.freeze()
- } else {
- Bytes::new()
- };
-
- frame_complete = true;
- crate::vsr::observe_metadata_reply(&metadata_watermark, &response_header);
- match crate::vsr::decode_response_split(&response_header, body)
- .map_err(|error| outcome_so_far.observe(error))
+ };
+ replay = true;
+ session_owner.observe_metadata_reply(&reply.header);
+ match crate::vsr::decode_response_split(&reply.header, reply.body)
+ .map_err(|error| outcome_so_far.observe(error))
+ {
+ Err(error) if !retry_transient => break Err(error),
+ // `TransientNotCommitted`: the op's outcome is unknown
+ // (e.g. a view change canceled it in flight) -- ONLY a
+ // same-session replay of the same request id is safe
+ // (the client-table serves it from cache if it did
+ // commit). Replay on this connection for the whole
+ // request budget; never hand it to the failover path,
+ // which re-issues under a fresh session and could
+ // double-apply a committed write.
+ Err(IggyError::TransientNotCommitted)
+ if code != iggy_binary_protocol::codes::POLL_MESSAGES_ON_PRIMARY_CODE
+ && tokio::time::Instant::now() < read_deadline =>
{
- Err(error) if !retry_transient => return Err(error),
- // `TransientNotCommitted`: the op's outcome is unknown
- // (e.g. a view change canceled it in flight) -- ONLY a
- // same-session replay of the same request id is safe
- // (the client-table serves it from cache if it did
- // commit). Replay on this connection for the whole
- // request budget; never hand it to the failover path,
- // which re-issues under a fresh session and could
- // double-apply a committed write.
- Err(IggyError::TransientNotCommitted)
- if code != iggy_binary_protocol::codes::POLL_MESSAGES_ON_PRIMARY_CODE
- && tokio::time::Instant::now() < read_deadline =>
- {
- let remaining = read_deadline
- .saturating_duration_since(tokio::time::Instant::now());
- tokio::time::sleep(NOT_READY_RETRY_INTERVAL.min(remaining)).await;
- }
- // `TransientNotAccepted`: the server never admitted the
- // request, so it is re-issuable anywhere. Replay here
- // briefly, then hand it back to the caller for a
- // leader recheck / failover.
- Err(IggyError::TransientNotAccepted)
- if tokio::time::Instant::now() < transient_deadline =>
- {
- let remaining = transient_deadline
- .saturating_duration_since(tokio::time::Instant::now());
- tokio::time::sleep(NOT_READY_RETRY_INTERVAL.min(remaining)).await;
- }
- other => return other,
+ let remaining =
+ read_deadline.saturating_duration_since(tokio::time::Instant::now());
+ tokio::time::sleep(NOT_READY_RETRY_INTERVAL.min(remaining)).await;
Review Comment:
Deferred to multi-leader PR5 with Rust driver activation. Release the link guard during retry waits while retaining the exact request identity and mutation-lane ownership; verify concurrent PING, replay and cancellation across TCP, WebSocket and QUIC.
##########
core/sdk/src/links/mux.rs:
##########
@@ -0,0 +1,815 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! One TCP or WebSocket connection that carries many requests at once.
+//!
+//! A writer task and a reader task own the two halves of the connection, and
+//! each reply goes to the request with its id. So a caller that goes away
+//! never leaves a frame half written or half read on the connection.
+
+use crate::links::{Frame, Reply};
+use crate::vsr::{decode_eviction, peek_command, reply_request_id, response_size};
+use bytes::{BufMut, BytesMut};
+use futures_util::stream::{SplitSink, SplitStream};
+use futures_util::{SinkExt, StreamExt};
+use iggy_binary_protocol::consensus::{Command, HEADER_SIZE};
+use iggy_common::IggyError;
+use std::collections::HashMap;
+use std::collections::hash_map::Entry;
+use std::future::Future;
+use std::io;
+use std::net::SocketAddr;
+use std::sync::{Arc, Mutex as StdMutex, MutexGuard, PoisonError};
+use std::time::Duration;
+use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt, BufReader, BufWriter};
+use tokio::sync::{Mutex, Semaphore, mpsc, oneshot, watch};
+use tokio::task::JoinHandle;
+use tokio::time::{Instant, timeout, timeout_at};
+use tokio_tungstenite::WebSocketStream;
+use tokio_tungstenite::tungstenite::Message;
+use tokio_tungstenite::tungstenite::protocol::CloseFrame;
+use tokio_tungstenite::tungstenite::protocol::frame::coding::CloseCode;
+use tracing::{debug, error, trace};
+
+/// Frames queued for the writer task before a sender waits for room.
+const LINK_WRITE_QUEUE: usize = 32;
+/// Requests that wait for a reply on one link at once. It matches
+/// `message_bus.client_queue_capacity`, the most requests the server runs at
+/// once for one connection.
+const MAX_PENDING_PER_LINK: usize = 256;
+/// A peer that stops reading cannot hold up a shutdown for longer than this.
+const SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(2);
+const WEBSOCKET_CLOSE_REASON: &str = "Client requested shutdown";
+
+type Waiter = oneshot::Sender<Result<Reply, IggyError>>;
+
+/// A connection that carries many requests at once. Dropping the link stops
+/// its tasks, which closes the connection.
+#[derive(Debug)]
+pub(crate) struct MuxLink {
+ shared: Arc<MuxShared>,
+ queue: mpsc::Sender<Frame>,
+ tasks: LinkTasks,
+}
+
+#[derive(Debug)]
+struct MuxShared {
+ peer: SocketAddr,
+ pending: StdMutex<Pending>,
+ permits: Semaphore,
+ /// Before bind, request ids repeat, so one such exchange runs at a time.
+ unbound: Mutex<()>,
+ /// Turns true once, when the link closes.
+ closed: watch::Sender<bool>,
+}
+
+#[derive(Debug)]
+struct Pending {
+ /// Cleared when the link closes. A closed link never opens again.
+ open: bool,
+ waiters: HashMap<u64, Waiter>,
+}
+
+#[derive(Debug)]
+struct LinkTasks {
+ writer: JoinHandle<()>,
+ reader: JoinHandle<()>,
+}
+
+impl Drop for LinkTasks {
+ fn drop(&mut self) {
+ self.writer.abort();
+ self.reader.abort();
+ }
+}
+
+/// A request that waits for its reply. Dropping it ends the wait.
+struct PendingEntry<'a> {
+ shared: &'a MuxShared,
+ request: u64,
+ /// Set for a repeated id that reached the queue: its late reply would
+ /// answer the next request with the same id.
+ close_if_abandoned: bool,
+}
+
+impl MuxLink {
+ /// A link over the two halves of a TCP connection, plain or TLS.
+ pub(crate) fn tcp<R, W>(reader: R, writer: W, peer: SocketAddr) -> Self
+ where
+ R: AsyncRead + Unpin + Send + 'static,
+ W: AsyncWrite + Unpin + Send + 'static,
+ {
+ Self::start(
+ peer,
+ |frames, shared| write_tcp(BufWriter::new(writer), frames, shared),
+ |shared| read_tcp(BufReader::new(reader), shared),
+ )
+ }
+
+ /// A link over a WebSocket connection, plain or TLS.
+ pub(crate) fn websocket<S>(stream: WebSocketStream<S>, peer: SocketAddr) -> Self
+ where
+ S: AsyncRead + AsyncWrite + Unpin + Send + 'static,
+ {
+ let (sink, stream) = stream.split();
+ Self::start(
+ peer,
+ |frames, shared| write_websocket(sink, frames, shared),
+ |shared| read_websocket(stream, shared),
+ )
+ }
+
+ /// Sends `frame` and waits for its reply until `deadline`. `sent` turns
+ /// true when the frame reaches the queue, so a caller that goes away
+ /// before the reply still knows that the frame left.
+ ///
+ /// `NotConnected` means the frame never reached the queue, so the request
+ /// can go again. `Disconnected` or an eviction reason means it did, and no
+ /// reply came.
+ pub(crate) async fn send(
+ &self,
+ frame: Frame,
+ deadline: Instant,
+ sent: &mut bool,
+ ) -> Result<Reply, IggyError> {
+ let request = frame.header.request;
+ let unbound = frame.header.session == 0;
+ let _unbound_turn = if unbound {
+ let turn = timeout_at(deadline, self.shared.unbound.lock()).await;
+ Some(turn.map_err(|_| IggyError::NotConnected)?)
+ } else {
+ None
+ };
+ let _permit = timeout_at(deadline, self.shared.permits.acquire())
+ .await
+ .map_err(|_| IggyError::NotConnected)?
+ .map_err(|_| IggyError::NotConnected)?;
+ let (waiter, reply) = oneshot::channel();
+ let mut entry = self.shared.wait_for(request, waiter)?;
+ if !matches!(
+ timeout_at(deadline, self.queue.send(frame)).await,
+ Ok(Ok(()))
+ ) {
+ return Err(IggyError::NotConnected);
+ }
+ *sent = true;
+ entry.close_if_abandoned = unbound;
+ match timeout_at(deadline, reply).await {
+ Ok(Ok(result)) => result,
+ Ok(Err(_)) => Err(IggyError::Disconnected),
+ Err(_) => {
+ error!(
+ "No reply to request {request} from {} before its deadline.",
+ self.shared.peer
+ );
+ Err(IggyError::Disconnected)
+ }
+ }
+ }
+
+ /// Fails every waiting request. The writer task then ends the connection
+ /// the way its protocol expects.
+ pub(crate) fn close(&self) {
+ self.shared.close(&IggyError::Disconnected);
+ }
+
+ /// Returns when the link closes, for any reason.
+ pub(crate) async fn closed(&self) {
+ let mut closed = self.shared.closed.subscribe();
+ // The sender lives as long as the link.
+ let _ = closed.wait_for(|closed| *closed).await;
+ }
+
+ /// Ends the connection the way its protocol expects: a FIN for TCP, a
+ /// Close message for WebSocket. The link counts as closed first, so the
+ /// reader does not report the end of the connection as an error.
+ pub(crate) async fn shutdown(self) {
+ let Self {
+ shared,
+ queue,
+ mut tasks,
+ } = self;
+ shared.close(&IggyError::Disconnected);
+ drop(queue);
+ let _ = timeout(SHUTDOWN_TIMEOUT, &mut tasks.writer).await;
+ }
+
+ fn start<Writer, Reader>(
+ peer: SocketAddr,
+ writer: impl FnOnce(mpsc::Receiver<Frame>, Arc<MuxShared>) -> Writer,
+ reader: impl FnOnce(Arc<MuxShared>) -> Reader,
+ ) -> Self
+ where
+ Writer: Future<Output = ()> + Send + 'static,
+ Reader: Future<Output = ()> + Send + 'static,
+ {
+ let shared = Arc::new(MuxShared {
+ peer,
+ pending: StdMutex::new(Pending {
+ open: true,
+ waiters: HashMap::new(),
+ }),
+ permits: Semaphore::new(MAX_PENDING_PER_LINK),
+ unbound: Mutex::new(()),
+ closed: watch::Sender::new(false),
+ });
+ let (queue, frames) = mpsc::channel(LINK_WRITE_QUEUE);
+ let tasks = LinkTasks {
+ writer: tokio::spawn(writer(frames, Arc::clone(&shared))),
+ reader: tokio::spawn(reader(Arc::clone(&shared))),
+ };
+ Self {
+ shared,
+ queue,
+ tasks,
+ }
+ }
+}
+
+impl MuxShared {
+ fn wait_for(&self, request: u64, waiter: Waiter) -> Result<PendingEntry<'_>, IggyError> {
+ let mut pending = self.lock_pending();
+ if !pending.open {
+ return Err(IggyError::NotConnected);
+ }
+ match pending.waiters.entry(request) {
+ // Two requests with one id could not tell their replies apart.
+ Entry::Occupied(_) => {
+ error!(
+ "Request {request} already waits for a reply from {}.",
+ self.peer
+ );
+ Err(IggyError::NotConnected)
+ }
+ Entry::Vacant(entry) => {
+ entry.insert(waiter);
+ Ok(PendingEntry {
+ shared: self,
+ request,
+ close_if_abandoned: false,
+ })
+ }
+ }
+ }
+
+ /// Cuts every whole frame off the front of `buffer` and dispatches it.
+ fn deliver_buffered(&self, buffer: &mut BytesMut) -> Result<(), IggyError> {
+ while let Some(&header) = buffer.first_chunk::<HEADER_SIZE>() {
+ let Ok(size) = response_size(&header) else {
+ error!(
+ "Received a frame shorter than its header from {}.",
+ self.peer
+ );
+ return Err(IggyError::Disconnected);
+ };
+ if buffer.len() < size {
+ break;
+ }
+ let body = buffer.split_to(size).freeze().slice(HEADER_SIZE..);
+ self.dispatch(Reply { header, body })?;
+ }
+ Ok(())
+ }
+
+ /// Gives a reply to the request that waits for it. An error is the reason
+ /// to close the link: an eviction ends the session, so it ends every
+ /// request.
+ fn dispatch(&self, reply: Reply) -> Result<(), IggyError> {
+ match peek_command(&reply.header) {
+ Command::Reply => {
+ let request = reply_request_id(&reply.header);
+ let waiter = self.lock_pending().waiters.remove(&request);
+ match waiter {
+ Some(waiter) => {
+ let _ = waiter.send(Ok(reply));
+ }
+ None => trace!(
+ "Dropped a reply to request {request} from {}, which nothing waits for.",
+ self.peer
+ ),
+ }
+ Ok(())
+ }
+ Command::Eviction => Err(decode_eviction(&reply.header)),
+ _ => {
+ error!(
+ "Received a frame that is neither a reply nor an eviction from {}.",
+ self.peer
+ );
+ Err(IggyError::Disconnected)
+ }
+ }
+ }
+
+ /// Fails every waiting request with `reason`. A request that comes later
+ /// fails with `NotConnected`.
+ fn close(&self, reason: &IggyError) {
+ let waiters = {
+ let mut pending = self.lock_pending();
+ pending.open = false;
+ std::mem::take(&mut pending.waiters)
+ };
+ self.permits.close();
+ self.closed.send_replace(true);
+ for waiter in waiters.into_values() {
+ let _ = waiter.send(Err(reason.clone()));
+ }
+ }
+
+ /// The next frame to write, or `None` once the link closed or every
+ /// sender went away.
+ async fn next_frame(&self, frames: &mut mpsc::Receiver<Frame>) -> Option<Frame> {
+ let mut closed = self.closed.subscribe();
+ tokio::select! {
+ biased;
+ _ = closed.wait_for(|closed| *closed) => None,
+ frame = frames.recv() => frame,
+ }
+ }
+
+ fn is_open(&self) -> bool {
+ self.lock_pending().open
+ }
+
+ fn lock_pending(&self) -> MutexGuard<'_, Pending> {
+ self.pending.lock().unwrap_or_else(PoisonError::into_inner)
+ }
+}
+
+impl Drop for PendingEntry<'_> {
+ fn drop(&mut self) {
+ let abandoned = self
+ .shared
+ .lock_pending()
+ .waiters
+ .remove(&self.request)
+ .is_some();
+ if abandoned && self.close_if_abandoned {
+ debug!(
+ "Closing the link to {}: request {} left before its reply.",
+ self.shared.peer, self.request
+ );
+ self.shared.close(&IggyError::Disconnected);
+ }
+ }
+}
+
+async fn write_tcp<W: AsyncWrite + Unpin>(
+ mut writer: W,
+ mut frames: mpsc::Receiver<Frame>,
+ shared: Arc<MuxShared>,
+) {
+ while let Some(frame) = shared.next_frame(&mut frames).await {
+ let written = async {
+ writer.write_all(bytemuck::bytes_of(&frame.header)).await?;
Review Comment:
Deferred to multi-leader PR6. Measure gather writes against the current buffering path across the 8 KiB boundary, including nodelay=false and delayed ACKs, and preserve partial-write and frame-boundary handling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4261453759
##########
core/message_bus/src/transports/quic.rs:
##########
@@ -326,12 +316,202 @@ fn reply_route(frame: &BusMessage) -> ReplyRoute {
if !is_reply {
return ReplyRoute::Unkeyed;
}
+ read_u64(bytes, offset_of!(ReplyHeader, request)).map_or(ReplyRoute::Unkeyed, ReplyRoute::Keyed)
+}
+
+/// The request id of a client frame, read as raw bytes like [`reply_route`],
+/// so a frame with an undeclared operation still has one.
+fn request_id(request: &Message<GenericHeader>) -> Option<u64> {
+ read_u64(request.as_slice(), offset_of!(RequestHeader, request))
+}
+
+fn read_u64(bytes: &[u8], offset: usize) -> Option<u64> {
bytes
- .get(..HEADER_SIZE)
- .and_then(|header| bytemuck::checked::try_from_bytes::<ReplyHeader>(header).ok())
- .map_or(ReplyRoute::Unkeyed, |header| {
- ReplyRoute::Keyed(header.request)
- })
+ .get(offset..offset + size_of::<u64>())
+ .and_then(|field| field.try_into().ok())
+ .map(u64::from_le_bytes)
+}
+
+/// Hands the outbound frames of one connection to the streams that wait
+/// for them.
+///
+/// A stream registers its waiter before it forwards its request, so a reply
+/// never comes before the stream it answers.
+struct ReplyRouter {
+ /// Waiting streams by request id, oldest first.
+ waiters: HashMap<u64, VecDeque<oneshot::Sender<BusMessage>>>,
+ /// Frames that came while no stream waited for them, oldest first.
+ unclaimed: VecDeque<BusMessage>,
+ /// When `unclaimed` holds this many frames, it drops its oldest one.
+ unclaimed_max: usize,
+}
+
+impl ReplyRouter {
+ fn new(unclaimed_max: usize) -> Self {
+ Self {
+ waiters: HashMap::new(),
+ unclaimed: VecDeque::new(),
+ unclaimed_max,
+ }
+ }
+
+ /// Register a stream that waits for the reply to `request_id`.
+ ///
+ /// The stream first takes the oldest unclaimed frame that has no request
+ /// id or has its own. The SDK replays an explicit transient with the same
+ /// request id on a new stream, so a late reply of the first attempt
+ /// answers the replay. The replies for other ids ahead of that frame are
+ /// dropped, because their streams already ended.
+ fn register(&mut self, request_id: u64) -> oneshot::Receiver<BusMessage> {
+ let (waiter, reply) = oneshot::channel();
Review Comment:
Deferred to multi-leader PR6. Profile QUIC waiter/map/timer overhead before changing it; any deadline sweep must preserve per-request expiry, cancellation cleanup and the bounded shutdown drain fixed in f90c6c858.
##########
core/simulator/src/workload/mod.rs:
##########
@@ -621,12 +758,34 @@ impl Workload {
// Effect-follows-actual: drive the shadow off the committed outcome, never
// the targeted one. Success mutates; a nonzero code is a committed no-op
// whose `predicted_effect` is `Effect::None`. Keeps the shadow correct
- // under at-least-once re-execution and races.
- let effect = ops::predicted_effect(&entry.input, &classified);
+ // under at-least-once re-execution and races. A partition-plane success
+ // sampled across a fixture replacement can belong to the deleted
+ // incarnation, whose state the shadow dropped with it.
+ let current = header.operation.is_metadata()
+ || self
+ .shadow
Review Comment:
The pending-delete shadow bug is fixed in 0dd5939c2, with tests for both refused and successful deletion. Simulator poll actions and consumer-liveness execution are deferred to multi-leader PR6 acceptance coverage. That follow-up must exercise duplicate-delivery and session-retirement failures, including crash/recovery runs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4269769190
##########
core/shard/src/lib.rs:
##########
@@ -5197,7 +5345,18 @@ where
let Some(partition) = planes.1.0.get_mut_by_ns(&namespace) else {
return;
};
- if !partition.consensus().is_normal() {
+ if !self.partition_incarnation_matches(
+ partition,
+ header.partition_incarnation,
+ Command::RequestPrepares,
+ ) {
+ return;
+ }
+ if !matches!(
Review Comment:
This arm now lets a partition replica in `ViewChange` serve `RequestPrepares`, and such a replica can hand an uncommitted op from an abandoned branch to a lagging backup that then commits it. A backup that times out into `ViewChange` never sets `view_log_unproven`, so `view_log_entry_proven` passes for every op and `repair_serve_ceiling` (`core/shard/src/lib.rs:11011`) serves through its own head. The requester trusts whatever arrives: `on_repair_prepare` checks only frame integrity (`core/shard/src/lib.rs:5638`), the canonical checksum comes only from `pending_view_log` and is not required at or below `commit_to_op` (`core/partitions/src/iggy_partition.rs:11651`), no repair path compares `parent` with the neighboring op, `RequestPreparesHeader.view` is never set or read, and `RepairDone` is not checked against `session.peer` (`core/shard/src/lib.rs:5901`). With 5 replicas: the view 1 primary sends op N only to R3 and crashes before its WAL write, R3 is cut off and times out into
`ViewChange`, R1 restarts and the others commit a different op at N while R4 drops prepare N, and R4's stalled repair rotates to R3 through `next_transfer_peer` (`core/shard/src/lib.rs:2855`). R4 journals and commits the abandoned op at N, later chunks with `parent` of the real op are accepted on top of it, and if R4 becomes primary its log wins at N, so polls return messages nobody acknowledged in place of acknowledged ones. A shard unit test that builds this sequence (R3 in `ViewChange` serving ops 1..=3 to R4 with `commit_max = 3`) ends with R4 at `commit_min = 3` holding the abandoned checksum at op 3. The requester-side trust predates this PR and needs 7 replicas at base, but restricting this arm to `is_normal()` is what kept the 5-replica case out.
##########
core/server/src/http/forward.rs:
##########
@@ -241,19 +360,108 @@ pub(in crate::http) async fn forward_to_primary(
SendWrapper::new(forward_or_pass(state, request, next)).await
}
-/// Route-layer fallback for acknowledged partition writes over HTTP.
-///
-/// HTTP has no persistent leader-aware connection to retarget. Execute on the
-/// contacted node first, then retry across the configured HTTP nodes within
-/// the deadline after a typed `TransientNotAccepted` denial. That
-/// denial proves the write never entered a partition pipeline. Every ambiguous
-/// outcome is returned without replay.
-pub(in crate::http) async fn forward_partition_write(
+/// Route layer of the partition routes: buffer the request once and attach it
+/// as a [`PartitionRequest`], so its handler can relay it to the partition
+/// primary with [`route_partition`]. No attempt runs here.
+pub(in crate::http) async fn forward_partition_request(
State(state): State<HttpState>,
request: Request,
next: Next,
) -> Response {
- SendWrapper::new(forward_partition_or_pass(state, request, next)).await
+ SendWrapper::new(attach_partition_request(state, request, next)).await
+}
+
+/// Decide where a partition request runs once its partition and context are
+/// fixed. `None` serves it on this node. `Some` is the final answer: the reply
+/// of the partition primary, or a refusal that proves nothing was admitted.
+///
+/// Until the deadline, every hop waits for a view with a primary, because a
+/// new partition starts on each replica at its own pace. A hop target never
+/// relays: it refuses with `NotPrimary` when another replica is the primary,
+/// so a stale view never chains hops. The contacted node starts another turn
+/// only after such a proven non-admission, a failed connect, or a read that
+/// failed after its send.
+pub(in crate::http) async fn route_partition(
+ state: &HttpInner,
+ request: &PartitionRequest,
+ namespace: IggyNamespace,
+ context: PartitionContext,
+ effect: PartitionEffect,
+) -> Option<Response> {
+ if request.hop == PartitionHop::Unrouted {
+ return None;
+ }
+ let Ok(partition_id) = u32::try_from(namespace.partition_id()) else {
+ return Some(CustomError::from(IggyError::InvalidCommand).into_response());
+ };
+ let mut guard = None;
+ loop {
+ if let Some(primary) = partition_primary(state, namespace).await {
+ if is_local_replica(state, primary) {
+ return None;
+ }
+ if matches!(request.hop, PartitionHop::Target(_)) {
+ return Some(CustomError::from(IggyError::NotPrimary).into_response());
+ }
+ if !state.forward.active {
+ return Some(not_admitted());
+ }
+ if let Some(socket) = primary_http_socket(&state.roster, primary) {
+ let remaining = request.deadline.saturating_duration_since(Instant::now());
+ // A target without time would dispatch a write it cannot wait
+ // for and answer an unknown outcome.
+ if remaining <= RELAY_MARGIN {
+ return Some(not_admitted());
+ }
+ if guard.is_none() {
+ guard = ForwardGuard::admit(&state.forward.in_flight);
+ if guard.is_none() {
+ return Some(forward_busy());
+ }
+ }
+ let route = PartitionRoute {
+ partition_id,
+ context,
+ budget: remaining.saturating_sub(RELAY_MARGIN),
+ };
+ let Ok(route) = HeaderValue::try_from(route.to_string()) else {
+ return Some(not_admitted());
+ };
+ let mut headers = request.headers.clone();
+ headers.insert(PARTITION_ROUTE_HEADER, route);
+ let url = format!(
+ "{}://{socket}{}",
+ state.forward.scheme, request.path_and_query
+ );
+ let outcome = attempt(
+ state,
+ &request.method,
+ headers,
+ &request.body,
+ &url,
+ RelayKind::Partition(effect),
+ remaining,
+ )
+ .await;
+ if let AttemptOutcome::Relay(response) = outcome {
+ return Some(match guard {
+ Some(guard)
+ if request.method == Method::GET && response.status().is_success() =>
+ {
+ let remaining =
+ request.deadline.saturating_duration_since(Instant::now());
+ retain_forward_guard(response, guard, remaining)
Review Comment:
The relayed body of a successful poll is now bounded by what is left of the 10s `PARTITION_REQUEST_DEADLINE`, and `retain_forward_guard` aborts the whole stream with `TimedOut` when it runs out, even while chunks still flow (`core/server/src/http/forward.rs:821-829`). The base gave the same span `FORWARD_ATTEMPT_TIMEOUT` (35s), and the local path at `core/server/src/http/handlers.rs:1340-1375` has no body timer at all. With `auto_commit` the primary advances the offset inside the poll completion before the reply leaves (`core/partitions/src/iggy_partition.rs:4042-4052`), so a client that reads a relayed page slower than the remaining budget loses messages that are already marked consumed, and the next poll skips them. HTTP poll pages have no byte cap and carry base64 payloads, so a page of 1000 messages of 10 KB (about 13 MB) is cut for any client below about 1.3 MB/s. This contradicts the `RESPONSE_BODY_LIMIT` doc (`core/server/src/http/forward.rs:146-151`), which exempts success
ful polls from the size cap because cutting them discards acknowledged data.
##########
core/server/src/dispatch/mod.rs:
##########
@@ -240,13 +284,153 @@ impl Drop for ActiveDrainSlot {
}
}
+/// The state that the drain task of one connection shares with the requests
+/// it spawned.
+///
+/// The drain takes the frames of the connection in arrival order and runs the
+/// prologue of each. Login, logout, `BIND_SESSION` and partition admission
+/// finish on the drain. Metadata commits run in tasks, but later non-PING
+/// frames wait in the queue until the metadata lane releases, before they
+/// capture session state or resolve a namespace. This orders dispatch, not local
+/// metadata catch-up: partition writes still resolve from this shard's STM
+/// without waiting for the connection watermark. PING can pass that queue.
+/// Stored-offset reads, Next/auto-commit polls and explicit offset writes share
+/// a lane per resolved namespace and consumer until their reply or wait expiry.
+/// Expiry releases only this connection's guard, not an admitted mutation, which
+/// may still commit after later progress. Independent requests can pass a queued
+/// conflict, but cannot pass a queued session or metadata barrier. A PING still waits
+/// behind a frame already suspended in admission or binding.
+///
+/// `capacity` caps those tasks at `message_bus.client_queue_capacity`, the
Review Comment:
The doc says the in-flight cap keeps replies from overflowing the outbound queue, but a read task drops its `InFlight` slot as soon as `send_to_client` enqueues the reply (`core/server/src/dispatch/mod.rs:842-853`), so the cap counts running tasks, not queued replies. Up to 256 reads now run at once per connection, where the base drain awaited each request before taking the next, so a client that pipelines past `client_queue_capacity` gets replies dropped as `Backpressure` by `try_send`, and `send_host_frame` only logs the drop and keeps the connection open (`core/server/src/dispatch/failure.rs:105`). The Java async SDK reaches this. It releases the channel lease right after the write for every command except login and logout (`foreign/java/java-sdk/src/main/java/org/apache/iggy/client/async/tcp/AsyncTcpConnection.java:626-631`) and tracks replies in an unbounded map, so a request whose reply was dropped waits for its timeout, and that timeout closes the channel and fails every ot
her request pending on it.
##########
core/simulator/src/workload/state_checker.rs:
##########
@@ -265,6 +390,85 @@ impl StateChecker {
}
}
+/// Membership is metadata state; the partition's ordered install barrier is
+/// what fences a departed member's later offset writes.
+fn check_partition_owner(
+ owners: &mut PartitionOwners,
+ header: &PrepareHeader,
+ body: &[u8],
+ seed: u64,
+) {
+ if header.operation == Operation::InstallConsumerGroupOwner {
+ let install = InstallConsumerGroupOwnerRequest::decode_from(body)
+ .expect("a committed owner installation decodes");
+ if !owners
+ .retirement
+ .as_ref()
+ .is_some_and(|catalog| catalog.retires(install.group_id))
+ {
+ owners.installed.insert(install.group_id, install.owner);
+ }
+ return;
+ }
+ if header.operation == Operation::RetireConsumerGroupOwners {
Review Comment:
This retirement check never runs end to end. The simulator calls only `reconcile_pending_revocations` and `reconcile_partition_lifecycles` (`core/simulator/src/lib.rs:1198`) and never `reconcile_consumer_group_offsets`, the only producer of `RetireConsumerGroupOwners` (`core/server/src/partition_reconciler.rs:1108`, `core/server/src/partition_reconciler.rs:1216`), so the branch is reached only by its own unit test at `core/simulator/src/workload/state_checker.rs:786`. Comment 4263203892 asked for this branch and it was added, and commit 8d3154989 says the simulator now checks owner retirement, but no workload drives it. Owner retirement is one of the #4456 gaps this PR says it closes.
##########
core/server/src/http/forward.rs:
##########
@@ -332,92 +536,69 @@ fn record_relayed_floor(state: &HttpInner, user_id: u32, response: &Response) {
state.metadata_watermarks.record(user_id, applied);
}
-async fn forward_partition_or_pass(state: HttpState, request: Request, next: Next) -> Response {
- if !state.forward.active || request.headers().contains_key(FORWARDED_HEADER) {
- return next.run(request).await;
- }
- // A read that does not move consumer progress is answered by whichever
- // replica the caller reached, which is the point of reading from one. Only
- // an automatic commit needs a replica that can originate the offset
- // operation, so only that poll pays for the buffering and the credential
- // resolution below.
- if request.method() == Method::GET && !wants_auto_commit(request.uri().query()) {
- return next.run(request).await;
- }
+async fn attach_partition_request(state: HttpState, request: Request, next: Next) -> Response {
+ let received = Instant::now();
+ let hop = match partition_hop(request.headers()) {
+ Ok(hop) => hop,
+ Err(error) => return CustomError::from(error).into_response(),
+ };
+ // Verify-only, like the metadata relay: an unauthenticated caller gets no
+ // buffered body and no relay. The handler authenticates again.
let bearer = match bearer_token(request.headers()) {
Ok(bearer) => bearer,
Err(error) => return CustomError::from(error).into_response(),
};
if let Err(rejection) = resolve_credential(&state, bearer).await {
return rejection.into_response();
}
-
- let (parts, request_body) = request.into_parts();
+ let _guard = if matches!(hop, PartitionHop::Target(_)) {
Review Comment:
Since this admit, a Target hop takes a slot of the shared 128-slot `forward.in_flight` pool after authentication only, before any stream or topic permission check, and holds it for up to `PARTITION_REQUEST_DEADLINE` (10s) while `hold_for_frontier` waits for a caller-chosen `metadata_op` (`core/server/src/dispatch/reads.rs:249`). Both headers are taken from the client request with no peer check (`core/server/src/http/forward.rs:943-954`), and nothing bounds the frontier against the local commit. A user with no permissions can send 128 concurrent requests with `iggy-forwarded: 1` and `iggy-partition-route: 0,0,0,18446744073709551615,10000` to any partition path, and for 10s that node answers 503 `forward_busy` to every metadata and partition relay it starts (`core/server/src/http/forward.rs:417`, `core/server/src/http/forward.rs:499`) and to every partition relay other nodes send to it, repeatable without limit. Comment 4268027195 asked for this admission, but sharing the pool with
outbound relays turns one user's slow requests into a node-wide stall of HTTP forwarding for every user.
##########
core/consensus/src/plane_helpers.rs:
##########
@@ -903,6 +903,84 @@ where
.await
}
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum ViewLogProof {
+ Proven,
+ /// Fetch `op`; retained entries above `unproven_through` match the view.
+ Missing {
+ op: u64,
+ unproven_through: u64,
+ },
+ Divergent(u64),
+}
+
+/// Prove held entries against the view's hash chain. Canonical headers can
+/// bridge missing bodies; a gap with no header leaves its lower prefix unknown.
+#[must_use]
+pub fn view_log_proof(
+ commit_floor: u64,
+ canonical: &[PrepareHeader],
+ op_head: u64,
+ header_at: impl Fn(u64) -> Option<PrepareHeader>,
+) -> ViewLogProof {
+ let proven = locally_proven_commit(commit_floor, &header_at);
+ if op_head <= proven {
+ return ViewLogProof::Proven;
+ }
+ let Some(head) = canonical.first().filter(|head| head.op == op_head) else {
+ return ViewLogProof::Missing {
+ op: op_head,
+ unproven_through: op_head,
+ };
+ };
+ let mut expected = head.checksum;
+ let mut missing = None;
+ for op in (proven + 1..=op_head).rev() {
+ let header = if let Some(local) = header_at(op) {
+ local
+ } else if let Some(header) = canonical.iter().find(|header| header.op == op) {
+ missing.get_or_insert(op);
+ *header
+ } else {
+ return ViewLogProof::Missing {
+ op,
+ unproven_through: op,
+ };
+ };
+ if header.checksum == CHECKSUM_UNSEALED || header.checksum != expected {
+ return ViewLogProof::Divergent(op);
+ }
+ expected = header.parent;
+ }
+ missing.map_or(ViewLogProof::Proven, |op| ViewLogProof::Missing {
+ op,
+ unproven_through: 0,
+ })
+}
+
+fn locally_proven_commit(
+ commit_floor: u64,
+ header_at: &impl Fn(u64) -> Option<PrepareHeader>,
+) -> u64 {
+ let mut proven = commit_floor;
+ let mut op = commit_floor;
+ let mut parent = None;
+ while let Some(next_op) = op.checked_add(1) {
+ let Some(local) = header_at(next_op) else {
+ break;
+ };
+ if local.checksum == CHECKSUM_UNSEALED
+ || parent.is_some_and(|checksum| local.parent != checksum)
+ {
+ break;
+ }
+ proven = proven.max(local.commit.min(op));
Review Comment:
No unit test reaches the commit-stamp proof any more. The d54da7a3 rewrite of the proof tests dropped the durable commit stamp case and the unsealed chain case, and the replacement test at `core/consensus/src/plane_helpers.rs:2603` stamps every header `commit: 2`, equal to the floor, so this line never raises `proven`. The shard StartView tests do not cover it either, because `primary_prepares` (`core/shard/src/partition_io.rs:4815`) never commits and every stamp is 0. This branch decides which held entries count as committed before a StartView proof settles.
##########
core/shard/src/lib.rs:
##########
@@ -6214,145 +6425,71 @@ where
.await;
}
- /// Compare this replica's log against the headers the view decided, and drop or
- /// report where they disagree.
- ///
- /// Without this, divergence is silent and permanent: the replica acks with its
- /// own checksum, the primary rejects the ack, and journal repair skips an op
- /// it already has a header for.
- ///
- /// Both roles. A backup runs it against the `StartView` suffix it adopted, the
- /// primary-elect against the merged log before the coverage scan in
- /// [`Self::advance_pending_metadata_view`]. The merge does NOT reconcile the
- /// primary's log for it: nothing installs the merged headers into the journal,
- /// `RebuildPipeline` reads the pipeline back out of it, and `CommitJournal`
- /// applies whatever sits at each op up to the merged commit point.
- ///
- /// Runs on every adoption, parked suffix or not: an EMPTY `StartView` suffix
- /// (`commit == op`, the steady case) parks no pending log, yet adoption still
- /// drops the head under any journaled relics above it, and the primary's next
- /// prepare would collide with them in `append` and poison the journal. With
- /// no pending log the divergence scan has nothing to walk and only the
- /// above-head sweep applies, with the head read off the adopted sequencer.
- ///
- /// The split at the announced commit point is what matters. Above it a
- /// disagreement is ordinary, so the entry is dropped and the primary's
- /// retransmission refills the range. At or below it, this replica applied
- /// something the view says was different, which only state transfer fixes, so
- /// it is reported and left alone.
- ///
- /// Truncation uses `Journal::truncate_from`, not `drain`: `drain` advances
- /// `snapshot_op` past what it removed, marking ops that must stay refillable
- /// as evictable.
+ // The shard pump serializes these rewrites with append. The journal's drain
+ // guard excludes concurrent compaction, but does not exclude append itself.
#[allow(clippy::future_not_send)]
async fn reconcile_metadata_view_divergence(&self)
where
B: MessageBus,
MJ: JournalHandle,
<MJ as JournalHandle>::Target:
Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>,
- M: MetadataStm,
{
let metadata = self.plane.metadata();
- let Some(ref consensus) = metadata.consensus else {
+ let (Some(consensus), Some(journal)) = (&metadata.consensus, &metadata.journal) else {
return;
};
- let pending = consensus.pending_view_log();
- let Some(journal) = metadata.journal.as_ref() else {
+ if !consensus.view_log_unproven() {
return;
- };
-
- // Truncation is safe only above what this replica has *applied*, which is
- // not the view's commit point: `pending.commit_max` is the new primary's
- // number and a backup can sit above it. Splitting on the view's number
- // would drop already-executed ops with no rollback, and silently.
- let announced_commit = pending.as_ref().map_or(0, |pending| pending.commit_max);
- let applied_floor = consensus.commit_min();
-
- let mut repairable_from: Option<u64> = None;
- for canonical in pending.as_ref().map_or(&[][..], |pending| &pending.headers) {
- let Some(local) = usize::try_from(canonical.op)
- .ok()
- .and_then(|slot| journal.handle().header(slot))
- else {
- continue;
- };
- if header_is_view_entry(&local, canonical) {
- continue;
- }
- if canonical.op <= applied_floor {
- tracing::error!(
- shard = self.id,
- op = canonical.op,
- view = consensus.view(),
- commit_max = announced_commit,
- commit_min = consensus.commit_min(),
- local_checksum = local.checksum,
- canonical_checksum = canonical.checksum,
- "committed op {} disagrees with the view that just started; this replica \
- applied a different op as committed and cannot be reconciled by log repair",
- canonical.op
- );
- continue;
- }
- repairable_from = Some(repairable_from.map_or(canonical.op, |op| op.min(canonical.op)));
}
-
- // A suffix ABOVE the announced head is named by nobody, so the loop cannot
- // see it, and it is exactly the log that AGREES in-window (restart and
- // re-adopt), where `repairable_from` never arms. Adoption drops the head
- // under it, and the next prepare at `op_head + 1` then collides in `append`,
- // which refuses the slot even when the ops match. Floored at the applied
- // point too: an executed op is not rollback-able whatever the head says.
- // With no parked suffix the adopted sequencer IS the announced head.
+ let pending = consensus.pending_view_log();
let op_head = pending.as_ref().map_or_else(
|| consensus.sequencer().current_sequence(),
|pending| pending.op_head,
);
- let above_head = op_head.max(applied_floor) + 1;
- if journal
- .handle()
- .last_op()
- .is_some_and(|last_op| last_op >= above_head)
+ let journal = journal.handle();
+ let above_head = op_head.max(consensus.commit_min()).saturating_add(1);
+ if consensus.view_log_repair_op().is_none()
+ && journal.last_op().is_some_and(|last| last >= above_head)
{
- repairable_from = Some(repairable_from.map_or(above_head, |op| op.min(above_head)));
- }
-
- let Some(from_op) = repairable_from else {
- return;
- };
- // SERIALIZATION: the drain guard excludes `truncate_from` against `drain`,
- // NOT against an append; that is metadata's private `journal_gate`. It holds
- // by call-site placement, not construction: the pump is single-threaded, and
- // both callers run with a view change parked, so no submit is admitted and no
- // repair prepare is in flight for these ops. Routing shard-side journal
- // mutations through gate-taking metadata methods would make it structural.
- match journal.handle().truncate_from(from_op).await {
- Ok(removed) => {
- // The snapshot's head and commit point do not move when entries
- // are removed under them, so without this the next `DoViewChange`
- // would advertise the dropped headers and offer bodies this replica
- // cannot serve.
- consensus.note_journal_mutation();
- tracing::warn!(
- shard = self.id,
- from_op,
- removed,
- op_head,
- view = consensus.view(),
- "dropped {removed} uncommitted entries from op {from_op} that disagreed with \
- the view's log; the primary's retransmission refills the range"
- );
+ if let Err(error) = journal.truncate_from(above_head).await {
+ tracing::error!(shard = self.id, above_head, %error, "could not truncate superseded metadata suffix");
+ return;
}
- Err(error) => {
- tracing::error!(
- shard = self.id,
- from_op,
- %error,
- "could not drop the diverging uncommitted entries from op {from_op}; journal \
- repair skips ops it already holds a header for, so this replica will not \
- converge at those ops until it is restarted"
- );
+ consensus.note_journal_mutation();
+ }
+ loop {
+ match consensus::view_log_proof(
+ consensus.commit_min(),
+ pending.as_ref().map_or(&[][..], |pending| &pending.headers),
+ op_head,
+ |op| {
+ usize::try_from(op)
+ .ok()
+ .and_then(|slot| journal.header(slot).map(|header| *header))
+ },
+ ) {
+ consensus::ViewLogProof::Proven => {
+ consensus.settle_view_log_proof();
+ return;
+ }
+ consensus::ViewLogProof::Missing {
+ op,
+ unproven_through,
+ } => {
+ consensus.fence_view_log(op, unproven_through);
+ return;
+ }
+ consensus::ViewLogProof::Divergent(op) => {
+ consensus.fence_view_log(op, op);
+ // The canonical tail may be this replica's only copy. Remove
+ // the proven mismatch without cutting entries above a gap.
+ if let Err(error) = journal.remove_range(op..=op).await {
Review Comment:
This loop removes one divergent op per pass, and every `remove_range` runs `rewrite_wal`, which writes every live entry again and fsyncs the temp file and the parent directory (`core/journal/src/prepare_journal.rs:883`). A stale branch of K ops (up to `PIPELINE_PREPARE_QUEUE_MAX`, 32) now costs K full WAL rewrites and 2K fsyncs on the view change path, where the base cut it with one `truncate_from`. The partition plane has the same shape at `core/shard/src/lib.rs:11664`. Each divergent op defers its own `op..=op` transition, which drains persistence and rebuilds the resident index, and `ViewApplied` re-proves to find the next one, so K ops cost K IO round trips although `PendingPartitionTransition::Truncate` already takes a range.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "spetz (via GitHub)" <gi...@apache.org>.
spetz commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4269769204
##########
core/consensus/src/plane_helpers.rs:
##########
@@ -903,6 +903,84 @@ where
.await
}
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum ViewLogProof {
+ Proven,
+ /// Fetch `op`; retained entries above `unproven_through` match the view.
+ Missing {
+ op: u64,
+ unproven_through: u64,
+ },
+ Divergent(u64),
+}
+
+/// Prove held entries against the view's hash chain. Canonical headers can
+/// bridge missing bodies; a gap with no header leaves its lower prefix unknown.
+#[must_use]
+pub fn view_log_proof(
+ commit_floor: u64,
+ canonical: &[PrepareHeader],
+ op_head: u64,
+ header_at: impl Fn(u64) -> Option<PrepareHeader>,
+) -> ViewLogProof {
+ let proven = locally_proven_commit(commit_floor, &header_at);
+ if op_head <= proven {
+ return ViewLogProof::Proven;
+ }
+ let Some(head) = canonical.first().filter(|head| head.op == op_head) else {
+ return ViewLogProof::Missing {
+ op: op_head,
+ unproven_through: op_head,
+ };
+ };
+ let mut expected = head.checksum;
+ let mut missing = None;
+ for op in (proven + 1..=op_head).rev() {
+ let header = if let Some(local) = header_at(op) {
+ local
+ } else if let Some(header) = canonical.iter().find(|header| header.op == op) {
+ missing.get_or_insert(op);
+ *header
+ } else {
+ return ViewLogProof::Missing {
+ op,
+ unproven_through: op,
+ };
+ };
+ if header.checksum == CHECKSUM_UNSEALED || header.checksum != expected {
Review Comment:
`view_log_proof` reads a blank canonical slot (`dvc_blank`, checksum 0, which equals `CHECKSUM_UNSEALED`) as `Divergent(op)`, while `view_header_at` and `header_is_view_entry` treat the same slot as no evidence. `build_dvc_suffix` writes that blank at the commit point when the sender holds no header there (`core/shard/src/lib.rs:11553-11570`, whose warning already names this state), and both the probe answer (`core/consensus/src/impls.rs:3489-3500`) and the stale-heartbeat answer (`core/consensus/src/impls.rs:3805`) ship that suffix. A replica that installs a metadata snapshot at op S on an idle cluster and becomes primary before S+1 commits answers every probe with `[dvc_blank(S)]`, and a prober that lacks S (a node rebuilt with an empty data directory, or a second lagging backup with 5 replicas) gets `Divergent(S)`. When the blank is the head, line 936 seeds `expected` with 0, so even a correct local header reads as divergent. On the metadata plane `reconcile_metadata_view_diver
gence` then calls `remove_range(S..=S)` at `core/shard/src/lib.rs:6487`, which returns `Ok(0)` before its only await (`core/journal/src/prepare_journal.rs:997`), and `pending` is cloned once, so the loop never yields and freezes the shard thread at 100% CPU. A restart does not help until the primary commits a later op. On the partition plane every `ViewApplied` re-defers the same one-op truncation, and the replica stays unproven and sends no PrepareOk. Unit tests at this head reproduce all three: the proof returns `Divergent(4)` for an op the backup does not hold, the metadata reconcile does not finish within 20s while the same scenario with an empty suffix passes in 0.02s, and the partition truncation is re-deferred 256 times. Both repair ingest gates (`core/shard/src/lib.rs:5521` and `core/partitions/src/iggy_partition.rs:11650`) also compare repaired prepares against the blank checksum 0 and drop every copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): route partition requests to their primaries [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4260704827
##########
core/sdk/src/session_owner/lanes.rs:
##########
@@ -0,0 +1,605 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! A lane admits one mutation at a time, in arrival order. There is one lane
+//! for metadata and one for each partition history. A mutation whose outcome
+//! is unknown stays retained in its lane, and the lane admits nothing else
+//! until a resolve or an abandon frees it. So a retained operation is always
+//! the newest operation of its lane, and its replay can never age out.
+
+use super::SessionOwner;
+use crate::driver::PartitionKey;
+use crate::links::Frame;
+use crate::vsr::encode_request_header;
+use bytes::Bytes;
+use iggy_binary_protocol::consensus::{Operation, operation_for_code};
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
+use iggy_common::IggyError;
+use std::collections::HashMap;
+use std::sync::Arc;
+use tokio::sync::{Mutex, OwnedMutexGuard};
+use tokio::time::{Instant, timeout_at};
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub(crate) enum LaneKey {
+ Metadata,
+ /// The incarnation names one history of the partition for every spelling
+ /// of its stream and topic.
+ Partition {
+ incarnation: u64,
+ partition_id: u32,
+ },
+}
+
+impl LaneKey {
+ /// The lane of `code`. A write to a partition takes the lane of the
+ /// incarnation that its context carries. Reads, Register and Logout take
+ /// no lane.
+ pub(crate) fn of(code: u32, context: &PartitionContext, partition_id: u32) -> Option<Self> {
+ match operation_for_code(code) {
+ Operation::NonReplicated | Operation::Register | Operation::Logout => None,
+ operation if operation.is_partition() => Some(Self::Partition {
+ incarnation: context.incarnation,
+ partition_id,
+ }),
+ _ => Some(Self::Metadata),
+ }
+ }
+}
+
+pub(super) type Lanes = HashMap<LaneKey, Lane>;
+
+#[derive(Debug, Default)]
+pub(super) struct Lane {
+ gate: Arc<Mutex<()>>,
+ /// The tickets that wait for the gate, hold it or resume. The lane leaves
+ /// the table when none is left and nothing is retained.
+ tickets: usize,
+ retained: Option<RetainedOperation>,
+}
+
+impl Lane {
+ fn holds(&self, op: u64) -> bool {
+ self.retained
+ .as_ref()
+ .is_some_and(|retained| retained.op() == op)
+ }
+}
+
+/// A frozen mutation. Every attempt and every replay sends the same frame.
+#[derive(Debug, Clone)]
+pub(crate) struct RetainedOperation {
+ pub(crate) code: u32,
+ pub(crate) frame: Frame,
+ /// The partition of a partition write, `None` for any other request. A
+ /// replay looks up its route by this key.
+ pub(crate) partition: Option<PartitionKey>,
+}
+
+impl RetainedOperation {
+ pub(crate) fn op(&self) -> u64 {
+ self.frame.header.request
+ }
+}
+
+/// The right of one call to send in its lane. Dropping the ticket frees the
+/// lane, except that a frame which left the client without a definitive
+/// outcome stays retained.
+#[derive(Debug)]
+pub(crate) struct LaneTicket {
+ owner: Arc<SessionOwner>,
+ /// `None` for a call that takes no lane.
+ key: Option<LaneKey>,
+ guard: Option<OwnedMutexGuard<()>>,
+ operation: Option<RetainedOperation>,
+ fence: u64,
+ /// A resume replays an operation that its lane still retains.
+ resumed: bool,
+ sent: bool,
+ done: bool,
+}
+
+impl SessionOwner {
+ /// Waits for the lane of `key` in arrival order. A lane that retains an
+ /// operation fails at once with `LaneBlocked`, and a lane still busy at
+ /// `deadline` fails with `TransientNotAccepted`. Neither failure takes a
+ /// request id.
+ pub(crate) async fn enter_lane(
+ self: &Arc<Self>,
+ key: Option<LaneKey>,
+ deadline: Instant,
+ ) -> Result<LaneTicket, IggyError> {
+ let Some(key) = key else {
+ return Ok(LaneTicket::new(Arc::clone(self), None, false));
+ };
+ let (mut ticket, gate) = {
+ let mut lanes = self.lanes();
+ let lane = lanes.entry(key).or_default();
Review Comment:
nit: `entry(key).or_default()` allocates a new `Arc<Mutex<()>>` for every uncontended mutation because the lane is removed when idle; `route_cache.rs:203` also scans up to 4096 entries per insert at capacity. Cold today (driver unused).
##########
core/sdk/src/driver/route_cache.rs:
##########
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The routes of one client. A route has no age limit. It goes stale after
+//! a refusal or a failure, and a stale route keeps its view as the lowest
+//! view that its next lookup accepts. So a lookup never goes back to an
+//! older primary of the same partition history.
+
+use iggy_binary_protocol::requests::messages::GetPartitionRouteRequest;
+use iggy_binary_protocol::responses::messages::PartitionRouteResponse;
+use iggy_common::wire_conversions::identifier_to_wire;
+use iggy_common::{Identifier, IggyError, PartitionTarget};
+use std::collections::HashMap;
+
+/// The most routes that one client keeps. A new route beyond it evicts the
+/// oldest lookup.
+pub(crate) const MAX_CACHED_ROUTES: usize = 4096;
+
+/// A partition as its caller spelled it. A stream or topic named by number
+/// and the same one named by name are two keys.
+#[derive(Debug, Clone, PartialEq, Eq, Hash)]
+pub(crate) struct PartitionKey {
+ pub(crate) stream_id: Identifier,
+ pub(crate) topic_id: Identifier,
+ pub(crate) partition_id: u32,
+}
+
+impl From<&PartitionTarget<'_>> for PartitionKey {
+ fn from(target: &PartitionTarget<'_>) -> Self {
+ Self {
+ stream_id: target.stream_id.clone(),
+ topic_id: target.topic_id.clone(),
+ partition_id: target.partition_id,
+ }
+ }
+}
+
+impl TryFrom<&PartitionKey> for GetPartitionRouteRequest {
+ type Error = IggyError;
+
+ fn try_from(key: &PartitionKey) -> Result<Self, Self::Error> {
+ Ok(Self {
+ stream_id: identifier_to_wire(&key.stream_id)?,
+ topic_id: identifier_to_wire(&key.topic_id)?,
+ partition_id: key.partition_id,
+ })
+ }
+}
+
+#[derive(Debug)]
+struct Entry {
+ route: PartitionRouteResponse,
+ /// The session fence at the lookup. A route serves only that session.
+ fence: u64,
+ stale: bool,
+ /// The lowest view that the next lookup accepts.
+ floor: u32,
+ /// The replica that the next lookup asks first. The ids after it follow.
+ ask_first: Option<u8>,
+ /// Orders the entries for eviction.
+ lookup: u64,
+}
+
+#[derive(Debug, Default)]
+pub(crate) struct RouteCache {
+ entries: HashMap<PartitionKey, Entry>,
+ lookups: u64,
+}
+
+impl RouteCache {
+ /// The route of `key`, unless it is stale, from another session, or
+ /// older than the metadata `watermark` that the session saw.
+ pub(crate) fn usable(
+ &self,
+ key: &PartitionKey,
+ fence: u64,
+ watermark: u64,
+ ) -> Option<PartitionRouteResponse> {
+ self.entries
Review Comment:
nit: `usable` requires `route.metadata_op >= watermark`, and every metadata reply the session observes raises the watermark, so one metadata write invalidates every cached route and each partition is looked up again. Fencing only on commits that touched the route's topic would avoid the extra round trips once the driver is live.
##########
core/common/src/traits/binary_transport.rs:
##########
@@ -33,6 +33,27 @@ use iggy_binary_protocol::requests::users::login_register::BindSecret;
use iggy_binary_protocol::responses::users::LoginRegisterResponse;
use std::sync::Arc;
+/// Where a binary client sends one request.
Review Comment:
**warning**: `RequestTarget` / `PartitionTarget` (and `Client::resolve_operation` / `abandon_operation` in `client.rs:84`) become stable public API with only the unused driver behind them (`driver.rs` is `allow(dead_code)`), every shipped client answers `FeatureUnavailable`, and `PartitionTarget` has pub fields without `#[non_exhaustive]`. Suggest `pub(crate)` in the sdk for the targets and `#[doc(hidden)]` on the two methods until the driver is wired.
##########
core/common/src/traits/client.rs:
##########
@@ -79,4 +80,25 @@ pub trait Client:
///
/// HTTP keeps no connection, so it publishes no events.
async fn subscribe_events(&self) -> Receiver<DiagnosticEvent>;
+
+ /// Replays the retained operation `op` until it commits or `deadline`
+ /// passes. `op` is the request id that `OperationUncertain` or
+ /// `LaneBlocked` carries.
+ ///
+ /// `Ok(())` means that the operation committed exactly once. After
+ /// `OperationUncertain(op)` the outcome is still unknown, and the
Review Comment:
nit: after a session reset (logout or reconnect under a new session) retained operations are dropped, so a later `resolve_operation(op)` returns `OperationNotRetained` while the outcome is still unknown. Worth saying here that after a reset it means "unknown and unrecoverable", not "settled".
##########
core/binary_protocol/src/consensus/header.rs:
##########
@@ -49,6 +49,8 @@
//! headers are sealed on neither side, so SDKs are untouched.
Review Comment:
nit: the compatibility note stops at command 30 and doesn't mention `SessionRetirementProgress` (31), the new `ForwardCompletion` (32) or the `partition_incarnation` carved from reserved bytes in 11 control headers, and "SDKs are untouched" no longer holds for this PR. Related: the PR breaks the client wire, the replica wire and the Rust SDK API, so the title wants a `!` and the `breaking:client-to-replica-protocol`, `breaking:replica-to-replica-protocol` and `breaking:rust-sdk` labels.
##########
core/binary_protocol/src/requests/messages/get_partition_route.rs:
##########
@@ -20,16 +20,17 @@ use bytes::{BufMut, BytesMut};
use crate::codec::{WireDecode, WireEncode, read_u32_le};
use crate::{WireError, WireIdentifier};
-/// Partition context discovery authorized by send permission, without exposing topic details.
-/// The response is a `PartitionContext` captured before a new send is admitted.
+/// Route lookup for one partition, authorized by send or poll permission on its
Review Comment:
nit: "without exposing topic details" isn't accurate: the reply carries `partitions_count`, the view and the primary replica, and poll permission alone is enough to ask.
##########
core/shard/src/lib.rs:
##########
@@ -4511,17 +4581,38 @@ where
}
}
+ fn partition_incarnation_matches(
+ &self,
+ partition: &IggyPartition<B, SB>,
+ partition_incarnation: u64,
+ frame: &'static str,
+ ) -> bool {
+ if partition_incarnation == partition.created_revision() {
+ return true;
+ }
+ tracing::debug!(
Review Comment:
nit: incarnation-mismatch drops are logged only at `debug!`, so an operator who skipped the all-at-once upgrade just sees a partition group that can't elect. A rate-limited counter (labelled for incarnation 0) would make it visible without flooding logs.
##########
core/shard/src/lib.rs:
##########
@@ -4841,10 +4934,14 @@ where
&& let Some(from_op) =
partition_view_divergence_from(self.id, partition, pending.as_ref())
{
+ partition.consensus().settle_view_log_proof();
Review Comment:
nit: the view-log proof is settled here (and at :6083) before the deferred truncate runs. If `truncate_resident_from` errors after `transition.take()`, `fail_io_work(.., false)` can return without fatal and `commit_journal` is no longer fenced over the unproven entries. Latent today; settling in the `ViewApplied` arm after a successful truncate keeps one owner of the proof bit.
##########
core/partitions/src/io.rs:
##########
@@ -1295,7 +1331,14 @@ impl OffsetIoJob {
impl OffsetDeleteIoJob {
#[allow(clippy::future_not_send)]
pub(crate) async fn execute(self) -> Result<bool, IggyError> {
- delete_persisted_offset(&self.path).await
+ let removed = delete_persisted_offset(&self.path).await?;
+ // After the unlink only as defense in depth. WAL replay of the delete,
+ // boot dropping a marker beside any offset file and the transfer bound
+ // `emptied_at <= commit_op` carry the crash safety.
+ if let Some((marker, op)) = &self.emptied_marker {
Review Comment:
nit: the marker is described as defense in depth, but a `replace_file` failure after the authoritative unlink propagates and goes fatal. The comment (also at :1269) overclaims too: WAL replay of the delete never rewrites the marker. Suggest warn-and-drop plus a corrected comment.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -9919,6 +10172,8 @@ where
self.publish_consumer_offset_effect(*effect);
self.apply_consumer_offset_commit(effect.pending);
}
+ self.durable_consumer_offsets
Review Comment:
nit: `emptied_at` is written back unconditionally from the snapshot taken in `select_commit`; a no-ack store that publishes in between gets overwritten with the stale value (safe, only refuses more, but memory, disk and offer diverge). Write it only when this run's plan set `emptied_at`.
##########
core/metadata/src/stm/lifecycle.rs:
##########
@@ -616,7 +592,7 @@ impl StreamsSnapshot {
if let Some(pending) = assignment.pending
&& ((pending.owner.client_id == 0) != (pending.owner.session == 0)
|| pending.owner.generation == 0
- || pending.owner.generation > group.generation
+ || pending.owner.generation != pending.metadata_op
Review Comment:
nit: owner generation now means the metadata op and validation requires it, but `SNAPSHOT_FORMAT_VERSION` in `stm/snapshot.rs` stays 11. Old data dirs are already refused by the storage format bump, so this is hygiene: bump to 12 with a doc line so a mismatched snapshot fails as `UnsupportedFormatVersion` rather than `InvalidState`.
##########
core/common/src/types/cluster/node.rs:
##########
@@ -23,6 +23,10 @@ use std::fmt::Display;
#[derive(Debug, Deserialize, Serialize, Clone)]
pub struct ClusterNode {
+ /// The node's configured VSR replica id. Key nodes by it: the position in
Review Comment:
nit: the doc says to key nodes by `replica_id`, but the C++ FFI `ClusterNode` drops it (`foreign/cpp/src/type_conversion.rs:296`) and the CLI `get_cluster_metadata` table doesn't print it. It is also a required serde field without `#[serde(default)]`, so a new Rust HTTP client can't read an older server's JSON.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server)!: route partition requests to their primaries [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4506:
URL: https://github.com/apache/iggy/pull/4506#discussion_r4270146506
##########
core/server/src/dispatch/mod.rs:
##########
@@ -240,13 +284,153 @@ impl Drop for ActiveDrainSlot {
}
}
+/// The state that the drain task of one connection shares with the requests
+/// it spawned.
+///
+/// The drain takes the frames of the connection in arrival order and runs the
+/// prologue of each. Login, logout, `BIND_SESSION` and partition admission
+/// finish on the drain. Metadata commits run in tasks, but later non-PING
+/// frames wait in the queue until the metadata lane releases, before they
+/// capture session state or resolve a namespace. This orders dispatch, not local
+/// metadata catch-up: partition writes still resolve from this shard's STM
+/// without waiting for the connection watermark. PING can pass that queue.
+/// Stored-offset reads, Next/auto-commit polls and explicit offset writes share
+/// a lane per resolved namespace and consumer until their reply or wait expiry.
+/// Expiry releases only this connection's guard, not an admitted mutation, which
+/// may still commit after later progress. Independent requests can pass a queued
+/// conflict, but cannot pass a queued session or metadata barrier. A PING still waits
+/// behind a frame already suspended in admission or binding.
+///
+/// `capacity` caps those tasks at `message_bus.client_queue_capacity`, the
Review Comment:
Deferred to PR6 as reply-delivery correctness work, not just a cap benchmark. B7 and the plan require bounded reply ownership through flush, waiting for bounded delivery, or explicit connection failure on enqueue refusal; logging and dropping a response while leaving its request pending is insufficient. Acceptance includes slow readers with Java async pipelines above queue capacity, bounded retained memory, and a response or explicit connection failure for every pending call. The ClientDrain comment now states that task permits end at enqueue and do not reserve outbound slots.
##########
core/server/src/http/forward.rs:
##########
@@ -332,92 +536,69 @@ fn record_relayed_floor(state: &HttpInner, user_id: u32, response: &Response) {
state.metadata_watermarks.record(user_id, applied);
}
-async fn forward_partition_or_pass(state: HttpState, request: Request, next: Next) -> Response {
- if !state.forward.active || request.headers().contains_key(FORWARDED_HEADER) {
- return next.run(request).await;
- }
- // A read that does not move consumer progress is answered by whichever
- // replica the caller reached, which is the point of reading from one. Only
- // an automatic commit needs a replica that can originate the offset
- // operation, so only that poll pays for the buffering and the credential
- // resolution below.
- if request.method() == Method::GET && !wants_auto_commit(request.uri().query()) {
- return next.run(request).await;
- }
+async fn attach_partition_request(state: HttpState, request: Request, next: Next) -> Response {
+ let received = Instant::now();
+ let hop = match partition_hop(request.headers()) {
+ Ok(hop) => hop,
+ Err(error) => return CustomError::from(error).into_response(),
+ };
+ // Verify-only, like the metadata relay: an unauthenticated caller gets no
+ // buffered body and no relay. The handler authenticates again.
let bearer = match bearer_token(request.headers()) {
Ok(bearer) => bearer,
Err(error) => return CustomError::from(error).into_response(),
};
if let Err(rejection) = resolve_credential(&state, bearer).await {
return rejection.into_response();
}
-
- let (parts, request_body) = request.into_parts();
+ let _guard = if matches!(hop, PartitionHop::Target(_)) {
Review Comment:
Deferred to PR6 as an admission-isolation correctness gate. The earlier target cap bounds the number of holds but does not prevent a caller without topic permissions from starving the shared outbound pool. B7 and the plan require trusted relay provenance and authorization before accepting untrusted frontier holds, plus capacity isolation/reservation for legitimate forwarding. The regression must hold 128 spoofed forwarded requests and prove unrelated authorized traffic still progresses. Rejecting only u64::MAX or changing the cap alone will not close this item.
##########
core/shard/src/lib.rs:
##########
@@ -6214,145 +6425,71 @@ where
.await;
}
- /// Compare this replica's log against the headers the view decided, and drop or
- /// report where they disagree.
- ///
- /// Without this, divergence is silent and permanent: the replica acks with its
- /// own checksum, the primary rejects the ack, and journal repair skips an op
- /// it already has a header for.
- ///
- /// Both roles. A backup runs it against the `StartView` suffix it adopted, the
- /// primary-elect against the merged log before the coverage scan in
- /// [`Self::advance_pending_metadata_view`]. The merge does NOT reconcile the
- /// primary's log for it: nothing installs the merged headers into the journal,
- /// `RebuildPipeline` reads the pipeline back out of it, and `CommitJournal`
- /// applies whatever sits at each op up to the merged commit point.
- ///
- /// Runs on every adoption, parked suffix or not: an EMPTY `StartView` suffix
- /// (`commit == op`, the steady case) parks no pending log, yet adoption still
- /// drops the head under any journaled relics above it, and the primary's next
- /// prepare would collide with them in `append` and poison the journal. With
- /// no pending log the divergence scan has nothing to walk and only the
- /// above-head sweep applies, with the head read off the adopted sequencer.
- ///
- /// The split at the announced commit point is what matters. Above it a
- /// disagreement is ordinary, so the entry is dropped and the primary's
- /// retransmission refills the range. At or below it, this replica applied
- /// something the view says was different, which only state transfer fixes, so
- /// it is reported and left alone.
- ///
- /// Truncation uses `Journal::truncate_from`, not `drain`: `drain` advances
- /// `snapshot_op` past what it removed, marking ops that must stay refillable
- /// as evictable.
+ // The shard pump serializes these rewrites with append. The journal's drain
+ // guard excludes concurrent compaction, but does not exclude append itself.
#[allow(clippy::future_not_send)]
async fn reconcile_metadata_view_divergence(&self)
where
B: MessageBus,
MJ: JournalHandle,
<MJ as JournalHandle>::Target:
Journal<Entry = Message<PrepareHeader>, Header = PrepareHeader>,
- M: MetadataStm,
{
let metadata = self.plane.metadata();
- let Some(ref consensus) = metadata.consensus else {
+ let (Some(consensus), Some(journal)) = (&metadata.consensus, &metadata.journal) else {
return;
};
- let pending = consensus.pending_view_log();
- let Some(journal) = metadata.journal.as_ref() else {
+ if !consensus.view_log_unproven() {
return;
- };
-
- // Truncation is safe only above what this replica has *applied*, which is
- // not the view's commit point: `pending.commit_max` is the new primary's
- // number and a backup can sit above it. Splitting on the view's number
- // would drop already-executed ops with no rollback, and silently.
- let announced_commit = pending.as_ref().map_or(0, |pending| pending.commit_max);
- let applied_floor = consensus.commit_min();
-
- let mut repairable_from: Option<u64> = None;
- for canonical in pending.as_ref().map_or(&[][..], |pending| &pending.headers) {
- let Some(local) = usize::try_from(canonical.op)
- .ok()
- .and_then(|slot| journal.handle().header(slot))
- else {
- continue;
- };
- if header_is_view_entry(&local, canonical) {
- continue;
- }
- if canonical.op <= applied_floor {
- tracing::error!(
- shard = self.id,
- op = canonical.op,
- view = consensus.view(),
- commit_max = announced_commit,
- commit_min = consensus.commit_min(),
- local_checksum = local.checksum,
- canonical_checksum = canonical.checksum,
- "committed op {} disagrees with the view that just started; this replica \
- applied a different op as committed and cannot be reconciled by log repair",
- canonical.op
- );
- continue;
- }
- repairable_from = Some(repairable_from.map_or(canonical.op, |op| op.min(canonical.op)));
}
-
- // A suffix ABOVE the announced head is named by nobody, so the loop cannot
- // see it, and it is exactly the log that AGREES in-window (restart and
- // re-adopt), where `repairable_from` never arms. Adoption drops the head
- // under it, and the next prepare at `op_head + 1` then collides in `append`,
- // which refuses the slot even when the ops match. Floored at the applied
- // point too: an executed op is not rollback-able whatever the head says.
- // With no parked suffix the adopted sequencer IS the announced head.
+ let pending = consensus.pending_view_log();
let op_head = pending.as_ref().map_or_else(
|| consensus.sequencer().current_sequence(),
|pending| pending.op_head,
);
- let above_head = op_head.max(applied_floor) + 1;
- if journal
- .handle()
- .last_op()
- .is_some_and(|last_op| last_op >= above_head)
+ let journal = journal.handle();
+ let above_head = op_head.max(consensus.commit_min()).saturating_add(1);
+ if consensus.view_log_repair_op().is_none()
+ && journal.last_op().is_some_and(|last| last >= above_head)
{
- repairable_from = Some(repairable_from.map_or(above_head, |op| op.min(above_head)));
- }
-
- let Some(from_op) = repairable_from else {
- return;
- };
- // SERIALIZATION: the drain guard excludes `truncate_from` against `drain`,
- // NOT against an append; that is metadata's private `journal_gate`. It holds
- // by call-site placement, not construction: the pump is single-threaded, and
- // both callers run with a view change parked, so no submit is admitted and no
- // repair prepare is in flight for these ops. Routing shard-side journal
- // mutations through gate-taking metadata methods would make it structural.
- match journal.handle().truncate_from(from_op).await {
- Ok(removed) => {
- // The snapshot's head and commit point do not move when entries
- // are removed under them, so without this the next `DoViewChange`
- // would advertise the dropped headers and offer bodies this replica
- // cannot serve.
- consensus.note_journal_mutation();
- tracing::warn!(
- shard = self.id,
- from_op,
- removed,
- op_head,
- view = consensus.view(),
- "dropped {removed} uncommitted entries from op {from_op} that disagreed with \
- the view's log; the primary's retransmission refills the range"
- );
+ if let Err(error) = journal.truncate_from(above_head).await {
+ tracing::error!(shard = self.id, above_head, %error, "could not truncate superseded metadata suffix");
+ return;
}
- Err(error) => {
- tracing::error!(
- shard = self.id,
- from_op,
- %error,
- "could not drop the diverging uncommitted entries from op {from_op}; journal \
- repair skips ops it already holds a header for, so this replica will not \
- converge at those ops until it is restarted"
- );
+ consensus.note_journal_mutation();
+ }
+ loop {
+ match consensus::view_log_proof(
+ consensus.commit_min(),
+ pending.as_ref().map_or(&[][..], |pending| &pending.headers),
+ op_head,
+ |op| {
+ usize::try_from(op)
+ .ok()
+ .and_then(|slot| journal.header(slot).map(|header| *header))
+ },
+ ) {
+ consensus::ViewLogProof::Proven => {
+ consensus.settle_view_log_proof();
+ return;
+ }
+ consensus::ViewLogProof::Missing {
+ op,
+ unproven_through,
+ } => {
+ consensus.fence_view_log(op, unproven_through);
+ return;
+ }
+ consensus::ViewLogProof::Divergent(op) => {
+ consensus.fence_view_log(op, op);
+ // The canonical tail may be this replica's only copy. Remove
+ // the proven mismatch without cutting entries above a gap.
+ if let Err(error) = journal.remove_range(op..=op).await {
Review Comment:
Deferred to PR6 in B5's recovery gates. Batch only a proven contiguous divergent range into one atomic metadata rewrite or partition continuation, preserving unknown gaps, canonical tails, commit floors and retry after failed I/O. Acceptance requires rewrite/fsync counts and interrupted-publication/restart tests. Dropping unknown entries to reduce the number of rewrites is not an acceptable shortcut.
##########
core/simulator/src/workload/state_checker.rs:
##########
@@ -265,6 +390,85 @@ impl StateChecker {
}
}
+/// Membership is metadata state; the partition's ordered install barrier is
+/// what fences a departed member's later offset writes.
+fn check_partition_owner(
+ owners: &mut PartitionOwners,
+ header: &PrepareHeader,
+ body: &[u8],
+ seed: u64,
+) {
+ if header.operation == Operation::InstallConsumerGroupOwner {
+ let install = InstallConsumerGroupOwnerRequest::decode_from(body)
+ .expect("a committed owner installation decodes");
+ if !owners
+ .retirement
+ .as_ref()
+ .is_some_and(|catalog| catalog.retires(install.group_id))
+ {
+ owners.installed.insert(install.group_id, install.owner);
+ }
+ return;
+ }
+ if header.operation == Operation::RetireConsumerGroupOwners {
Review Comment:
Deferred to PR6. The consolidated simulator gates, B6 and the plan now require calling the actual reconcile_consumer_group_offsets producer in a group-retirement/recreation workload. Acceptance requires nonzero RetireConsumerGroupOwners delivery and retirement comparisons, vacuity failure when either is absent, and an end-to-end mutation that the strict retirement oracle catches. The PR description now calls the existing retirement check unit coverage only; session revocation and generation-refusal counters are separate evidence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org