You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by hg...@apache.org on 2026/07/01 23:39:59 UTC
(iggy) branch http-listener created (now b50ff02de)
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a change to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
at b50ff02de test(server-ng): add HTTP data-plane integration tests
This branch includes the following new commits:
new 95bb74d8e feat(message_bus): add in-process reply routing for HTTP listener
new 5c3762dd3 feat(server-ng): add HTTP listener with login and session binding
new 6dcc84125 feat(server-ng): route HTTP create-stream through VSR consensus
new b3422d95e feat(server-ng): add HTTP stream/topic update, delete, purge routes
new 466077bb4 feat(server-ng): add HTTP create-topic route
new f5f435210 feat(server-ng): gate HTTP write routes to root user
new 5cef9e79b feat(server-ng): add HTTP user-management write routes
new 0633cdb60 feat(server-ng): add HTTP personal-access-token create/delete routes
new 5e10f8c42 feat(server-ng): add HTTP consumer-group create/delete routes
new 949b3d20a feat(server-ng): add HTTP read routes
new 33e396759 feat(server-ng): add HTTP cluster-metadata route and view header
new 4976321aa feat(server-ng): redirect linearizable follower reads to primary
new 547e760d8 feat(server-ng): add in-process reply slots for HTTP data plane
new cdd4bea14 feat(server-ng): add HTTP produce route
new 0a9284514 feat(server-ng): add HTTP poll and consumer-offset read routes
new 0944a78c6 feat(server-ng): add HTTP consumer-offset write routes
new 8285c8511 feat(server-ng): add HTTP admission control
new aa3132db9 fix(server-ng): make HTTP write id-advance cancellation-safe
new b50ff02de test(server-ng): add HTTP data-plane integration tests
The 19 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
(iggy) 07/19: feat(server-ng): add HTTP user-management write routes
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 5cef9e79bbdba3106c6ef777cdb584d8bdabd016
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 15:32:59 2026 +0200
feat(server-ng): add HTTP user-management write routes
Add the five user-management write routes to server-ng's shard-0 HTTP
listener: POST /users, PUT/DELETE /users/{id}, PUT /users/{id}/password,
and PUT /users/{id}/permissions. This brings the REST control plane to
parity with the legacy server's user surface. CreateUser renders the
committed reply as UserInfoDetails (201 Created); the rest answer 204.
CreateUser and ChangePassword carry plaintext passwords in their JSON
bodies. They are hashed on shard 0 through the shared submit_write helper
via maybe_rewrite_user_password_request before the request enters
consensus, mirroring the TCP dispatch path. Because Argon2 salting is
non-deterministic, hashing must happen once on the primary so every
replica receives the identical hash; a raw password must never be
persisted or replicated. Placing the rewrite in submit_write makes the
invariant hold for every current and future user write.
All five routes flow through submit_write and therefore inherit its
interim root-only authorization gate until server-ng grows an RBAC
permissioner; a non-root credential is rejected with 403.
---
core/server-ng/src/http.rs | 168 ++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 166 insertions(+), 2 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index be7dcd9b5..719219c2f 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -43,23 +43,32 @@ use iggy_binary_protocol::requests::streams::{
use iggy_binary_protocol::requests::topics::{
CreateTopicRequest, DeleteTopicRequest, PurgeTopicRequest, UpdateTopicRequest,
};
+use iggy_binary_protocol::requests::users::{
+ ChangePasswordRequest, CreateUserRequest, DeleteUserRequest, UpdatePermissionsRequest,
+ UpdateUserRequest,
+};
use iggy_binary_protocol::responses::streams::get_stream::GetStreamResponse;
use iggy_binary_protocol::responses::topics::get_topic::GetTopicResponse;
+use iggy_binary_protocol::responses::users::get_user::UserDetailsResponse;
use iggy_binary_protocol::version::IGGY_PROTOCOL_VERSION;
use iggy_binary_protocol::{
GenericHeader, Operation, RequestHeader, WireDecode, WireEncode, WireName,
};
+use iggy_common::change_password::ChangePassword;
use iggy_common::create_stream::CreateStream;
use iggy_common::create_topic::CreateTopic;
+use iggy_common::create_user::CreateUser;
use iggy_common::defaults::DEFAULT_ROOT_USER_ID;
use iggy_common::login_user::LoginUser;
use iggy_common::login_with_personal_access_token::LoginWithPersonalAccessToken;
+use iggy_common::update_permissions::UpdatePermissions;
use iggy_common::update_stream::UpdateStream;
use iggy_common::update_topic::UpdateTopic;
-use iggy_common::wire_conversions::identifier_to_wire;
+use iggy_common::update_user::UpdateUser;
+use iggy_common::wire_conversions::{identifier_to_wire, permissions_to_wire};
use iggy_common::{
Identifier, IdentityInfo, IggyError, IggyTimestamp, StreamDetails, TokenInfo, TopicDetails,
- Validatable,
+ UserInfoDetails, Validatable,
};
use message_bus::client_listener;
use secrecy::ExposeSecret;
@@ -76,6 +85,7 @@ use crate::http::extractor::Authenticated;
use crate::http::jwt::JwtManager;
use crate::login_register::LoginRegisterError;
use crate::server_error::ServerNgError;
+use crate::users::maybe_rewrite_user_password_request;
/// `GET /ping` response body, matching the legacy HTTP server's health probe.
const PONG: &str = "pong";
@@ -353,6 +363,10 @@ fn router(state: HttpState) -> Router {
post(login_with_personal_access_token),
)
.route("/users/me", get(get_me))
+ .route("/users", post(create_user))
+ .route("/users/{user_id}", put(update_user).delete(delete_user))
+ .route("/users/{user_id}/password", put(change_password))
+ .route("/users/{user_id}/permissions", put(update_permissions))
.route("/streams", post(create_stream))
.route(
"/streams/{stream_id}",
@@ -620,6 +634,142 @@ async fn purge_topic(
Ok(StatusCode::NO_CONTENT)
}
+/// `POST /users`: create a user and render the committed reply as the same
+/// `UserInfoDetails` JSON the legacy server returns.
+///
+/// The plaintext password rides the JSON body; [`submit_write`] hashes it on
+/// shard 0 before the request enters consensus (see
+/// [`maybe_rewrite_user_password_request`]), so no plaintext is ever replicated.
+async fn create_user(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Json(command): Json<CreateUser>,
+) -> Result<Json<UserInfoDetails>, WriteError> {
+ // Rejects empty/oversized username or password before any consensus work.
+ command.validate().map_err(WriteError::Rejected)?;
+ let request = CreateUserRequest {
+ username: WireName::new(&command.username)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidUsername))?,
+ password: command.password.expose_secret().to_string(),
+ status: command.status.as_code(),
+ permissions: command.permissions.as_ref().map(permissions_to_wire),
+ };
+ let body = request.to_bytes();
+ let payload = SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::CreateUser,
+ &body,
+ ))
+ .await?;
+ Ok(Json(decode_user_details(&payload)?))
+}
+
+/// `PUT /users/{user_id}`: update a user's username and/or status. Returns 204.
+async fn update_user(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path(user_id): Path<String>,
+ Json(command): Json<UpdateUser>,
+) -> Result<StatusCode, WriteError> {
+ let user_id = Identifier::from_str_value(&user_id).map_err(WriteError::Rejected)?;
+ // Rejects an oversized replacement username; a no-op when username is absent.
+ command.validate().map_err(WriteError::Rejected)?;
+ let request = UpdateUserRequest {
+ user_id: identifier_to_wire(&user_id).map_err(WriteError::Rejected)?,
+ username: command
+ .username
+ .as_deref()
+ .map(WireName::new)
+ .transpose()
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidUsername))?,
+ status: command.status.map(|status| status.as_code()),
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::UpdateUser,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// `DELETE /users/{user_id}`: delete a user. Returns 204.
+async fn delete_user(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path(user_id): Path<String>,
+) -> Result<StatusCode, WriteError> {
+ let user_id = Identifier::from_str_value(&user_id).map_err(WriteError::Rejected)?;
+ let request = DeleteUserRequest {
+ user_id: identifier_to_wire(&user_id).map_err(WriteError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::DeleteUser,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// `PUT /users/{user_id}/password`: change a user's password. Returns 204.
+///
+/// Like [`create_user`], the new password rides the JSON body in plaintext and
+/// is hashed by [`submit_write`] on shard 0 before replication.
+async fn change_password(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path(user_id): Path<String>,
+ Json(command): Json<ChangePassword>,
+) -> Result<StatusCode, WriteError> {
+ let user_id = Identifier::from_str_value(&user_id).map_err(WriteError::Rejected)?;
+ // Rejects empty/oversized current or new password before any consensus work.
+ command.validate().map_err(WriteError::Rejected)?;
+ let request = ChangePasswordRequest {
+ user_id: identifier_to_wire(&user_id).map_err(WriteError::Rejected)?,
+ current_password: command.current_password.expose_secret().to_string(),
+ new_password: command.new_password.expose_secret().to_string(),
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::ChangePassword,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// `PUT /users/{user_id}/permissions`: replace a user's permissions. Returns 204.
+async fn update_permissions(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path(user_id): Path<String>,
+ Json(command): Json<UpdatePermissions>,
+) -> Result<StatusCode, WriteError> {
+ let user_id = Identifier::from_str_value(&user_id).map_err(WriteError::Rejected)?;
+ command.validate().map_err(WriteError::Rejected)?;
+ let request = UpdatePermissionsRequest {
+ user_id: identifier_to_wire(&user_id).map_err(WriteError::Rejected)?,
+ permissions: command.permissions.as_ref().map(permissions_to_wire),
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::UpdatePermissions,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
/// Run one authenticated control-plane write end to end and return the committed
/// reply's typed payload. Shared by every write route: `create_stream` decodes
/// the payload into an entity, the update/delete/purge routes ignore it (it is
@@ -658,6 +808,11 @@ async fn submit_write(
*next_request_id,
body,
);
+ // Hash raw passwords on shard 0 before replication (CreateUser /
+ // ChangePassword); a no-op for every other operation. Mirrors the TCP
+ // dispatch path so plaintext never enters consensus. On decode failure the
+ // gate is released without advancing, leaving the request id free to retry.
+ let message = maybe_rewrite_user_password_request(message).map_err(WriteError::Rejected)?;
let reply = submit_client_request_on_owner(&state.shard, message).await;
let Some(reply) = reply else {
return Err(WriteError::Unavailable);
@@ -732,6 +887,15 @@ fn decode_topic_details(payload: &[u8]) -> Result<TopicDetails, WriteError> {
TopicDetails::try_from(response).map_err(WriteError::Rejected)
}
+/// Decode the `UserDetailsResponse` payload of a committed create-user reply into
+/// `UserInfoDetails`. `payload` is the slice past the result section that
+/// [`submit_write`] already validated as a success.
+fn decode_user_details(payload: &[u8]) -> Result<UserInfoDetails, WriteError> {
+ let response = UserDetailsResponse::decode_from(payload)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidCommand))?;
+ UserInfoDetails::try_from(response).map_err(WriteError::Rejected)
+}
+
/// Map an eviction frame to the same typed [`IggyError`] the SDK's
/// `decode_eviction` produces, so an HTTP caller sees the identical status a TCP
/// caller would (session-terminal reasons render as 401 -> re-authenticate).
(iggy) 15/19: feat(server-ng): add HTTP poll and consumer-offset read routes
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 0a9284514462a5f508d8dcfc5733b9769c45a1e1
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 23:54:42 2026 +0200
feat(server-ng): add HTTP poll and consumer-offset read routes
Poll and get-consumer-offset are non-replicated reads served in band:
no VSR session, no reply slot, no consensus. Both routes run the same
resolution the TCP dispatch uses (its decode helpers split into shared
resolver cores), read the owning shard over the mesh, and decode the
re-encoded legacy wire body with the SDK's own
PolledMessages::from_bytes, so HTTP and TCP answers stay
field-identical.
HTTP reads carry VSR client id 0, which can never name a group member,
so a group-kind poll fences closed to the re-sync sentinel exactly like
a stale TCP member; the pinned query shape cannot express a group poll
anyway (Consumer.kind is serde-skipped), matching legacy HTTP. A
missing stored offset renders as the legacy 404, and an unanswered
mesh read as 504 through the shared gateway-timeout body.
---
core/server-ng/src/dispatch.rs | 28 ++-
core/server-ng/src/http.rs | 401 ++++++++++++++++++++++++++++++++++++++++-
2 files changed, 416 insertions(+), 13 deletions(-)
diff --git a/core/server-ng/src/dispatch.rs b/core/server-ng/src/dispatch.rs
index f942ebefb..50b3a1e65 100644
--- a/core/server-ng/src/dispatch.rs
+++ b/core/server-ng/src/dispatch.rs
@@ -1305,15 +1305,27 @@ fn empty_polled_messages_body(partition_id: u32) -> Bytes {
Bytes::from(body)
}
-type DecodedPollRequest = (IggyNamespace, u32, PollingConsumer, PollingArgs);
+pub(crate) type DecodedPollRequest = (IggyNamespace, u32, PollingConsumer, PollingArgs);
-#[allow(clippy::cast_possible_truncation)]
fn decode_poll_request(
shard: &Rc<ServerNgShard>,
request: &Message<RequestHeader>,
) -> Result<DecodedPollRequest, IggyError> {
let wire = PollMessagesRequest::decode_from(request_body(request))
.map_err(|_| IggyError::InvalidCommand)?;
+ resolve_poll_request(shard, &wire, request.header().client)
+}
+
+/// Resolve a decoded poll request into its owning-shard read: namespace,
+/// partition, polling consumer, and args. Shared by the TCP dispatch (client
+/// id = the connection's bound VSR client) and the HTTP route (client id 0,
+/// which fences group polls closed).
+#[allow(clippy::cast_possible_truncation)]
+pub(crate) fn resolve_poll_request(
+ shard: &Rc<ServerNgShard>,
+ wire: &PollMessagesRequest,
+ client_id: u128,
+) -> Result<DecodedPollRequest, IggyError> {
let strategy = polling_strategy_from_wire(&wire.strategy)?;
let args = PollingArgs::new(strategy, wire.count, wire.auto_commit);
@@ -1325,7 +1337,6 @@ fn decode_poll_request(
// both use, so `next()` reads back the offset it just committed.
if wire.consumer.kind == KIND_CONSUMER_GROUP {
let partition_id = wire.partition_id.ok_or(IggyError::InvalidIdentifier)?;
- let client_id = request.header().client;
let group_id = shard
.plane
.metadata()
@@ -1369,6 +1380,17 @@ fn decode_consumer_offset_request(
) -> Result<(IggyNamespace, u32, PollingConsumer), IggyError> {
let wire = GetConsumerOffsetRequest::decode_from(request_body(request))
.map_err(|_| IggyError::InvalidCommand)?;
+ resolve_consumer_offset_request(shard, &wire)
+}
+
+/// Resolve a decoded consumer-offset read into its owning-shard read:
+/// namespace, partition, and polling consumer. Shared by the TCP dispatch and
+/// the HTTP route; needs no client id because offset reads are not fenced
+/// (any client may read a group's offset, member or not).
+pub(crate) fn resolve_consumer_offset_request(
+ shard: &Rc<ServerNgShard>,
+ wire: &GetConsumerOffsetRequest,
+) -> Result<(IggyNamespace, u32, PollingConsumer), IggyError> {
let partition_id = wire.partition_id.ok_or(IggyError::InvalidIdentifier)?;
let namespace =
resolve_partition_namespace(shard, &wire.stream_id, &wire.topic_id, Some(partition_id))?;
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index ab7c6053a..c3a406d93 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -46,11 +46,16 @@ use iggy_binary_protocol::codes::{
use iggy_binary_protocol::consensus::{
Command2, EvictionHeader, EvictionReason, HEADER_SIZE, result_code, result_section_len,
};
+use iggy_binary_protocol::primitives::consumer::WireConsumer;
+use iggy_binary_protocol::primitives::polling_strategy::WirePollingStrategy;
use iggy_binary_protocol::requests::consumer_groups::{
CreateConsumerGroupRequest, DeleteConsumerGroupRequest, GetConsumerGroupRequest,
GetConsumerGroupsRequest,
};
-use iggy_binary_protocol::requests::messages::{RawMessage, SendMessagesEncoder};
+use iggy_binary_protocol::requests::consumer_offsets::GetConsumerOffsetRequest;
+use iggy_binary_protocol::requests::messages::{
+ PollMessagesRequest, RawMessage, SendMessagesEncoder,
+};
use iggy_binary_protocol::requests::personal_access_tokens::{
CreatePersonalAccessTokenRequest, DeletePersonalAccessTokenRequest,
};
@@ -91,8 +96,10 @@ use iggy_common::create_stream::CreateStream;
use iggy_common::create_topic::CreateTopic;
use iggy_common::create_user::CreateUser;
use iggy_common::defaults::DEFAULT_ROOT_USER_ID;
+use iggy_common::get_consumer_offset::GetConsumerOffset;
use iggy_common::login_user::LoginUser;
use iggy_common::login_with_personal_access_token::LoginWithPersonalAccessToken;
+use iggy_common::poll_messages::DEFAULT_PARTITION_ID;
use iggy_common::update_permissions::UpdatePermissions;
use iggy_common::update_stream::UpdateStream;
use iggy_common::update_topic::UpdateTopic;
@@ -103,8 +110,9 @@ use iggy_common::wire_conversions::{
};
use iggy_common::{
ClientInfo, ClientInfoDetails, ClusterMetadata, ClusterNode, ClusterNodeRole,
- ClusterNodeStatus, ConsumerGroup, ConsumerGroupDetails, Identifier, IdentityInfo, IggyError,
- IggyMessageView, IggyTimestamp, RawPersonalAccessToken, SendMessages, Stats, Stream,
+ ClusterNodeStatus, ConsumerGroup, ConsumerGroupDetails, ConsumerOffsetInfo, Identifier,
+ IdentityInfo, IggyError, IggyMessageView, IggyTimestamp, PollMessages, PolledMessages,
+ RESYNC_REQUIRED_PARTITION_SENTINEL, RawPersonalAccessToken, SendMessages, Stats, Stream,
StreamDetails, TokenInfo, Topic, TopicDetails, TransportEndpoints, UserInfo, UserInfoDetails,
Validatable,
};
@@ -115,21 +123,23 @@ use send_wrapper::SendWrapper;
use serde::Deserialize;
use server::http::error::{CustomError, ErrorResponse};
use server_common::Message;
+use shard::{PartitionRead, PartitionReadReply};
use tokio::sync::Mutex;
use tracing::{error, info, warn};
use crate::auth::{verify_login_credentials, verify_pat_credentials};
use crate::bootstrap::ServerNgShard;
use crate::dispatch::{
- dispatch_partition_request, submit_client_request_on_owner, submit_register_on_owner,
+ dispatch_partition_request, resolve_consumer_offset_request, resolve_poll_request,
+ submit_client_request_on_owner, submit_register_on_owner,
};
use crate::http::extractor::{Authenticated, Identity};
use crate::http::jwt::JwtManager;
use crate::login_register::LoginRegisterError;
use crate::pat::rewrite_pat_request_for_user;
use crate::responses::{
- NonReplicatedResponse, build_non_replicated_response, build_raw_pat_reply,
- connected_client_to_response,
+ NonReplicatedResponse, build_non_replicated_response, build_polled_messages_body,
+ build_raw_pat_reply, connected_client_to_response,
};
use crate::server_error::ServerNgError;
use crate::users::maybe_rewrite_user_password_request;
@@ -156,6 +166,15 @@ const FIRST_REQUEST_ID: u64 = 1;
/// wait gave up, so the server never retries on the caller's behalf.
const PRODUCE_REPLY_TIMEOUT: Duration = Duration::from_secs(10);
+/// VSR client id stamped on HTTP data-plane reads (poll / consumer-offset).
+/// HTTP reads never Register a VSR client, and shard-0 client ids are minted
+/// from 1, so 0 can never name a live consumer-group member: a group-kind
+/// poll fences closed with `ConsumerGroupPartitionNotOwned` and answers the
+/// re-sync sentinel empty poll - the same failure a stale TCP member sees.
+/// Legacy HTTP polls with client id 0 for the same reason (no persistent
+/// sessions, so no group membership).
+const HTTP_READ_CLIENT_ID: u128 = 0;
+
/// Response header attesting what durability a produce response proves:
/// [`DURABILITY_REPLICATED_MEMORY`] after an awaited quorum commit,
/// [`DURABILITY_NONE`] for a `?ack=none` fire-and-forget.
@@ -314,12 +333,23 @@ impl IntoResponse for ProduceError {
/// 504 body for a produce whose commit outcome is unknown. Shaped like every
/// other HTTP error (`ErrorResponse`) so clients parse one error schema.
fn produce_timeout_response() -> Response {
+ gateway_timeout_response(
+ "produce_timeout",
+ "produce was not acknowledged in time; the write may still commit",
+ )
+}
+
+/// Shared 504 rendering for an in-band request the partition plane did not
+/// answer in time, shaped like every other HTTP error (`ErrorResponse`) so
+/// clients parse one error schema. Consumed by the produce reply wait and the
+/// partition reads ([`ReadError::Timeout`]).
+fn gateway_timeout_response(code: &str, reason: &str) -> Response {
(
StatusCode::GATEWAY_TIMEOUT,
Json(ErrorResponse {
id: StatusCode::GATEWAY_TIMEOUT.as_u16().into(),
- code: "produce_timeout".to_owned(),
- reason: "produce was not acknowledged in time; the write may still commit".to_owned(),
+ code: code.to_owned(),
+ reason: reason.to_owned(),
field: None,
}),
)
@@ -404,6 +434,10 @@ enum ReadError {
/// query, so the caller re-issues the read against the leader (see
/// [`primary_redirect_response`]).
RedirectToPrimary(String),
+ /// A partition read (poll / consumer-offset) got no reply from the owning
+ /// shard within the mesh budget. 504 like a produce timeout: the outcome is
+ /// unknown (the abandoned read may still be running), so the caller retries.
+ Timeout,
}
impl IntoResponse for ReadError {
@@ -415,6 +449,10 @@ impl IntoResponse for ReadError {
Self::NotFound => CustomError::ResourceNotFound.into_response(),
Self::NotPrimary => not_primary_response(),
Self::RedirectToPrimary(location) => primary_redirect_response(&location),
+ Self::Timeout => gateway_timeout_response(
+ "partition_read_timeout",
+ "the partition owner did not answer the read in time; retry",
+ ),
}
}
}
@@ -750,7 +788,11 @@ fn router(state: HttpState) -> Router {
)
.route(
"/streams/{stream_id}/topics/{topic_id}/messages",
- post(send_messages),
+ get(poll_messages).post(send_messages),
+ )
+ .route(
+ "/streams/{stream_id}/topics/{topic_id}/consumer-offsets",
+ get(get_consumer_offset),
)
.route(
"/streams/{stream_id}/topics/{topic_id}/consumer-groups",
@@ -1312,6 +1354,112 @@ async fn purge_topic(
Ok(StatusCode::NO_CONTENT)
}
+/// `GET /streams/{stream_id}/topics/{topic_id}/messages`: poll a batch of
+/// messages as the same `PolledMessages` JSON the legacy server returns. The
+/// query is the same flattened `PollMessages` shape the legacy server accepts
+/// (`consumer_id`, `partition_id`, strategy `kind`+`value`, `count`,
+/// `auto_commit`); stream and topic come from the path.
+///
+/// A non-replicated read served in band: the same resolution the TCP dispatch
+/// runs ([`resolve_poll_request`]), then a mesh read on the owning shard and a
+/// re-encode of the stored batches into the legacy wire body, decoded here by
+/// the SDK's own [`PolledMessages::from_bytes`] so the JSON is field-identical
+/// to a TCP poll. `auto_commit` rides [`resolve_poll_request`]'s args and is
+/// honored by the owning shard's poll plan; no extra HTTP work.
+async fn poll_messages(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+ Query(query): Query<PollMessages>,
+ Query(consistency): Query<ConsistencyQuery>,
+) -> Result<Json<PolledMessages>, ReadError> {
+ authorize_read(&state, &identity, consistency.consistency)?;
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(ReadError::Rejected)?;
+ let wire = poll_wire_request(&stream_id, &topic_id, &query).map_err(ReadError::Rejected)?;
+ let (namespace, partition_id, consumer, args) =
+ match resolve_poll_request(&state.shard, &wire, HTTP_READ_CLIENT_ID) {
+ Ok(decoded) => decoded,
+ // TCP parity: a fenced group poll answers 200 with the re-sync
+ // sentinel partition, not an error (see [`HTTP_READ_CLIENT_ID`]).
+ Err(IggyError::ConsumerGroupPartitionNotOwned(..)) => {
+ return Ok(Json(resync_required_polled_messages()));
+ }
+ // The remaining resolver failures are STM lookups that came up
+ // empty (unknown stream, topic, partition, or consumer group), so
+ // they render as the legacy 404 body.
+ Err(_) => return Err(ReadError::NotFound),
+ };
+ let reply = SendWrapper::new(
+ state
+ .shard
+ .partition_read(namespace, PartitionRead::Poll { consumer, args }),
+ )
+ .await;
+ match reply {
+ Some(PartitionReadReply::Poll {
+ fragments,
+ current_offset,
+ }) => {
+ let body = build_polled_messages_body(partition_id, current_offset, fragments)
+ .map_err(ReadError::Rejected)?;
+ Ok(Json(
+ PolledMessages::from_bytes(body).map_err(ReadError::Rejected)?,
+ ))
+ }
+ Some(PartitionReadReply::NotFound) => Err(ReadError::NotFound),
+ Some(_) => Err(ReadError::Rejected(IggyError::InvalidCommand)),
+ None => Err(ReadError::Timeout),
+ }
+}
+
+/// `GET /streams/{stream_id}/topics/{topic_id}/consumer-offsets`: fetch a
+/// consumer's stored offset as the same `ConsumerOffsetInfo` JSON the legacy
+/// server returns. The query is the same flattened `GetConsumerOffset` shape
+/// the legacy server accepts (`consumer_id`, optional `partition_id`).
+///
+/// A non-replicated read served in band, mirroring [`poll_messages`]. A
+/// missing offset (never stored, or the partition unknown to its owner) is
+/// the legacy 404: the TCP path replies an empty body the SDK decodes as
+/// `None`, and the legacy HTTP server renders that `None` as
+/// `CustomError::ResourceNotFound`.
+async fn get_consumer_offset(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+ Query(query): Query<GetConsumerOffset>,
+ Query(consistency): Query<ConsistencyQuery>,
+) -> Result<Json<ConsumerOffsetInfo>, ReadError> {
+ authorize_read(&state, &identity, consistency.consistency)?;
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(ReadError::Rejected)?;
+ let wire =
+ consumer_offset_wire_request(&stream_id, &topic_id, &query).map_err(ReadError::Rejected)?;
+ let (namespace, partition_id, consumer) =
+ resolve_consumer_offset_request(&state.shard, &wire).map_err(|_| ReadError::NotFound)?;
+ let reply = SendWrapper::new(
+ state
+ .shard
+ .partition_read(namespace, PartitionRead::ConsumerOffset { consumer }),
+ )
+ .await;
+ match reply {
+ Some(PartitionReadReply::ConsumerOffset {
+ stored: Some(stored_offset),
+ current_offset,
+ }) => Ok(Json(ConsumerOffsetInfo {
+ partition_id,
+ current_offset,
+ stored_offset,
+ })),
+ Some(
+ PartitionReadReply::ConsumerOffset { stored: None, .. } | PartitionReadReply::NotFound,
+ ) => Err(ReadError::NotFound),
+ Some(_) => Err(ReadError::Rejected(IggyError::InvalidCommand)),
+ None => Err(ReadError::Timeout),
+ }
+}
+
/// `POST /streams/{stream_id}/topics/{topic_id}/messages`: produce a batch of
/// messages to a topic. The JSON body is the same `SendMessages` shape the
/// legacy server accepts (partitioning + base64 messages); stream and topic
@@ -1757,6 +1905,65 @@ fn encode_send_messages(
Ok(buf.freeze())
}
+/// Map a validated HTTP poll query onto the wire `PollMessagesRequest` the
+/// shared TCP resolver consumes, so both transports resolve one request shape.
+/// The query's consumer kind is structurally always `Consumer`
+/// (`Consumer::kind` is `#[serde(skip)]` - the flattened `kind` param names
+/// the polling strategy), matching the legacy HTTP server.
+fn poll_wire_request(
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ query: &PollMessages,
+) -> Result<PollMessagesRequest, IggyError> {
+ Ok(PollMessagesRequest {
+ consumer: WireConsumer {
+ kind: query.consumer.kind.as_code(),
+ id: identifier_to_wire(&query.consumer.id)?,
+ },
+ stream_id: identifier_to_wire(stream_id)?,
+ topic_id: identifier_to_wire(topic_id)?,
+ partition_id: query.partition_id,
+ strategy: WirePollingStrategy {
+ kind: query.strategy.kind.as_code(),
+ value: query.strategy.value,
+ },
+ count: query.count,
+ auto_commit: query.auto_commit,
+ })
+}
+
+/// Map a validated HTTP consumer-offset query onto the wire
+/// `GetConsumerOffsetRequest` the shared TCP resolver consumes. An omitted
+/// `partition_id` defaults to partition 0, matching the legacy server's
+/// `resolve_consumer_with_partition_id` (`partition_id.unwrap_or(0)`).
+fn consumer_offset_wire_request(
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ query: &GetConsumerOffset,
+) -> Result<GetConsumerOffsetRequest, IggyError> {
+ Ok(GetConsumerOffsetRequest {
+ consumer: WireConsumer {
+ kind: query.consumer.kind.as_code(),
+ id: identifier_to_wire(&query.consumer.id)?,
+ },
+ stream_id: identifier_to_wire(stream_id)?,
+ topic_id: identifier_to_wire(topic_id)?,
+ partition_id: Some(query.partition_id.unwrap_or(DEFAULT_PARTITION_ID)),
+ })
+}
+
+/// The empty `PolledMessages` a fenced consumer-group poll answers, carrying
+/// the re-sync sentinel in `partition_id` exactly as the TCP dispatch replies
+/// it, so an SDK re-syncs its assignment instead of reading end-of-partition.
+const fn resync_required_polled_messages() -> PolledMessages {
+ PolledMessages {
+ partition_id: RESYNC_REQUIRED_PARTITION_SENTINEL,
+ current_offset: 0,
+ count: 0,
+ messages: Vec::new(),
+ }
+}
+
/// Run one awaited produce end to end: install the reply slot, dispatch into
/// the partition plane, and wait (bounded) for the committed reply.
///
@@ -2166,9 +2373,20 @@ const fn login_error_to_iggy(error: &LoginRegisterError) -> IggyError {
#[cfg(test)]
mod tests {
+ use axum::http::Uri;
use configs::cluster::TransportPorts;
use iggy_binary_protocol::PrepareHeader;
- use iggy_common::{IggyMessage, IggyMessagesBatch, Partitioning, PartitioningKind};
+ use iggy_common::{
+ Consumer, ConsumerKind, IggyMessage, IggyMessagesBatch, Partitioning, PartitioningKind,
+ PollingKind, PollingStrategy,
+ };
+ use partitions::{Fragment, PollFragments};
+ use server_common::MESSAGE_ALIGN;
+ use server_common::iobuf::Owned;
+ use server_common::send_messages2::{
+ COMMAND_HEADER_SIZE, IggyMessage2, IggyMessage2Header, IggyMessages2, SendMessages2Header,
+ SendMessages2Owned,
+ };
use super::*;
use crate::responses::build_empty_reply;
@@ -2319,6 +2537,169 @@ mod tests {
assert!(command.validate().is_err());
}
+ // -- poll / consumer-offset query parsing and wire mapping --
+
+ #[test]
+ fn poll_query_parses_with_documented_defaults() {
+ let uri: Uri = "/streams/1/topics/1/messages?consumer_id=42"
+ .parse()
+ .expect("valid uri");
+ let Query(query) = Query::<PollMessages>::try_from_uri(&uri).expect("parses");
+ assert_eq!(query.consumer.kind, ConsumerKind::Consumer);
+ assert_eq!(
+ query.consumer.id,
+ Identifier::numeric(42).expect("valid id")
+ );
+ assert_eq!(query.partition_id, Some(0));
+ assert_eq!(query.strategy, PollingStrategy::offset(0));
+ assert_eq!(query.count, 10);
+ assert!(!query.auto_commit);
+ }
+
+ #[test]
+ fn poll_query_parses_explicit_strategy_count_and_tolerates_consistency_param() {
+ let uri: Uri = "/x?consumer_id=app&partition_id=3&kind=timestamp&value=42&count=5&auto_commit=true&consistency=linearizable"
+ .parse()
+ .expect("valid uri");
+ let Query(query) = Query::<PollMessages>::try_from_uri(&uri).expect("parses");
+ assert_eq!(
+ query.consumer.id,
+ Identifier::named("app").expect("valid id")
+ );
+ assert_eq!(query.partition_id, Some(3));
+ assert_eq!(
+ query.strategy,
+ PollingStrategy::timestamp(IggyTimestamp::from(42))
+ );
+ assert_eq!(query.count, 5);
+ assert!(query.auto_commit);
+ let Query(consistency) = Query::<ConsistencyQuery>::try_from_uri(&uri).expect("parses");
+ assert!(consistency.consistency == Consistency::Linearizable);
+ }
+
+ #[test]
+ fn poll_wire_request_maps_query_onto_tcp_request_shape() {
+ let stream_id = Identifier::from_str_value("orders").expect("valid stream id");
+ let topic_id = Identifier::from_str_value("1").expect("valid topic id");
+ let query = PollMessages {
+ consumer: Consumer {
+ kind: ConsumerKind::Consumer,
+ id: Identifier::numeric(9).expect("valid id"),
+ },
+ partition_id: Some(4),
+ strategy: PollingStrategy::next(),
+ count: 25,
+ auto_commit: true,
+ ..Default::default()
+ };
+ let wire = poll_wire_request(&stream_id, &topic_id, &query).expect("maps");
+ assert_eq!(wire.consumer.kind, 1);
+ assert_eq!(
+ wire.consumer.id,
+ identifier_to_wire(&query.consumer.id).unwrap()
+ );
+ assert_eq!(wire.stream_id, identifier_to_wire(&stream_id).unwrap());
+ assert_eq!(wire.topic_id, identifier_to_wire(&topic_id).unwrap());
+ assert_eq!(wire.partition_id, Some(4));
+ assert_eq!(wire.strategy.kind, PollingKind::Next.as_code());
+ assert_eq!(wire.strategy.value, 0);
+ assert_eq!(wire.count, 25);
+ assert!(wire.auto_commit);
+ }
+
+ #[test]
+ fn consumer_offset_wire_request_defaults_omitted_partition_to_zero() {
+ let stream_id = Identifier::numeric(1).expect("valid stream id");
+ let topic_id = Identifier::numeric(1).expect("valid topic id");
+ let query = GetConsumerOffset {
+ consumer: Consumer::new(Identifier::numeric(7).expect("valid id")),
+ partition_id: None,
+ };
+ let wire = consumer_offset_wire_request(&stream_id, &topic_id, &query).expect("maps");
+ assert_eq!(wire.partition_id, Some(DEFAULT_PARTITION_ID));
+ assert_eq!(wire.consumer.kind, 1);
+ }
+
+ // -- polled messages body decode --
+
+ /// Wrap one stored `SendMessages2` batch (`[256B header][blob]`) as the
+ /// poll fragment the owning shard replies, the shape
+ /// `build_polled_messages_body` consumes.
+ fn fragment_from_stored_batch(header: &SendMessages2Header, blob: &[u8]) -> PollFragments {
+ let mut buffer = Owned::<MESSAGE_ALIGN>::zeroed(COMMAND_HEADER_SIZE + blob.len());
+ header.encode_into(buffer.as_mut_slice());
+ buffer.as_mut_slice()[COMMAND_HEADER_SIZE..].copy_from_slice(blob);
+ let mut fragments = PollFragments::new();
+ fragments.push(Fragment::whole(buffer.into()));
+ fragments
+ }
+
+ /// Round-trip the poll route's encode/decode seam: the store's own batch
+ /// writer (`SendMessages2Owned::from_messages`) is the encoder oracle,
+ /// `build_polled_messages_body` re-encodes to the legacy wire body, and
+ /// the SDK's `PolledMessages::from_bytes` must read back every field.
+ #[test]
+ fn polled_messages_body_decodes_into_common_polled_messages() {
+ let mut messages = IggyMessages2::with_capacity(2);
+ messages.push(IggyMessage2 {
+ header: IggyMessage2Header {
+ id: 7,
+ origin_timestamp: 1_000,
+ ..Default::default()
+ },
+ payload: Bytes::from_static(b"first"),
+ user_headers: None,
+ });
+ messages.push(IggyMessage2 {
+ header: IggyMessage2Header {
+ id: 8,
+ origin_timestamp: 1_050,
+ ..Default::default()
+ },
+ payload: Bytes::from_static(b"second"),
+ user_headers: Some(Bytes::from_static(b"raw-header-bytes")),
+ });
+ let namespace = server_common::sharding::IggyNamespace::new(0, 0, 3);
+ let stored =
+ SendMessages2Owned::from_messages(namespace, &messages).expect("encodes stored batch");
+ // The store stamps these on append; `from_messages` leaves them zero.
+ let mut header = stored.header;
+ header.base_offset = 41;
+ header.base_timestamp = 999_999;
+
+ let body =
+ build_polled_messages_body(3, 42, fragment_from_stored_batch(&header, &stored.blob))
+ .expect("re-encodes wire body");
+ let polled = PolledMessages::from_bytes(body).expect("decodes as the SDK does");
+
+ assert_eq!(polled.partition_id, 3);
+ assert_eq!(polled.current_offset, 42);
+ assert_eq!(polled.count, 2);
+ assert_eq!(polled.messages.len(), 2);
+ assert_eq!(polled.messages[0].header.id, 7);
+ assert_eq!(polled.messages[0].header.offset, 41);
+ assert_eq!(polled.messages[0].header.timestamp, 999_999);
+ assert_eq!(polled.messages[0].header.origin_timestamp, 1_000);
+ assert_eq!(polled.messages[0].payload.as_ref(), b"first");
+ assert!(polled.messages[0].user_headers.is_none());
+ assert_eq!(polled.messages[1].header.id, 8);
+ assert_eq!(polled.messages[1].header.offset, 42);
+ assert_eq!(polled.messages[1].header.origin_timestamp, 1_050);
+ assert_eq!(polled.messages[1].payload.as_ref(), b"second");
+ assert_eq!(
+ polled.messages[1].user_headers.as_deref(),
+ Some(b"raw-header-bytes".as_ref())
+ );
+ }
+
+ #[test]
+ fn resync_required_polled_messages_carries_sentinel_partition() {
+ let polled = resync_required_polled_messages();
+ assert_eq!(polled.partition_id, RESYNC_REQUIRED_PARTITION_SENTINEL);
+ assert_eq!(polled.count, 0);
+ assert!(polled.messages.is_empty());
+ }
+
// -- classify_partition_reply --
fn frozen(reply: Message<iggy_binary_protocol::ReplyHeader>) -> BusMessage {
(iggy) 04/19: feat(server-ng): add HTTP stream/topic update, delete, purge routes
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit b3422d95ef55f10da1c95b99bad5e16ce3f2bcad
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 13:27:38 2026 +0200
feat(server-ng): add HTTP stream/topic update, delete, purge routes
Extend the server-ng HTTP control plane beyond create-stream with the
data-adjacent stream and topic writes, all sharing one reply path.
Adds PUT/DELETE/DELETE-purge for streams and PUT/DELETE/DELETE-purge
for topics, each parsing the dual-addressed {stream_id}/{topic_id}
path segments into wire identifiers and returning 204 on the empty
committed reply, matching the legacy HTTP status.
The eviction-aware reply interpretation, per-session gate advance, and
result-code discrimination are factored into a single submit_write
helper and a generic request-message builder now shared by every
write route including create-stream, so the logic lives once.
---
core/server-ng/src/http.rs | 311 ++++++++++++++++++++++++++++++++++++---------
1 file changed, 249 insertions(+), 62 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index a2de4b11c..f2f52868b 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -27,16 +27,22 @@ use std::collections::HashMap;
use std::net::SocketAddr;
use std::rc::Rc;
-use axum::extract::State;
+use axum::extract::{Path, State};
use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
-use axum::routing::{get, post};
+use axum::routing::{delete, get, post, put};
use axum::{Json, Router};
+use bytes::Bytes;
use configs::http::HttpJwtConfig;
use iggy_binary_protocol::consensus::{
Command2, EvictionHeader, EvictionReason, HEADER_SIZE, result_code, result_section_len,
};
-use iggy_binary_protocol::requests::streams::CreateStreamRequest;
+use iggy_binary_protocol::requests::streams::{
+ CreateStreamRequest, DeleteStreamRequest, PurgeStreamRequest, UpdateStreamRequest,
+};
+use iggy_binary_protocol::requests::topics::{
+ DeleteTopicRequest, PurgeTopicRequest, UpdateTopicRequest,
+};
use iggy_binary_protocol::responses::streams::get_stream::GetStreamResponse;
use iggy_binary_protocol::version::IGGY_PROTOCOL_VERSION;
use iggy_binary_protocol::{
@@ -45,7 +51,12 @@ use iggy_binary_protocol::{
use iggy_common::create_stream::CreateStream;
use iggy_common::login_user::LoginUser;
use iggy_common::login_with_personal_access_token::LoginWithPersonalAccessToken;
-use iggy_common::{IdentityInfo, IggyError, IggyTimestamp, StreamDetails, TokenInfo};
+use iggy_common::update_stream::UpdateStream;
+use iggy_common::update_topic::UpdateTopic;
+use iggy_common::wire_conversions::identifier_to_wire;
+use iggy_common::{
+ Identifier, IdentityInfo, IggyError, IggyTimestamp, StreamDetails, TokenInfo, Validatable,
+};
use message_bus::client_listener;
use secrecy::ExposeSecret;
use send_wrapper::SendWrapper;
@@ -340,6 +351,19 @@ fn router(state: HttpState) -> Router {
)
.route("/users/me", get(get_me))
.route("/streams", post(create_stream))
+ .route(
+ "/streams/{stream_id}",
+ put(update_stream).delete(delete_stream),
+ )
+ .route("/streams/{stream_id}/purge", delete(purge_stream))
+ .route(
+ "/streams/{stream_id}/topics/{topic_id}",
+ put(update_topic).delete(delete_topic),
+ )
+ .route(
+ "/streams/{stream_id}/topics/{topic_id}/purge",
+ delete(purge_topic),
+ )
.with_state(state)
}
@@ -383,8 +407,7 @@ async fn get_me(identity: Authenticated) -> Json<IdentityInfo> {
})
}
-/// `POST /streams`: create a stream through the per-session gate and the
-/// in-process VSR dispatch spine, then render the committed reply as the same
+/// `POST /streams`: create a stream and render the committed reply as the same
/// `StreamDetails` JSON the legacy server returns.
///
/// The accepted body is name-only (`{"name": ...}`), matching the legacy
@@ -399,94 +422,258 @@ async fn create_stream(
name: WireName::new(command.name)
.map_err(|_| WriteError::Rejected(IggyError::InvalidStreamName))?,
};
+ let body = request.to_bytes();
+ let payload = SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::CreateStream,
+ &body,
+ ))
+ .await?;
+ Ok(Json(decode_stream_details(&payload)?))
+}
+
+/// `PUT /streams/{stream_id}`: rename a stream. A committed write returns 204
+/// with no body, matching the legacy server.
+async fn update_stream(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path(stream_id): Path<String>,
+ Json(command): Json<UpdateStream>,
+) -> Result<StatusCode, WriteError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(WriteError::Rejected)?;
+ let request = UpdateStreamRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(WriteError::Rejected)?,
+ name: WireName::new(command.name)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidStreamName))?,
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::UpdateStream,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// `DELETE /streams/{stream_id}`: delete a stream. Returns 204 on commit.
+async fn delete_stream(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path(stream_id): Path<String>,
+) -> Result<StatusCode, WriteError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(WriteError::Rejected)?;
+ let request = DeleteStreamRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(WriteError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::DeleteStream,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// `DELETE /streams/{stream_id}/purge`: drop a stream's messages. Returns 204.
+async fn purge_stream(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path(stream_id): Path<String>,
+) -> Result<StatusCode, WriteError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(WriteError::Rejected)?;
+ let request = PurgeStreamRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(WriteError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::PurgeStream,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// `PUT /streams/{stream_id}/topics/{topic_id}`: update a topic. Returns 204.
+async fn update_topic(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+ Json(command): Json<UpdateTopic>,
+) -> Result<StatusCode, WriteError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(WriteError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(WriteError::Rejected)?;
+ // Also rejects replication_factor == Some(0), which `WireName` cannot see.
+ command.validate().map_err(WriteError::Rejected)?;
+ let request = UpdateTopicRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(WriteError::Rejected)?,
+ topic_id: identifier_to_wire(&topic_id).map_err(WriteError::Rejected)?,
+ compression_algorithm: command.compression_algorithm.as_code(),
+ message_expiry: command.message_expiry.into(),
+ max_topic_size: command.max_topic_size.into(),
+ replication_factor: command.replication_factor.unwrap_or(0),
+ name: WireName::new(command.name)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidTopicName))?,
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::UpdateTopic,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// `DELETE /streams/{stream_id}/topics/{topic_id}`: delete a topic. Returns 204.
+async fn delete_topic(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+) -> Result<StatusCode, WriteError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(WriteError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(WriteError::Rejected)?;
+ let request = DeleteTopicRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(WriteError::Rejected)?,
+ topic_id: identifier_to_wire(&topic_id).map_err(WriteError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::DeleteTopic,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// `DELETE /streams/{stream_id}/topics/{topic_id}/purge`: drop a topic's
+/// messages. Returns 204.
+async fn purge_topic(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+) -> Result<StatusCode, WriteError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(WriteError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(WriteError::Rejected)?;
+ let request = PurgeTopicRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(WriteError::Rejected)?,
+ topic_id: identifier_to_wire(&topic_id).map_err(WriteError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::PurgeTopic,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
- let client_id = identity.session.client_id;
- let session_id = identity.session.session;
-
- // Serialize this session's writes behind the gate and take the next VSR
- // request id. The gate is held across the commit so ids reach the primary
- // strictly in order, and it advances only once the request commits (a
- // `Reply` frame): a transient failure or an eviction leaves the id free for
- // the caller's retry, which the depth-1 dedup requires (the next accepted
- // request must be `committed_request + 1`, so a consumed-but-uncommitted id
- // would wedge the session on `RequestGap`).
- let mut next_request_id = identity.session.gate.lock().await;
- let message = build_create_stream_message(client_id, session_id, *next_request_id, &request);
- // `submit_client_request_on_owner` is `!Send` (Rc-based); `SendWrapper`
- // bridges it onto axum's `Send` handler future, sound because the whole
- // handler runs pinned to shard 0's compio thread.
- let reply = SendWrapper::new(submit_client_request_on_owner(&state.shard, message)).await;
+/// Run one authenticated control-plane write end to end and return the committed
+/// reply's typed payload. Shared by every write route: `create_stream` decodes
+/// the payload into an entity, the update/delete/purge routes ignore it (it is
+/// empty) and answer 204.
+///
+/// Serializes this session's writes behind its gate and holds it across the
+/// submit so request ids reach the primary strictly in order. The gate advances
+/// only on a committed `Reply`: a transient failure or an eviction leaves the id
+/// free for the caller's retry, which the depth-1 dedup requires (the next
+/// accepted request must be `committed + 1`, so a consumed-but-uncommitted id
+/// would wedge the session on `RequestGap`).
+///
+/// Reply discrimination mirrors the SDK's `split_metadata_result`: the body
+/// leads with a result section (`Some(0)` is success followed by the typed
+/// payload; a nonzero first result is a committed business rejection carrying an
+/// `IggyError` code). An eviction frame carries no result section and means the
+/// session is dead (re-authenticate); a missing or short result section is a
+/// malformed committed reply, mapped to an error rather than a false success.
+async fn submit_write(
+ state: &HttpInner,
+ session: &HttpSession,
+ operation: Operation,
+ body: &[u8],
+) -> Result<Bytes, WriteError> {
+ let mut next_request_id = session.gate.lock().await;
+ let message = build_request_message(
+ operation,
+ session.client_id,
+ session.session,
+ *next_request_id,
+ body,
+ );
+ let reply = submit_client_request_on_owner(&state.shard, message).await;
let Some(reply) = reply else {
return Err(WriteError::Unavailable);
};
- // Peek the reply frame before decoding: an eviction carries no result
- // section and means the session is dead (re-authenticate), so unlike a
- // committed `Reply` it must not consume this request id.
match reply.header().command {
Command2::Reply => {
- // Committed: the cluster consumed this id, so advance the gate before
- // rendering, whether or not the body decodes.
*next_request_id += 1;
drop(next_request_id);
- Ok(Json(decode_committed_stream(&reply)?))
+ let size = reply.header().size as usize;
+ let reply_body = reply.as_slice().get(HEADER_SIZE..size).unwrap_or_default();
+ match result_code(reply_body) {
+ Some(0) => {
+ let payload_start = result_section_len(reply_body)
+ .ok_or(WriteError::Rejected(IggyError::InvalidCommand))?;
+ let payload = reply_body
+ .get(payload_start..)
+ .ok_or(WriteError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Bytes::copy_from_slice(payload))
+ }
+ Some(code) => Err(WriteError::Rejected(IggyError::from_code(code))),
+ None => Err(WriteError::Rejected(IggyError::InvalidCommand)),
+ }
}
Command2::Eviction => Err(WriteError::Rejected(eviction_error(&reply))),
_ => Err(WriteError::Rejected(IggyError::InvalidCommand)),
}
}
-/// Build the `Message<RequestHeader>` for a create-stream by filling a zeroed
-/// `#[repr(C)]` header, mirroring `wire::rewrite_request_body` and the
-/// partition reconciler's prepare builder.
-fn build_create_stream_message(
+/// Build a `Message<RequestHeader>` for a control-plane write by filling a zeroed
+/// `#[repr(C)]` header, mirroring `wire::rewrite_request_body` and the partition
+/// reconciler's prepare builder. `body` is the already-encoded wire request,
+/// copied in after the header.
+fn build_request_message(
+ operation: Operation,
client_id: u128,
session_id: u64,
request_id: u64,
- request: &CreateStreamRequest,
+ body: &[u8],
) -> Message<RequestHeader> {
- let body = request.to_bytes();
let total = HEADER_SIZE + body.len();
let mut message = Message::<RequestHeader>::new(total);
- message.as_mut_slice()[HEADER_SIZE..].copy_from_slice(&body);
+ message.as_mut_slice()[HEADER_SIZE..].copy_from_slice(body);
let header = bytemuck::checked::try_from_bytes_mut::<RequestHeader>(
&mut message.as_mut_slice()[..HEADER_SIZE],
)
.expect("zeroed bytes form a valid RequestHeader");
header.command = Command2::Request;
- header.operation = Operation::CreateStream;
+ header.operation = operation;
header.client = client_id;
header.session = session_id;
header.request = request_id;
- header.size = u32::try_from(total).expect("create-stream message size fits u32");
+ header.size = u32::try_from(total).expect("control-plane message size fits u32");
message
}
-/// Decode a committed `Reply` frame's body into `StreamDetails`, mirroring the
-/// SDK's `split_metadata_result`. A metadata reply body leads with a result
-/// section (`[count][{index, result}]*`): `count == 0` is success followed by
-/// the `GetStreamResponse` payload, and a nonzero first `result` is a committed
-/// business rejection carrying an `IggyError` code (e.g. duplicate name). The
-/// caller has already confirmed the frame is a `Reply`, so a missing result
-/// section is a malformed committed reply, mapped to an error rather than a 200
-/// with a stale body.
-fn decode_committed_stream(reply: &Message<GenericHeader>) -> Result<StreamDetails, WriteError> {
- let size = reply.header().size as usize;
- let body = reply.as_slice().get(HEADER_SIZE..size).unwrap_or_default();
- match result_code(body) {
- Some(0) => {
- let payload_start =
- result_section_len(body).ok_or(WriteError::Rejected(IggyError::InvalidCommand))?;
- let payload = body
- .get(payload_start..)
- .ok_or(WriteError::Rejected(IggyError::InvalidCommand))?;
- let response = GetStreamResponse::decode_from(payload)
- .map_err(|_| WriteError::Rejected(IggyError::InvalidCommand))?;
- StreamDetails::try_from(response).map_err(WriteError::Rejected)
- }
- Some(code) => Err(WriteError::Rejected(IggyError::from_code(code))),
- None => Err(WriteError::Rejected(IggyError::InvalidCommand)),
- }
+/// Decode the `GetStreamResponse` payload of a committed create-stream reply into
+/// `StreamDetails`. `payload` is the slice past the result section that
+/// [`submit_write`] already validated as a success.
+fn decode_stream_details(payload: &[u8]) -> Result<StreamDetails, WriteError> {
+ let response = GetStreamResponse::decode_from(payload)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidCommand))?;
+ StreamDetails::try_from(response).map_err(WriteError::Rejected)
}
/// Map an eviction frame to the same typed [`IggyError`] the SDK's
(iggy) 12/19: feat(server-ng): redirect linearizable follower reads to primary
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 4976321aa20cec679933b70a634d517a49e5a0f5
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 20:07:58 2026 +0200
feat(server-ng): redirect linearizable follower reads to primary
When a linearizable read hits a non-primary node, return 307 with a Location pointing at the current primary's HTTP address resolved from the cluster roster, instead of 503, so a client can follow it to the leader. Falls back to 503 when the primary address is unresolvable.
Dormant on a single node (always primary) and client-inert until an SDK follows the redirect; covered by unit tests on the pure location builder, including IPv6 bracketing.
---
core/server-ng/src/http.rs | 232 +++++++++++++++++++++++++----------
core/server-ng/src/http/extractor.rs | 15 ++-
2 files changed, 181 insertions(+), 66 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index c21fe5c85..7d1799f74 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -24,10 +24,11 @@ mod jwt;
use std::cell::RefCell;
use std::collections::HashMap;
-use std::net::SocketAddr;
+use std::net::{IpAddr, SocketAddr};
use std::rc::Rc;
use axum::extract::{Path, Query, State};
+use axum::http::header::LOCATION;
use axum::http::{HeaderName, HeaderValue, StatusCode};
use axum::middleware::map_response;
use axum::response::{IntoResponse, Response};
@@ -279,10 +280,15 @@ enum ReadError {
Rejected(IggyError),
/// Requested entity is absent -> 404 with the legacy not-found body.
NotFound,
- /// A linearizable read reached a follower. Transient from the caller's view
- /// -> 503, retryable; becomes a 307-to-leader once the roster + redirect
- /// land (see [`not_primary_response`]).
+ /// A linearizable read reached a follower and the primary's HTTP address was
+ /// not resolvable from the roster. Fail-closed 503, retryable against the
+ /// leader (see [`not_primary_response`]).
NotPrimary,
+ /// A linearizable read reached a follower and the current VSR primary's HTTP
+ /// address resolved: 307 to that address carrying the original path and
+ /// query, so the caller re-issues the read against the leader (see
+ /// [`primary_redirect_response`]).
+ RedirectToPrimary(String),
}
impl IntoResponse for ReadError {
@@ -293,15 +299,17 @@ impl IntoResponse for ReadError {
// the legacy server's `CustomError::ResourceNotFound` does.
Self::NotFound => CustomError::ResourceNotFound.into_response(),
Self::NotPrimary => not_primary_response(),
+ Self::RedirectToPrimary(location) => primary_redirect_response(&location),
}
}
}
-/// The 503 body for a linearizable read that reached a follower. No node roster
-/// exists at the HTTP layer yet (F7), so this node cannot 307-redirect to the
-/// primary nor emit a `Location` (F3); the caller must retry, ideally against
-/// the leader. Once both land this branch becomes a 307. Rendered as an
-/// `ErrorResponse` so the body shape matches every other HTTP error.
+/// The 503 fail-closed body for a linearizable read that reached a follower
+/// whose primary HTTP address could not be resolved (absent consensus, a roster
+/// with no node at the primary index, or a port-less node). The resolvable case
+/// is a 307 via [`primary_redirect_response`] instead. Rendered as an
+/// `ErrorResponse` so the body shape matches every other HTTP error; the caller
+/// retries against the leader.
fn not_primary_response() -> Response {
(
StatusCode::SERVICE_UNAVAILABLE,
@@ -315,6 +323,45 @@ fn not_primary_response() -> Response {
.into_response()
}
+/// 307 Temporary Redirect to the current VSR primary for a linearizable read
+/// that reached a follower. `Location` is the primary's HTTP base plus the
+/// original path and query, so the caller re-issues the identical read against
+/// the leader. Dormant on a single node (always primary) and followed by no SDK
+/// yet. A `Location` that is not a valid header value falls back to the 503.
+fn primary_redirect_response(location: &str) -> Response {
+ HeaderValue::from_str(location).map_or_else(
+ |_| not_primary_response(),
+ |value| {
+ let mut response = StatusCode::TEMPORARY_REDIRECT.into_response();
+ response.headers_mut().insert(LOCATION, value);
+ response
+ },
+ )
+}
+
+/// Build the `Location` for a 307 redirect of a linearizable read to the VSR
+/// primary: `http://<host>:<http-port><path_and_query>`. `None` when the roster
+/// has no node at `primary_index`, that node exposes no HTTP port, or its `ip`
+/// is not a valid address, so the caller fails closed to a 503 rather than
+/// pointing at an unreachable target. Formats through [`SocketAddr`] so an IPv6
+/// host is bracketed (`http://[::1]:8080/...`) rather than left ambiguous. Pure
+/// (no consensus or axum dependency) so the redirect target is unit-tested in
+/// isolation.
+fn primary_redirect_location(
+ roster: &ClusterRoster,
+ primary_index: u8,
+ path_and_query: &str,
+) -> Option<String> {
+ let node = roster
+ .nodes
+ .iter()
+ .find(|node| node.replica_id == primary_index)?;
+ let http_port = node.ports.http?;
+ let ip = node.ip.parse::<IpAddr>().ok()?;
+ let socket = SocketAddr::new(ip, http_port);
+ Some(format!("http://{socket}{path_and_query}"))
+}
+
/// Owned snapshot of the cluster topology `GET /cluster/metadata` reports.
///
/// The roster lives only in `ServerNgConfig` and is not reachable from
@@ -361,6 +408,25 @@ impl HttpInner {
.is_some_and(VsrConsensus::is_primary)
}
+ /// Grade a linearizable read that reached a follower: redirect (307) to the
+ /// current VSR primary's HTTP address when it resolves from the roster, else
+ /// fail closed to the 503. The target is the roster node whose `replica_id`
+ /// equals `primary_index(view)`; an absent consensus, an unmatched id, or a
+ /// port-less node all fall back to [`ReadError::NotPrimary`].
+ fn not_primary_read_error(&self, path_and_query: &str) -> ReadError {
+ let location = self
+ .shard
+ .plane
+ .metadata()
+ .consensus
+ .as_ref()
+ .and_then(|consensus| {
+ let primary_index = consensus.primary_index(consensus.view());
+ primary_redirect_location(&self.roster, primary_index, path_and_query)
+ });
+ location.map_or(ReadError::NotPrimary, ReadError::RedirectToPrimary)
+ }
+
/// Resolve the VSR session for `key`, minting and Registering one on first
/// use. Every later request bearing the same credential reuses it.
///
@@ -623,7 +689,7 @@ async fn get_streams(
let body = GetStreamsRequest.to_bytes();
let bytes = read_local(
&state,
- identity.user_id,
+ &identity,
query.consistency,
GET_STREAMS_CODE,
&body,
@@ -647,13 +713,7 @@ async fn get_stream(
stream_id: identifier_to_wire(&stream_id).map_err(ReadError::Rejected)?,
};
let body = request.to_bytes();
- let bytes = read_local(
- &state,
- identity.user_id,
- query.consistency,
- GET_STREAM_CODE,
- &body,
- )?;
+ let bytes = read_local(&state, &identity, query.consistency, GET_STREAM_CODE, &body)?;
let response = GetStreamResponse::decode_from(&bytes)
.map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
Ok(Json(
@@ -675,13 +735,7 @@ async fn get_topics(
stream_id: identifier_to_wire(&stream_id).map_err(ReadError::Rejected)?,
};
let body = request.to_bytes();
- let bytes = read_local(
- &state,
- identity.user_id,
- query.consistency,
- GET_TOPICS_CODE,
- &body,
- )?;
+ let bytes = read_local(&state, &identity, query.consistency, GET_TOPICS_CODE, &body)?;
let response = GetTopicsResponse::decode_from(&bytes)
.map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
Ok(Json(
@@ -705,13 +759,7 @@ async fn get_topic(
topic_id: identifier_to_wire(&topic_id).map_err(ReadError::Rejected)?,
};
let body = request.to_bytes();
- let bytes = read_local(
- &state,
- identity.user_id,
- query.consistency,
- GET_TOPIC_CODE,
- &body,
- )?;
+ let bytes = read_local(&state, &identity, query.consistency, GET_TOPIC_CODE, &body)?;
let response = GetTopicResponse::decode_from(&bytes)
.map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
Ok(Json(
@@ -727,13 +775,7 @@ async fn get_users(
Query(query): Query<ConsistencyQuery>,
) -> Result<Json<Vec<UserInfo>>, ReadError> {
let body = GetUsersRequest.to_bytes();
- let bytes = read_local(
- &state,
- identity.user_id,
- query.consistency,
- GET_USERS_CODE,
- &body,
- )?;
+ let bytes = read_local(&state, &identity, query.consistency, GET_USERS_CODE, &body)?;
let response = GetUsersResponse::decode_from(&bytes)
.map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
Ok(Json(
@@ -755,13 +797,7 @@ async fn get_user(
user_id: identifier_to_wire(&user_id).map_err(ReadError::Rejected)?,
};
let body = request.to_bytes();
- let bytes = read_local(
- &state,
- identity.user_id,
- query.consistency,
- GET_USER_CODE,
- &body,
- )?;
+ let bytes = read_local(&state, &identity, query.consistency, GET_USER_CODE, &body)?;
let response = UserDetailsResponse::decode_from(&bytes)
.map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
Ok(Json(
@@ -787,7 +823,7 @@ async fn get_cgs(
let body = request.to_bytes();
let bytes = read_local(
&state,
- identity.user_id,
+ &identity,
query.consistency,
GET_CONSUMER_GROUPS_CODE,
&body,
@@ -818,7 +854,7 @@ async fn get_cg(
let body = request.to_bytes();
let bytes = read_local(
&state,
- identity.user_id,
+ &identity,
query.consistency,
GET_CONSUMER_GROUP_CODE,
&body,
@@ -838,13 +874,7 @@ async fn get_stats(
Query(query): Query<ConsistencyQuery>,
) -> Result<Json<Stats>, ReadError> {
let body = GetStatsRequest.to_bytes();
- let bytes = read_local(
- &state,
- identity.user_id,
- query.consistency,
- GET_STATS_CODE,
- &body,
- )?;
+ let bytes = read_local(&state, &identity, query.consistency, GET_STATS_CODE, &body)?;
let response = StatsResponse::decode_from(&bytes)
.map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
Ok(Json(Stats::from(response)))
@@ -879,7 +909,7 @@ async fn get_clients(
identity: Identity,
Query(query): Query<ConsistencyQuery>,
) -> Result<Json<Vec<ClientInfo>>, ReadError> {
- authorize_read(&state, identity.user_id, query.consistency)?;
+ authorize_read(&state, &identity, query.consistency)?;
let infos = SendWrapper::new(state.shard.list_all_clients()).await;
let response = GetClientsResponse {
clients: infos
@@ -904,7 +934,7 @@ async fn get_client(
Path(client_id): Path<u32>,
Query(query): Query<ConsistencyQuery>,
) -> Result<Json<ClientInfoDetails>, ReadError> {
- authorize_read(&state, identity.user_id, query.consistency)?;
+ authorize_read(&state, &identity, query.consistency)?;
let infos = SendWrapper::new(state.shard.list_all_clients()).await;
// The wire client id is the u32 seq tail of the u128 transport id.
#[allow(clippy::cast_possible_truncation)]
@@ -1505,20 +1535,20 @@ async fn submit_write(
///
/// Root-only: until server-ng has an RBAC permissioner, every read is root-only,
/// mirroring the write gate in [`submit_committed`]. A non-root credential is
-/// authenticated but unprivileged -> 403. Linearizable reads must come from the
-/// primary; on a follower we cannot yet 307-redirect (no roster: F7) nor emit
-/// the Location (F3), so 503 and let the client retry. This becomes a
-/// 307-to-leader once both land.
+/// authenticated but unprivileged -> 403. A linearizable read must come from the
+/// primary; on a follower it redirects (307) to the primary's HTTP address when
+/// resolvable, else fails closed to a 503 (see
+/// [`HttpInner::not_primary_read_error`]).
fn authorize_read(
state: &HttpInner,
- user_id: u32,
+ identity: &Identity,
consistency: Consistency,
) -> Result<(), ReadError> {
- if user_id != DEFAULT_ROOT_USER_ID {
+ if identity.user_id != DEFAULT_ROOT_USER_ID {
return Err(ReadError::Rejected(IggyError::Unauthorized));
}
if consistency == Consistency::Linearizable && !state.is_metadata_primary() {
- return Err(ReadError::NotPrimary);
+ return Err(state.not_primary_read_error(&identity.path_and_query));
}
Ok(())
}
@@ -1537,12 +1567,12 @@ fn authorize_read(
/// whose entity can be missing shares that not-found shape.
fn read_local(
state: &HttpInner,
- user_id: u32,
+ identity: &Identity,
consistency: Consistency,
code: u32,
body: &[u8],
) -> Result<Bytes, ReadError> {
- authorize_read(state, user_id, consistency)?;
+ authorize_read(state, identity, consistency)?;
match build_non_replicated_response(&state.shard, code, body).map_err(ReadError::Rejected)? {
NonReplicatedResponse::Empty => Err(ReadError::NotFound),
NonReplicatedResponse::Bytes(bytes) => Ok(bytes),
@@ -1785,3 +1815,75 @@ const fn login_error_to_iggy(error: &LoginRegisterError) -> IggyError {
_ => IggyError::Unauthenticated,
}
}
+
+#[cfg(test)]
+mod tests {
+ use configs::cluster::TransportPorts;
+
+ use super::*;
+
+ const READ_PATH: &str = "/streams?consistency=linearizable";
+
+ fn node(replica_id: u8, ip: &str, http: Option<u16>) -> ClusterNodeConfig {
+ ClusterNodeConfig {
+ name: format!("node-{replica_id}"),
+ ip: ip.to_owned(),
+ replica_id,
+ ports: TransportPorts {
+ tcp: None,
+ quic: None,
+ http,
+ websocket: None,
+ tcp_replica: None,
+ },
+ }
+ }
+
+ fn roster(nodes: Vec<ClusterNodeConfig>) -> ClusterRoster {
+ ClusterRoster {
+ enabled: true,
+ name: "test-cluster".to_owned(),
+ nodes,
+ http_addr: "127.0.0.1:3000".parse().expect("valid socket addr"),
+ }
+ }
+
+ #[test]
+ fn primary_redirect_location_targets_primary_http_addr_with_path_passthrough() {
+ let roster = roster(vec![
+ node(0, "10.0.0.1", Some(8080)),
+ node(1, "10.0.0.2", Some(8090)),
+ ]);
+ assert_eq!(
+ primary_redirect_location(&roster, 1, READ_PATH),
+ Some("http://10.0.0.2:8090/streams?consistency=linearizable".to_owned())
+ );
+ }
+
+ #[test]
+ fn primary_redirect_location_is_none_when_no_node_matches_primary_index() {
+ let roster = roster(vec![node(0, "10.0.0.1", Some(8080))]);
+ assert_eq!(primary_redirect_location(&roster, 2, READ_PATH), None);
+ }
+
+ #[test]
+ fn primary_redirect_location_is_none_when_primary_has_no_http_port() {
+ let roster = roster(vec![node(0, "10.0.0.1", None)]);
+ assert_eq!(primary_redirect_location(&roster, 0, READ_PATH), None);
+ }
+
+ #[test]
+ fn primary_redirect_location_is_none_for_empty_roster() {
+ let roster = roster(Vec::new());
+ assert_eq!(primary_redirect_location(&roster, 0, READ_PATH), None);
+ }
+
+ #[test]
+ fn primary_redirect_location_brackets_ipv6_host() {
+ let roster = roster(vec![node(0, "::1", Some(8080))]);
+ assert_eq!(
+ primary_redirect_location(&roster, 0, READ_PATH),
+ Some("http://[::1]:8080/streams?consistency=linearizable".to_owned())
+ );
+ }
+}
diff --git a/core/server-ng/src/http/extractor.rs b/core/server-ng/src/http/extractor.rs
index 96815a56b..c3e235370 100644
--- a/core/server-ng/src/http/extractor.rs
+++ b/core/server-ng/src/http/extractor.rs
@@ -90,6 +90,11 @@ impl FromRequestParts<HttpState> for Authenticated {
/// [`resolve_credential`] chokepoint, so a JWT and a PAT are honored identically.
pub struct Identity {
pub user_id: u32,
+ /// Original request path + query (e.g. `/streams?consistency=linearizable`),
+ /// captured so a linearizable read that reaches a follower can build the
+ /// `Location` for its 307 redirect to the primary. Empty only when the URI
+ /// carries neither, which a routed read never is.
+ pub path_and_query: String,
}
impl FromRequestParts<HttpState> for Identity {
@@ -112,7 +117,15 @@ impl FromRequestParts<HttpState> for Identity {
// no `SendWrapper` bridge (contrast [`Authenticated`], which awaits the
// `!Send` `resolve_session`).
let (_key, user_id, _expiry) = resolve_credential(state, bearer)?;
- Ok(Self { user_id })
+ let path_and_query = parts
+ .uri
+ .path_and_query()
+ .map(|value| value.as_str().to_owned())
+ .unwrap_or_default();
+ Ok(Self {
+ user_id,
+ path_and_query,
+ })
}
}
(iggy) 19/19: test(server-ng): add HTTP data-plane integration tests
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit b50ff02de017d89d54bacd04cdec6f35452c2f19
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Thu Jul 2 01:13:55 2026 +0200
test(server-ng): add HTTP data-plane integration tests
Gate the new shard-0 HTTP surface end to end over raw reqwest rather
than the SDK client, so the wire contract itself is what regresses:
201 + x-iggy-durability on committed produces, 202 + async commit for
ack=none, 413 on the 2 MB body cap without poisoning the listener,
consumer-offset store/get/delete round trip, and 16 concurrent
produces on one login session polled back with no loss and no
cross-partition leakage.
---
core/integration/tests/server/http_vsr.rs | 546 ++++++++++++++++++++++++++++++
core/integration/tests/server/mod.rs | 3 +
2 files changed, 549 insertions(+)
diff --git a/core/integration/tests/server/http_vsr.rs b/core/integration/tests/server/http_vsr.rs
new file mode 100644
index 000000000..eb68dd355
--- /dev/null
+++ b/core/integration/tests/server/http_vsr.rs
@@ -0,0 +1,546 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! HTTP data-plane gate for server-ng: produce, poll, and consumer-offset
+//! routes exercised over raw `reqwest` (not the SDK HTTP client) so the wire
+//! contract itself is under test - exact status codes, the
+//! `x-iggy-durability` header, the body-size cap, and cross-request isolation
+//! of concurrent produces on one login session.
+
+use futures::future::join_all;
+use iggy::prelude::*;
+use iggy_common::create_stream::CreateStream;
+use iggy_common::create_topic::CreateTopic;
+use iggy_common::store_consumer_offset::StoreConsumerOffset;
+use iggy_common::{ConsumerOffsetInfo, IggyMessagesBatch};
+use integration::harness::TestHarness;
+use integration::iggy_harness;
+use reqwest::{Response, StatusCode};
+use std::collections::{BTreeMap, HashMap, HashSet};
+use std::str::FromStr;
+use std::time::{Duration, Instant};
+use tokio::time::sleep;
+
+// server-ng partition ids are 0-based (CreateTopic assigns them from 0).
+const PARTITION_ID: u32 = 0;
+
+/// Explicit consumer id shared by the offset store body and the read/delete
+/// query, so both sides address the same offset-table key
+/// (`Consumer::default()` would carry numeric id 0, not 1).
+const CONSUMER_ID: u32 = 1;
+
+const DURABILITY_HEADER: &str = "x-iggy-durability";
+const DURABILITY_REPLICATED_MEMORY: &str = "replicated-memory";
+const DURABILITY_NONE: &str = "none";
+
+/// The harness readiness gate waits for the dumped runtime config, which the
+/// server writes before binding the HTTP listener; a bounded login retry
+/// bridges that gap plus single-node consensus warmup.
+const LOGIN_TIMEOUT: Duration = Duration::from_secs(15);
+const LOGIN_RETRY_INTERVAL: Duration = Duration::from_millis(50);
+
+/// `?ack=none` answers before the commit; poll until visible, never unbounded.
+const ASYNC_COMMIT_TIMEOUT: Duration = Duration::from_secs(15);
+const ASYNC_COMMIT_RETRY_INTERVAL: Duration = Duration::from_millis(100);
+
+const REQUEST_TIMEOUT: Duration = Duration::from_secs(30);
+
+/// One authenticated root session against the test server's HTTP listener.
+struct HttpSession {
+ client: reqwest::Client,
+ base_url: String,
+ token: String,
+}
+
+impl HttpSession {
+ async fn login_root(harness: &TestHarness) -> Self {
+ let addr = harness
+ .server()
+ .http_addr()
+ .expect("HTTP transport not configured on test server");
+ let base_url = format!("http://{addr}");
+ let client = reqwest::Client::builder()
+ .timeout(REQUEST_TIMEOUT)
+ .build()
+ .expect("build reqwest client");
+
+ let body = serde_json::json!({
+ "username": DEFAULT_ROOT_USERNAME,
+ "password": DEFAULT_ROOT_PASSWORD,
+ });
+ let deadline = Instant::now() + LOGIN_TIMEOUT;
+ loop {
+ let attempt = client
+ .post(format!("{base_url}/users/login"))
+ .json(&body)
+ .send()
+ .await;
+ match attempt {
+ Ok(response) if response.status().is_success() => {
+ let identity: IdentityInfo =
+ response.json().await.expect("decode IdentityInfo");
+ let token = identity
+ .access_token
+ .expect("login must return an access token")
+ .token;
+ return Self {
+ client,
+ base_url,
+ token,
+ };
+ }
+ // Listener not bound yet or consensus still warming up; any
+ // non-5xx rejection (e.g. 401) is a real failure.
+ Ok(response) if response.status().is_server_error() => {}
+ Ok(response) => panic!("login rejected: {}", response.status()),
+ Err(error) if error.is_connect() => {}
+ Err(error) => panic!("login request failed: {error}"),
+ }
+ assert!(
+ Instant::now() < deadline,
+ "HTTP login did not succeed within {LOGIN_TIMEOUT:?}"
+ );
+ sleep(LOGIN_RETRY_INTERVAL).await;
+ }
+ }
+
+ fn url(&self, path: &str) -> String {
+ format!("{}{path}", self.base_url)
+ }
+
+ async fn create_stream_and_topic(&self, stream: &str, topic: &str, partitions: u32) {
+ let response = self
+ .client
+ .post(self.url("/streams"))
+ .bearer_auth(&self.token)
+ .json(&CreateStream {
+ name: stream.to_string(),
+ })
+ .send()
+ .await
+ .expect("create stream request");
+ assert!(
+ response.status().is_success(),
+ "create stream failed: {}",
+ response.status()
+ );
+
+ let command = CreateTopic {
+ stream_id: Identifier::default(),
+ partitions_count: partitions,
+ compression_algorithm: CompressionAlgorithm::None,
+ message_expiry: IggyExpiry::NeverExpire,
+ max_topic_size: MaxTopicSize::ServerDefault,
+ replication_factor: None,
+ name: topic.to_string(),
+ };
+ let response = self
+ .client
+ .post(self.url(&format!("/streams/{stream}/topics")))
+ .bearer_auth(&self.token)
+ .json(&command)
+ .send()
+ .await
+ .expect("create topic request");
+ assert!(
+ response.status().is_success(),
+ "create topic failed: {}",
+ response.status()
+ );
+ }
+
+ async fn produce(
+ &self,
+ stream: &str,
+ topic: &str,
+ partition_id: u32,
+ messages: Vec<IggyMessage>,
+ ) -> Response {
+ self.produce_with_query(stream, topic, partition_id, messages, "")
+ .await
+ }
+
+ async fn produce_with_query(
+ &self,
+ stream: &str,
+ topic: &str,
+ partition_id: u32,
+ messages: Vec<IggyMessage>,
+ query: &str,
+ ) -> Response {
+ let body = SendMessages {
+ metadata_length: 0,
+ stream_id: Identifier::default(),
+ topic_id: Identifier::default(),
+ partitioning: Partitioning::partition_id(partition_id),
+ batch: IggyMessagesBatch::from(&messages),
+ };
+ self.client
+ .post(self.url(&format!("/streams/{stream}/topics/{topic}/messages{query}")))
+ .bearer_auth(&self.token)
+ .json(&body)
+ .send()
+ .await
+ .expect("produce request")
+ }
+
+ async fn poll(
+ &self,
+ stream: &str,
+ topic: &str,
+ partition_id: u32,
+ offset: u64,
+ count: u32,
+ ) -> PolledMessages {
+ let path = format!(
+ "/streams/{stream}/topics/{topic}/messages\
+ ?consumer_id={CONSUMER_ID}&partition_id={partition_id}\
+ &kind=offset&value={offset}&count={count}&auto_commit=false"
+ );
+ let response = self
+ .client
+ .get(self.url(&path))
+ .bearer_auth(&self.token)
+ .send()
+ .await
+ .expect("poll request");
+ assert_eq!(response.status(), StatusCode::OK, "poll must answer 200");
+ response.json().await.expect("decode PolledMessages")
+ }
+}
+
+fn durability(response: &Response) -> &str {
+ response
+ .headers()
+ .get(DURABILITY_HEADER)
+ .expect("produce response must carry the durability header")
+ .to_str()
+ .expect("durability header is ASCII")
+}
+
+fn text_message(id: u128, payload: String) -> IggyMessage {
+ IggyMessage::builder()
+ .id(id)
+ .payload(payload.into())
+ .build()
+ .expect("message build")
+}
+
+#[iggy_harness]
+async fn given_http_session_when_producing_and_polling_should_round_trip(harness: &TestHarness) {
+ let http = HttpSession::login_root(harness).await;
+ http.create_stream_and_topic("http-stream", "http-topic", 1)
+ .await;
+
+ let header_key = HeaderKey::try_from("trace-id").expect("header key");
+ let header_value = HeaderValue::from_str("trace-42").expect("header value");
+ let messages: Vec<IggyMessage> = (0..3u32)
+ .map(|i| {
+ let builder = IggyMessage::builder()
+ .id(u128::from(i + 1))
+ .payload(format!("round-trip-{i}").into());
+ if i == 0 {
+ let user_headers = BTreeMap::from([(header_key.clone(), header_value.clone())]);
+ builder
+ .user_headers(user_headers)
+ .build()
+ .expect("message build")
+ } else {
+ builder.build().expect("message build")
+ }
+ })
+ .collect();
+
+ let response = http
+ .produce("http-stream", "http-topic", PARTITION_ID, messages)
+ .await;
+ assert_eq!(
+ response.status(),
+ StatusCode::CREATED,
+ "produce must commit"
+ );
+ assert_eq!(durability(&response), DURABILITY_REPLICATED_MEMORY);
+
+ let polled = http
+ .poll("http-stream", "http-topic", PARTITION_ID, 0, 10)
+ .await;
+ assert_eq!(polled.messages.len(), 3, "all sent messages must come back");
+ for (i, message) in polled.messages.iter().enumerate() {
+ assert_eq!(message.header.offset, i as u64, "offsets must be dense");
+ assert_eq!(
+ message.payload,
+ bytes::Bytes::from(format!("round-trip-{i}")),
+ "payload round trip for message {i}"
+ );
+ }
+ let user_headers = polled.messages[0]
+ .user_headers_map()
+ .expect("headers decode")
+ .expect("first message carries user headers");
+ assert_eq!(
+ user_headers.get(&header_key),
+ Some(&header_value),
+ "user header must survive the round trip"
+ );
+}
+
+const CONCURRENT_PRODUCES: usize = 16;
+const CONCURRENT_PARTITIONS: u32 = 4;
+
+#[iggy_harness]
+async fn given_one_session_when_producing_concurrently_should_not_cross_talk(
+ harness: &TestHarness,
+) {
+ let http = HttpSession::login_root(harness).await;
+ http.create_stream_and_topic("http-concurrent", "hammer", CONCURRENT_PARTITIONS)
+ .await;
+
+ let produces = (0..CONCURRENT_PRODUCES).map(|i| {
+ let http = &http;
+ let partition_id = (i as u32) % CONCURRENT_PARTITIONS;
+ async move {
+ let message = text_message(i as u128 + 1, format!("concurrent-{i}"));
+ let response = http
+ .produce("http-concurrent", "hammer", partition_id, vec![message])
+ .await;
+ (i, response.status(), durability(&response).to_string())
+ }
+ });
+ let outcomes = join_all(produces).await;
+
+ for (i, status, durability) in &outcomes {
+ assert_eq!(
+ *status,
+ StatusCode::CREATED,
+ "concurrent produce {i} must commit"
+ );
+ assert_eq!(
+ durability, DURABILITY_REPLICATED_MEMORY,
+ "concurrent produce {i} must attest a replicated commit"
+ );
+ }
+
+ let mut produced: HashMap<u32, HashSet<String>> = HashMap::new();
+ for i in 0..CONCURRENT_PRODUCES {
+ produced
+ .entry((i as u32) % CONCURRENT_PARTITIONS)
+ .or_default()
+ .insert(format!("concurrent-{i}"));
+ }
+
+ // Every partition must return exactly its own produced payloads: nothing
+ // lost, nothing foreign. The produce path is at-least-once, so duplicates
+ // are tolerated but must be byte-identical to a produced payload.
+ let mut polled_union: HashSet<String> = HashSet::new();
+ for partition_id in 0..CONCURRENT_PARTITIONS {
+ let polled = http
+ .poll("http-concurrent", "hammer", partition_id, 0, 100)
+ .await;
+ let expected = &produced[&partition_id];
+ let mut seen: HashSet<String> = HashSet::new();
+ for message in &polled.messages {
+ let payload = String::from_utf8(message.payload.to_vec()).expect("payload is UTF-8");
+ assert!(
+ expected.contains(&payload),
+ "partition {partition_id} returned foreign payload {payload:?}"
+ );
+ seen.insert(payload);
+ }
+ assert_eq!(
+ &seen, expected,
+ "partition {partition_id} must return exactly its own produced payloads"
+ );
+ polled_union.extend(seen);
+ }
+ assert_eq!(
+ polled_union.len(),
+ CONCURRENT_PRODUCES,
+ "every produced payload must be polled back"
+ );
+}
+
+#[iggy_harness]
+async fn given_consumer_offset_when_stored_and_deleted_should_round_trip(harness: &TestHarness) {
+ let http = HttpSession::login_root(harness).await;
+ http.create_stream_and_topic("http-offsets", "offsets", 1)
+ .await;
+
+ // Two committed messages so offset 1 names a real position.
+ let messages = (0..2u32)
+ .map(|i| text_message(u128::from(i + 1), format!("offset-fodder-{i}")))
+ .collect();
+ let response = http
+ .produce("http-offsets", "offsets", PARTITION_ID, messages)
+ .await;
+ assert_eq!(
+ response.status(),
+ StatusCode::CREATED,
+ "produce must commit"
+ );
+
+ let offsets_path = "/streams/http-offsets/topics/offsets/consumer-offsets";
+ let store = StoreConsumerOffset {
+ consumer: Consumer::new(Identifier::numeric(CONSUMER_ID).expect("consumer id")),
+ partition_id: Some(PARTITION_ID),
+ offset: 1,
+ };
+ let response = http
+ .client
+ .put(http.url(offsets_path))
+ .bearer_auth(&http.token)
+ .json(&store)
+ .send()
+ .await
+ .expect("store offset request");
+ assert_eq!(
+ response.status(),
+ StatusCode::NO_CONTENT,
+ "store must commit"
+ );
+
+ let get_path = format!("{offsets_path}?consumer_id={CONSUMER_ID}&partition_id={PARTITION_ID}");
+ let response = http
+ .client
+ .get(http.url(&get_path))
+ .bearer_auth(&http.token)
+ .send()
+ .await
+ .expect("get offset request");
+ assert_eq!(
+ response.status(),
+ StatusCode::OK,
+ "stored offset must be readable"
+ );
+ let info: ConsumerOffsetInfo = response.json().await.expect("decode ConsumerOffsetInfo");
+ assert_eq!(info.stored_offset, 1, "stored offset must round trip");
+ assert_eq!(info.partition_id, PARTITION_ID);
+
+ let delete_path = format!("{offsets_path}/{CONSUMER_ID}?partition_id={PARTITION_ID}");
+ let response = http
+ .client
+ .delete(http.url(&delete_path))
+ .bearer_auth(&http.token)
+ .send()
+ .await
+ .expect("delete offset request");
+ assert_eq!(
+ response.status(),
+ StatusCode::NO_CONTENT,
+ "delete must commit"
+ );
+
+ let response = http
+ .client
+ .get(http.url(&get_path))
+ .bearer_auth(&http.token)
+ .send()
+ .await
+ .expect("get offset request after delete");
+ assert_eq!(
+ response.status(),
+ StatusCode::NOT_FOUND,
+ "deleted offset must be gone"
+ );
+}
+
+#[iggy_harness]
+async fn given_oversized_body_when_producing_should_reject_413(harness: &TestHarness) {
+ let http = HttpSession::login_root(harness).await;
+ http.create_stream_and_topic("http-oversized", "big", 1)
+ .await;
+
+ // 3 MiB payload -> ~4 MiB JSON after base64: over the 2 MB body cap under
+ // either decimal or binary megabyte semantics.
+ let oversized = IggyMessage::builder()
+ .payload(vec![0x42u8; 3 * 1024 * 1024].into())
+ .build()
+ .expect("message build");
+ let response = http
+ .produce("http-oversized", "big", PARTITION_ID, vec![oversized])
+ .await;
+ assert_eq!(
+ response.status(),
+ StatusCode::PAYLOAD_TOO_LARGE,
+ "oversized body must be rejected"
+ );
+
+ // The rejection must not poison the listener: a normal produce still commits.
+ let normal = text_message(1, "small-after-large".to_string());
+ let response = http
+ .produce("http-oversized", "big", PARTITION_ID, vec![normal])
+ .await;
+ assert_eq!(
+ response.status(),
+ StatusCode::CREATED,
+ "server must stay healthy after a 413"
+ );
+ let polled = http
+ .poll("http-oversized", "big", PARTITION_ID, 0, 10)
+ .await;
+ assert_eq!(polled.messages.len(), 1, "normal produce must be pollable");
+ assert_eq!(
+ polled.messages[0].payload,
+ bytes::Bytes::from("small-after-large")
+ );
+}
+
+#[iggy_harness]
+async fn given_ack_none_when_producing_should_return_202_and_commit(harness: &TestHarness) {
+ let http = HttpSession::login_root(harness).await;
+ http.create_stream_and_topic("http-ack-none", "fire", 1)
+ .await;
+
+ let message = text_message(1, "fire-and-forget".to_string());
+ let response = http
+ .produce_with_query(
+ "http-ack-none",
+ "fire",
+ PARTITION_ID,
+ vec![message],
+ "?ack=none",
+ )
+ .await;
+ assert_eq!(
+ response.status(),
+ StatusCode::ACCEPTED,
+ "ack=none must answer before the commit"
+ );
+ assert_eq!(durability(&response), DURABILITY_NONE);
+
+ // The commit still happens, just asynchronously: poll bounded until the
+ // message becomes visible.
+ let deadline = Instant::now() + ASYNC_COMMIT_TIMEOUT;
+ loop {
+ let polled = http
+ .poll("http-ack-none", "fire", PARTITION_ID, 0, 10)
+ .await;
+ if !polled.messages.is_empty() {
+ assert_eq!(polled.messages.len(), 1, "exactly one message was produced");
+ assert_eq!(
+ polled.messages[0].payload,
+ bytes::Bytes::from("fire-and-forget"),
+ "ack=none payload must round trip"
+ );
+ break;
+ }
+ assert!(
+ Instant::now() < deadline,
+ "ack=none produce did not become pollable within {ASYNC_COMMIT_TIMEOUT:?}"
+ );
+ sleep(ASYNC_COMMIT_RETRY_INTERVAL).await;
+ }
+}
diff --git a/core/integration/tests/server/mod.rs b/core/integration/tests/server/mod.rs
index 53e8c70a4..b2f1d8351 100644
--- a/core/integration/tests/server/mod.rs
+++ b/core/integration/tests/server/mod.rs
@@ -24,6 +24,9 @@ mod cg;
// The ported round-robin membership join scenario runs against server-ng.
#[cfg(feature = "vsr")]
mod cg_vsr;
+// Raw-HTTP data-plane contract against server-ng's shard-0 listener.
+#[cfg(feature = "vsr")]
+mod http_vsr;
// 80-case race matrix with hardcoded HTTP variants (test_matrix bypasses
// the harness transport filter); revisit under vsr once basics are green.
#[cfg(not(feature = "vsr"))]
(iggy) 11/19: feat(server-ng): add HTTP cluster-metadata route and view header
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 33e396759910669bc5ce3e450deb9511dda8eb2e
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 19:40:40 2026 +0200
feat(server-ng): add HTTP cluster-metadata route and view header
Serve GET /cluster/metadata from the live config roster and consensus view instead of the hardcoded shared builder, and stamp X-Iggy-View on responses so clients can observe the current VSR view.
Auth-only like legacy but bypassing the consistency gate so followers still answer for leader discovery. The roster is captured from config at http::start; a single self-node is synthesized when the cluster is disabled, and the leader is marked per primary_index(view).
---
core/server-ng/src/bootstrap.rs | 2 +-
core/server-ng/src/http.rs | 136 ++++++++++++++++++++++++++++++++++++++--
2 files changed, 133 insertions(+), 5 deletions(-)
diff --git a/core/server-ng/src/bootstrap.rs b/core/server-ng/src/bootstrap.rs
index aa3d4fc0b..a3910964f 100644
--- a/core/server-ng/src/bootstrap.rs
+++ b/core/server-ng/src/bootstrap.rs
@@ -1946,7 +1946,7 @@ async fn start_tcp_runtime(
// caller of this function.
if config.http.enabled {
let http_addr = parse_socket_addr("http.address", &config.http.address)?;
- http::start(shard, http_addr, &config.http.jwt).await?;
+ http::start(shard, http_addr, &config.http.jwt, &config.cluster).await?;
}
Ok(())
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index dc1de5f62..c21fe5c85 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -28,11 +28,13 @@ use std::net::SocketAddr;
use std::rc::Rc;
use axum::extract::{Path, Query, State};
-use axum::http::StatusCode;
+use axum::http::{HeaderName, HeaderValue, StatusCode};
+use axum::middleware::map_response;
use axum::response::{IntoResponse, Response};
use axum::routing::{delete, get, post, put};
use axum::{Json, Router};
use bytes::Bytes;
+use configs::cluster::{ClusterConfig, ClusterNodeConfig};
use configs::http::HttpJwtConfig;
use consensus::{MetadataHandle, VsrConsensus};
use iggy_binary_protocol::codes::{
@@ -97,9 +99,10 @@ use iggy_common::wire_conversions::{
streams_from_wire, topics_from_wire, users_from_wire,
};
use iggy_common::{
- ClientInfo, ClientInfoDetails, ConsumerGroup, ConsumerGroupDetails, Identifier, IdentityInfo,
- IggyError, IggyTimestamp, RawPersonalAccessToken, Stats, Stream, StreamDetails, TokenInfo,
- Topic, TopicDetails, UserInfo, UserInfoDetails, Validatable,
+ ClientInfo, ClientInfoDetails, ClusterMetadata, ClusterNode, ClusterNodeRole,
+ ClusterNodeStatus, ConsumerGroup, ConsumerGroupDetails, Identifier, IdentityInfo, IggyError,
+ IggyTimestamp, RawPersonalAccessToken, Stats, Stream, StreamDetails, TokenInfo, Topic,
+ TopicDetails, TransportEndpoints, UserInfo, UserInfoDetails, Validatable,
};
use message_bus::client_listener;
use metadata::impls::metadata::StreamsFrontend;
@@ -139,6 +142,18 @@ const MAX_HTTP_SESSIONS: usize = 100_000;
/// are 1-based and strictly increasing within a session.
const FIRST_REQUEST_ID: u64 = 1;
+/// Node name reported for the synthesized single self-node, matching the legacy
+/// server's single-node synthesis.
+const SELF_NODE_NAME: &str = "iggy-node";
+
+/// Cluster name reported when no roster is configured, matching the legacy
+/// server's single-node synthesis.
+const SINGLE_NODE_CLUSTER_NAME: &str = "single-node";
+
+/// Response header carrying the current VSR view number, set on every response
+/// while this shard-0 node has live consensus.
+const VIEW_HEADER: HeaderName = HeaderName::from_static("x-iggy-view");
+
/// One VSR session established for a single login credential (a JWT `jti` or a
/// PAT). Shared via `Rc` by every concurrent request bearing that credential,
/// so the session granularity is per-login.
@@ -300,6 +315,20 @@ fn not_primary_response() -> Response {
.into_response()
}
+/// Owned snapshot of the cluster topology `GET /cluster/metadata` reports.
+///
+/// The roster lives only in `ServerNgConfig` and is not reachable from
+/// [`ServerNgShard`], so [`start`] copies the minimal pieces here at listener
+/// start. Owned so the handler stays synchronous and never borrows config.
+struct ClusterRoster {
+ enabled: bool,
+ name: String,
+ nodes: Vec<ClusterNodeConfig>,
+ /// This node's bound HTTP address, used to synthesize the sole self-node
+ /// when no roster is configured (the only transport port known here).
+ http_addr: SocketAddr,
+}
+
/// Shared shard-0 HTTP state.
///
/// Groups the shard handle, the JWT issuer/verifier, and the per-credential VSR
@@ -313,6 +342,7 @@ struct HttpInner {
/// bridge tolerates the `!Sync` interior - but the guard must never be held
/// across an `.await` (see [`HttpInner::resolve_session`]).
sessions: RefCell<HashMap<String, Rc<HttpSession>>>,
+ roster: ClusterRoster,
}
impl HttpInner {
@@ -448,6 +478,7 @@ pub async fn start(
shard: &Rc<ServerNgShard>,
addr: SocketAddr,
jwt_config: &HttpJwtConfig,
+ cluster: &ClusterConfig,
) -> Result<(), ServerNgError> {
let jwt = JwtManager::build(jwt_config)?;
let (listener, bound_addr) = client_listener::tcp::bind(addr).await?;
@@ -457,6 +488,12 @@ pub async fn start(
shard: Rc::clone(shard),
jwt,
sessions: RefCell::new(HashMap::new()),
+ roster: ClusterRoster {
+ enabled: cluster.enabled,
+ name: cluster.name.clone(),
+ nodes: cluster.nodes.clone(),
+ http_addr: bound_addr,
+ },
}));
let router = router(state);
@@ -477,6 +514,9 @@ pub async fn start(
/// Assemble the shard-0 router: unauthenticated health + login routes, plus
/// one authenticated probe that exercises the [`Authenticated`] extractor.
fn router(state: HttpState) -> Router {
+ // Cloned for the response layer so `X-Iggy-View` reads the live view per
+ // response; the original `state` is moved into `with_state` below.
+ let view_source = state.clone();
Router::new()
.route("/ping", get(ping))
.route("/users/login", post(login_user))
@@ -523,9 +563,14 @@ fn router(state: HttpState) -> Router {
.route("/personal-access-tokens", post(create_pat))
.route("/personal-access-tokens/{name}", delete(delete_pat))
.route("/stats", get(get_stats))
+ .route("/cluster/metadata", get(get_cluster_metadata))
.route("/clients", get(get_clients))
.route("/clients/{client_id}", get(get_client))
.with_state(state)
+ .layer(map_response(move |response: Response| {
+ let view_source = view_source.clone();
+ async move { insert_view_header(&view_source, response) }
+ }))
}
/// Extracting the state here proves at compile time that the `!Send` state
@@ -805,6 +850,20 @@ async fn get_stats(
Ok(Json(Stats::from(response)))
}
+/// `GET /cluster/metadata`: report the live cluster topology as the same
+/// `ClusterMetadata` JSON the legacy server returns.
+///
+/// Auth-only: any valid token serves. Unlike the entity reads it bypasses both
+/// the root gate and the consistency gate, and serves from the roster captured
+/// at listener start plus the sync consensus getters, so it never touches the
+/// metadata STM, consensus, or a VSR session and stays fully synchronous.
+async fn get_cluster_metadata(
+ State(state): State<HttpState>,
+ _identity: Identity,
+) -> Json<ClusterMetadata> {
+ Json(build_cluster_metadata(&state))
+}
+
/// `GET /clients`: list every connected client across all shards as the same
/// `Vec<ClientInfo>` JSON the legacy server returns.
///
@@ -1490,6 +1549,75 @@ fn read_local(
}
}
+/// Build the live [`ClusterMetadata`] for `GET /cluster/metadata`.
+///
+/// Mirrors the legacy `get_cluster_metadata`: with a configured, non-empty
+/// roster and live consensus, emit one node per roster entry and derive each
+/// role from the current VSR view - the node at `primary_index(view)` leads, the
+/// rest follow. Otherwise (cluster disabled, empty roster, or absent consensus)
+/// fail closed to a synthesized self-node as the sole leader.
+fn build_cluster_metadata(state: &HttpInner) -> ClusterMetadata {
+ let roster = &state.roster;
+ match state.shard.plane.metadata().consensus.as_ref() {
+ Some(consensus) if roster.enabled && !roster.nodes.is_empty() => {
+ let primary = consensus.primary_index(consensus.view());
+ let nodes = roster
+ .nodes
+ .iter()
+ .map(|node| ClusterNode {
+ name: node.name.clone(),
+ ip: node.ip.clone(),
+ endpoints: TransportEndpoints::new(
+ node.ports.tcp.unwrap_or(0),
+ node.ports.quic.unwrap_or(0),
+ node.ports.http.unwrap_or(0),
+ node.ports.websocket.unwrap_or(0),
+ ),
+ role: if node.replica_id == primary {
+ ClusterNodeRole::Leader
+ } else {
+ ClusterNodeRole::Follower
+ },
+ status: ClusterNodeStatus::Healthy,
+ })
+ .collect();
+ ClusterMetadata {
+ name: roster.name.clone(),
+ nodes,
+ }
+ }
+ _ => self_node_metadata(roster),
+ }
+}
+
+/// Synthesize a single self-node [`ClusterMetadata`], mirroring the legacy
+/// single-node path. Only this node's bound HTTP port is known at the HTTP layer
+/// (the roster is not threaded onto the shard), so the other transports report 0.
+fn self_node_metadata(roster: &ClusterRoster) -> ClusterMetadata {
+ ClusterMetadata {
+ name: SINGLE_NODE_CLUSTER_NAME.to_owned(),
+ nodes: vec![ClusterNode {
+ name: SELF_NODE_NAME.to_owned(),
+ ip: roster.http_addr.ip().to_string(),
+ endpoints: TransportEndpoints::new(0, 0, roster.http_addr.port(), 0),
+ role: ClusterNodeRole::Leader,
+ status: ClusterNodeStatus::Healthy,
+ }],
+ }
+}
+
+/// Set the [`VIEW_HEADER`] to the current VSR view on `response`. Omits the
+/// header when this node has no live consensus: a missing header is
+/// unambiguous, whereas a fabricated view number would mislead.
+fn insert_view_header(state: &HttpInner, mut response: Response) -> Response {
+ if let Some(consensus) = state.shard.plane.metadata().consensus.as_ref() {
+ response
+ .headers_mut()
+ .insert(VIEW_HEADER, HeaderValue::from(consensus.view()));
+ }
+ response
+}
+
/// Classify a committed reply's leading result section and return the typed
/// payload slice on success. Mirrors the SDK's `split_metadata_result`:
/// `Some(0)` is success and the payload follows the result section; a nonzero
(iggy) 18/19: fix(server-ng): make HTTP write id-advance cancellation-safe
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit aa3132db944555b2d048836dd04b817d2e1f392f
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Thu Jul 2 00:48:46 2026 +0200
fix(server-ng): make HTTP write id-advance cancellation-safe
Axum drops a write handler's future the moment the HTTP client
disconnects, and that drop could land between the pump-driven consensus
commit and the session gate's id advance. The next write then reused the
id, collided into the depth-1 dedup as Duplicate, and silently received
the prior op's cached reply (or wedged the session near RequestGap on a
single node).
submit_committed now spawns the gate-locked mint/rewrite/submit/advance
sequence (submit_gated) as a detached shard-0 compio task and awaits its
outcome over a oneshot, so a disconnect only drops the receiver half; a
failed result send is ignored because the advance already happened.
---
core/server-ng/src/http.rs | 118 ++++++++++++++++++++++++++++++++++-----------
1 file changed, 90 insertions(+), 28 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index e36f4b813..121e37ce0 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -39,6 +39,7 @@ use bytes::{Bytes, BytesMut};
use configs::cluster::{ClusterConfig, ClusterNodeConfig};
use configs::http::HttpConfig;
use consensus::{MetadataHandle, VsrConsensus};
+use futures::channel::oneshot;
use iggy_binary_protocol::codes::{
GET_CONSUMER_GROUP_CODE, GET_CONSUMER_GROUPS_CODE, GET_STATS_CODE, GET_STREAM_CODE,
GET_STREAMS_CODE, GET_TOPIC_CODE, GET_TOPICS_CODE, GET_USER_CODE, GET_USERS_CODE,
@@ -1975,23 +1976,18 @@ async fn delete_pat(
/// reply body for the stream/topic/user routes, while [`create_pat`] needs the
/// raw `Message` + request header to substitute the one-time token.
///
-/// Enforces the root-only admin gate, then serializes this session's writes
-/// behind its gate and holds it across the submit so request ids reach the
-/// primary strictly in order. The gate advances only on a committed `Reply`: a
-/// transient failure or an eviction leaves the id free for the caller's retry,
-/// which the depth-1 dedup requires (the next accepted request must be
-/// `committed + 1`, so a consumed-but-uncommitted id would wedge the session on
-/// `RequestGap`). An eviction means the session is dead, mapped to 401.
-///
-/// Both shard-0 request rewrites run here before consensus, mirroring the TCP
-/// dispatch path: the PAT rewrite mints a raw token and replicates only its hash
-/// (`CreatePersonalAccessToken`), and the user-password rewrite hashes plaintext
-/// (`CreateUser` / `ChangePassword`). Both are no-ops for every other operation,
-/// so plaintext secrets never enter consensus on any write. On either rewrite's
-/// decode failure the gate is released without advancing, leaving the id free.
+/// Enforces the root-only admin gate, then runs the gate-locked submit
+/// ([`submit_gated`]) on a detached shard-0 task and awaits its outcome over a
+/// oneshot. Axum drops this handler future the moment the HTTP client
+/// disconnects; were the gate held here, that drop could land between the
+/// pump-driven consensus commit and the id advance, leaving the committed op's
+/// reply cached under an id the session still considers unused - the next
+/// write would collide into `Duplicate` and silently receive the prior op's
+/// reply. Detaching moves the whole critical section out of the cancellable
+/// future; a disconnect now only drops the receiver half.
async fn submit_committed(
state: &HttpInner,
- session: &HttpSession,
+ session: &Rc<HttpSession>,
operation: Operation,
body: &[u8],
) -> Result<(RequestHeader, Message<GenericHeader>, Option<String>), WriteError> {
@@ -2001,17 +1997,45 @@ async fn submit_committed(
if session.user_id != DEFAULT_ROOT_USER_ID {
return Err(WriteError::Rejected(IggyError::Unauthorized));
}
- // Per-session gate: serialize this session's request ids across the commit
- // await, advancing the id only after an observed Reply so a transient
- // failure leaves it free to retry.
- //
- // TODO(hubcio): cancellation-unsafe. If the client disconnects while parked
- // on the await below, this handler is dropped without advancing the id, yet
- // the prepare still commits pump-driven. The next write on this session then
- // reuses the id, gets the prior op's cached reply (Duplicate), silently loses
- // its own write, and returns a foreign success. Fix by moving the submit into
- // a detached shard-0 task that owns the id advance and signals the handler
- // via a oneshot, so the commit is recorded regardless of client liveness.
+ let (result_slot, committed) = oneshot::channel();
+ let shard = Rc::clone(&state.shard);
+ let session = Rc::clone(session);
+ let body = body.to_vec();
+ // Detached so a client disconnect cannot split a commit from its id
+ // advance; both happen inside the task regardless of handler liveness.
+ compio::runtime::spawn(async move {
+ let result = submit_gated(&shard, &session, operation, &body).await;
+ // A failed send means the handler died mid-await. The id advance
+ // already happened above, which is the invariant that matters.
+ let _ = result_slot.send(result);
+ })
+ .detach();
+ // `Canceled` = the task was dropped before sending (runtime teardown), a
+ // transient server condition like an unanswered submit.
+ committed.await.map_err(|_| WriteError::Unavailable)?
+}
+
+/// Gate-locked core of [`submit_committed`], run on a task the HTTP client
+/// cannot cancel: serializes this session's writes behind its gate and holds
+/// it across the submit so request ids reach the primary strictly in order.
+/// The gate advances only on a committed `Reply`: a transient failure or an
+/// eviction leaves the id free for the caller's retry, which the depth-1
+/// dedup requires (the next accepted request must be `committed + 1`, so a
+/// consumed-but-uncommitted id would wedge the session on `RequestGap`). An
+/// eviction means the session is dead, mapped to 401.
+///
+/// Both shard-0 request rewrites run here before consensus, mirroring the TCP
+/// dispatch path: the PAT rewrite mints a raw token and replicates only its hash
+/// (`CreatePersonalAccessToken`), and the user-password rewrite hashes plaintext
+/// (`CreateUser` / `ChangePassword`). Both are no-ops for every other operation,
+/// so plaintext secrets never enter consensus on any write. On either rewrite's
+/// decode failure the gate is released without advancing, leaving the id free.
+async fn submit_gated(
+ shard: &Rc<ServerNgShard>,
+ session: &HttpSession,
+ operation: Operation,
+ body: &[u8],
+) -> Result<(RequestHeader, Message<GenericHeader>, Option<String>), WriteError> {
let mut next_request_id = session.gate.lock().await;
let message = build_request_message(
operation,
@@ -2024,7 +2048,7 @@ async fn submit_committed(
rewrite_pat_request_for_user(session.user_id, message).map_err(WriteError::Rejected)?;
let message = maybe_rewrite_user_password_request(message).map_err(WriteError::Rejected)?;
let request_header = *message.header();
- let reply = submit_client_request_on_owner(&state.shard, message).await;
+ let reply = submit_client_request_on_owner(shard, message).await;
let Some(reply) = reply else {
return Err(WriteError::Unavailable);
};
@@ -2046,7 +2070,7 @@ async fn submit_committed(
/// the update/delete/purge routes ignore it (it is empty) and answer 204.
async fn submit_write(
state: &HttpInner,
- session: &HttpSession,
+ session: &Rc<HttpSession>,
operation: Operation,
body: &[u8],
) -> Result<Bytes, WriteError> {
@@ -3209,4 +3233,42 @@ mod tests {
assert_eq!(global.get(), MAX_IN_FLIGHT_WRITES_GLOBAL);
drop(readmitted);
}
+
+ // -- detached control-plane submit --
+
+ /// Pins the platform contract `submit_committed`'s cancellation safety
+ /// rests on: a detached compio task keeps running after the handler side
+ /// is gone, the gate advance it performs sticks, and sending the result
+ /// into a dropped oneshot receiver is an ignorable `Err`, never a panic.
+ /// The full hazard window (client disconnect between the consensus commit
+ /// and the id advance) needs a live commit round-trip, so it is covered by
+ /// construction plus the live cancellation smoke, not faked here.
+ #[compio::test]
+ async fn detached_task_advances_gate_and_ignores_dead_receiver() {
+ let session = Rc::new(HttpSession {
+ client_id: 7,
+ session: 1,
+ user_id: DEFAULT_ROOT_USER_ID,
+ expiry: u64::MAX,
+ gate: Mutex::new(FIRST_REQUEST_ID),
+ data_request: Cell::new(FIRST_REQUEST_ID),
+ registry_token: Cell::new(None),
+ in_flight_writes: Cell::new(0),
+ });
+ let (result_slot, committed) = oneshot::channel::<u64>();
+ // The handler future dies (client disconnect) before the task runs.
+ drop(committed);
+ let (done_slot, done) = oneshot::channel::<()>();
+ let task_session = Rc::clone(&session);
+ compio::runtime::spawn(async move {
+ let mut next_request_id = task_session.gate.lock().await;
+ *next_request_id += 1;
+ let _ = result_slot.send(*next_request_id);
+ drop(next_request_id);
+ let _ = done_slot.send(());
+ })
+ .detach();
+ done.await.expect("detached task must run to completion");
+ assert_eq!(*session.gate.lock().await, FIRST_REQUEST_ID + 1);
+ }
}
(iggy) 17/19: feat(server-ng): add HTTP admission control
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 8285c851124eb6bd4829ebc67d8a51ea7f2fa004
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Thu Jul 2 00:30:21 2026 +0200
feat(server-ng): add HTTP admission control
HTTP handlers run on shard 0's single-threaded compio runtime, so
unbounded concurrent writes park unbounded handlers whose buffered
bodies and per-request CPU compete with the consensus pump. Three
bounds close that: the legacy DefaultBodyLimit (413 past
http.max_request_size) caps the per-request term, a per-session
in-flight cap (429) keeps one credential from monopolizing the shared
budget, and a global in-flight budget (503 server_busy) bounds the
multiplier, so budget x max_request_size caps worst-case buffered
bytes and budget x per-request CPU caps pump starvation. One RAII
guard owns both counter releases, so success, errors, the reply
timeout, and client disconnect all release admission through the same
Drop. ack=none produces stay outside admission: they never park a
handler, so their whole cost is the synchronous handler body.
---
core/server-ng/src/bootstrap.rs | 2 +-
core/server-ng/src/http.rs | 245 ++++++++++++++++++++++++++++++++++++++--
2 files changed, 237 insertions(+), 10 deletions(-)
diff --git a/core/server-ng/src/bootstrap.rs b/core/server-ng/src/bootstrap.rs
index a3910964f..87ba615da 100644
--- a/core/server-ng/src/bootstrap.rs
+++ b/core/server-ng/src/bootstrap.rs
@@ -1946,7 +1946,7 @@ async fn start_tcp_runtime(
// caller of this function.
if config.http.enabled {
let http_addr = parse_socket_addr("http.address", &config.http.address)?;
- http::start(shard, http_addr, &config.http.jwt, &config.cluster).await?;
+ http::start(shard, http_addr, &config.http, &config.cluster).await?;
}
Ok(())
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index 596f00a94..e36f4b813 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -28,7 +28,7 @@ use std::net::{IpAddr, SocketAddr};
use std::rc::Rc;
use std::time::Duration;
-use axum::extract::{Path, Query, State};
+use axum::extract::{DefaultBodyLimit, Path, Query, State};
use axum::http::header::LOCATION;
use axum::http::{HeaderName, HeaderValue, StatusCode};
use axum::middleware::map_response;
@@ -37,7 +37,7 @@ use axum::routing::{delete, get, post, put};
use axum::{Json, Router};
use bytes::{Bytes, BytesMut};
use configs::cluster::{ClusterConfig, ClusterNodeConfig};
-use configs::http::HttpJwtConfig;
+use configs::http::HttpConfig;
use consensus::{MetadataHandle, VsrConsensus};
use iggy_binary_protocol::codes::{
GET_CONSUMER_GROUP_CODE, GET_CONSUMER_GROUPS_CODE, GET_STATS_CODE, GET_STREAM_CODE,
@@ -173,6 +173,25 @@ const FIRST_REQUEST_ID: u64 = 1;
/// caller's behalf.
const PARTITION_WRITE_REPLY_TIMEOUT: Duration = Duration::from_secs(10);
+/// Per-session cap on concurrently awaited partition writes (produce /
+/// consumer-offset). Bounds how much of [`MAX_IN_FLIGHT_WRITES_GLOBAL`] one
+/// credential can occupy, so a session that outruns its own commits saturates
+/// itself (429, its own backpressure signal) before it can starve every other
+/// session out of the shared budget.
+const MAX_IN_FLIGHT_WRITES_PER_SESSION: u32 = 32;
+
+/// Shard-0-global budget for concurrently awaited partition writes, across all
+/// sessions. Every admitted write parks a handler for up to
+/// [`PARTITION_WRITE_REPLY_TIMEOUT`] while pinning its buffered body, and its
+/// decode/encode/HS256 CPU runs on the same single-threaded core that pumps
+/// consensus. The budget therefore bounds both starvation terms: budget x
+/// `max_request_size` bounds the worst-case buffered bytes, and budget x
+/// per-request CPU bounds how far admitted HTTP work can delay the consensus
+/// pump. `?ack=none` produces sit outside the budget by design: they never
+/// park a handler, so their whole cost is the synchronous handler body itself,
+/// which admission could not shed anyway.
+const MAX_IN_FLIGHT_WRITES_GLOBAL: u32 = 128;
+
/// VSR client id stamped on HTTP data-plane reads (poll / consumer-offset).
/// HTTP reads never Register a VSR client, and shard-0 client ids are minted
/// from 1, so 0 can never name a live consumer-group member: a group-kind
@@ -234,6 +253,10 @@ struct HttpSession {
/// session eviction can tear the registry entry down fenced by the same
/// token.
registry_token: Cell<Option<InstanceToken>>,
+ /// Awaited partition writes currently in flight on this session, gated by
+ /// [`MAX_IN_FLIGHT_WRITES_PER_SESSION`]. Only [`InFlightWriteGuard`]
+ /// touches it, so every admission is paired with exactly one release.
+ in_flight_writes: Cell<u32>,
}
impl HttpSession {
@@ -309,6 +332,7 @@ impl IntoResponse for WriteError {
/// reply distinguishable only by header (see [`classify_partition_reply`]),
/// and an unanswered write is a distinct outcome the caller must treat as
/// unknown rather than failed.
+#[derive(Debug)]
enum PartitionWriteError {
/// Caller-side rejection (bad identifier, oversized batch, the interim
/// non-root denial) or a malformed reply frame, rendered through the
@@ -321,6 +345,15 @@ enum PartitionWriteError {
/// The in-process reply slot could not be installed. Transient server
/// condition -> the shared 503, retryable.
Unavailable,
+ /// This session is already at [`MAX_IN_FLIGHT_WRITES_PER_SESSION`]
+ /// awaited writes. 429: the caller's own concurrency is the problem, so
+ /// it must drain its outstanding writes before submitting more.
+ TooManyInFlight,
+ /// Shard 0 is already at [`MAX_IN_FLIGHT_WRITES_GLOBAL`] awaited writes
+ /// across all sessions. 503 with its own code (distinct from the shared
+ /// consensus-unavailable body) so an operator can tell admission shedding
+ /// from a consensus outage.
+ ServerBusy,
/// No committed reply within [`PARTITION_WRITE_REPLY_TIMEOUT`], or the
/// session's reply target was torn down mid-wait. 504: the commit may
/// still land (at-least-once), so this is a hard "outcome unknown", not a
@@ -335,6 +368,8 @@ impl IntoResponse for PartitionWriteError {
Self::Rejected(error) => CustomError::from(error).into_response(),
Self::NotFound => CustomError::ResourceNotFound.into_response(),
Self::Unavailable => service_unavailable(),
+ Self::TooManyInFlight => too_many_in_flight_response(),
+ Self::ServerBusy => server_busy_response(),
Self::Timeout(operation) => partition_write_timeout_response(operation),
}
}
@@ -389,6 +424,41 @@ fn service_unavailable() -> Response {
.into_response()
}
+/// 429 for a session at [`MAX_IN_FLIGHT_WRITES_PER_SESSION`] awaited partition
+/// writes. Shaped like every other HTTP error (`ErrorResponse`) so clients
+/// parse one error schema; the remedy is the caller's own: let outstanding
+/// writes finish, then retry.
+fn too_many_in_flight_response() -> Response {
+ (
+ StatusCode::TOO_MANY_REQUESTS,
+ Json(ErrorResponse {
+ id: StatusCode::TOO_MANY_REQUESTS.as_u16().into(),
+ code: "too_many_in_flight_writes".to_owned(),
+ reason: "session reached its in-flight write cap; await outstanding writes and retry"
+ .to_owned(),
+ field: None,
+ }),
+ )
+ .into_response()
+}
+
+/// 503 for shard 0 at [`MAX_IN_FLIGHT_WRITES_GLOBAL`] awaited partition writes
+/// across all sessions. A distinct `server_busy` code (unlike the shared
+/// consensus-unavailable 503) so admission shedding is tellable from a
+/// consensus outage; retry with backoff.
+fn server_busy_response() -> Response {
+ (
+ StatusCode::SERVICE_UNAVAILABLE,
+ Json(ErrorResponse {
+ id: StatusCode::SERVICE_UNAVAILABLE.as_u16().into(),
+ code: "server_busy".to_owned(),
+ reason: "shard is at its in-flight write budget; retry with backoff".to_owned(),
+ field: None,
+ }),
+ )
+ .into_response()
+}
+
/// Read consistency selected by the `?consistency=` query param.
///
/// `serializable` (the default) serves from this node's local metadata STM:
@@ -562,6 +632,10 @@ struct HttpInner {
/// across an `.await` (see [`HttpInner::resolve_session`]).
sessions: RefCell<HashMap<String, Rc<HttpSession>>>,
roster: ClusterRoster,
+ /// Awaited partition writes currently in flight across all sessions, gated
+ /// by [`MAX_IN_FLIGHT_WRITES_GLOBAL`]. Only [`InFlightWriteGuard`] touches
+ /// it, so every admission is paired with exactly one release.
+ in_flight_writes: Cell<u32>,
}
impl HttpInner {
@@ -696,6 +770,7 @@ impl HttpInner {
gate: Mutex::new(FIRST_REQUEST_ID),
data_request: Cell::new(FIRST_REQUEST_ID),
registry_token: Cell::new(None),
+ in_flight_writes: Cell::new(0),
}))
}
}
@@ -728,14 +803,14 @@ type HttpState = SendWrapper<Rc<HttpInner>>;
/// # Errors
///
/// Returns [`ServerNgError`] if the JWT manager cannot be built from
-/// `jwt_config` or the listener cannot bind to `addr`.
+/// `http_config.jwt` or the listener cannot bind to `addr`.
pub async fn start(
shard: &Rc<ServerNgShard>,
addr: SocketAddr,
- jwt_config: &HttpJwtConfig,
+ http_config: &HttpConfig,
cluster: &ClusterConfig,
) -> Result<(), ServerNgError> {
- let jwt = JwtManager::build(jwt_config)?;
+ let jwt = JwtManager::build(&http_config.jwt)?;
let (listener, bound_addr) = client_listener::tcp::bind(addr).await?;
info!(address = %bound_addr, "server-ng HTTP listener started");
@@ -749,8 +824,13 @@ pub async fn start(
nodes: cluster.nodes.clone(),
http_addr: bound_addr,
},
+ in_flight_writes: Cell::new(0),
}));
- let router = router(state);
+ // Saturating: a configured limit past the pointer width (32-bit target,
+ // >4 GiB value) clamps to the largest enforceable cap instead of wrapping.
+ let max_request_size =
+ usize::try_from(http_config.max_request_size.as_bytes_u64()).unwrap_or(usize::MAX);
+ let router = router(state, max_request_size);
let shutdown = shard.bus.token();
let handle = compio::runtime::spawn(async move {
@@ -768,7 +848,12 @@ pub async fn start(
/// Assemble the shard-0 router: unauthenticated health + login routes, plus
/// one authenticated probe that exercises the [`Authenticated`] extractor.
-fn router(state: HttpState) -> Router {
+///
+/// `max_request_size` becomes the router-wide `DefaultBodyLimit` (413 past
+/// it), exactly like the legacy server: it bounds the per-request term of the
+/// admission math - what one body may cost in bytes and decode CPU - while
+/// the in-flight caps bound the multiplier.
+fn router(state: HttpState, max_request_size: usize) -> Router {
// Cloned for the response layer so `X-Iggy-View` reads the live view per
// response; the original `state` is moved into `with_state` below.
let view_source = state.clone();
@@ -834,6 +919,7 @@ fn router(state: HttpState) -> Router {
.route("/clients", get(get_clients))
.route("/clients/{client_id}", get(get_client))
.with_state(state)
+ .layer(DefaultBodyLimit::max(max_request_size))
.layer(map_response(move |response: Response| {
let view_source = view_source.clone();
async move { insert_view_header(&view_source, response) }
@@ -2109,9 +2195,47 @@ const fn resync_required_polled_messages() -> PolledMessages {
}
}
+/// In-flight admission token for one awaited partition write. One guard owns
+/// both releases (session + global) so success, every error return, the reply
+/// timeout, and handler cancellation (the client hanging up mid-await drops
+/// this future) all decrement through the same `Drop`.
+struct InFlightWriteGuard<'a> {
+ session_in_flight: &'a Cell<u32>,
+ global_in_flight: &'a Cell<u32>,
+}
+
+impl Drop for InFlightWriteGuard<'_> {
+ fn drop(&mut self) {
+ self.session_in_flight.set(self.session_in_flight.get() - 1);
+ self.global_in_flight.set(self.global_in_flight.get() - 1);
+ }
+}
+
+/// Admit one awaited partition write against the per-session cap and the
+/// shard-0 global budget, incrementing both counters only when both pass. The
+/// session cap is checked first so a session that saturates itself reads as
+/// its own 429 rather than as server-wide pressure.
+fn admit_partition_write<'a>(
+ session_in_flight: &'a Cell<u32>,
+ global_in_flight: &'a Cell<u32>,
+) -> Result<InFlightWriteGuard<'a>, PartitionWriteError> {
+ if session_in_flight.get() >= MAX_IN_FLIGHT_WRITES_PER_SESSION {
+ return Err(PartitionWriteError::TooManyInFlight);
+ }
+ if global_in_flight.get() >= MAX_IN_FLIGHT_WRITES_GLOBAL {
+ return Err(PartitionWriteError::ServerBusy);
+ }
+ session_in_flight.set(session_in_flight.get() + 1);
+ global_in_flight.set(global_in_flight.get() + 1);
+ Ok(InFlightWriteGuard {
+ session_in_flight,
+ global_in_flight,
+ })
+}
+
/// Run one awaited partition write (produce / consumer-offset write) end to
-/// end: install the reply slot, dispatch into the partition plane, and wait
-/// (bounded) for the committed reply.
+/// end: admit against the in-flight caps, install the reply slot, dispatch
+/// into the partition plane, and wait (bounded) for the committed reply.
///
/// Slot-before-dispatch is load-bearing: every pre-dispatch gate failure
/// inside [`dispatch_partition_request`] replies through `send_to_client`,
@@ -2124,6 +2248,13 @@ async fn partition_write_replicated(
operation: Operation,
body: &[u8],
) -> Result<(), PartitionWriteError> {
+ // Admission sits here rather than before body decode: axum's extractors
+ // already buffered and deserialized the body (bounded by the router-wide
+ // `DefaultBodyLimit`) before the handler ran, so the caps gate what is
+ // actually unbounded - the slot install, the dispatch, and the parked
+ // reply await that pins this request's buffers for up to the reply
+ // timeout. Held across every exit below; released by `Drop`.
+ let _in_flight = admit_partition_write(&session.in_flight_writes, &state.in_flight_writes)?;
ensure_in_process_reply_target(state, session);
let request_id = session.next_data_request_id();
let message = build_request_message(
@@ -2982,4 +3113,100 @@ mod tests {
Err(PartitionWriteError::Rejected(IggyError::InvalidCommand))
));
}
+
+ #[test]
+ fn in_flight_write_guard_decrements_both_counters_on_drop() {
+ let session = Cell::new(0);
+ let global = Cell::new(0);
+ let guard = admit_partition_write(&session, &global).expect("below both caps");
+ assert_eq!(session.get(), 1);
+ assert_eq!(global.get(), 1);
+ drop(guard);
+ assert_eq!(session.get(), 0);
+ assert_eq!(global.get(), 0);
+ }
+
+ #[test]
+ fn admission_at_session_cap_rejects_with_too_many_in_flight() {
+ let session = Cell::new(MAX_IN_FLIGHT_WRITES_PER_SESSION);
+ let global = Cell::new(0);
+ assert!(matches!(
+ admit_partition_write(&session, &global),
+ Err(PartitionWriteError::TooManyInFlight)
+ ));
+ // A refusal must not leak a partial increment on either counter.
+ assert_eq!(session.get(), MAX_IN_FLIGHT_WRITES_PER_SESSION);
+ assert_eq!(global.get(), 0);
+ }
+
+ #[test]
+ fn admission_at_global_budget_rejects_with_server_busy() {
+ let session = Cell::new(0);
+ let global = Cell::new(MAX_IN_FLIGHT_WRITES_GLOBAL);
+ assert!(matches!(
+ admit_partition_write(&session, &global),
+ Err(PartitionWriteError::ServerBusy)
+ ));
+ assert_eq!(session.get(), 0);
+ assert_eq!(global.get(), MAX_IN_FLIGHT_WRITES_GLOBAL);
+ }
+
+ #[test]
+ fn interleaved_admission_reopens_exactly_released_session_slots() {
+ let session = Cell::new(0);
+ let global = Cell::new(0);
+ let mut guards = Vec::new();
+ for _ in 0..MAX_IN_FLIGHT_WRITES_PER_SESSION {
+ guards.push(admit_partition_write(&session, &global).expect("below both caps"));
+ }
+ assert!(matches!(
+ admit_partition_write(&session, &global),
+ Err(PartitionWriteError::TooManyInFlight)
+ ));
+ let released = 3;
+ guards.truncate((MAX_IN_FLIGHT_WRITES_PER_SESSION - released) as usize);
+ assert_eq!(session.get(), MAX_IN_FLIGHT_WRITES_PER_SESSION - released);
+ for _ in 0..released {
+ guards.push(admit_partition_write(&session, &global).expect("released slots"));
+ }
+ assert!(matches!(
+ admit_partition_write(&session, &global),
+ Err(PartitionWriteError::TooManyInFlight)
+ ));
+ drop(guards);
+ assert_eq!(session.get(), 0);
+ assert_eq!(global.get(), 0);
+ }
+
+ #[test]
+ fn global_budget_spans_sessions_and_reopens_after_release() {
+ let global = Cell::new(0);
+ let session_count =
+ MAX_IN_FLIGHT_WRITES_GLOBAL.div_ceil(MAX_IN_FLIGHT_WRITES_PER_SESSION) as usize;
+ let sessions: Vec<Cell<u32>> = (0..session_count).map(|_| Cell::new(0)).collect();
+ let mut guards = Vec::new();
+ 'fill: for session in &sessions {
+ for _ in 0..MAX_IN_FLIGHT_WRITES_PER_SESSION {
+ match admit_partition_write(session, &global) {
+ Ok(guard) => guards.push(guard),
+ Err(PartitionWriteError::ServerBusy) => break 'fill,
+ Err(other) => {
+ panic!("only the global budget may refuse this fill, got {other:?}")
+ }
+ }
+ }
+ }
+ assert_eq!(global.get(), MAX_IN_FLIGHT_WRITES_GLOBAL);
+ // A fresh session is refused on the shared budget, not its own cap.
+ let fresh = Cell::new(0);
+ assert!(matches!(
+ admit_partition_write(&fresh, &global),
+ Err(PartitionWriteError::ServerBusy)
+ ));
+ drop(guards.pop());
+ let readmitted = admit_partition_write(&fresh, &global).expect("budget slot released");
+ assert_eq!(fresh.get(), 1);
+ assert_eq!(global.get(), MAX_IN_FLIGHT_WRITES_GLOBAL);
+ drop(readmitted);
+ }
}
(iggy) 02/19: feat(server-ng): add HTTP listener with login and session binding
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 5c3762dd31aa726dd914181785ebac5346ce6eae
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 11:18:19 2026 +0200
feat(server-ng): add HTTP listener with login and session binding
server-ng had no HTTP/REST surface; management and data access went
through the native binary transports only.
This adds a cyper-axum listener on shard 0 (mirroring the existing
QUIC/TLS placement) serving GET /ping plus JWT and personal-access-
token login. The dispatch spine drops any request without a bound VSR
session, so an authenticated caller cannot submit until a Register
commit binds one. The listener establishes that session lazily on the
first authenticated request and caches it per credential (JWT jti /
PAT hash) rather than per connection, because the physical HTTP socket
is the wrong unit under connection pooling and HTTP/2 multiplexing.
Credential verification is stateless; the Rc-based, !Send session
future is bridged to axum's Send bound with SendWrapper, sound because
compio pins every handler to shard 0's thread. Control writes and the
data plane are not wired yet.
---
core/server-ng/Cargo.toml | 1 +
core/server-ng/src/auth.rs | 18 +-
core/server-ng/src/bootstrap.rs | 33 ++-
core/server-ng/src/http.rs | 379 +++++++++++++++++++++++++++++++++++
core/server-ng/src/http/extractor.rs | 111 ++++++++++
core/server-ng/src/http/jwt.rs | 175 ++++++++++++++++
core/server-ng/src/lib.rs | 1 +
7 files changed, 706 insertions(+), 12 deletions(-)
diff --git a/core/server-ng/Cargo.toml b/core/server-ng/Cargo.toml
index bf42650ed..fd9193c5b 100644
--- a/core/server-ng/Cargo.toml
+++ b/core/server-ng/Cargo.toml
@@ -155,6 +155,7 @@ strum = { workspace = true }
sysinfo = { workspace = true }
tempfile = { workspace = true }
thiserror = { workspace = true }
+tokio = { workspace = true }
toml = { workspace = true }
tower-http = { workspace = true }
tracing = { workspace = true }
diff --git a/core/server-ng/src/auth.rs b/core/server-ng/src/auth.rs
index b8aa393f6..bcfd37de9 100644
--- a/core/server-ng/src/auth.rs
+++ b/core/server-ng/src/auth.rs
@@ -74,6 +74,17 @@ pub(crate) fn verify_pat_credentials(
shard: &Rc<ServerNgShard>,
token: &str,
) -> Result<u32, LoginRegisterError> {
+ verify_pat_credentials_with_expiry(shard, token).map(|(user_id, _)| user_id)
+}
+
+/// Like [`verify_pat_credentials`] but also surfaces the token's expiry (unix
+/// seconds, `u64::MAX` when the PAT never expires). The HTTP extractor keys a
+/// per-token VSR session table on this expiry for lazy eviction; the wire and
+/// login paths only need the user id and go through [`verify_pat_credentials`].
+pub(crate) fn verify_pat_credentials_with_expiry(
+ shard: &Rc<ServerNgShard>,
+ token: &str,
+) -> Result<(u32, u64), LoginRegisterError> {
let token_hash = PersonalAccessToken::hash_token(token);
let now = IggyTimestamp::now();
shard.plane.metadata().mux_stm.users().read(|users| {
@@ -98,7 +109,12 @@ pub(crate) fn verify_pat_credentials(
if user.status != UserStatus::Active {
return Err(LoginRegisterError::UserInactive);
}
- Ok(user.id)
+ // `expiry_at == None` is a never-expiring PAT; map it to `u64::MAX` so
+ // the HTTP session table never expiry-evicts its entry.
+ let expiry = pat
+ .expiry_at
+ .map_or(u64::MAX, |expiry_at| expiry_at.to_secs());
+ Ok((user.id, expiry))
})
}
diff --git a/core/server-ng/src/bootstrap.rs b/core/server-ng/src/bootstrap.rs
index ab82265cc..aa3d4fc0b 100644
--- a/core/server-ng/src/bootstrap.rs
+++ b/core/server-ng/src/bootstrap.rs
@@ -21,6 +21,7 @@ use crate::dispatch::{
make_deferred_replica_message_handler, make_list_clients_handler, make_metadata_submit_handler,
make_partition_read_handler,
};
+use crate::http;
use crate::partition_helpers::{
configure_consumer_offsets, ensure_initial_segment, validate_namespace_bounds,
};
@@ -1919,7 +1920,7 @@ async fn start_tcp_runtime(
accepted_clients: LocalClientAcceptFns,
) -> Result<(), ServerNgError> {
if config.tcp.enabled && !config.tcp.tls.enabled {
- return start_via_replica_io(
+ start_via_replica_io(
shard,
config,
topology,
@@ -1927,18 +1928,28 @@ async fn start_tcp_runtime(
dialed_replica,
accepted_clients,
)
- .await;
+ .await?;
+ } else {
+ start_manual_runtime(
+ shard,
+ config,
+ topology,
+ accepted_replica,
+ dialed_replica,
+ accepted_clients,
+ )
+ .await?;
}
- start_manual_runtime(
- shard,
- config,
- topology,
- accepted_replica,
- dialed_replica,
- accepted_clients,
- )
- .await
+ // HTTP is served over TCP but sits outside the replica_io / manual client
+ // reactor, so it binds independently. Shard-0 gating comes from the sole
+ // caller of this function.
+ if config.http.enabled {
+ let http_addr = parse_socket_addr("http.address", &config.http.address)?;
+ http::start(shard, http_addr, &config.http.jwt).await?;
+ }
+
+ Ok(())
}
async fn start_via_replica_io(
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
new file mode 100644
index 000000000..e4f2d58f4
--- /dev/null
+++ b/core/server-ng/src/http.rs
@@ -0,0 +1,379 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Shard-0 HTTP/REST listener: state bridge, router, and the auth-issuance
+//! half (login + bearer extraction). Management and data-plane routes land
+//! later; this module owns only health, login, and the authenticated probe.
+
+mod extractor;
+mod jwt;
+
+use std::cell::RefCell;
+use std::collections::HashMap;
+use std::net::SocketAddr;
+use std::rc::Rc;
+
+use axum::extract::State;
+use axum::http::StatusCode;
+use axum::response::{IntoResponse, Response};
+use axum::routing::{get, post};
+use axum::{Json, Router};
+use configs::http::HttpJwtConfig;
+use iggy_common::login_user::LoginUser;
+use iggy_common::login_with_personal_access_token::LoginWithPersonalAccessToken;
+use iggy_common::{IdentityInfo, IggyError, IggyTimestamp, TokenInfo};
+use message_bus::client_listener;
+use secrecy::ExposeSecret;
+use send_wrapper::SendWrapper;
+use server::http::error::{CustomError, ErrorResponse};
+use tokio::sync::Mutex;
+use tracing::{error, info, warn};
+
+use crate::auth::{verify_login_credentials, verify_pat_credentials};
+use crate::bootstrap::ServerNgShard;
+use crate::dispatch::submit_register_on_owner;
+use crate::http::extractor::Authenticated;
+use crate::http::jwt::JwtManager;
+use crate::login_register::LoginRegisterError;
+use crate::server_error::ServerNgError;
+
+/// `GET /ping` response body, matching the legacy HTTP server's health probe.
+const PONG: &str = "pong";
+
+/// Hard cap on live per-credential sessions. A leak-guard, not a tuning knob:
+/// reaching it means this many distinct live tokens are in flight at once
+/// (pathological at P1 volume). New sessions past the cap are refused with a
+/// transient 503 rather than evicting a live one; the client retries. Expired
+/// entries are dropped first, so the cap only bites on live oversubscription.
+const MAX_HTTP_SESSIONS: usize = 100_000;
+
+/// First per-session request id the write path (2c) will hand out. VSR request
+/// numbers are 1-based and strictly increasing within a session.
+const FIRST_REQUEST_ID: u64 = 1;
+
+/// One VSR session established for a single login credential (a JWT `jti` or a
+/// PAT). Shared via `Rc` by every concurrent request bearing that credential,
+/// so the session granularity is per-login.
+struct HttpSession {
+ /// Shard-0 client id minted for this credential; its top 16 bits are 0, so
+ /// it shares the shard-0 id space with TCP virtual clients without
+ /// colliding. Consumed by the write path (2c).
+ #[expect(
+ dead_code,
+ reason = "read by the write path (2c) to fill RequestHeader.client"
+ )]
+ client_id: u128,
+ /// Cluster session number returned by the VSR `Register` commit. Consumed
+ /// by the write path (2c).
+ #[expect(
+ dead_code,
+ reason = "read by the write path (2c) to fill RequestHeader.session"
+ )]
+ session: u64,
+ /// User the credential authenticated as. Consumed by the write path (2c).
+ #[expect(dead_code, reason = "read by the write path (2c) for authorization")]
+ user_id: u32,
+ /// Credential expiry in unix seconds (`u64::MAX` = never). Drives lazy
+ /// eviction of stale table entries.
+ expiry: u64,
+ /// Serializes the write path (2c): the guarded value is the NEXT request
+ /// id. A `tokio::sync::Mutex` because 2c holds it across the submit `.await`
+ /// so each session's request numbers stay monotonic and gap-free.
+ #[expect(
+ dead_code,
+ reason = "locked by the write path (2c) to serialize per-session writes"
+ )]
+ gate: Mutex<u64>,
+}
+
+/// Rejection for protected routes.
+///
+/// Two failure classes get two statuses: a missing, invalid, or expired
+/// credential is the caller's fault (401, rendered exactly like the legacy
+/// server so SDK error bodies match), while a VSR `Register` that cannot commit
+/// right now is a transient server condition (503) and must never masquerade as
+/// an auth failure.
+pub enum AuthError {
+ Unauthenticated(IggyError),
+ SessionUnavailable,
+}
+
+impl From<IggyError> for AuthError {
+ fn from(error: IggyError) -> Self {
+ Self::Unauthenticated(error)
+ }
+}
+
+impl IntoResponse for AuthError {
+ fn into_response(self) -> Response {
+ match self {
+ // Reuse the legacy `IggyError -> CustomError` rendering so 401
+ // bodies stay byte-identical to what the SDKs are tested against.
+ Self::Unauthenticated(error) => CustomError::from(error).into_response(),
+ // Register could not commit (no caught-up primary, pipeline full, or
+ // a view-change cancel). Transient server condition -> 503, retryable.
+ Self::SessionUnavailable => (
+ StatusCode::SERVICE_UNAVAILABLE,
+ Json(ErrorResponse::from_error(
+ IggyError::CannotEstablishConnection,
+ )),
+ )
+ .into_response(),
+ }
+ }
+}
+
+/// Shared shard-0 HTTP state.
+///
+/// Groups the shard handle, the JWT issuer/verifier, and the per-credential VSR
+/// session table so every handler and the [`Authenticated`] extractor reach
+/// them through one axum `State`.
+struct HttpInner {
+ shard: Rc<ServerNgShard>,
+ jwt: JwtManager,
+ /// Per-credential VSR sessions keyed by JWT `jti` / PAT hash. `RefCell` is
+ /// sound here - shard 0 is single-threaded and the `SendWrapper` state
+ /// bridge tolerates the `!Sync` interior - but the guard must never be held
+ /// across an `.await` (see [`HttpInner::resolve_session`]).
+ sessions: RefCell<HashMap<String, Rc<HttpSession>>>,
+}
+
+impl HttpInner {
+ /// Resolve the VSR session for `key`, minting and Registering one on first
+ /// use. Every later request bearing the same credential reuses it.
+ ///
+ /// Borrow discipline: the `RefCell` table guard is taken, read, and dropped
+ /// WITHOUT crossing the `.await`. Holding it across the Register suspend
+ /// would panic the moment a sibling shard-0 task borrowed the table while
+ /// this one is parked (single-threaded `RefCell` + cooperative scheduling).
+ async fn resolve_session(
+ &self,
+ key: String,
+ user_id: u32,
+ expiry: u64,
+ ) -> Result<Rc<HttpSession>, AuthError> {
+ let now = IggyTimestamp::now().to_secs();
+ if let Some(session) = self.live_session(&key, now) {
+ return Ok(session);
+ }
+
+ // Miss: mint + Register with no borrow held (an async VSR commit).
+ let fresh = self.register_session(user_id, expiry).await?;
+
+ let mut table = self.sessions.borrow_mut();
+ // A concurrent first request for the same key may have Registered and
+ // inserted while we were parked. Last-writer-wins: reuse the installed
+ // entry and drop `fresh`, whose client id is then orphaned on the peers
+ // until they evict it - an accepted rare cost at P1 volume.
+ if let Some(session) = live_entry(&table, &key, now) {
+ return Ok(session);
+ }
+ table.retain(|_, session| session.expiry > now);
+ if table.len() >= MAX_HTTP_SESSIONS {
+ // Still full after dropping expired entries: too many genuinely live
+ // sessions. Refuse rather than evict a live one; the client retries.
+ return Err(AuthError::SessionUnavailable);
+ }
+ table.insert(key, Rc::clone(&fresh));
+ Ok(fresh)
+ }
+
+ /// Clone the live (non-expired) entry for `key`, if present. Confines the
+ /// shared `RefCell` borrow to this call so it can never span an `.await`.
+ fn live_session(&self, key: &str, now_secs: u64) -> Option<Rc<HttpSession>> {
+ live_entry(&self.sessions.borrow(), key, now_secs)
+ }
+
+ /// Mint a shard-0 client id and run the VSR `Register` for a fresh session.
+ /// Holds no table borrow; the caller inserts the result.
+ async fn register_session(
+ &self,
+ user_id: u32,
+ expiry: u64,
+ ) -> Result<Rc<HttpSession>, AuthError> {
+ let coordinator = self
+ .shard
+ .coordinator()
+ .ok_or(AuthError::SessionUnavailable)?;
+ // Reuse the TCP accept path's minter: it draws from the same shard-0
+ // `client_seq`, so an HTTP session id can never collide with a TCP
+ // virtual client's and the shard-0 tag (top 16 bits == 0) is preserved.
+ let client_id = coordinator.mint_shard_zero_client_id();
+ // The minter seeds at 1, so 0 is only reachable after a 2^112 wrap.
+ // Guard anyway: `submit_register_in_process` asserts `client_id != 0`,
+ // and an assert on this request path would be a panic.
+ if client_id == 0 {
+ return Err(AuthError::SessionUnavailable);
+ }
+ // Shared Register entry point; on shard 0 (always, for HTTP) it runs
+ // `submit_register_in_process` directly on the metadata owner.
+ let session = submit_register_on_owner(&self.shard, client_id)
+ .await
+ .map_err(|error| {
+ warn!(?error, "server-ng HTTP: VSR Register submit failed");
+ AuthError::SessionUnavailable
+ })?;
+ Ok(Rc::new(HttpSession {
+ client_id,
+ session,
+ user_id,
+ expiry,
+ gate: Mutex::new(FIRST_REQUEST_ID),
+ }))
+ }
+}
+
+/// Borrow-and-clone a live table entry, or `None` if missing or expired. Shared
+/// by the fast path and the post-Register re-check so neither leaks a guard.
+fn live_entry(
+ table: &HashMap<String, Rc<HttpSession>>,
+ key: &str,
+ now_secs: u64,
+) -> Option<Rc<HttpSession>> {
+ table
+ .get(key)
+ .filter(|session| session.expiry > now_secs)
+ .map(Rc::clone)
+}
+
+/// Axum router state: shard-0's [`HttpInner`] behind an `Rc`, `!Send` yet
+/// bridged into axum's `Send + Sync` requirement by `SendWrapper`. Sound
+/// because the listener and every handler run on shard 0's compio thread - the
+/// same thread that builds this state. Never touch it off that thread.
+type HttpState = SendWrapper<Rc<HttpInner>>;
+
+/// Bind the shard-0 HTTP listener and spawn the `cyper-axum` serve loop as a
+/// background task on shard 0's compio runtime.
+///
+/// The caller gates this to shard 0 and to `http.enabled`; the listener stops
+/// when the bus shutdown token fires.
+///
+/// # Errors
+///
+/// Returns [`ServerNgError`] if the JWT manager cannot be built from
+/// `jwt_config` or the listener cannot bind to `addr`.
+pub async fn start(
+ shard: &Rc<ServerNgShard>,
+ addr: SocketAddr,
+ jwt_config: &HttpJwtConfig,
+) -> Result<(), ServerNgError> {
+ let jwt = JwtManager::build(jwt_config)?;
+ let (listener, bound_addr) = client_listener::tcp::bind(addr).await?;
+ info!(address = %bound_addr, "server-ng HTTP listener started");
+
+ let state: HttpState = SendWrapper::new(Rc::new(HttpInner {
+ shard: Rc::clone(shard),
+ jwt,
+ sessions: RefCell::new(HashMap::new()),
+ }));
+ let router = router(state);
+
+ let shutdown = shard.bus.token();
+ let handle = compio::runtime::spawn(async move {
+ if let Err(error) = cyper_axum::serve(listener, router)
+ .with_graceful_shutdown(async move { shutdown.wait().await })
+ .await
+ {
+ error!(%error, "server-ng HTTP listener terminated with error");
+ }
+ });
+ shard.bus.track_background(handle);
+
+ Ok(())
+}
+
+/// Assemble the shard-0 router: unauthenticated health + login routes, plus
+/// one authenticated probe that exercises the [`Authenticated`] extractor.
+fn router(state: HttpState) -> Router {
+ Router::new()
+ .route("/ping", get(ping))
+ .route("/users/login", post(login_user))
+ .route(
+ "/personal-access-tokens/login",
+ post(login_with_personal_access_token),
+ )
+ .route("/users/me", get(get_me))
+ .with_state(state)
+}
+
+/// Extracting the state here proves at compile time that the `!Send` state
+/// bridges into axum's `Send + Sync` router state on shard 0's compio thread.
+/// Ping needs no state, so it is discarded.
+async fn ping(State(_state): State<HttpState>) -> &'static str {
+ PONG
+}
+
+async fn login_user(
+ State(state): State<HttpState>,
+ Json(command): Json<LoginUser>,
+) -> Result<Json<IdentityInfo>, CustomError> {
+ let user_id = verify_login_credentials(
+ &state.shard,
+ &command.username,
+ command.password.expose_secret(),
+ )
+ .map_err(|error| login_error_to_iggy(&error))?;
+ issue_identity(&state, user_id)
+}
+
+async fn login_with_personal_access_token(
+ State(state): State<HttpState>,
+ Json(command): Json<LoginWithPersonalAccessToken>,
+) -> Result<Json<IdentityInfo>, CustomError> {
+ let user_id = verify_pat_credentials(&state.shard, command.token.expose_secret())
+ .map_err(|error| login_error_to_iggy(&error))?;
+ issue_identity(&state, user_id)
+}
+
+/// Interim authenticated probe: echoes the caller's resolved `user_id` so the
+/// [`Authenticated`] extractor is covered end to end. Iggy exposes no "current
+/// identity" route to reuse. This path also shadows a future
+/// `GET /users/{user_id}` and must be reconciled when that route is added.
+async fn get_me(identity: Authenticated) -> Json<IdentityInfo> {
+ Json(IdentityInfo {
+ user_id: identity.user_id,
+ access_token: None,
+ })
+}
+
+/// Issue a fresh access token for `user_id` and wrap it in the exact
+/// `IdentityInfo` shape the SDKs pin: numeric `user_id` plus an `access_token`
+/// carrying the token string and its unix-seconds expiry.
+fn issue_identity(inner: &HttpInner, user_id: u32) -> Result<Json<IdentityInfo>, CustomError> {
+ let generated = inner.jwt.generate(user_id)?;
+ Ok(Json(IdentityInfo {
+ user_id: generated.user_id,
+ access_token: Some(TokenInfo {
+ token: generated.access_token,
+ expiry: generated.access_token_expiry,
+ }),
+ }))
+}
+
+/// Grade a credential-verification failure onto the `IggyError` the legacy
+/// HTTP error map already maps to a status + body, so `CustomError` renders
+/// what the SDKs are tested against. `verify_*` only yield the first three
+/// variants; the tail is unreachable but kept terminal (401) for the
+/// `#[non_exhaustive]` enum.
+const fn login_error_to_iggy(error: &LoginRegisterError) -> IggyError {
+ match error {
+ LoginRegisterError::InvalidCredentials => IggyError::InvalidCredentials,
+ LoginRegisterError::InvalidToken => IggyError::InvalidPersonalAccessToken,
+ LoginRegisterError::UserInactive => IggyError::UserInactive,
+ _ => IggyError::Unauthenticated,
+ }
+}
diff --git a/core/server-ng/src/http/extractor.rs b/core/server-ng/src/http/extractor.rs
new file mode 100644
index 000000000..fd2e90051
--- /dev/null
+++ b/core/server-ng/src/http/extractor.rs
@@ -0,0 +1,111 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Bearer-credential extractor for protected shard-0 HTTP routes.
+
+use std::rc::Rc;
+
+use axum::extract::FromRequestParts;
+use axum::http::header::AUTHORIZATION;
+use axum::http::request::Parts;
+use iggy_common::{IggyError, PersonalAccessToken};
+use send_wrapper::SendWrapper;
+
+use super::{AuthError, HttpSession, HttpState};
+use crate::auth::verify_pat_credentials_with_expiry;
+
+/// Bearer scheme prefix in the `Authorization` header.
+const BEARER: &str = "Bearer ";
+
+/// Key-space prefixes: a JWT `jti` and a PAT hash live in disjoint namespaces
+/// so the two credential kinds can never collide on the same table key.
+const JWT_KEY_PREFIX: &str = "jwt:";
+const PAT_KEY_PREFIX: &str = "pat:";
+
+/// Resolved caller identity for a protected route: the numeric user id plus the
+/// shared VSR session established once per presenting credential.
+pub struct Authenticated {
+ pub user_id: u32,
+ /// Per-credential VSR session. The write path (2c) reads its client id and
+ /// session number and serializes writes through its gate; `get_me` and
+ /// other read-only probes ignore it.
+ ///
+ /// Wrapped because axum requires every handler argument to be `Send` and
+ /// `Rc` is not; this mirrors the `HttpState` bridge. Sound for the same
+ /// reason: the extractor mints it on shard 0's compio thread and every
+ /// handler runs on that same thread, so the wrapper is never crossed.
+ #[expect(
+ dead_code,
+ reason = "read by the write path (2c) via client id, session, and gate"
+ )]
+ pub session: SendWrapper<Rc<HttpSession>>,
+}
+
+impl FromRequestParts<HttpState> for Authenticated {
+ type Rejection = AuthError;
+
+ async fn from_request_parts(
+ parts: &mut Parts,
+ state: &HttpState,
+ ) -> Result<Self, Self::Rejection> {
+ let bearer = parts
+ .headers
+ .get(AUTHORIZATION)
+ .and_then(|value| value.to_str().ok())
+ .and_then(|value| value.strip_prefix(BEARER))
+ .ok_or(IggyError::AccessTokenMissing)?;
+
+ let (key, user_id, expiry) = resolve_credential(state, bearer)?;
+
+ // `resolve_session` is `Rc`-based and `!Send`, yet axum requires this
+ // extractor future to be `Send`. `SendWrapper` bridges the gap: sound
+ // because compio pins the future to shard 0's thread - the only thread
+ // that ever touches the session table (mirrors legacy `HttpSafeShard`).
+ let session = SendWrapper::new(state.resolve_session(key, user_id, expiry)).await?;
+ Ok(Self {
+ user_id,
+ session: SendWrapper::new(session),
+ })
+ }
+}
+
+/// Map a bearer credential to its `(session key, user id, expiry secs)`.
+///
+/// A JWT is the primary credential, keyed by its `jti`. A raw PAT presented as
+/// the bearer is the documented fallback, keyed by its SHA-256 hash - the same
+/// digest Iggy already indexes PATs by, so the key is stable and collision-free
+/// while the raw secret never enters the table. Both checks are local and
+/// synchronous, so no borrow is held across an await here.
+fn resolve_credential(state: &HttpState, bearer: &str) -> Result<(String, u32, u64), AuthError> {
+ if let Ok(claims) = state.jwt.decode(bearer)
+ && let Ok(user_id) = claims.sub.parse::<u32>()
+ {
+ return Ok((
+ format!("{JWT_KEY_PREFIX}{}", claims.jti),
+ user_id,
+ claims.exp,
+ ));
+ }
+
+ let (user_id, expiry) = verify_pat_credentials_with_expiry(&state.shard, bearer)
+ .map_err(|_| IggyError::Unauthenticated)?;
+ let key = format!(
+ "{PAT_KEY_PREFIX}{}",
+ PersonalAccessToken::hash_token(bearer)
+ );
+ Ok((key, user_id, expiry))
+}
diff --git a/core/server-ng/src/http/jwt.rs b/core/server-ng/src/http/jwt.rs
new file mode 100644
index 000000000..e8d61b337
--- /dev/null
+++ b/core/server-ng/src/http/jwt.rs
@@ -0,0 +1,175 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Minimal JWT issuer/verifier for the shard-0 HTTP listener.
+//!
+//! Ported from the legacy `server::http::jwt::jwt_manager::JwtManager`, reduced
+//! to the issue + verify half: no revoked-token persistence, no JWKS /
+//! trusted-issuer path, no refresh. The claim set (`JwtClaims`, including the
+//! `jti`) and the `IggyError -> HTTP` grading are reused verbatim so issued
+//! tokens and error bodies stay identical to the legacy server.
+
+use std::ops::Range;
+
+use configs::http::HttpJwtConfig;
+use iggy_common::{IggyDuration, IggyError, IggyExpiry, IggyTimestamp, UserId};
+use jsonwebtoken::{Algorithm, DecodingKey, EncodingKey, Header, Validation, decode, encode};
+use server::http::jwt::json_web_token::{Audience, GeneratedToken, JwtClaims};
+use server::streaming::utils::crypto;
+use tracing::warn;
+use uuid::Uuid;
+
+/// Length window for the random secret minted when no secret is configured.
+/// Matches the legacy server so both behave identically on an empty secret.
+const GENERATED_SECRET_LEN: Range<usize> = 32..64;
+
+/// Expiry stamp used for a non-expiring token: far enough out to never trip
+/// `exp` validation, small enough to fit `u32`. Mirrors the legacy server.
+const NEVER_EXPIRE_SECS: u32 = 1_000_000_000;
+
+/// Fixed-lifetime issuer/verifier for HTTP bearer tokens on shard 0.
+pub struct JwtManager {
+ algorithm: Algorithm,
+ issuer: String,
+ audience: String,
+ access_token_expiry: IggyExpiry,
+ not_before: IggyDuration,
+ encoding_key: EncodingKey,
+ decoding_key: DecodingKey,
+ validation: Validation,
+}
+
+impl JwtManager {
+ /// Build the manager from the `[http.jwt]` config, normalizing an
+ /// unconfigured secret the same way the legacy server does (mint a random
+ /// ephemeral secret, or mirror whichever half is set).
+ ///
+ /// # Errors
+ ///
+ /// Returns [`IggyError`] if the configured algorithm is unsupported or a
+ /// secret cannot be turned into an encoding/decoding key.
+ pub fn build(config: &HttpJwtConfig) -> Result<Self, IggyError> {
+ let config = normalize_secrets(config.clone());
+ let algorithm = config.get_algorithm()?;
+ let encoding_key = config.get_encoding_key()?;
+ let decoding_key = config.get_decoding_key()?;
+ let mut validation = Validation::new(algorithm);
+ validation.set_issuer(&config.valid_issuers);
+ validation.set_audience(&config.valid_audiences);
+ validation.leeway = u64::from(config.clock_skew.as_secs());
+ Ok(Self {
+ algorithm,
+ issuer: config.issuer,
+ audience: config.audience,
+ access_token_expiry: config.access_token_expiry,
+ not_before: config.not_before,
+ encoding_key,
+ decoding_key,
+ validation,
+ })
+ }
+
+ /// Issue a signed access token for `user_id` carrying a unique `jti` plus
+ /// issued-at / not-before / expiry stamps derived from the config.
+ ///
+ /// # Errors
+ ///
+ /// Returns [`IggyError::CannotGenerateJwt`] if signing fails.
+ pub fn generate(&self, user_id: UserId) -> Result<GeneratedToken, IggyError> {
+ let header = Header::new(self.algorithm);
+ let iat = IggyTimestamp::now().to_secs();
+ let expiry_secs = match self.access_token_expiry {
+ IggyExpiry::NeverExpire => NEVER_EXPIRE_SECS,
+ IggyExpiry::ServerDefault => 0,
+ IggyExpiry::ExpireDuration(duration) => duration.as_secs(),
+ };
+ let exp = iat + u64::from(expiry_secs);
+ let nbf = iat + u64::from(self.not_before.as_secs());
+ let claims = JwtClaims {
+ jti: Uuid::now_v7().to_string(),
+ sub: user_id.to_string(),
+ aud: Audience::from(self.audience.clone()),
+ iss: self.issuer.clone(),
+ iat,
+ exp,
+ nbf,
+ };
+ let access_token = encode::<JwtClaims>(&header, &claims, &self.encoding_key)
+ .map_err(|_| IggyError::CannotGenerateJwt)?;
+ Ok(GeneratedToken {
+ user_id,
+ access_token,
+ access_token_expiry: exp,
+ })
+ }
+
+ /// Verify a token's signature and registered claims, returning its claim
+ /// set (the `jti` is what a session table keys on).
+ ///
+ /// # Errors
+ ///
+ /// Returns [`IggyError::Unauthenticated`] if the token is malformed,
+ /// expired, or fails issuer/audience/signature validation.
+ pub fn decode(&self, token: &str) -> Result<JwtClaims, IggyError> {
+ decode::<JwtClaims>(token, &self.decoding_key, &self.validation)
+ .map(|data| data.claims)
+ .map_err(|_| IggyError::Unauthenticated)
+ }
+}
+
+/// Fill in an unconfigured JWT secret exactly like the legacy HTTP server:
+/// both empty -> one random ephemeral secret; one empty -> mirror the other;
+/// both set but different under an HMAC algorithm -> warn (they must match).
+fn normalize_secrets(mut config: HttpJwtConfig) -> HttpJwtConfig {
+ match (
+ config.encoding_secret.is_empty(),
+ config.decoding_secret.is_empty(),
+ ) {
+ (true, true) => {
+ let secret = crypto::generate_secret(GENERATED_SECRET_LEN);
+ let redacted: String = secret.chars().take(3).collect();
+ warn!(
+ "JWT encoding and decoding secrets are not configured - generated a random secret: {redacted}***. JWT tokens will be invalidated on server restart. Set 'encoding_secret' and 'decoding_secret' in the config to use persistent secrets."
+ );
+ config.encoding_secret.clone_from(&secret);
+ config.decoding_secret = secret;
+ }
+ (true, false) => {
+ warn!(
+ "JWT encoding secret is not configured but decoding secret is set - using decoding secret for both. Set 'encoding_secret' in the config to avoid this warning."
+ );
+ config.encoding_secret = config.decoding_secret.clone();
+ }
+ (false, true) => {
+ warn!(
+ "JWT decoding secret is not configured but encoding secret is set - using encoding secret for both. Set 'decoding_secret' in the config to avoid this warning."
+ );
+ config.decoding_secret = config.encoding_secret.clone();
+ }
+ (false, false) => {
+ if config.encoding_secret != config.decoding_secret
+ && config.algorithm.starts_with("HS")
+ {
+ warn!(
+ "JWT encoding and decoding secrets are different but algorithm is {} (HMAC) - both secrets must be identical for symmetric algorithms.",
+ config.algorithm
+ );
+ }
+ }
+ }
+ config
+}
diff --git a/core/server-ng/src/lib.rs b/core/server-ng/src/lib.rs
index 78d9847b8..cd212e98b 100644
--- a/core/server-ng/src/lib.rs
+++ b/core/server-ng/src/lib.rs
@@ -22,6 +22,7 @@ pub mod bootstrap;
pub mod config_writer;
pub mod consumer_group;
pub mod dispatch;
+pub(crate) mod http;
pub mod login_register;
pub mod partition_helpers;
pub mod partition_reconciler;
(iggy) 14/19: feat(server-ng): add HTTP produce route
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit cdd4bea1441fa29b336d22d2fe3e94790313ad13
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 23:22:52 2026 +0200
feat(server-ng): add HTTP produce route
An HTTP caller holds no consensus socket, so it could not produce:
partition replies fire post-commit through send_to_client and the
control-plane submit path never touches the partition groups.
POST /streams/{stream_id}/topics/{topic_id}/messages accepts the
legacy SendMessages JSON and encodes it through the SDK's own wire
path (identifier/partitioning conversion + SendMessagesEncoder), so
HTTP and TCP produce byte-identical request bodies. The handler
installs a request-keyed in-process reply slot before dispatching
through the shared partition gates; one slot catches the committed
reply and every pre-dispatch gate failure. Partition replies carry
no error code - success and gate failure are both empty-bodied - so
the reply's op discriminates: a committed reply carries its
prepare's op (>= 1), a gate failure replies via build_empty_reply
with 0 and renders as the legacy 404.
The default answers 201 + X-Iggy-Durability: replicated-memory after
the quorum commit, bounded at 10s (504 = outcome unknown: the
partition plane is at-least-once, the prepare may still land).
?ack=none answers 202 right after dispatch; its reply sheds at the
bus. Data-plane request ids ride a separate gate-free per-session
counter because partition ops have no dedup and concurrent produces
are legal - they must not serialize behind the control-plane gate.
Session eviction tears the reply target down token-fenced,
cancelling parked waiters.
---
core/server-ng/src/http.rs | 491 ++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 481 insertions(+), 10 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index 7d1799f74..ab7c6053a 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -22,10 +22,11 @@
mod extractor;
mod jwt;
-use std::cell::RefCell;
+use std::cell::{Cell, RefCell};
use std::collections::HashMap;
use std::net::{IpAddr, SocketAddr};
use std::rc::Rc;
+use std::time::Duration;
use axum::extract::{Path, Query, State};
use axum::http::header::LOCATION;
@@ -34,7 +35,7 @@ use axum::middleware::map_response;
use axum::response::{IntoResponse, Response};
use axum::routing::{delete, get, post, put};
use axum::{Json, Router};
-use bytes::Bytes;
+use bytes::{Bytes, BytesMut};
use configs::cluster::{ClusterConfig, ClusterNodeConfig};
use configs::http::HttpJwtConfig;
use consensus::{MetadataHandle, VsrConsensus};
@@ -49,6 +50,7 @@ use iggy_binary_protocol::requests::consumer_groups::{
CreateConsumerGroupRequest, DeleteConsumerGroupRequest, GetConsumerGroupRequest,
GetConsumerGroupsRequest,
};
+use iggy_binary_protocol::requests::messages::{RawMessage, SendMessagesEncoder};
use iggy_binary_protocol::requests::personal_access_tokens::{
CreatePersonalAccessTokenRequest, DeletePersonalAccessTokenRequest,
};
@@ -80,7 +82,7 @@ use iggy_binary_protocol::responses::users::get_user::UserDetailsResponse;
use iggy_binary_protocol::responses::users::get_users::GetUsersResponse;
use iggy_binary_protocol::version::IGGY_PROTOCOL_VERSION;
use iggy_binary_protocol::{
- GenericHeader, Operation, RequestHeader, WireDecode, WireEncode, WireName,
+ GenericHeader, Operation, ReplyHeader, RequestHeader, WireDecode, WireEncode, WireName,
};
use iggy_common::change_password::ChangePassword;
use iggy_common::create_consumer_group::CreateConsumerGroup;
@@ -96,16 +98,17 @@ use iggy_common::update_stream::UpdateStream;
use iggy_common::update_topic::UpdateTopic;
use iggy_common::update_user::UpdateUser;
use iggy_common::wire_conversions::{
- clients_from_wire, consumer_groups_from_wire, identifier_to_wire, permissions_to_wire,
- streams_from_wire, topics_from_wire, users_from_wire,
+ clients_from_wire, consumer_groups_from_wire, identifier_to_wire, partitioning_to_wire,
+ permissions_to_wire, streams_from_wire, topics_from_wire, users_from_wire,
};
use iggy_common::{
ClientInfo, ClientInfoDetails, ClusterMetadata, ClusterNode, ClusterNodeRole,
ClusterNodeStatus, ConsumerGroup, ConsumerGroupDetails, Identifier, IdentityInfo, IggyError,
- IggyTimestamp, RawPersonalAccessToken, Stats, Stream, StreamDetails, TokenInfo, Topic,
- TopicDetails, TransportEndpoints, UserInfo, UserInfoDetails, Validatable,
+ IggyMessageView, IggyTimestamp, RawPersonalAccessToken, SendMessages, Stats, Stream,
+ StreamDetails, TokenInfo, Topic, TopicDetails, TransportEndpoints, UserInfo, UserInfoDetails,
+ Validatable,
};
-use message_bus::client_listener;
+use message_bus::{BusMessage, InstanceToken, client_listener};
use metadata::impls::metadata::StreamsFrontend;
use secrecy::ExposeSecret;
use send_wrapper::SendWrapper;
@@ -117,7 +120,9 @@ use tracing::{error, info, warn};
use crate::auth::{verify_login_credentials, verify_pat_credentials};
use crate::bootstrap::ServerNgShard;
-use crate::dispatch::{submit_client_request_on_owner, submit_register_on_owner};
+use crate::dispatch::{
+ dispatch_partition_request, submit_client_request_on_owner, submit_register_on_owner,
+};
use crate::http::extractor::{Authenticated, Identity};
use crate::http::jwt::JwtManager;
use crate::login_register::LoginRegisterError;
@@ -143,6 +148,21 @@ const MAX_HTTP_SESSIONS: usize = 100_000;
/// are 1-based and strictly increasing within a session.
const FIRST_REQUEST_ID: u64 = 1;
+/// Bound on a produce's wait for its committed partition reply. Long enough to
+/// ride out a view change (plus the dispatch gates' own routable-wait budget),
+/// short enough not to pin HTTP connections behind a dead consensus group. On
+/// expiry the caller gets 504 and must treat the outcome as unknown: the
+/// partition plane is at-least-once and the prepare may still commit after the
+/// wait gave up, so the server never retries on the caller's behalf.
+const PRODUCE_REPLY_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// Response header attesting what durability a produce response proves:
+/// [`DURABILITY_REPLICATED_MEMORY`] after an awaited quorum commit,
+/// [`DURABILITY_NONE`] for a `?ack=none` fire-and-forget.
+const DURABILITY_HEADER: HeaderName = HeaderName::from_static("x-iggy-durability");
+const DURABILITY_REPLICATED_MEMORY: &str = "replicated-memory";
+const DURABILITY_NONE: &str = "none";
+
/// Node name reported for the synthesized single self-node, matching the legacy
/// server's single-node synthesis.
const SELF_NODE_NAME: &str = "iggy-node";
@@ -177,6 +197,27 @@ struct HttpSession {
/// submit `.await` so each session's request numbers reach the primary in
/// order and stay gap-free for the depth-1 consensus dedup.
gate: Mutex<u64>,
+ /// Next data-plane request id. A separate, gate-free counter: partition ops
+ /// are at-least-once with no consensus dedup, so the id only correlates the
+ /// in-process reply slot and concurrent produces on one session are legal.
+ /// A plain `Cell` suffices on single-threaded shard 0; ids are minted
+ /// monotonically and never reused, which the slot-guard contract requires.
+ data_request: Cell<u64>,
+ /// Registry token of this session's lazily-installed in-process reply
+ /// target (`None` until the first awaited produce). Stored so session
+ /// eviction can tear the registry entry down fenced by the same token.
+ registry_token: Cell<Option<InstanceToken>>,
+}
+
+impl HttpSession {
+ /// Mint the next data-plane request id. Also consumed by the `?ack=none`
+ /// path, which installs no slot: sharing one counter keeps a shed reply's
+ /// id from ever colliding with a live awaited slot on this session.
+ fn next_data_request_id(&self) -> u64 {
+ let id = self.data_request.get();
+ self.data_request.set(id + 1);
+ id
+ }
}
/// Rejection for protected routes.
@@ -233,6 +274,58 @@ impl IntoResponse for WriteError {
}
}
+/// Rejection for a data-plane produce (`POST .../messages`).
+///
+/// Split differently from [`WriteError`] because the partition plane replies
+/// carry no committed error code: a pre-dispatch gate failure is an empty
+/// reply distinguishable only by header (see [`classify_partition_reply`]),
+/// and an unanswered produce is a distinct outcome the caller must treat as
+/// unknown rather than failed.
+enum ProduceError {
+ /// Caller-side rejection (bad identifier, oversized batch, the interim
+ /// non-root denial) or a malformed reply frame, rendered through the
+ /// legacy `IggyError -> status` map for SDK-identical bodies.
+ Rejected(IggyError),
+ /// The dispatch gates could not route the produce: the stream, topic, or
+ /// partition does not resolve (or never materialised within the routable
+ /// budget). Rendered as the legacy 404 body.
+ NotFound,
+ /// The in-process reply slot could not be installed. Transient server
+ /// condition -> the shared 503, retryable.
+ Unavailable,
+ /// No committed reply within [`PRODUCE_REPLY_TIMEOUT`], or the session's
+ /// reply target was torn down mid-wait. 504: the commit may still land
+ /// (at-least-once), so this is a hard "outcome unknown", not a failure
+ /// the server may transparently retry.
+ Timeout,
+}
+
+impl IntoResponse for ProduceError {
+ fn into_response(self) -> Response {
+ match self {
+ Self::Rejected(error) => CustomError::from(error).into_response(),
+ Self::NotFound => CustomError::ResourceNotFound.into_response(),
+ Self::Unavailable => service_unavailable(),
+ Self::Timeout => produce_timeout_response(),
+ }
+ }
+}
+
+/// 504 body for a produce whose commit outcome is unknown. Shaped like every
+/// other HTTP error (`ErrorResponse`) so clients parse one error schema.
+fn produce_timeout_response() -> Response {
+ (
+ StatusCode::GATEWAY_TIMEOUT,
+ Json(ErrorResponse {
+ id: StatusCode::GATEWAY_TIMEOUT.as_u16().into(),
+ code: "produce_timeout".to_owned(),
+ reason: "produce was not acknowledged in time; the write may still commit".to_owned(),
+ field: None,
+ }),
+ )
+ .into_response()
+}
+
/// The shared 503 body for a request that could not commit right now: no
/// caught-up primary, a full pipeline, or a view-change cancel. Retryable, and
/// rendered with the `CannotEstablishConnection` code the SDKs treat as a
@@ -269,6 +362,28 @@ struct ConsistencyQuery {
consistency: Consistency,
}
+/// Produce acknowledgement selected by the `?ack=` query param.
+///
+/// `replicated` (the default) answers 201 only after the partition group's
+/// quorum commit. `none` is fire-and-forget: the request is validated,
+/// dispatched, and answered 202 immediately; the commit still happens, but its
+/// reply is shed at the bus (no reply slot is installed).
+#[derive(Clone, Copy, Default, PartialEq, Eq, Deserialize)]
+#[serde(rename_all = "lowercase")]
+enum ProduceAck {
+ #[default]
+ Replicated,
+ None,
+}
+
+/// `?ack=` query wrapper. An absent param defaults to
+/// [`ProduceAck::Replicated`]; an unrecognized value is a 400 (axum `Query`).
+#[derive(Default, Deserialize)]
+struct ProduceQuery {
+ #[serde(default)]
+ ack: ProduceAck,
+}
+
/// Rejection for an authenticated read route (`GET /streams`,
/// `GET /streams/{id}`, and the reads that follow). Three classes, three
/// renderings, mirroring [`WriteError`]'s split so error bodies stay
@@ -456,7 +571,22 @@ impl HttpInner {
if let Some(session) = live_entry(&table, &key, now) {
return Ok(session);
}
- table.retain(|_, session| session.expiry > now);
+ table.retain(|_, session| {
+ if session.expiry > now {
+ return true;
+ }
+ // Evicting the session also tears down its in-process reply
+ // target (if a produce ever installed one), cancelling any
+ // still-parked reply waiters. Token-fenced so a stale sweep can
+ // never remove a later occupant of the key.
+ if let Some(token) = session.registry_token.get() {
+ self.shard
+ .bus
+ .clients()
+ .remove_if_token_matches(session.client_id, token);
+ }
+ false
+ });
if table.len() >= MAX_HTTP_SESSIONS {
// Still full after dropping expired entries: too many genuinely live
// sessions. Refuse rather than evict a live one; the client retries.
@@ -507,6 +637,8 @@ impl HttpInner {
user_id,
expiry,
gate: Mutex::new(FIRST_REQUEST_ID),
+ data_request: Cell::new(FIRST_REQUEST_ID),
+ registry_token: Cell::new(None),
}))
}
}
@@ -616,6 +748,10 @@ fn router(state: HttpState) -> Router {
"/streams/{stream_id}/topics/{topic_id}/purge",
delete(purge_topic),
)
+ .route(
+ "/streams/{stream_id}/topics/{topic_id}/messages",
+ post(send_messages),
+ )
.route(
"/streams/{stream_id}/topics/{topic_id}/consumer-groups",
get(get_cgs).post(create_cg),
@@ -1176,6 +1312,59 @@ async fn purge_topic(
Ok(StatusCode::NO_CONTENT)
}
+/// `POST /streams/{stream_id}/topics/{topic_id}/messages`: produce a batch of
+/// messages to a topic. The JSON body is the same `SendMessages` shape the
+/// legacy server accepts (partitioning + base64 messages); stream and topic
+/// come from the path.
+///
+/// Data plane, not control plane: the batch rides the partition group's own
+/// consensus (at-least-once, no dedup, no session gate - concurrent produces
+/// on one credential are legal), and the committed reply comes back through
+/// the session's in-process reply slot rather than a submit return value.
+/// The default answers 201 + `X-Iggy-Durability: replicated-memory` only
+/// after the quorum commit; `?ack=none` answers 202 + `X-Iggy-Durability:
+/// none` immediately after dispatch.
+async fn send_messages(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+ Query(query): Query<ProduceQuery>,
+ Json(command): Json<SendMessages>,
+) -> Result<Response, ProduceError> {
+ // Interim authorization: root-only until server-ng has an RBAC
+ // permissioner, mirroring the control-plane gate in `submit_committed`.
+ if identity.session.user_id != DEFAULT_ROOT_USER_ID {
+ return Err(ProduceError::Rejected(IggyError::Unauthorized));
+ }
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(ProduceError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(ProduceError::Rejected)?;
+ // Rejects an oversized partitioning key and an empty or oversized batch.
+ command.validate().map_err(ProduceError::Rejected)?;
+ let body =
+ encode_send_messages(&stream_id, &topic_id, &command).map_err(ProduceError::Rejected)?;
+ match query.ack {
+ ProduceAck::Replicated => {
+ SendWrapper::new(produce_replicated(&state, &identity.session, &body)).await?;
+ Ok((
+ StatusCode::CREATED,
+ [(
+ DURABILITY_HEADER,
+ HeaderValue::from_static(DURABILITY_REPLICATED_MEMORY),
+ )],
+ )
+ .into_response())
+ }
+ ProduceAck::None => {
+ SendWrapper::new(produce_unacked(&state, &identity.session, &body)).await;
+ Ok((
+ StatusCode::ACCEPTED,
+ [(DURABILITY_HEADER, HeaderValue::from_static(DURABILITY_NONE))],
+ )
+ .into_response())
+ }
+ }
+}
+
/// `POST /streams/{stream_id}/topics/{topic_id}/consumer-groups`: create a
/// consumer group under a topic and render the committed reply as the same
/// `ConsumerGroupDetails` JSON the legacy server returns.
@@ -1526,6 +1715,165 @@ async fn submit_write(
Ok(Bytes::copy_from_slice(committed_payload(&reply)?))
}
+/// Encode a validated HTTP `SendMessages` into the `SendMessagesRequest` wire
+/// body, mirroring the SDK's TCP produce encode: identifier + partitioning
+/// wire conversion, then `RawMessage` borrows into [`SendMessagesEncoder`].
+/// Balanced / messages-key partitioning passes through untouched; the
+/// dispatch gates resolve it to a concrete partition server-side.
+fn encode_send_messages(
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ command: &SendMessages,
+) -> Result<Bytes, IggyError> {
+ let wire_stream_id = identifier_to_wire(stream_id)?;
+ let wire_topic_id = identifier_to_wire(topic_id)?;
+ let wire_partitioning = partitioning_to_wire(&command.partitioning)?;
+ // Two passes because the view accessors' return borrows are tied to the
+ // view value, not the batch buffer, so the views must outlive the borrows.
+ let views: Vec<IggyMessageView<'_>> = command.batch.iter().collect();
+ let raw_messages: Vec<RawMessage<'_>> = views
+ .iter()
+ .map(|view| RawMessage {
+ id: view.header().id(),
+ origin_timestamp: view.header().origin_timestamp(),
+ headers: view.user_headers(),
+ payload: view.payload(),
+ })
+ .collect();
+ let size = SendMessagesEncoder::encoded_size(
+ &wire_stream_id,
+ &wire_topic_id,
+ &wire_partitioning,
+ &raw_messages,
+ );
+ let mut buf = BytesMut::with_capacity(size);
+ SendMessagesEncoder::encode(
+ &mut buf,
+ &wire_stream_id,
+ &wire_topic_id,
+ &wire_partitioning,
+ &raw_messages,
+ );
+ Ok(buf.freeze())
+}
+
+/// Run one awaited produce end to end: install the reply slot, dispatch into
+/// the partition plane, and wait (bounded) for the committed reply.
+///
+/// Slot-before-dispatch is load-bearing: every pre-dispatch gate failure
+/// inside [`dispatch_partition_request`] replies through `send_to_client`,
+/// which fires an installed slot, so one slot catches every exit. The slot
+/// guard borrows the registry, which is why this whole future runs inside
+/// the caller's `SendWrapper` on shard 0.
+async fn produce_replicated(
+ state: &HttpInner,
+ session: &HttpSession,
+ body: &[u8],
+) -> Result<(), ProduceError> {
+ ensure_in_process_reply_target(state, session);
+ let request_id = session.next_data_request_id();
+ let message = build_request_message(
+ Operation::SendMessages,
+ session.client_id,
+ session.session,
+ request_id,
+ body,
+ );
+ let (guard, receiver) = state
+ .shard
+ .bus
+ .clients()
+ .install_reply_slot(session.client_id, request_id)
+ .map_err(|error| {
+ warn!(?error, "server-ng HTTP: produce reply slot install failed");
+ ProduceError::Unavailable
+ })?;
+ dispatch_partition_request(
+ &state.shard,
+ message,
+ session.client_id,
+ session.session,
+ session.client_id,
+ )
+ .await;
+ let outcome = compio::time::timeout(PRODUCE_REPLY_TIMEOUT, receiver).await;
+ // Removes the slot unless the reply already fired, so a late commit
+ // reply after a timeout sheds at the bus instead of leaking a waiter.
+ drop(guard);
+ match outcome {
+ Ok(Ok(reply)) => classify_partition_reply(&reply),
+ // Cancelled (reply target torn down by session eviction mid-wait) or
+ // elapsed: same caller contract either way - outcome unknown, 504.
+ Ok(Err(_)) | Err(_) => Err(ProduceError::Timeout),
+ }
+}
+
+/// Fire-and-forget produce (`?ack=none`): no reply slot, no wait. The commit
+/// still happens; its reply (and any gate-failure reply) targets a request id
+/// with no slot installed and is shed at the bus by design.
+async fn produce_unacked(state: &HttpInner, session: &HttpSession, body: &[u8]) {
+ let request_id = session.next_data_request_id();
+ let message = build_request_message(
+ Operation::SendMessages,
+ session.client_id,
+ session.session,
+ request_id,
+ body,
+ );
+ dispatch_partition_request(
+ &state.shard,
+ message,
+ session.client_id,
+ session.session,
+ session.client_id,
+ )
+ .await;
+}
+
+/// Install this session's in-process reply target on first data-plane use.
+///
+/// The registry key is the session's shard-0 client id - the same id stamped
+/// into `RequestHeader.client` - so a partition reply routed through
+/// `send_to_client` lands on this entry and resolves the request-keyed slot.
+/// `None` from the registry means the key is already occupied; treat it as
+/// installed but leave the token unset so this session never tears down an
+/// entry it does not own.
+fn ensure_in_process_reply_target(state: &HttpInner, session: &HttpSession) {
+ if session.registry_token.get().is_some() {
+ return;
+ }
+ if let Some(token) = state
+ .shard
+ .bus
+ .clients()
+ .insert_in_process(session.client_id)
+ {
+ session.registry_token.set(Some(token));
+ }
+}
+
+/// Discriminate a produce reply. Partition replies carry no result section
+/// and no error code - success and gate failure are both empty-bodied - so
+/// the only wire discriminator is the reply's `op`: a committed reply is
+/// built from its prepare header, whose op (the partition group's commit
+/// number) is always >= 1, while the pre-dispatch gate failures reply through
+/// `build_empty_reply` with 0 in that field. The gate reply cannot name which
+/// entity was missing, hence the generic legacy 404 body.
+fn classify_partition_reply(reply: &BusMessage) -> Result<(), ProduceError> {
+ let header = reply
+ .as_slice()
+ .get(..HEADER_SIZE)
+ .and_then(|bytes| bytemuck::checked::try_from_bytes::<ReplyHeader>(bytes).ok())
+ .ok_or(ProduceError::Rejected(IggyError::InvalidCommand))?;
+ if header.command != Command2::Reply {
+ return Err(ProduceError::Rejected(IggyError::InvalidCommand));
+ }
+ if header.op == 0 {
+ return Err(ProduceError::NotFound);
+ }
+ Ok(())
+}
+
/// The two cross-cutting gates every authenticated read enforces before it
/// touches state. Factored out of [`read_local`] so the cross-shard client
/// reads (`get_clients` / `get_client`) - which serve from the shard session
@@ -1819,8 +2167,11 @@ const fn login_error_to_iggy(error: &LoginRegisterError) -> IggyError {
#[cfg(test)]
mod tests {
use configs::cluster::TransportPorts;
+ use iggy_binary_protocol::PrepareHeader;
+ use iggy_common::{IggyMessage, IggyMessagesBatch, Partitioning, PartitioningKind};
use super::*;
+ use crate::responses::build_empty_reply;
const READ_PATH: &str = "/streams?consistency=linearizable";
@@ -1886,4 +2237,124 @@ mod tests {
Some("http://[::1]:8080/streams?consistency=linearizable".to_owned())
);
}
+
+ // -- encode_send_messages --
+
+ fn produce_command(partitioning: Partitioning) -> SendMessages {
+ let first = IggyMessage::builder()
+ .id(7)
+ .payload(Bytes::from_static(b"first"))
+ .build()
+ .expect("valid message");
+ // Raw pre-encoded user headers, mirroring the HTTP deserializer's
+ // base64 branch.
+ let mut second = IggyMessage::builder()
+ .id(8)
+ .payload(Bytes::from_static(b"second"))
+ .build()
+ .expect("valid message");
+ let raw_headers = Bytes::from_static(b"raw-header-bytes");
+ second.header.user_headers_length =
+ u32::try_from(raw_headers.len()).expect("test headers fit u32");
+ second.user_headers = Some(raw_headers);
+ let messages = vec![first, second];
+ SendMessages {
+ partitioning,
+ batch: IggyMessagesBatch::from(&messages),
+ ..Default::default()
+ }
+ }
+
+ #[test]
+ fn encode_send_messages_round_trips_through_wire_decoders() {
+ use iggy_binary_protocol::WireMessageIterator;
+ use iggy_binary_protocol::message_layout::WIRE_MESSAGE_INDEX_SIZE;
+ use iggy_binary_protocol::requests::messages::SendMessagesHeader;
+
+ let stream_id = Identifier::from_str_value("1").expect("valid stream id");
+ let topic_id = Identifier::from_str_value("orders").expect("valid topic id");
+ let command = produce_command(Partitioning::partition_id(3));
+ let origin_timestamps: Vec<u64> = command
+ .batch
+ .iter()
+ .map(|view| view.header().origin_timestamp())
+ .collect();
+
+ let bytes = encode_send_messages(&stream_id, &topic_id, &command).expect("encodes");
+
+ let metadata_length =
+ u32::from_le_bytes(bytes[..4].try_into().expect("length prefix")) as usize;
+ let (header, consumed) =
+ SendMessagesHeader::decode(&bytes[4..4 + metadata_length]).expect("valid metadata");
+ assert_eq!(consumed, metadata_length);
+ assert_eq!(header.stream_id, identifier_to_wire(&stream_id).unwrap());
+ assert_eq!(header.topic_id, identifier_to_wire(&topic_id).unwrap());
+ assert_eq!(
+ header.partitioning,
+ partitioning_to_wire(&command.partitioning).unwrap()
+ );
+ assert_eq!(header.messages_count, 2);
+
+ let data_offset = 4 + metadata_length + 2 * WIRE_MESSAGE_INDEX_SIZE;
+ let views: Vec<_> = WireMessageIterator::new(&bytes[data_offset..], 2)
+ .collect::<Result<Vec<_>, _>>()
+ .expect("valid message frames");
+ assert_eq!(views[0].id(), 7);
+ assert_eq!(views[0].payload(), b"first");
+ assert_eq!(views[0].user_headers(), b"");
+ assert_eq!(views[0].origin_timestamp(), origin_timestamps[0]);
+ assert_eq!(views[1].id(), 8);
+ assert_eq!(views[1].payload(), b"second");
+ assert_eq!(views[1].user_headers(), b"raw-header-bytes");
+ assert_eq!(views[1].origin_timestamp(), origin_timestamps[1]);
+ }
+
+ #[test]
+ fn send_messages_validate_rejects_oversized_partitioning_key() {
+ let command = produce_command(Partitioning {
+ kind: PartitioningKind::MessagesKey,
+ length: 0,
+ value: vec![0u8; 256],
+ });
+ assert!(command.validate().is_err());
+ }
+
+ // -- classify_partition_reply --
+
+ fn frozen(reply: Message<iggy_binary_protocol::ReplyHeader>) -> BusMessage {
+ reply.into_generic().into_frozen()
+ }
+
+ #[test]
+ fn committed_partition_reply_classifies_as_success() {
+ let prepare = PrepareHeader {
+ command: Command2::Prepare,
+ operation: Operation::SendMessages,
+ client: 42,
+ op: 1,
+ request: 1,
+ ..Default::default()
+ };
+ let reply = frozen(consensus::build_reply_message(&prepare, &Bytes::new()));
+ assert!(classify_partition_reply(&reply).is_ok());
+ }
+
+ #[test]
+ fn gate_failure_empty_reply_classifies_as_not_found() {
+ let request = build_request_message(Operation::SendMessages, 42, 7, 1, &[]);
+ let reply = frozen(build_empty_reply(request.header(), 42, 0, 9));
+ assert!(matches!(
+ classify_partition_reply(&reply),
+ Err(ProduceError::NotFound)
+ ));
+ }
+
+ #[test]
+ fn non_reply_frame_classifies_as_rejected() {
+ let request = build_request_message(Operation::SendMessages, 42, 7, 1, &[]);
+ assert!(matches!(
+ classify_partition_reply(&request.into_generic().into_frozen()),
+ Err(ProduceError::Rejected(IggyError::InvalidCommand))
+ ));
+ }
}
(iggy) 01/19: feat(message_bus): add in-process reply routing for HTTP listener
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 95bb74d8e614ac2fbf6244026d0028bb55cdd1d0
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 09:12:38 2026 +0200
feat(message_bus): add in-process reply routing for HTTP listener
server-ng routes every client reply through a per-connection
socket in the connection registry, correlating by client id
only. An in-process caller (the upcoming HTTP listener) holds
no consensus socket: it awaits a single reply that must be
matched by request id on the server side, with no way to
target such a waiter today.
Reify the registry entry's reply target as an enum. Socket
keeps today's per-peer queue unchanged; InProcess holds a
request-id-keyed map of oneshot senders. send_to_client now
routes through ReplyRoute so the request-id decode stays off
the socket path, which stays byte-identical for TCP, WS and
QUIC. Only Socket is constructed here; the InProcess arm is
wired when the HTTP data plane lands.
---
core/message_bus/src/lib.rs | 36 ++++-
.../src/lifecycle/connection_registry.rs | 169 +++++++++++++++------
core/message_bus/src/lifecycle/mod.rs | 2 +-
3 files changed, 153 insertions(+), 54 deletions(-)
diff --git a/core/message_bus/src/lib.rs b/core/message_bus/src/lib.rs
index fc96d7afb..ea50e4cd1 100644
--- a/core/message_bus/src/lib.rs
+++ b/core/message_bus/src/lib.rs
@@ -95,13 +95,13 @@ pub use installer::conn_info::{
};
pub use lifecycle::{
BusMessage, BusReceiver, BusSender, ConnectionRegistry, DrainOutcome, FusedShutdown,
- ReplicaRegistry, Shutdown, ShutdownToken,
+ ReplicaRegistry, ReplyRoute, Shutdown, ShutdownToken,
};
pub use transports::tls::TlsServerCredentials;
pub use compio::runtime::JoinHandle;
use configs::server_ng::ServerNgConfig;
-use iggy_binary_protocol::GenericHeader;
+use iggy_binary_protocol::{GenericHeader, ReplyHeader};
use server_common::{MESSAGE_ALIGN, Message, iobuf::Frozen};
use std::array;
use std::cell::{Cell, OnceCell, RefCell};
@@ -1225,13 +1225,21 @@ impl MessageBus for IggyMessageBus {
// Owning shard is encoded in the top 16 bits of client_id.
let owning_shard = client_id_owning_shard(client_id);
if owning_shard == self.shard_id {
- // Fast path: move `message` straight into `try_send`. On no-slot
- // the registry hands it back; we drop it and surface
- // ClientNotFound (matches prior behaviour: SendError did not
- // preserve payload either).
+ // Route by the entry's reply target. Socket is the unchanged
+ // fast path and never decodes. InProcess resolves the waiting
+ // request id from the reply header and fires its oneshot; that
+ // decode stays off the socket path. No slot surfaces
+ // ClientNotFound (prior behaviour: SendError did not preserve
+ // the payload either).
return match self.clients.try_send_or_return(client_id, message) {
- Ok(send_result) => send_result.map_err(map_try_send_err),
- Err(_msg) => Err(SendError::ClientNotFound(client_id)),
+ ReplyRoute::Delivered(send_result) => send_result.map_err(map_try_send_err),
+ ReplyRoute::InProcess(message) => {
+ let request = reply_request_id(&message);
+ self.clients
+ .fire_in_process(client_id, request, message)
+ .map_err(|_| SendError::ClientNotFound(client_id))
+ }
+ ReplyRoute::NoSlot(_message) => Err(SendError::ClientNotFound(client_id)),
};
}
let forward = self
@@ -1325,6 +1333,18 @@ fn map_try_send_err(e: async_channel::TrySendError<Frozen<MESSAGE_ALIGN>>) -> Se
}
}
+/// Peek `ReplyHeader.request` (the originating request id) from a reply
+/// buffer at its fixed header offset. Only the in-process reply path calls
+/// this; the socket path never decodes.
+fn reply_request_id(reply: &Frozen<MESSAGE_ALIGN>) -> u64 {
+ const OFFSET: usize = std::mem::offset_of!(ReplyHeader, request);
+ reply
+ .as_slice()
+ .get(OFFSET..OFFSET + std::mem::size_of::<u64>())
+ .and_then(|bytes| bytes.try_into().ok())
+ .map_or(0, u64::from_ne_bytes)
+}
+
#[cfg(test)]
mod tests {
use super::*;
diff --git a/core/message_bus/src/lifecycle/connection_registry.rs b/core/message_bus/src/lifecycle/connection_registry.rs
index 162d139df..6a01110b8 100644
--- a/core/message_bus/src/lifecycle/connection_registry.rs
+++ b/core/message_bus/src/lifecycle/connection_registry.rs
@@ -33,6 +33,7 @@
//! [`ShutdownToken`]: crate::lifecycle::ShutdownToken
use compio::runtime::JoinHandle;
+use futures::channel::oneshot;
use server_common::{MESSAGE_ALIGN, iobuf::Frozen};
use std::cell::{Cell, RefCell};
use std::collections::HashMap;
@@ -128,9 +129,50 @@ pub struct DrainOutcome {
pub background_force: usize,
}
+/// Where a client reply is delivered.
+///
+/// `Socket` is the wire path taken by every TCP / WS / QUIC client: the
+/// reply is pushed into the per-peer queue exactly as before. `InProcess`
+/// reifies an in-process reply target keyed by request id so a caller
+/// holding the matching [`oneshot::Receiver`] can await its reply without a
+/// socket. Both ends are shard-0-local, hence a single-threaded oneshot.
+#[derive(Debug)]
+enum ReplyTarget {
+ Socket(BusSender),
+ #[allow(dead_code)] // wired in P2 (HTTP data plane)
+ InProcess {
+ by_request: HashMap<u64, oneshot::Sender<BusMessage>>,
+ },
+}
+
+impl ReplyTarget {
+ /// Wire sender for socket targets; `None` for in-process targets.
+ const fn as_socket(&self) -> Option<&BusSender> {
+ match self {
+ Self::Socket(sender) => Some(sender),
+ Self::InProcess { .. } => None,
+ }
+ }
+}
+
+/// Outcome of routing a client reply through an [`Entry`]'s reply target.
+///
+/// `Delivered` is the socket fast path: `try_send` was attempted and its
+/// result is carried through unchanged. `InProcess` hands the message back
+/// so the caller can decode the request id and fire the matching oneshot
+/// via [`ConnectionRegistry::fire_in_process`], keeping that decode off the
+/// socket path. `NoSlot` means no entry exists for the key.
+#[derive(Debug)]
+#[must_use]
+pub enum ReplyRoute {
+ Delivered(Result<(), async_channel::TrySendError<BusMessage>>),
+ InProcess(BusMessage),
+ NoSlot(BusMessage),
+}
+
#[derive(Debug)]
struct Entry {
- sender: BusSender,
+ reply: ReplyTarget,
writer_handle: Option<JoinHandle<()>>,
reader_handle: Option<JoinHandle<()>>,
/// Generation token minted by the registry on insert. Entries that
@@ -215,39 +257,58 @@ where
/// `close_peer`.
pub fn with_sender<R>(&self, key: K, f: impl FnOnce(&BusSender) -> R) -> Option<R> {
let entries = self.entries.borrow();
- entries.get(&key).map(|entry| f(&entry.sender))
+ entries
+ .get(&key)
+ .and_then(|entry| entry.reply.as_socket())
+ .map(f)
}
- /// Try to send `msg` to the per-peer queue, returning the message back
- /// when no slot exists.
+ /// Route `msg` to the entry's reply target, handing it back when no
+ /// live target accepts it.
///
- /// Lets `send_to_*` skip the unconditional `Frozen::clone()` the
- /// `with_sender` shape forced (closure consumes a clone, outer caller
- /// retains the original for the slow path the borrow checker cannot
- /// prove unreachable). The consuming variant moves `msg` into
- /// `try_send` on the fast path; on no-slot the message is handed back
- /// so the caller can route it via the slow path or drop it.
+ /// Lets `send_to_client` skip the unconditional `Frozen::clone()` the
+ /// `with_sender` shape forced. Socket targets move `msg` straight into
+ /// `try_send`; in-process targets hand it back via
+ /// [`ReplyRoute::InProcess`] so the caller can resolve the request id
+ /// and fire the oneshot without decoding on the socket path.
///
/// Returns:
- /// - `Ok(Ok(()))`: slot exists and the queue accepted `msg`.
- /// - `Ok(Err(TrySendError))`: slot exists but `try_send` failed
- /// (`Full` or `Closed`); the inner error carries `msg` back per
- /// `async_channel`'s contract.
- /// - `Err(msg)`: no slot for `key`; `msg` returned to the caller.
+ /// - [`ReplyRoute::Delivered`]: socket target; carries the `try_send`
+ /// result (`Full` / `Closed` errors carry `msg` back per
+ /// `async_channel`'s contract).
+ /// - [`ReplyRoute::InProcess`]: in-process target; `msg` handed back.
+ /// - [`ReplyRoute::NoSlot`]: no entry for `key`; `msg` handed back.
+ pub fn try_send_or_return(&self, key: K, msg: BusMessage) -> ReplyRoute {
+ let entries = self.entries.borrow();
+ let Some(entry) = entries.get(&key) else {
+ return ReplyRoute::NoSlot(msg);
+ };
+ match &entry.reply {
+ ReplyTarget::Socket(sender) => ReplyRoute::Delivered(sender.try_send(msg)),
+ ReplyTarget::InProcess { .. } => ReplyRoute::InProcess(msg),
+ }
+ }
+
+ /// Fire the in-process reply registered under `request` for `key`.
+ ///
+ /// Companion to [`ReplyRoute::InProcess`]; the socket path never reaches
+ /// here. Removes the matching oneshot and delivers `msg` on it.
///
/// # Errors
///
- /// The outer `Err(msg)` reports that no entry exists for `key`. The
- /// inner `Err(TrySendError)` reports that the entry's queue is full
- /// or closed.
- pub fn try_send_or_return(
- &self,
- key: K,
- msg: BusMessage,
- ) -> Result<Result<(), async_channel::TrySendError<BusMessage>>, BusMessage> {
- let entries = self.entries.borrow();
- match entries.get(&key) {
- Some(entry) => Ok(entry.sender.try_send(msg)),
+ /// Hands `msg` back when `key` has no in-process target or no waiter is
+ /// registered for `request`, so the caller surfaces the same not-found
+ /// outcome as a missing socket.
+ pub fn fire_in_process(&self, key: K, request: u64, msg: BusMessage) -> Result<(), BusMessage> {
+ let mut entries = self.entries.borrow_mut();
+ let Some(entry) = entries.get_mut(&key) else {
+ return Err(msg);
+ };
+ let ReplyTarget::InProcess { by_request } = &mut entry.reply else {
+ return Err(msg);
+ };
+ match by_request.remove(&request) {
+ Some(reply_tx) => reply_tx.send(msg),
None => Err(msg),
}
}
@@ -294,7 +355,7 @@ where
entries.insert(
key,
Entry {
- sender,
+ reply: ReplyTarget::Socket(sender),
writer_handle: Some(writer_handle),
reader_handle: Some(reader_handle),
token,
@@ -360,7 +421,9 @@ where
let Some(mut entry) = self.entries.borrow_mut().remove(&key) else {
return;
};
- entry.sender.close();
+ if let Some(sender) = entry.reply.as_socket() {
+ sender.close();
+ }
if let Some(writer_handle) = entry.writer_handle.take() {
let _ = compio::time::timeout(timeout, writer_handle).await;
}
@@ -389,7 +452,9 @@ where
}
slot.remove()
};
- entry.sender.close();
+ if let Some(sender) = entry.reply.as_socket() {
+ sender.close();
+ }
if let Some(writer_handle) = entry.writer_handle.take() {
let _ = compio::time::timeout(timeout, writer_handle).await;
}
@@ -447,7 +512,9 @@ where
// the shutdown token has not been triggered. Writer is
// awaited before the reader handle is dropped so a
// mid-writev cancellation cannot truncate a frame.
- entry.sender.close();
+ if let Some(sender) = entry.reply.as_socket() {
+ sender.close();
+ }
// Writer gets the full shared deadline so a legitimately-
// flushing `write_vectored_all` never loses budget to any
// other entry in the same drain batch.
@@ -581,7 +648,8 @@ impl ReplicaRegistry {
let slots = self.slots.borrow();
slots[usize::from(key)]
.as_ref()
- .map(|entry| f(&entry.sender))
+ .and_then(|entry| entry.reply.as_socket())
+ .map(f)
}
/// See [`ConnectionRegistry::try_send_or_return`].
@@ -595,8 +663,11 @@ impl ReplicaRegistry {
msg: BusMessage,
) -> Result<Result<(), async_channel::TrySendError<BusMessage>>, BusMessage> {
let slots = self.slots.borrow();
- match slots[usize::from(key)].as_ref() {
- Some(entry) => Ok(entry.sender.try_send(msg)),
+ match slots[usize::from(key)]
+ .as_ref()
+ .and_then(|entry| entry.reply.as_socket())
+ {
+ Some(sender) => Ok(sender.try_send(msg)),
None => Err(msg),
}
}
@@ -627,7 +698,7 @@ impl ReplicaRegistry {
}
let token = self.mint_token();
*slot = Some(Entry {
- sender,
+ reply: ReplyTarget::Socket(sender),
writer_handle: Some(writer_handle),
reader_handle: Some(reader_handle),
token,
@@ -672,7 +743,9 @@ impl ReplicaRegistry {
self.len.set(self.len.get() - 1);
entry
};
- entry.sender.close();
+ if let Some(sender) = entry.reply.as_socket() {
+ sender.close();
+ }
if let Some(writer_handle) = entry.writer_handle.take() {
let _ = compio::time::timeout(timeout, writer_handle).await;
}
@@ -697,7 +770,9 @@ impl ReplicaRegistry {
self.len.set(self.len.get() - 1);
removed_entry
};
- entry.sender.close();
+ if let Some(sender) = entry.reply.as_socket() {
+ sender.close();
+ }
if let Some(writer_handle) = entry.writer_handle.take() {
let _ = compio::time::timeout(timeout, writer_handle).await;
}
@@ -1065,7 +1140,7 @@ mod tests {
.expect("insert ok");
let outer = reg.try_send_or_return(1u8, make_bus_msg());
- assert!(matches!(outer, Ok(Ok(()))));
+ assert!(matches!(outer, ReplyRoute::Delivered(Ok(()))));
// Receiver drains the message; closing the writer's rx end via
// close_peer is not necessary for this assertion.
let received = rx.recv().await.expect("queue had message");
@@ -1081,7 +1156,9 @@ mod tests {
let original_len = msg.len();
let outcome = reg.try_send_or_return(99u8, msg);
- let returned = outcome.expect_err("missing slot should return msg");
+ let ReplyRoute::NoSlot(returned) = outcome else {
+ panic!("missing slot should return msg");
+ };
assert_eq!(returned.len(), original_len);
}
@@ -1104,14 +1181,16 @@ mod tests {
.expect("insert ok");
// Saturate the queue.
- reg.try_send_or_return(1u8, make_bus_msg())
- .expect("slot present")
- .expect("first send accepted");
+ assert!(matches!(
+ reg.try_send_or_return(1u8, make_bus_msg()),
+ ReplyRoute::Delivered(Ok(()))
+ ));
// Second send: slot present, queue full.
- let outcome = reg
- .try_send_or_return(1u8, make_bus_msg())
- .expect("slot still present");
- assert!(matches!(outcome, Err(async_channel::TrySendError::Full(_))));
+ let outcome = reg.try_send_or_return(1u8, make_bus_msg());
+ assert!(matches!(
+ outcome,
+ ReplyRoute::Delivered(Err(async_channel::TrySendError::Full(_)))
+ ));
}
/// Mirror of `try_send_or_return_returns_msg_when_slot_missing` for the
diff --git a/core/message_bus/src/lifecycle/mod.rs b/core/message_bus/src/lifecycle/mod.rs
index 4fa15b9e0..e80bec1f7 100644
--- a/core/message_bus/src/lifecycle/mod.rs
+++ b/core/message_bus/src/lifecycle/mod.rs
@@ -26,6 +26,6 @@ pub mod shutdown;
pub use connection_registry::{
BusMessage, BusReceiver, BusSender, ConnectionRegistry, DrainOutcome, InstanceToken,
- RejectedRegistration, ReplicaRegistry,
+ RejectedRegistration, ReplicaRegistry, ReplyRoute,
};
pub use shutdown::{FusedShutdown, Shutdown, ShutdownToken};
(iggy) 03/19: feat(server-ng): route HTTP create-stream through VSR consensus
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 6dcc841253f797143daffd54bca76478c0cfe323
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 12:47:55 2026 +0200
feat(server-ng): route HTTP create-stream through VSR consensus
The server-ng HTTP listener could authenticate and bind a session
but had no way to perform a control-plane write, leaving every
mutating route unserved.
POST /streams now builds a create-stream request header and submits
it on the owning shard via the in-process dispatch spine, the same
path TCP and QUIC use, so the write is replicated and linearized
identically. A per-session gate serializes writes and assigns the
gap-free request ids the depth-1 consensus dedup requires; the id
advances only once a Reply commits, so a transient failure or an
eviction leaves it free for the caller's retry.
The committed reply is decoded by frame command: a Reply's result
section renders the parity JSON or a committed business rejection
(duplicate name -> 400), while an Eviction maps to the same typed
401 a TCP client receives, so the caller re-authenticates rather
than retrying a dead session.
---
core/server-ng/src/dispatch.rs | 2 +-
core/server-ng/src/http.rs | 241 ++++++++++++++++++++++++++++++-----
core/server-ng/src/http/extractor.rs | 6 +-
3 files changed, 212 insertions(+), 37 deletions(-)
diff --git a/core/server-ng/src/dispatch.rs b/core/server-ng/src/dispatch.rs
index 74b81e92b..3bf5fa33b 100644
--- a/core/server-ng/src/dispatch.rs
+++ b/core/server-ng/src/dispatch.rs
@@ -1624,7 +1624,7 @@ fn submit_disconnect_logout(shard: Rc<ServerNgShard>, vsr_client_id: u128, sessi
/// `client` id (it's the VSR id, not the transport/home-shard-encoding id).
/// `None` = transient submit failure (SDK read-timeout replays).
#[allow(clippy::future_not_send)]
-async fn submit_client_request_on_owner(
+pub(crate) async fn submit_client_request_on_owner(
shard: &Rc<ServerNgShard>,
request: Message<RequestHeader>,
) -> Option<Message<GenericHeader>> {
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index e4f2d58f4..a2de4b11c 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -33,19 +33,30 @@ use axum::response::{IntoResponse, Response};
use axum::routing::{get, post};
use axum::{Json, Router};
use configs::http::HttpJwtConfig;
+use iggy_binary_protocol::consensus::{
+ Command2, EvictionHeader, EvictionReason, HEADER_SIZE, result_code, result_section_len,
+};
+use iggy_binary_protocol::requests::streams::CreateStreamRequest;
+use iggy_binary_protocol::responses::streams::get_stream::GetStreamResponse;
+use iggy_binary_protocol::version::IGGY_PROTOCOL_VERSION;
+use iggy_binary_protocol::{
+ GenericHeader, Operation, RequestHeader, WireDecode, WireEncode, WireName,
+};
+use iggy_common::create_stream::CreateStream;
use iggy_common::login_user::LoginUser;
use iggy_common::login_with_personal_access_token::LoginWithPersonalAccessToken;
-use iggy_common::{IdentityInfo, IggyError, IggyTimestamp, TokenInfo};
+use iggy_common::{IdentityInfo, IggyError, IggyTimestamp, StreamDetails, TokenInfo};
use message_bus::client_listener;
use secrecy::ExposeSecret;
use send_wrapper::SendWrapper;
use server::http::error::{CustomError, ErrorResponse};
+use server_common::Message;
use tokio::sync::Mutex;
use tracing::{error, info, warn};
use crate::auth::{verify_login_credentials, verify_pat_credentials};
use crate::bootstrap::ServerNgShard;
-use crate::dispatch::submit_register_on_owner;
+use crate::dispatch::{submit_client_request_on_owner, submit_register_on_owner};
use crate::http::extractor::Authenticated;
use crate::http::jwt::JwtManager;
use crate::login_register::LoginRegisterError;
@@ -61,8 +72,8 @@ const PONG: &str = "pong";
/// entries are dropped first, so the cap only bites on live oversubscription.
const MAX_HTTP_SESSIONS: usize = 100_000;
-/// First per-session request id the write path (2c) will hand out. VSR request
-/// numbers are 1-based and strictly increasing within a session.
+/// First per-session request id the write path hands out. VSR request numbers
+/// are 1-based and strictly increasing within a session.
const FIRST_REQUEST_ID: u64 = 1;
/// One VSR session established for a single login credential (a JWT `jti` or a
@@ -71,32 +82,22 @@ const FIRST_REQUEST_ID: u64 = 1;
struct HttpSession {
/// Shard-0 client id minted for this credential; its top 16 bits are 0, so
/// it shares the shard-0 id space with TCP virtual clients without
- /// colliding. Consumed by the write path (2c).
- #[expect(
- dead_code,
- reason = "read by the write path (2c) to fill RequestHeader.client"
- )]
+ /// colliding. Fills `RequestHeader.client` on every write.
client_id: u128,
- /// Cluster session number returned by the VSR `Register` commit. Consumed
- /// by the write path (2c).
- #[expect(
- dead_code,
- reason = "read by the write path (2c) to fill RequestHeader.session"
- )]
+ /// Cluster session number returned by the VSR `Register` commit. Fills
+ /// `RequestHeader.session` on every write.
session: u64,
- /// User the credential authenticated as. Consumed by the write path (2c).
- #[expect(dead_code, reason = "read by the write path (2c) for authorization")]
+ /// User the credential authenticated as. Consumed by the write path for
+ /// authorization.
+ #[expect(dead_code, reason = "read once authorization is wired for writes")]
user_id: u32,
/// Credential expiry in unix seconds (`u64::MAX` = never). Drives lazy
/// eviction of stale table entries.
expiry: u64,
- /// Serializes the write path (2c): the guarded value is the NEXT request
- /// id. A `tokio::sync::Mutex` because 2c holds it across the submit `.await`
- /// so each session's request numbers stay monotonic and gap-free.
- #[expect(
- dead_code,
- reason = "locked by the write path (2c) to serialize per-session writes"
- )]
+ /// Serializes this session's writes: the guarded value is the NEXT request
+ /// id. A `tokio::sync::Mutex` because the write path holds it across the
+ /// submit `.await` so each session's request numbers reach the primary in
+ /// order and stay gap-free for the depth-1 consensus dedup.
gate: Mutex<u64>,
}
@@ -126,17 +127,48 @@ impl IntoResponse for AuthError {
Self::Unauthenticated(error) => CustomError::from(error).into_response(),
// Register could not commit (no caught-up primary, pipeline full, or
// a view-change cancel). Transient server condition -> 503, retryable.
- Self::SessionUnavailable => (
- StatusCode::SERVICE_UNAVAILABLE,
- Json(ErrorResponse::from_error(
- IggyError::CannotEstablishConnection,
- )),
- )
- .into_response(),
+ Self::SessionUnavailable => service_unavailable(),
}
}
}
+/// Rejection for an authenticated control-plane write (`POST /streams` and the
+/// writes that follow it).
+///
+/// Same two-class split as [`AuthError`], for the same reasons: a caller-side
+/// validation failure or a committed business rejection (e.g. a duplicate
+/// stream name) renders through the legacy `IggyError -> CustomError` map so
+/// SDK error bodies stay byte-identical, while a write that cannot commit right
+/// now is a transient server condition (503) and must never surface as a
+/// business error or, worse, a 200 with a stale body.
+enum WriteError {
+ Rejected(IggyError),
+ Unavailable,
+}
+
+impl IntoResponse for WriteError {
+ fn into_response(self) -> Response {
+ match self {
+ Self::Rejected(error) => CustomError::from(error).into_response(),
+ Self::Unavailable => service_unavailable(),
+ }
+ }
+}
+
+/// The shared 503 body for a request that could not commit right now: no
+/// caught-up primary, a full pipeline, or a view-change cancel. Retryable, and
+/// rendered with the `CannotEstablishConnection` code the SDKs treat as a
+/// connection-level retry rather than a terminal error.
+fn service_unavailable() -> Response {
+ (
+ StatusCode::SERVICE_UNAVAILABLE,
+ Json(ErrorResponse::from_error(
+ IggyError::CannotEstablishConnection,
+ )),
+ )
+ .into_response()
+}
+
/// Shared shard-0 HTTP state.
///
/// Groups the shard handle, the JWT issuer/verifier, and the per-credential VSR
@@ -307,6 +339,7 @@ fn router(state: HttpState) -> Router {
post(login_with_personal_access_token),
)
.route("/users/me", get(get_me))
+ .route("/streams", post(create_stream))
.with_state(state)
}
@@ -350,6 +383,152 @@ async fn get_me(identity: Authenticated) -> Json<IdentityInfo> {
})
}
+/// `POST /streams`: create a stream through the per-session gate and the
+/// in-process VSR dispatch spine, then render the committed reply as the same
+/// `StreamDetails` JSON the legacy server returns.
+///
+/// The accepted body is name-only (`{"name": ...}`), matching the legacy
+/// request; server-ng's wire `CreateStreamRequest` is likewise name-only and
+/// auto-assigns the id, so there is no client-supplied stream id to honor.
+async fn create_stream(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Json(command): Json<CreateStream>,
+) -> Result<Json<StreamDetails>, WriteError> {
+ let request = CreateStreamRequest {
+ name: WireName::new(command.name)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidStreamName))?,
+ };
+
+ let client_id = identity.session.client_id;
+ let session_id = identity.session.session;
+
+ // Serialize this session's writes behind the gate and take the next VSR
+ // request id. The gate is held across the commit so ids reach the primary
+ // strictly in order, and it advances only once the request commits (a
+ // `Reply` frame): a transient failure or an eviction leaves the id free for
+ // the caller's retry, which the depth-1 dedup requires (the next accepted
+ // request must be `committed_request + 1`, so a consumed-but-uncommitted id
+ // would wedge the session on `RequestGap`).
+ let mut next_request_id = identity.session.gate.lock().await;
+ let message = build_create_stream_message(client_id, session_id, *next_request_id, &request);
+ // `submit_client_request_on_owner` is `!Send` (Rc-based); `SendWrapper`
+ // bridges it onto axum's `Send` handler future, sound because the whole
+ // handler runs pinned to shard 0's compio thread.
+ let reply = SendWrapper::new(submit_client_request_on_owner(&state.shard, message)).await;
+ let Some(reply) = reply else {
+ return Err(WriteError::Unavailable);
+ };
+
+ // Peek the reply frame before decoding: an eviction carries no result
+ // section and means the session is dead (re-authenticate), so unlike a
+ // committed `Reply` it must not consume this request id.
+ match reply.header().command {
+ Command2::Reply => {
+ // Committed: the cluster consumed this id, so advance the gate before
+ // rendering, whether or not the body decodes.
+ *next_request_id += 1;
+ drop(next_request_id);
+ Ok(Json(decode_committed_stream(&reply)?))
+ }
+ Command2::Eviction => Err(WriteError::Rejected(eviction_error(&reply))),
+ _ => Err(WriteError::Rejected(IggyError::InvalidCommand)),
+ }
+}
+
+/// Build the `Message<RequestHeader>` for a create-stream by filling a zeroed
+/// `#[repr(C)]` header, mirroring `wire::rewrite_request_body` and the
+/// partition reconciler's prepare builder.
+fn build_create_stream_message(
+ client_id: u128,
+ session_id: u64,
+ request_id: u64,
+ request: &CreateStreamRequest,
+) -> Message<RequestHeader> {
+ let body = request.to_bytes();
+ let total = HEADER_SIZE + body.len();
+ let mut message = Message::<RequestHeader>::new(total);
+ message.as_mut_slice()[HEADER_SIZE..].copy_from_slice(&body);
+ let header = bytemuck::checked::try_from_bytes_mut::<RequestHeader>(
+ &mut message.as_mut_slice()[..HEADER_SIZE],
+ )
+ .expect("zeroed bytes form a valid RequestHeader");
+ header.command = Command2::Request;
+ header.operation = Operation::CreateStream;
+ header.client = client_id;
+ header.session = session_id;
+ header.request = request_id;
+ header.size = u32::try_from(total).expect("create-stream message size fits u32");
+ message
+}
+
+/// Decode a committed `Reply` frame's body into `StreamDetails`, mirroring the
+/// SDK's `split_metadata_result`. A metadata reply body leads with a result
+/// section (`[count][{index, result}]*`): `count == 0` is success followed by
+/// the `GetStreamResponse` payload, and a nonzero first `result` is a committed
+/// business rejection carrying an `IggyError` code (e.g. duplicate name). The
+/// caller has already confirmed the frame is a `Reply`, so a missing result
+/// section is a malformed committed reply, mapped to an error rather than a 200
+/// with a stale body.
+fn decode_committed_stream(reply: &Message<GenericHeader>) -> Result<StreamDetails, WriteError> {
+ let size = reply.header().size as usize;
+ let body = reply.as_slice().get(HEADER_SIZE..size).unwrap_or_default();
+ match result_code(body) {
+ Some(0) => {
+ let payload_start =
+ result_section_len(body).ok_or(WriteError::Rejected(IggyError::InvalidCommand))?;
+ let payload = body
+ .get(payload_start..)
+ .ok_or(WriteError::Rejected(IggyError::InvalidCommand))?;
+ let response = GetStreamResponse::decode_from(payload)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidCommand))?;
+ StreamDetails::try_from(response).map_err(WriteError::Rejected)
+ }
+ Some(code) => Err(WriteError::Rejected(IggyError::from_code(code))),
+ None => Err(WriteError::Rejected(IggyError::InvalidCommand)),
+ }
+}
+
+/// Map an eviction frame to the same typed [`IggyError`] the SDK's
+/// `decode_eviction` produces, so an HTTP caller sees the identical status a TCP
+/// caller would (session-terminal reasons render as 401 -> re-authenticate).
+/// Reuses the shared [`EvictionHeader`] primitive rather than hand-decoding
+/// offsets; an unreadable frame falls back to re-authentication.
+fn eviction_error(reply: &Message<GenericHeader>) -> IggyError {
+ let Some(eviction) = reply
+ .as_slice()
+ .get(..HEADER_SIZE)
+ .and_then(|bytes| bytemuck::checked::try_from_bytes::<EvictionHeader>(bytes).ok())
+ else {
+ return IggyError::Unauthenticated;
+ };
+ match eviction.reason {
+ EvictionReason::InvalidCredentials => IggyError::InvalidCredentials,
+ EvictionReason::InvalidToken => IggyError::InvalidPersonalAccessToken,
+ EvictionReason::UserInactive
+ | EvictionReason::SessionError
+ | EvictionReason::NoSession
+ | EvictionReason::SessionTooLow
+ | EvictionReason::SessionReleaseMismatch => IggyError::Unauthenticated,
+ EvictionReason::StaleClient => IggyError::StaleClient,
+ EvictionReason::IncompatibleProtocol => {
+ let server_max = eviction.server_protocol_version;
+ let server_min = eviction.server_protocol_version_min;
+ if server_min == 0 || server_max < server_min {
+ IggyError::Unauthenticated
+ } else {
+ IggyError::IncompatibleProtocolVersion(
+ IGGY_PROTOCOL_VERSION,
+ server_min,
+ server_max,
+ )
+ }
+ }
+ EvictionReason::MalformedLogin => IggyError::InvalidFormat,
+ _ => IggyError::InvalidCommand,
+ }
+}
+
/// Issue a fresh access token for `user_id` and wrap it in the exact
/// `IdentityInfo` shape the SDKs pin: numeric `user_id` plus an `access_token`
/// carrying the token string and its unix-seconds expiry.
diff --git a/core/server-ng/src/http/extractor.rs b/core/server-ng/src/http/extractor.rs
index fd2e90051..87d8921ff 100644
--- a/core/server-ng/src/http/extractor.rs
+++ b/core/server-ng/src/http/extractor.rs
@@ -40,7 +40,7 @@ const PAT_KEY_PREFIX: &str = "pat:";
/// shared VSR session established once per presenting credential.
pub struct Authenticated {
pub user_id: u32,
- /// Per-credential VSR session. The write path (2c) reads its client id and
+ /// Per-credential VSR session. The write path reads its client id and
/// session number and serializes writes through its gate; `get_me` and
/// other read-only probes ignore it.
///
@@ -48,10 +48,6 @@ pub struct Authenticated {
/// `Rc` is not; this mirrors the `HttpState` bridge. Sound for the same
/// reason: the extractor mints it on shard 0's compio thread and every
/// handler runs on that same thread, so the wrapper is never crossed.
- #[expect(
- dead_code,
- reason = "read by the write path (2c) via client id, session, and gate"
- )]
pub session: SendWrapper<Rc<HttpSession>>,
}
(iggy) 09/19: feat(server-ng): add HTTP consumer-group create/delete routes
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 5e10f8c423470f5d13aaa1510f36b4a56d6f44e4
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 16:42:11 2026 +0200
feat(server-ng): add HTTP consumer-group create/delete routes
Add HTTP consumer-group create and delete under the topic path. Both
are mechanical clones of the create-topic / delete-topic handlers routed
through the shared submit_write helper, so they inherit the root-only
admin gate and the committed result-code business-error mapping for
free. Create decodes the committed ConsumerGroupDetailsResponse into
ConsumerGroupDetails (HTTP 200); delete returns 204.
---
core/server-ng/src/http.rs | 86 ++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 84 insertions(+), 2 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index ad3f3968f..7d3c7b467 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -37,6 +37,9 @@ use configs::http::HttpJwtConfig;
use iggy_binary_protocol::consensus::{
Command2, EvictionHeader, EvictionReason, HEADER_SIZE, result_code, result_section_len,
};
+use iggy_binary_protocol::requests::consumer_groups::{
+ CreateConsumerGroupRequest, DeleteConsumerGroupRequest,
+};
use iggy_binary_protocol::requests::personal_access_tokens::{
CreatePersonalAccessTokenRequest, DeletePersonalAccessTokenRequest,
};
@@ -50,6 +53,7 @@ use iggy_binary_protocol::requests::users::{
ChangePasswordRequest, CreateUserRequest, DeleteUserRequest, UpdatePermissionsRequest,
UpdateUserRequest,
};
+use iggy_binary_protocol::responses::consumer_groups::get_consumer_group::ConsumerGroupDetailsResponse;
use iggy_binary_protocol::responses::personal_access_tokens::RawPersonalAccessTokenResponse;
use iggy_binary_protocol::responses::streams::get_stream::GetStreamResponse;
use iggy_binary_protocol::responses::topics::get_topic::GetTopicResponse;
@@ -59,6 +63,7 @@ use iggy_binary_protocol::{
GenericHeader, Operation, RequestHeader, WireDecode, WireEncode, WireName,
};
use iggy_common::change_password::ChangePassword;
+use iggy_common::create_consumer_group::CreateConsumerGroup;
use iggy_common::create_personal_access_token::CreatePersonalAccessToken;
use iggy_common::create_stream::CreateStream;
use iggy_common::create_topic::CreateTopic;
@@ -72,8 +77,8 @@ use iggy_common::update_topic::UpdateTopic;
use iggy_common::update_user::UpdateUser;
use iggy_common::wire_conversions::{identifier_to_wire, permissions_to_wire};
use iggy_common::{
- Identifier, IdentityInfo, IggyError, IggyTimestamp, RawPersonalAccessToken, StreamDetails,
- TokenInfo, TopicDetails, UserInfoDetails, Validatable,
+ ConsumerGroupDetails, Identifier, IdentityInfo, IggyError, IggyTimestamp,
+ RawPersonalAccessToken, StreamDetails, TokenInfo, TopicDetails, UserInfoDetails, Validatable,
};
use message_bus::client_listener;
use secrecy::ExposeSecret;
@@ -389,6 +394,14 @@ fn router(state: HttpState) -> Router {
"/streams/{stream_id}/topics/{topic_id}/purge",
delete(purge_topic),
)
+ .route(
+ "/streams/{stream_id}/topics/{topic_id}/consumer-groups",
+ post(create_cg),
+ )
+ .route(
+ "/streams/{stream_id}/topics/{topic_id}/consumer-groups/{group_id}",
+ delete(delete_cg),
+ )
.route("/personal-access-tokens", post(create_pat))
.route("/personal-access-tokens/{name}", delete(delete_pat))
.with_state(state)
@@ -643,6 +656,65 @@ async fn purge_topic(
Ok(StatusCode::NO_CONTENT)
}
+/// `POST /streams/{stream_id}/topics/{topic_id}/consumer-groups`: create a
+/// consumer group under a topic and render the committed reply as the same
+/// `ConsumerGroupDetails` JSON the legacy server returns.
+///
+/// The stream and topic come from the path; the JSON body is name-only
+/// (`{"name": ...}`), matching the legacy request. The submitted op is a plain
+/// `CreateConsumerGroup`; the metadata owner assigns the group id, so this
+/// handler never allocates one itself.
+async fn create_cg(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+ Json(command): Json<CreateConsumerGroup>,
+) -> Result<Json<ConsumerGroupDetails>, WriteError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(WriteError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(WriteError::Rejected)?;
+ let request = CreateConsumerGroupRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(WriteError::Rejected)?,
+ topic_id: identifier_to_wire(&topic_id).map_err(WriteError::Rejected)?,
+ name: WireName::new(command.name)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidConsumerGroupName))?,
+ };
+ let body = request.to_bytes();
+ let payload = SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::CreateConsumerGroup,
+ &body,
+ ))
+ .await?;
+ Ok(Json(decode_consumer_group_details(&payload)?))
+}
+
+/// `DELETE /streams/{stream_id}/topics/{topic_id}/consumer-groups/{group_id}`:
+/// delete a consumer group. Returns 204 on commit.
+async fn delete_cg(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path((stream_id, topic_id, group_id)): Path<(String, String, String)>,
+) -> Result<StatusCode, WriteError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(WriteError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(WriteError::Rejected)?;
+ let group_id = Identifier::from_str_value(&group_id).map_err(WriteError::Rejected)?;
+ let request = DeleteConsumerGroupRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(WriteError::Rejected)?,
+ topic_id: identifier_to_wire(&topic_id).map_err(WriteError::Rejected)?,
+ group_id: identifier_to_wire(&group_id).map_err(WriteError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::DeleteConsumerGroup,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
/// `POST /users`: create a user and render the committed reply as the same
/// `UserInfoDetails` JSON the legacy server returns.
///
@@ -1001,6 +1073,16 @@ fn decode_user_details(payload: &[u8]) -> Result<UserInfoDetails, WriteError> {
UserInfoDetails::try_from(response).map_err(WriteError::Rejected)
}
+/// Decode the `ConsumerGroupDetailsResponse` payload of a committed
+/// create-consumer-group reply into `ConsumerGroupDetails`. `payload` is the
+/// slice past the result section that [`submit_write`] already validated as a
+/// success. The wire-to-domain conversion is infallible.
+fn decode_consumer_group_details(payload: &[u8]) -> Result<ConsumerGroupDetails, WriteError> {
+ let response = ConsumerGroupDetailsResponse::decode_from(payload)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidCommand))?;
+ Ok(ConsumerGroupDetails::from(response))
+}
+
/// Extract the raw one-time token from a [`build_raw_pat_reply`] output. That
/// reply's body is a bare `RawPersonalAccessTokenResponse` (no leading result
/// section, unlike a committed metadata reply), so decode it straight past the
(iggy) 10/19: feat(server-ng): add HTTP read routes
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 949b3d20ab7cd465e42238718584d8912939bd89
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 18:52:59 2026 +0200
feat(server-ng): add HTTP read routes
The server-ng HTTP listener could create and manage entities but
had no way to read them back; every GET returned 404/405. This
adds the metadata read surface (streams, topics, users, consumer
groups, stats, clients - list and detail).
Reads are served locally from the shard-0 state machine via the
existing build_non_replicated_response path, bypassing consensus
and needing no bound VSR session - only credential verification.
A shared authorize_read gate applies the interim root-only authz
(consistent with the write routes) and the consistency policy:
the default serializable-local read is served from local state,
while ?consistency=linearizable is served only when this node is
the current VSR primary and otherwise returns 503 (a placeholder
for the leader redirect that lands with the node roster).
GET /personal-access-tokens is deferred: listing a user's tokens
needs a new iteration-safe state-machine accessor that server-ng
does not yet expose (the TCP path lacks it too).
---
core/server-ng/src/http.rs | 533 +++++++++++++++++++++++++++++++++--
core/server-ng/src/http/extractor.rs | 37 +++
2 files changed, 551 insertions(+), 19 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index 7d3c7b467..dc1de5f62 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -27,37 +27,54 @@ use std::collections::HashMap;
use std::net::SocketAddr;
use std::rc::Rc;
-use axum::extract::{Path, State};
+use axum::extract::{Path, Query, State};
use axum::http::StatusCode;
use axum::response::{IntoResponse, Response};
use axum::routing::{delete, get, post, put};
use axum::{Json, Router};
use bytes::Bytes;
use configs::http::HttpJwtConfig;
+use consensus::{MetadataHandle, VsrConsensus};
+use iggy_binary_protocol::codes::{
+ GET_CONSUMER_GROUP_CODE, GET_CONSUMER_GROUPS_CODE, GET_STATS_CODE, GET_STREAM_CODE,
+ GET_STREAMS_CODE, GET_TOPIC_CODE, GET_TOPICS_CODE, GET_USER_CODE, GET_USERS_CODE,
+};
use iggy_binary_protocol::consensus::{
Command2, EvictionHeader, EvictionReason, HEADER_SIZE, result_code, result_section_len,
};
use iggy_binary_protocol::requests::consumer_groups::{
- CreateConsumerGroupRequest, DeleteConsumerGroupRequest,
+ CreateConsumerGroupRequest, DeleteConsumerGroupRequest, GetConsumerGroupRequest,
+ GetConsumerGroupsRequest,
};
use iggy_binary_protocol::requests::personal_access_tokens::{
CreatePersonalAccessTokenRequest, DeletePersonalAccessTokenRequest,
};
use iggy_binary_protocol::requests::streams::{
- CreateStreamRequest, DeleteStreamRequest, PurgeStreamRequest, UpdateStreamRequest,
+ CreateStreamRequest, DeleteStreamRequest, GetStreamRequest, GetStreamsRequest,
+ PurgeStreamRequest, UpdateStreamRequest,
};
+use iggy_binary_protocol::requests::system::GetStatsRequest;
use iggy_binary_protocol::requests::topics::{
- CreateTopicRequest, DeleteTopicRequest, PurgeTopicRequest, UpdateTopicRequest,
+ CreateTopicRequest, DeleteTopicRequest, GetTopicRequest, GetTopicsRequest, PurgeTopicRequest,
+ UpdateTopicRequest,
};
use iggy_binary_protocol::requests::users::{
- ChangePasswordRequest, CreateUserRequest, DeleteUserRequest, UpdatePermissionsRequest,
- UpdateUserRequest,
+ ChangePasswordRequest, CreateUserRequest, DeleteUserRequest, GetUserRequest, GetUsersRequest,
+ UpdatePermissionsRequest, UpdateUserRequest,
};
+use iggy_binary_protocol::responses::clients::client_response::ConsumerGroupInfoResponse;
+use iggy_binary_protocol::responses::clients::get_client::ClientDetailsResponse;
+use iggy_binary_protocol::responses::clients::get_clients::GetClientsResponse;
use iggy_binary_protocol::responses::consumer_groups::get_consumer_group::ConsumerGroupDetailsResponse;
+use iggy_binary_protocol::responses::consumer_groups::get_consumer_groups::GetConsumerGroupsResponse;
use iggy_binary_protocol::responses::personal_access_tokens::RawPersonalAccessTokenResponse;
use iggy_binary_protocol::responses::streams::get_stream::GetStreamResponse;
+use iggy_binary_protocol::responses::streams::get_streams::GetStreamsResponse;
+use iggy_binary_protocol::responses::system::get_stats::StatsResponse;
use iggy_binary_protocol::responses::topics::get_topic::GetTopicResponse;
+use iggy_binary_protocol::responses::topics::get_topics::GetTopicsResponse;
use iggy_binary_protocol::responses::users::get_user::UserDetailsResponse;
+use iggy_binary_protocol::responses::users::get_users::GetUsersResponse;
use iggy_binary_protocol::version::IGGY_PROTOCOL_VERSION;
use iggy_binary_protocol::{
GenericHeader, Operation, RequestHeader, WireDecode, WireEncode, WireName,
@@ -75,14 +92,20 @@ use iggy_common::update_permissions::UpdatePermissions;
use iggy_common::update_stream::UpdateStream;
use iggy_common::update_topic::UpdateTopic;
use iggy_common::update_user::UpdateUser;
-use iggy_common::wire_conversions::{identifier_to_wire, permissions_to_wire};
+use iggy_common::wire_conversions::{
+ clients_from_wire, consumer_groups_from_wire, identifier_to_wire, permissions_to_wire,
+ streams_from_wire, topics_from_wire, users_from_wire,
+};
use iggy_common::{
- ConsumerGroupDetails, Identifier, IdentityInfo, IggyError, IggyTimestamp,
- RawPersonalAccessToken, StreamDetails, TokenInfo, TopicDetails, UserInfoDetails, Validatable,
+ ClientInfo, ClientInfoDetails, ConsumerGroup, ConsumerGroupDetails, Identifier, IdentityInfo,
+ IggyError, IggyTimestamp, RawPersonalAccessToken, Stats, Stream, StreamDetails, TokenInfo,
+ Topic, TopicDetails, UserInfo, UserInfoDetails, Validatable,
};
use message_bus::client_listener;
+use metadata::impls::metadata::StreamsFrontend;
use secrecy::ExposeSecret;
use send_wrapper::SendWrapper;
+use serde::Deserialize;
use server::http::error::{CustomError, ErrorResponse};
use server_common::Message;
use tokio::sync::Mutex;
@@ -91,11 +114,14 @@ use tracing::{error, info, warn};
use crate::auth::{verify_login_credentials, verify_pat_credentials};
use crate::bootstrap::ServerNgShard;
use crate::dispatch::{submit_client_request_on_owner, submit_register_on_owner};
-use crate::http::extractor::Authenticated;
+use crate::http::extractor::{Authenticated, Identity};
use crate::http::jwt::JwtManager;
use crate::login_register::LoginRegisterError;
use crate::pat::rewrite_pat_request_for_user;
-use crate::responses::build_raw_pat_reply;
+use crate::responses::{
+ NonReplicatedResponse, build_non_replicated_response, build_raw_pat_reply,
+ connected_client_to_response,
+};
use crate::server_error::ServerNgError;
use crate::users::maybe_rewrite_user_password_request;
@@ -205,6 +231,75 @@ fn service_unavailable() -> Response {
.into_response()
}
+/// Read consistency selected by the `?consistency=` query param.
+///
+/// `serializable` (the default) serves from this node's local metadata STM:
+/// correct and consensus-free, but may trail the primary by the replication
+/// delay. `linearizable` demands the freshest committed state and is honored
+/// only on the primary; a follower answers 503 (see [`read_local`]).
+#[derive(Clone, Copy, Default, PartialEq, Eq, Deserialize)]
+#[serde(rename_all = "lowercase")]
+enum Consistency {
+ #[default]
+ Serializable,
+ Linearizable,
+}
+
+/// `?consistency=` query wrapper. An absent param defaults to
+/// [`Consistency::Serializable`]; an unrecognized value is a 400 (axum `Query`).
+#[derive(Default, Deserialize)]
+struct ConsistencyQuery {
+ #[serde(default)]
+ consistency: Consistency,
+}
+
+/// Rejection for an authenticated read route (`GET /streams`,
+/// `GET /streams/{id}`, and the reads that follow). Three classes, three
+/// renderings, mirroring [`WriteError`]'s split so error bodies stay
+/// SDK-identical:
+enum ReadError {
+ /// Caller-side or STM rejection (bad identifier, unsupported op, or the
+ /// interim non-root authz denial) graded through the legacy
+ /// `IggyError -> status` map so SDK error bodies stay byte-identical.
+ Rejected(IggyError),
+ /// Requested entity is absent -> 404 with the legacy not-found body.
+ NotFound,
+ /// A linearizable read reached a follower. Transient from the caller's view
+ /// -> 503, retryable; becomes a 307-to-leader once the roster + redirect
+ /// land (see [`not_primary_response`]).
+ NotPrimary,
+}
+
+impl IntoResponse for ReadError {
+ fn into_response(self) -> Response {
+ match self {
+ Self::Rejected(error) => CustomError::from(error).into_response(),
+ // Reuse the legacy 404 body so a missing stream renders exactly as
+ // the legacy server's `CustomError::ResourceNotFound` does.
+ Self::NotFound => CustomError::ResourceNotFound.into_response(),
+ Self::NotPrimary => not_primary_response(),
+ }
+ }
+}
+
+/// The 503 body for a linearizable read that reached a follower. No node roster
+/// exists at the HTTP layer yet (F7), so this node cannot 307-redirect to the
+/// primary nor emit a `Location` (F3); the caller must retry, ideally against
+/// the leader. Once both land this branch becomes a 307. Rendered as an
+/// `ErrorResponse` so the body shape matches every other HTTP error.
+fn not_primary_response() -> Response {
+ (
+ StatusCode::SERVICE_UNAVAILABLE,
+ Json(ErrorResponse {
+ id: StatusCode::SERVICE_UNAVAILABLE.as_u16().into(),
+ code: "not_primary".to_owned(),
+ reason: "linearizable read requires the primary; retry against the leader".to_owned(),
+ field: None,
+ }),
+ )
+ .into_response()
+}
+
/// Shared shard-0 HTTP state.
///
/// Groups the shard handle, the JWT issuer/verifier, and the per-credential VSR
@@ -221,6 +316,21 @@ struct HttpInner {
}
impl HttpInner {
+ /// True when this shard-0 node is the current VSR metadata primary, i.e.
+ /// `primary_index(current_view) == own_replica_id` - the check consensus
+ /// `is_primary` already encapsulates over the live view and this replica's
+ /// id. Absent consensus (never on shard 0 under VSR, only a no-replica
+ /// build) is treated as not-primary so a linearizable read fails closed
+ /// rather than serving possibly-stale local state as authoritative.
+ fn is_metadata_primary(&self) -> bool {
+ self.shard
+ .plane
+ .metadata()
+ .consensus
+ .as_ref()
+ .is_some_and(VsrConsensus::is_primary)
+ }
+
/// Resolve the VSR session for `key`, minting and Registering one on first
/// use. Every later request bearing the same credential reuses it.
///
@@ -375,20 +485,26 @@ fn router(state: HttpState) -> Router {
post(login_with_personal_access_token),
)
.route("/users/me", get(get_me))
- .route("/users", post(create_user))
- .route("/users/{user_id}", put(update_user).delete(delete_user))
+ .route("/users", get(get_users).post(create_user))
+ .route(
+ "/users/{user_id}",
+ get(get_user).put(update_user).delete(delete_user),
+ )
.route("/users/{user_id}/password", put(change_password))
.route("/users/{user_id}/permissions", put(update_permissions))
- .route("/streams", post(create_stream))
+ .route("/streams", get(get_streams).post(create_stream))
.route(
"/streams/{stream_id}",
- put(update_stream).delete(delete_stream),
+ get(get_stream).put(update_stream).delete(delete_stream),
)
.route("/streams/{stream_id}/purge", delete(purge_stream))
- .route("/streams/{stream_id}/topics", post(create_topic))
+ .route(
+ "/streams/{stream_id}/topics",
+ get(get_topics).post(create_topic),
+ )
.route(
"/streams/{stream_id}/topics/{topic_id}",
- put(update_topic).delete(delete_topic),
+ get(get_topic).put(update_topic).delete(delete_topic),
)
.route(
"/streams/{stream_id}/topics/{topic_id}/purge",
@@ -396,14 +512,19 @@ fn router(state: HttpState) -> Router {
)
.route(
"/streams/{stream_id}/topics/{topic_id}/consumer-groups",
- post(create_cg),
+ get(get_cgs).post(create_cg),
)
.route(
"/streams/{stream_id}/topics/{topic_id}/consumer-groups/{group_id}",
- delete(delete_cg),
+ get(get_cg).delete(delete_cg),
)
+ // TODO(hubcio): GET /personal-access-tokens (list) deferred - needs a server-ng STM
+ // list accessor (PATs sit behind the "Never iterate" apply invariant; TCP-ng lacks it too).
.route("/personal-access-tokens", post(create_pat))
.route("/personal-access-tokens/{name}", delete(delete_pat))
+ .route("/stats", get(get_stats))
+ .route("/clients", get(get_clients))
+ .route("/clients/{client_id}", get(get_client))
.with_state(state)
}
@@ -447,6 +568,316 @@ async fn get_me(identity: Authenticated) -> Json<IdentityInfo> {
})
}
+/// `GET /streams`: list every stream as the same `Vec<Stream>` JSON the legacy
+/// server returns. A consensus-free local STM read via [`read_local`].
+async fn get_streams(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<Vec<Stream>>, ReadError> {
+ let body = GetStreamsRequest.to_bytes();
+ let bytes = read_local(
+ &state,
+ identity.user_id,
+ query.consistency,
+ GET_STREAMS_CODE,
+ &body,
+ )?;
+ let response = GetStreamsResponse::decode_from(&bytes)
+ .map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Json(streams_from_wire(response)))
+}
+
+/// `GET /streams/{stream_id}`: fetch one stream by numeric id or name as the
+/// same `StreamDetails` JSON the legacy server returns; 404 when absent. A
+/// consensus-free local STM read via [`read_local`].
+async fn get_stream(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Path(stream_id): Path<String>,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<StreamDetails>, ReadError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
+ let request = GetStreamRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(ReadError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ let bytes = read_local(
+ &state,
+ identity.user_id,
+ query.consistency,
+ GET_STREAM_CODE,
+ &body,
+ )?;
+ let response = GetStreamResponse::decode_from(&bytes)
+ .map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Json(
+ StreamDetails::try_from(response).map_err(ReadError::Rejected)?,
+ ))
+}
+
+/// `GET /streams/{stream_id}/topics`: list a stream's topics as the same
+/// `Vec<Topic>` JSON the legacy server returns. A consensus-free local STM read
+/// via [`read_local`].
+async fn get_topics(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Path(stream_id): Path<String>,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<Vec<Topic>>, ReadError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
+ let request = GetTopicsRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(ReadError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ let bytes = read_local(
+ &state,
+ identity.user_id,
+ query.consistency,
+ GET_TOPICS_CODE,
+ &body,
+ )?;
+ let response = GetTopicsResponse::decode_from(&bytes)
+ .map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Json(
+ topics_from_wire(response).map_err(ReadError::Rejected)?,
+ ))
+}
+
+/// `GET /streams/{stream_id}/topics/{topic_id}`: fetch one topic by numeric id
+/// or name as the same `TopicDetails` JSON the legacy server returns; 404 when
+/// absent. A consensus-free local STM read via [`read_local`].
+async fn get_topic(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<TopicDetails>, ReadError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(ReadError::Rejected)?;
+ let request = GetTopicRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(ReadError::Rejected)?,
+ topic_id: identifier_to_wire(&topic_id).map_err(ReadError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ let bytes = read_local(
+ &state,
+ identity.user_id,
+ query.consistency,
+ GET_TOPIC_CODE,
+ &body,
+ )?;
+ let response = GetTopicResponse::decode_from(&bytes)
+ .map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Json(
+ TopicDetails::try_from(response).map_err(ReadError::Rejected)?,
+ ))
+}
+
+/// `GET /users`: list every user as the same `Vec<UserInfo>` JSON the legacy
+/// server returns. A consensus-free local STM read via [`read_local`].
+async fn get_users(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<Vec<UserInfo>>, ReadError> {
+ let body = GetUsersRequest.to_bytes();
+ let bytes = read_local(
+ &state,
+ identity.user_id,
+ query.consistency,
+ GET_USERS_CODE,
+ &body,
+ )?;
+ let response = GetUsersResponse::decode_from(&bytes)
+ .map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Json(
+ users_from_wire(response).map_err(ReadError::Rejected)?,
+ ))
+}
+
+/// `GET /users/{user_id}`: fetch one user by numeric id or name as the same
+/// `UserInfoDetails` JSON the legacy server returns; 404 when absent. A
+/// consensus-free local STM read via [`read_local`].
+async fn get_user(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Path(user_id): Path<String>,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<UserInfoDetails>, ReadError> {
+ let user_id = Identifier::from_str_value(&user_id).map_err(ReadError::Rejected)?;
+ let request = GetUserRequest {
+ user_id: identifier_to_wire(&user_id).map_err(ReadError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ let bytes = read_local(
+ &state,
+ identity.user_id,
+ query.consistency,
+ GET_USER_CODE,
+ &body,
+ )?;
+ let response = UserDetailsResponse::decode_from(&bytes)
+ .map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Json(
+ UserInfoDetails::try_from(response).map_err(ReadError::Rejected)?,
+ ))
+}
+
+/// `GET /streams/{stream_id}/topics/{topic_id}/consumer-groups`: list a topic's
+/// consumer groups as the same `Vec<ConsumerGroup>` JSON the legacy server
+/// returns. A consensus-free local STM read via [`read_local`].
+async fn get_cgs(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<Vec<ConsumerGroup>>, ReadError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(ReadError::Rejected)?;
+ let request = GetConsumerGroupsRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(ReadError::Rejected)?,
+ topic_id: identifier_to_wire(&topic_id).map_err(ReadError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ let bytes = read_local(
+ &state,
+ identity.user_id,
+ query.consistency,
+ GET_CONSUMER_GROUPS_CODE,
+ &body,
+ )?;
+ let response = GetConsumerGroupsResponse::decode_from(&bytes)
+ .map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Json(consumer_groups_from_wire(response)))
+}
+
+/// `GET /streams/{stream_id}/topics/{topic_id}/consumer-groups/{group_id}`:
+/// fetch one consumer group by numeric id or name as the same
+/// `ConsumerGroupDetails` JSON the legacy server returns; 404 when absent. The
+/// wire-to-domain conversion is infallible.
+async fn get_cg(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Path((stream_id, topic_id, group_id)): Path<(String, String, String)>,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<ConsumerGroupDetails>, ReadError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(ReadError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(ReadError::Rejected)?;
+ let group_id = Identifier::from_str_value(&group_id).map_err(ReadError::Rejected)?;
+ let request = GetConsumerGroupRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(ReadError::Rejected)?,
+ topic_id: identifier_to_wire(&topic_id).map_err(ReadError::Rejected)?,
+ group_id: identifier_to_wire(&group_id).map_err(ReadError::Rejected)?,
+ };
+ let body = request.to_bytes();
+ let bytes = read_local(
+ &state,
+ identity.user_id,
+ query.consistency,
+ GET_CONSUMER_GROUP_CODE,
+ &body,
+ )?;
+ let response = ConsumerGroupDetailsResponse::decode_from(&bytes)
+ .map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Json(ConsumerGroupDetails::from(response)))
+}
+
+/// `GET /stats`: server + storage counters as the same `Stats` JSON the legacy
+/// server returns. `GET_STATS` is served by `build_non_replicated_response` like
+/// the entity reads, so it flows through [`read_local`] unchanged rather than a
+/// dedicated builder. No entity can be missing, so no 404 branch.
+async fn get_stats(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<Stats>, ReadError> {
+ let body = GetStatsRequest.to_bytes();
+ let bytes = read_local(
+ &state,
+ identity.user_id,
+ query.consistency,
+ GET_STATS_CODE,
+ &body,
+ )?;
+ let response = StatsResponse::decode_from(&bytes)
+ .map_err(|_| ReadError::Rejected(IggyError::InvalidCommand))?;
+ Ok(Json(Stats::from(response)))
+}
+
+/// `GET /clients`: list every connected client across all shards as the same
+/// `Vec<ClientInfo>` JSON the legacy server returns.
+///
+/// Unlike the entity reads, connections live in each shard's session manager,
+/// not the metadata STM, so this scatter-gathers over the shard mesh
+/// (`list_all_clients`) instead of going through [`read_local`]. It still runs
+/// the identical root-only + consistency gate via [`authorize_read`], so its
+/// authorization matches every metadata read. The gather future is `!Send`,
+/// bridged onto shard 0's thread by `SendWrapper` exactly as the write path
+/// bridges its submit.
+async fn get_clients(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<Vec<ClientInfo>>, ReadError> {
+ authorize_read(&state, identity.user_id, query.consistency)?;
+ let infos = SendWrapper::new(state.shard.list_all_clients()).await;
+ let response = GetClientsResponse {
+ clients: infos
+ .iter()
+ .map(|info| connected_client_to_response(&state.shard, info))
+ .collect(),
+ };
+ Ok(Json(clients_from_wire(response)))
+}
+
+/// `GET /clients/{client_id}`: fetch one connected client as the same
+/// `ClientInfoDetails` JSON the legacy server returns; 404 when absent.
+///
+/// The path id is the `u32` wire client id (the seq tail of the u128 transport
+/// id), matching the legacy route's `Path<u32>`. There is no reverse map from
+/// that id to a home shard, so this gathers every shard's clients and filters -
+/// the same fan-out-and-filter as [`get_clients`] and the TCP `get_client`
+/// dispatch. The wire-to-domain conversion is infallible.
+async fn get_client(
+ State(state): State<HttpState>,
+ identity: Identity,
+ Path(client_id): Path<u32>,
+ Query(query): Query<ConsistencyQuery>,
+) -> Result<Json<ClientInfoDetails>, ReadError> {
+ authorize_read(&state, identity.user_id, query.consistency)?;
+ let infos = SendWrapper::new(state.shard.list_all_clients()).await;
+ // The wire client id is the u32 seq tail of the u128 transport id.
+ #[allow(clippy::cast_possible_truncation)]
+ let info = infos
+ .iter()
+ .find(|info| info.client_id as u32 == client_id)
+ .ok_or(ReadError::NotFound)?;
+ let consumer_groups = info.vsr_client_id.map_or_else(Vec::new, |vsr_client_id| {
+ state
+ .shard
+ .plane
+ .metadata()
+ .mux_stm
+ .streams()
+ .consumer_group_memberships(vsr_client_id)
+ .into_iter()
+ .map(
+ |(stream_id, topic_id, group_id)| ConsumerGroupInfoResponse {
+ stream_id,
+ topic_id,
+ group_id,
+ },
+ )
+ .collect()
+ });
+ let response = ClientDetailsResponse {
+ client: connected_client_to_response(&state.shard, info),
+ consumer_groups,
+ };
+ Ok(Json(ClientInfoDetails::from(response)))
+}
+
/// `POST /streams`: create a stream and render the committed reply as the same
/// `StreamDetails` JSON the legacy server returns.
///
@@ -952,6 +1383,17 @@ async fn submit_committed(
if session.user_id != DEFAULT_ROOT_USER_ID {
return Err(WriteError::Rejected(IggyError::Unauthorized));
}
+ // Per-session gate: serialize this session's request ids across the commit
+ // await, advancing the id only after an observed Reply so a transient
+ // failure leaves it free to retry.
+ //
+ // TODO(hubcio): cancellation-unsafe. If the client disconnects while parked
+ // on the await below, this handler is dropped without advancing the id, yet
+ // the prepare still commits pump-driven. The next write on this session then
+ // reuses the id, gets the prior op's cached reply (Duplicate), silently loses
+ // its own write, and returns a foreign success. Fix by moving the submit into
+ // a detached shard-0 task that owns the id advance and signals the handler
+ // via a oneshot, so the commit is recorded regardless of client liveness.
let mut next_request_id = session.gate.lock().await;
let message = build_request_message(
operation,
@@ -995,6 +1437,59 @@ async fn submit_write(
Ok(Bytes::copy_from_slice(committed_payload(&reply)?))
}
+/// The two cross-cutting gates every authenticated read enforces before it
+/// touches state. Factored out of [`read_local`] so the cross-shard client
+/// reads (`get_clients` / `get_client`) - which serve from the shard session
+/// managers, not the local STM, and so cannot use [`read_local`] - still pass
+/// the identical gate. Keeping it in one place is what guarantees no read route
+/// can silently skip authz or answer a linearizable request on a follower.
+///
+/// Root-only: until server-ng has an RBAC permissioner, every read is root-only,
+/// mirroring the write gate in [`submit_committed`]. A non-root credential is
+/// authenticated but unprivileged -> 403. Linearizable reads must come from the
+/// primary; on a follower we cannot yet 307-redirect (no roster: F7) nor emit
+/// the Location (F3), so 503 and let the client retry. This becomes a
+/// 307-to-leader once both land.
+fn authorize_read(
+ state: &HttpInner,
+ user_id: u32,
+ consistency: Consistency,
+) -> Result<(), ReadError> {
+ if user_id != DEFAULT_ROOT_USER_ID {
+ return Err(ReadError::Rejected(IggyError::Unauthorized));
+ }
+ if consistency == Consistency::Linearizable && !state.is_metadata_primary() {
+ return Err(ReadError::NotPrimary);
+ }
+ Ok(())
+}
+
+/// Serve one authenticated read from the local metadata STM and hand back the
+/// wire response body. Shared chokepoint for every read route whose data lives
+/// in the metadata STM: it runs the shared [`authorize_read`] gate, then
+/// delegates to [`build_non_replicated_response`], the SAME local-read entry the
+/// TCP dispatch spine uses (`handle_default_non_replicated`), so an HTTP read and
+/// a TCP read of the same entity return byte-identical bodies.
+///
+/// Reads never touch consensus or a VSR session: `build_non_replicated_response`
+/// is a pure STM read. It is synchronous, so this helper is too - no submit
+/// await, no gate, no `SendWrapper`. An absent entity surfaces as
+/// [`NonReplicatedResponse::Empty`], mapped to 404 here because every REST read
+/// whose entity can be missing shares that not-found shape.
+fn read_local(
+ state: &HttpInner,
+ user_id: u32,
+ consistency: Consistency,
+ code: u32,
+ body: &[u8],
+) -> Result<Bytes, ReadError> {
+ authorize_read(state, user_id, consistency)?;
+ match build_non_replicated_response(&state.shard, code, body).map_err(ReadError::Rejected)? {
+ NonReplicatedResponse::Empty => Err(ReadError::NotFound),
+ NonReplicatedResponse::Bytes(bytes) => Ok(bytes),
+ }
+}
+
/// Classify a committed reply's leading result section and return the typed
/// payload slice on success. Mirrors the SDK's `split_metadata_result`:
/// `Some(0)` is success and the payload follows the result section; a nonzero
diff --git a/core/server-ng/src/http/extractor.rs b/core/server-ng/src/http/extractor.rs
index 87d8921ff..96815a56b 100644
--- a/core/server-ng/src/http/extractor.rs
+++ b/core/server-ng/src/http/extractor.rs
@@ -79,6 +79,43 @@ impl FromRequestParts<HttpState> for Authenticated {
}
}
+/// Read-only caller identity for a protected read route: the authenticated
+/// user id and nothing else.
+///
+/// Unlike [`Authenticated`], it verifies the bearer WITHOUT minting or
+/// Registering a VSR session. Reads are served from the local metadata STM and
+/// never enter consensus, so a session (and the `Register` commit round-trip it
+/// costs) would be dead weight. Same 401 rejection surface as [`Authenticated`]
+/// (missing/invalid/expired credential), and the verify runs through the same
+/// [`resolve_credential`] chokepoint, so a JWT and a PAT are honored identically.
+pub struct Identity {
+ pub user_id: u32,
+}
+
+impl FromRequestParts<HttpState> for Identity {
+ type Rejection = AuthError;
+
+ async fn from_request_parts(
+ parts: &mut Parts,
+ state: &HttpState,
+ ) -> Result<Self, Self::Rejection> {
+ let bearer = parts
+ .headers
+ .get(AUTHORIZATION)
+ .and_then(|value| value.to_str().ok())
+ .and_then(|value| value.strip_prefix(BEARER))
+ .ok_or(IggyError::AccessTokenMissing)?;
+
+ // Verify only. The session key and expiry `resolve_credential` also
+ // returns feed the write path's session table; a read discards them.
+ // No `.await` and no session borrow here, so the extractor future needs
+ // no `SendWrapper` bridge (contrast [`Authenticated`], which awaits the
+ // `!Send` `resolve_session`).
+ let (_key, user_id, _expiry) = resolve_credential(state, bearer)?;
+ Ok(Self { user_id })
+ }
+}
+
/// Map a bearer credential to its `(session key, user id, expiry secs)`.
///
/// A JWT is the primary credential, keyed by its `jti`. A raw PAT presented as
(iggy) 16/19: feat(server-ng): add HTTP consumer-offset write routes
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 0944a78c637136ebde127312f7e5f86a9c98d194
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Thu Jul 2 00:13:08 2026 +0200
feat(server-ng): add HTTP consumer-offset write routes
The HTTP listener could read consumer offsets but not store or
delete them, so an HTTP consumer could not persist its position.
Add legacy-parity PUT /consumer-offsets and DELETE
/consumer-offsets/{consumer_id}, both returning 204 on commit.
Offset writes are partition-plane ops, not metadata writes: encode
the v2 wire requests (StoreConsumerOffset2/DeleteConsumerOffset2)
with ack pinned to Quorum and ride the same reply-slot machinery
the produce route proved - slot before dispatch, bounded wait,
op==0 gate reply as the generic 404. The produce-named shared
pieces are renamed plane-neutral (PartitionWriteError,
partition_write_replicated) instead of duplicated; the 504 body
keeps a per-kind code (produce_timeout / offset_write_timeout) so
callers can tell which write surface timed out.
---
core/server-ng/src/http.rs | 376 +++++++++++++++++++++++++++++++++++++--------
1 file changed, 310 insertions(+), 66 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index c3a406d93..596f00a94 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -52,7 +52,9 @@ use iggy_binary_protocol::requests::consumer_groups::{
CreateConsumerGroupRequest, DeleteConsumerGroupRequest, GetConsumerGroupRequest,
GetConsumerGroupsRequest,
};
-use iggy_binary_protocol::requests::consumer_offsets::GetConsumerOffsetRequest;
+use iggy_binary_protocol::requests::consumer_offsets::{
+ DeleteConsumerOffset2Request, GetConsumerOffsetRequest, StoreConsumerOffset2Request,
+};
use iggy_binary_protocol::requests::messages::{
PollMessagesRequest, RawMessage, SendMessagesEncoder,
};
@@ -87,7 +89,8 @@ use iggy_binary_protocol::responses::users::get_user::UserDetailsResponse;
use iggy_binary_protocol::responses::users::get_users::GetUsersResponse;
use iggy_binary_protocol::version::IGGY_PROTOCOL_VERSION;
use iggy_binary_protocol::{
- GenericHeader, Operation, ReplyHeader, RequestHeader, WireDecode, WireEncode, WireName,
+ AckLevel, GenericHeader, Operation, ReplyHeader, RequestHeader, WireDecode, WireEncode,
+ WireName,
};
use iggy_common::change_password::ChangePassword;
use iggy_common::create_consumer_group::CreateConsumerGroup;
@@ -96,25 +99,28 @@ use iggy_common::create_stream::CreateStream;
use iggy_common::create_topic::CreateTopic;
use iggy_common::create_user::CreateUser;
use iggy_common::defaults::DEFAULT_ROOT_USER_ID;
+use iggy_common::delete_consumer_offset::DeleteConsumerOffset;
use iggy_common::get_consumer_offset::GetConsumerOffset;
use iggy_common::login_user::LoginUser;
use iggy_common::login_with_personal_access_token::LoginWithPersonalAccessToken;
use iggy_common::poll_messages::DEFAULT_PARTITION_ID;
+use iggy_common::store_consumer_offset::StoreConsumerOffset;
use iggy_common::update_permissions::UpdatePermissions;
use iggy_common::update_stream::UpdateStream;
use iggy_common::update_topic::UpdateTopic;
use iggy_common::update_user::UpdateUser;
use iggy_common::wire_conversions::{
- clients_from_wire, consumer_groups_from_wire, identifier_to_wire, partitioning_to_wire,
- permissions_to_wire, streams_from_wire, topics_from_wire, users_from_wire,
+ clients_from_wire, consumer_groups_from_wire, consumer_to_wire, identifier_to_wire,
+ partitioning_to_wire, permissions_to_wire, streams_from_wire, topics_from_wire,
+ users_from_wire,
};
use iggy_common::{
ClientInfo, ClientInfoDetails, ClusterMetadata, ClusterNode, ClusterNodeRole,
- ClusterNodeStatus, ConsumerGroup, ConsumerGroupDetails, ConsumerOffsetInfo, Identifier,
- IdentityInfo, IggyError, IggyMessageView, IggyTimestamp, PollMessages, PolledMessages,
- RESYNC_REQUIRED_PARTITION_SENTINEL, RawPersonalAccessToken, SendMessages, Stats, Stream,
- StreamDetails, TokenInfo, Topic, TopicDetails, TransportEndpoints, UserInfo, UserInfoDetails,
- Validatable,
+ ClusterNodeStatus, Consumer, ConsumerGroup, ConsumerGroupDetails, ConsumerOffsetInfo,
+ Identifier, IdentityInfo, IggyError, IggyMessageView, IggyTimestamp, PollMessages,
+ PolledMessages, RESYNC_REQUIRED_PARTITION_SENTINEL, RawPersonalAccessToken, SendMessages,
+ Stats, Stream, StreamDetails, TokenInfo, Topic, TopicDetails, TransportEndpoints, UserInfo,
+ UserInfoDetails, Validatable,
};
use message_bus::{BusMessage, InstanceToken, client_listener};
use metadata::impls::metadata::StreamsFrontend;
@@ -158,13 +164,14 @@ const MAX_HTTP_SESSIONS: usize = 100_000;
/// are 1-based and strictly increasing within a session.
const FIRST_REQUEST_ID: u64 = 1;
-/// Bound on a produce's wait for its committed partition reply. Long enough to
-/// ride out a view change (plus the dispatch gates' own routable-wait budget),
-/// short enough not to pin HTTP connections behind a dead consensus group. On
-/// expiry the caller gets 504 and must treat the outcome as unknown: the
-/// partition plane is at-least-once and the prepare may still commit after the
-/// wait gave up, so the server never retries on the caller's behalf.
-const PRODUCE_REPLY_TIMEOUT: Duration = Duration::from_secs(10);
+/// Bound on a partition write's (produce / consumer-offset write) wait for its
+/// committed reply. Long enough to ride out a view change (plus the dispatch
+/// gates' own routable-wait budget), short enough not to pin HTTP connections
+/// behind a dead consensus group. On expiry the caller gets 504 and must treat
+/// the outcome as unknown: the partition plane is at-least-once and the prepare
+/// may still commit after the wait gave up, so the server never retries on the
+/// caller's behalf.
+const PARTITION_WRITE_REPLY_TIMEOUT: Duration = Duration::from_secs(10);
/// VSR client id stamped on HTTP data-plane reads (poll / consumer-offset).
/// HTTP reads never Register a VSR client, and shard-0 client ids are minted
@@ -223,8 +230,9 @@ struct HttpSession {
/// monotonically and never reused, which the slot-guard contract requires.
data_request: Cell<u64>,
/// Registry token of this session's lazily-installed in-process reply
- /// target (`None` until the first awaited produce). Stored so session
- /// eviction can tear the registry entry down fenced by the same token.
+ /// target (`None` until the first awaited partition write). Stored so
+ /// session eviction can tear the registry entry down fenced by the same
+ /// token.
registry_token: Cell<Option<InstanceToken>>,
}
@@ -293,56 +301,67 @@ impl IntoResponse for WriteError {
}
}
-/// Rejection for a data-plane produce (`POST .../messages`).
+/// Rejection for a data-plane partition write (`POST .../messages` produce and
+/// the `PUT`/`DELETE .../consumer-offsets` writes).
///
/// Split differently from [`WriteError`] because the partition plane replies
/// carry no committed error code: a pre-dispatch gate failure is an empty
/// reply distinguishable only by header (see [`classify_partition_reply`]),
-/// and an unanswered produce is a distinct outcome the caller must treat as
+/// and an unanswered write is a distinct outcome the caller must treat as
/// unknown rather than failed.
-enum ProduceError {
+enum PartitionWriteError {
/// Caller-side rejection (bad identifier, oversized batch, the interim
/// non-root denial) or a malformed reply frame, rendered through the
/// legacy `IggyError -> status` map for SDK-identical bodies.
Rejected(IggyError),
- /// The dispatch gates could not route the produce: the stream, topic, or
+ /// The dispatch gates could not route the write: the stream, topic, or
/// partition does not resolve (or never materialised within the routable
/// budget). Rendered as the legacy 404 body.
NotFound,
/// The in-process reply slot could not be installed. Transient server
/// condition -> the shared 503, retryable.
Unavailable,
- /// No committed reply within [`PRODUCE_REPLY_TIMEOUT`], or the session's
- /// reply target was torn down mid-wait. 504: the commit may still land
- /// (at-least-once), so this is a hard "outcome unknown", not a failure
- /// the server may transparently retry.
- Timeout,
+ /// No committed reply within [`PARTITION_WRITE_REPLY_TIMEOUT`], or the
+ /// session's reply target was torn down mid-wait. 504: the commit may
+ /// still land (at-least-once), so this is a hard "outcome unknown", not a
+ /// failure the server may transparently retry. Carries the write's
+ /// operation so the 504 body names which write kind timed out.
+ Timeout(Operation),
}
-impl IntoResponse for ProduceError {
+impl IntoResponse for PartitionWriteError {
fn into_response(self) -> Response {
match self {
Self::Rejected(error) => CustomError::from(error).into_response(),
Self::NotFound => CustomError::ResourceNotFound.into_response(),
Self::Unavailable => service_unavailable(),
- Self::Timeout => produce_timeout_response(),
+ Self::Timeout(operation) => partition_write_timeout_response(operation),
}
}
}
-/// 504 body for a produce whose commit outcome is unknown. Shaped like every
-/// other HTTP error (`ErrorResponse`) so clients parse one error schema.
-fn produce_timeout_response() -> Response {
- gateway_timeout_response(
- "produce_timeout",
- "produce was not acknowledged in time; the write may still commit",
- )
+/// 504 body for a partition write whose commit outcome is unknown, coded per
+/// write kind so a caller can tell a produce timeout from an offset-write
+/// timeout. Shaped like every other HTTP error (`ErrorResponse`) so clients
+/// parse one error schema.
+fn partition_write_timeout_response(operation: Operation) -> Response {
+ let (code, reason) = match operation {
+ Operation::SendMessages => (
+ "produce_timeout",
+ "produce was not acknowledged in time; the write may still commit",
+ ),
+ _ => (
+ "offset_write_timeout",
+ "consumer-offset write was not acknowledged in time; the write may still commit",
+ ),
+ };
+ gateway_timeout_response(code, reason)
}
/// Shared 504 rendering for an in-band request the partition plane did not
/// answer in time, shaped like every other HTTP error (`ErrorResponse`) so
-/// clients parse one error schema. Consumed by the produce reply wait and the
-/// partition reads ([`ReadError::Timeout`]).
+/// clients parse one error schema. Consumed by the partition-write reply wait
+/// and the partition reads ([`ReadError::Timeout`]).
fn gateway_timeout_response(code: &str, reason: &str) -> Response {
(
StatusCode::GATEWAY_TIMEOUT,
@@ -614,7 +633,7 @@ impl HttpInner {
return true;
}
// Evicting the session also tears down its in-process reply
- // target (if a produce ever installed one), cancelling any
+ // target (if a partition write ever installed one), cancelling any
// still-parked reply waiters. Token-fenced so a stale sweep can
// never remove a later occupant of the key.
if let Some(token) = session.registry_token.get() {
@@ -792,7 +811,11 @@ fn router(state: HttpState) -> Router {
)
.route(
"/streams/{stream_id}/topics/{topic_id}/consumer-offsets",
- get(get_consumer_offset),
+ get(get_consumer_offset).put(store_consumer_offset),
+ )
+ .route(
+ "/streams/{stream_id}/topics/{topic_id}/consumer-offsets/{consumer_id}",
+ delete(delete_consumer_offset),
)
.route(
"/streams/{stream_id}/topics/{topic_id}/consumer-groups",
@@ -1478,21 +1501,28 @@ async fn send_messages(
Path((stream_id, topic_id)): Path<(String, String)>,
Query(query): Query<ProduceQuery>,
Json(command): Json<SendMessages>,
-) -> Result<Response, ProduceError> {
+) -> Result<Response, PartitionWriteError> {
// Interim authorization: root-only until server-ng has an RBAC
// permissioner, mirroring the control-plane gate in `submit_committed`.
if identity.session.user_id != DEFAULT_ROOT_USER_ID {
- return Err(ProduceError::Rejected(IggyError::Unauthorized));
+ return Err(PartitionWriteError::Rejected(IggyError::Unauthorized));
}
- let stream_id = Identifier::from_str_value(&stream_id).map_err(ProduceError::Rejected)?;
- let topic_id = Identifier::from_str_value(&topic_id).map_err(ProduceError::Rejected)?;
+ let stream_id =
+ Identifier::from_str_value(&stream_id).map_err(PartitionWriteError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(PartitionWriteError::Rejected)?;
// Rejects an oversized partitioning key and an empty or oversized batch.
- command.validate().map_err(ProduceError::Rejected)?;
- let body =
- encode_send_messages(&stream_id, &topic_id, &command).map_err(ProduceError::Rejected)?;
+ command.validate().map_err(PartitionWriteError::Rejected)?;
+ let body = encode_send_messages(&stream_id, &topic_id, &command)
+ .map_err(PartitionWriteError::Rejected)?;
match query.ack {
ProduceAck::Replicated => {
- SendWrapper::new(produce_replicated(&state, &identity.session, &body)).await?;
+ SendWrapper::new(partition_write_replicated(
+ &state,
+ &identity.session,
+ Operation::SendMessages,
+ &body,
+ ))
+ .await?;
Ok((
StatusCode::CREATED,
[(
@@ -1513,6 +1543,82 @@ async fn send_messages(
}
}
+/// `PUT /streams/{stream_id}/topics/{topic_id}/consumer-offsets`: store a
+/// consumer's offset. The JSON body is the same `StoreConsumerOffset` shape the
+/// legacy server accepts (flattened `consumer_id`, optional `partition_id`,
+/// `offset`); stream and topic come from the path. Returns 204 on commit,
+/// matching the legacy server.
+///
+/// Data plane like a produce: the offset write is a replicated op on the
+/// partition group's own consensus, awaited through the session's in-process
+/// reply slot ([`partition_write_replicated`]). The v2 wire op is pinned to
+/// `ack = Quorum` - `?ack=none` is a produce-only surface. The consumer
+/// identifier passes through on the wire; the dispatch resolvers hash named
+/// consumers and rewrite group ids server-side, identically to TCP.
+async fn store_consumer_offset(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path((stream_id, topic_id)): Path<(String, String)>,
+ Json(command): Json<StoreConsumerOffset>,
+) -> Result<StatusCode, PartitionWriteError> {
+ // Interim authorization: root-only until server-ng has an RBAC
+ // permissioner, mirroring the control-plane gate in `submit_committed`.
+ if identity.session.user_id != DEFAULT_ROOT_USER_ID {
+ return Err(PartitionWriteError::Rejected(IggyError::Unauthorized));
+ }
+ let stream_id =
+ Identifier::from_str_value(&stream_id).map_err(PartitionWriteError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(PartitionWriteError::Rejected)?;
+ let request = store_offset_wire_request(&stream_id, &topic_id, &command)
+ .map_err(PartitionWriteError::Rejected)?;
+ let body = request.to_bytes();
+ SendWrapper::new(partition_write_replicated(
+ &state,
+ &identity.session,
+ Operation::StoreConsumerOffset2,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// `DELETE /streams/{stream_id}/topics/{topic_id}/consumer-offsets/{consumer_id}`:
+/// delete a consumer's stored offset. The consumer comes from the path and the
+/// optional `partition_id` from the query, the same `DeleteConsumerOffset`
+/// shape the legacy server accepts. Returns 204 on commit, matching the legacy
+/// server. Same replicated partition write as [`store_consumer_offset`].
+async fn delete_consumer_offset(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path((stream_id, topic_id, consumer_id)): Path<(String, String, String)>,
+ Query(query): Query<DeleteConsumerOffset>,
+) -> Result<StatusCode, PartitionWriteError> {
+ // Interim authorization: root-only until server-ng has an RBAC
+ // permissioner, mirroring the control-plane gate in `submit_committed`.
+ if identity.session.user_id != DEFAULT_ROOT_USER_ID {
+ return Err(PartitionWriteError::Rejected(IggyError::Unauthorized));
+ }
+ let stream_id =
+ Identifier::from_str_value(&stream_id).map_err(PartitionWriteError::Rejected)?;
+ let topic_id = Identifier::from_str_value(&topic_id).map_err(PartitionWriteError::Rejected)?;
+ // `Consumer::new` fixes the kind to `Consumer`, exactly as the legacy
+ // handler does; HTTP cannot express a group-kind offset op.
+ let consumer = Consumer::new(
+ Identifier::from_str_value(&consumer_id).map_err(PartitionWriteError::Rejected)?,
+ );
+ let request = delete_offset_wire_request(&stream_id, &topic_id, &consumer, query.partition_id)
+ .map_err(PartitionWriteError::Rejected)?;
+ let body = request.to_bytes();
+ SendWrapper::new(partition_write_replicated(
+ &state,
+ &identity.session,
+ Operation::DeleteConsumerOffset2,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
/// `POST /streams/{stream_id}/topics/{topic_id}/consumer-groups`: create a
/// consumer group under a topic and render the committed reply as the same
/// `ConsumerGroupDetails` JSON the legacy server returns.
@@ -1952,6 +2058,45 @@ fn consumer_offset_wire_request(
})
}
+/// Map a validated HTTP store-offset body onto the v2 wire request
+/// (`StoreConsumerOffset2Request`), `ack` pinned to `Quorum` so the route can
+/// await the committed reply. The body's consumer kind is structurally always
+/// `Consumer` (`Consumer::kind` is `#[serde(skip)]`), matching the legacy HTTP
+/// server; `partition_id` passes through as the wire `Option` (flag byte +
+/// u32) for the server-side resolvers to ground.
+fn store_offset_wire_request(
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ command: &StoreConsumerOffset,
+) -> Result<StoreConsumerOffset2Request, IggyError> {
+ Ok(StoreConsumerOffset2Request {
+ consumer: consumer_to_wire(&command.consumer)?,
+ stream_id: identifier_to_wire(stream_id)?,
+ topic_id: identifier_to_wire(topic_id)?,
+ partition_id: command.partition_id,
+ offset: command.offset,
+ ack: AckLevel::Quorum,
+ })
+}
+
+/// Map a validated HTTP delete-offset request onto the v2 wire request
+/// (`DeleteConsumerOffset2Request`), `ack` pinned to `Quorum` like
+/// [`store_offset_wire_request`].
+fn delete_offset_wire_request(
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ consumer: &Consumer,
+ partition_id: Option<u32>,
+) -> Result<DeleteConsumerOffset2Request, IggyError> {
+ Ok(DeleteConsumerOffset2Request {
+ consumer: consumer_to_wire(consumer)?,
+ stream_id: identifier_to_wire(stream_id)?,
+ topic_id: identifier_to_wire(topic_id)?,
+ partition_id,
+ ack: AckLevel::Quorum,
+ })
+}
+
/// The empty `PolledMessages` a fenced consumer-group poll answers, carrying
/// the re-sync sentinel in `partition_id` exactly as the TCP dispatch replies
/// it, so an SDK re-syncs its assignment instead of reading end-of-partition.
@@ -1964,23 +2109,25 @@ const fn resync_required_polled_messages() -> PolledMessages {
}
}
-/// Run one awaited produce end to end: install the reply slot, dispatch into
-/// the partition plane, and wait (bounded) for the committed reply.
+/// Run one awaited partition write (produce / consumer-offset write) end to
+/// end: install the reply slot, dispatch into the partition plane, and wait
+/// (bounded) for the committed reply.
///
/// Slot-before-dispatch is load-bearing: every pre-dispatch gate failure
/// inside [`dispatch_partition_request`] replies through `send_to_client`,
/// which fires an installed slot, so one slot catches every exit. The slot
/// guard borrows the registry, which is why this whole future runs inside
/// the caller's `SendWrapper` on shard 0.
-async fn produce_replicated(
+async fn partition_write_replicated(
state: &HttpInner,
session: &HttpSession,
+ operation: Operation,
body: &[u8],
-) -> Result<(), ProduceError> {
+) -> Result<(), PartitionWriteError> {
ensure_in_process_reply_target(state, session);
let request_id = session.next_data_request_id();
let message = build_request_message(
- Operation::SendMessages,
+ operation,
session.client_id,
session.session,
request_id,
@@ -1992,8 +2139,12 @@ async fn produce_replicated(
.clients()
.install_reply_slot(session.client_id, request_id)
.map_err(|error| {
- warn!(?error, "server-ng HTTP: produce reply slot install failed");
- ProduceError::Unavailable
+ warn!(
+ ?error,
+ ?operation,
+ "server-ng HTTP: partition write reply slot install failed"
+ );
+ PartitionWriteError::Unavailable
})?;
dispatch_partition_request(
&state.shard,
@@ -2003,7 +2154,7 @@ async fn produce_replicated(
session.client_id,
)
.await;
- let outcome = compio::time::timeout(PRODUCE_REPLY_TIMEOUT, receiver).await;
+ let outcome = compio::time::timeout(PARTITION_WRITE_REPLY_TIMEOUT, receiver).await;
// Removes the slot unless the reply already fired, so a late commit
// reply after a timeout sheds at the bus instead of leaking a waiter.
drop(guard);
@@ -2011,7 +2162,7 @@ async fn produce_replicated(
Ok(Ok(reply)) => classify_partition_reply(&reply),
// Cancelled (reply target torn down by session eviction mid-wait) or
// elapsed: same caller contract either way - outcome unknown, 504.
- Ok(Err(_)) | Err(_) => Err(ProduceError::Timeout),
+ Ok(Err(_)) | Err(_) => Err(PartitionWriteError::Timeout(operation)),
}
}
@@ -2059,24 +2210,24 @@ fn ensure_in_process_reply_target(state: &HttpInner, session: &HttpSession) {
}
}
-/// Discriminate a produce reply. Partition replies carry no result section
-/// and no error code - success and gate failure are both empty-bodied - so
-/// the only wire discriminator is the reply's `op`: a committed reply is
+/// Discriminate a partition write reply. Partition replies carry no result
+/// section and no error code (success and gate failure are both empty-bodied),
+/// so the only wire discriminator is the reply's `op`: a committed reply is
/// built from its prepare header, whose op (the partition group's commit
/// number) is always >= 1, while the pre-dispatch gate failures reply through
/// `build_empty_reply` with 0 in that field. The gate reply cannot name which
/// entity was missing, hence the generic legacy 404 body.
-fn classify_partition_reply(reply: &BusMessage) -> Result<(), ProduceError> {
+fn classify_partition_reply(reply: &BusMessage) -> Result<(), PartitionWriteError> {
let header = reply
.as_slice()
.get(..HEADER_SIZE)
.and_then(|bytes| bytemuck::checked::try_from_bytes::<ReplyHeader>(bytes).ok())
- .ok_or(ProduceError::Rejected(IggyError::InvalidCommand))?;
+ .ok_or(PartitionWriteError::Rejected(IggyError::InvalidCommand))?;
if header.command != Command2::Reply {
- return Err(ProduceError::Rejected(IggyError::InvalidCommand));
+ return Err(PartitionWriteError::Rejected(IggyError::InvalidCommand));
}
if header.op == 0 {
- return Err(ProduceError::NotFound);
+ return Err(PartitionWriteError::NotFound);
}
Ok(())
}
@@ -2620,6 +2771,83 @@ mod tests {
assert_eq!(wire.consumer.kind, 1);
}
+ // -- consumer-offset write wire mapping --
+
+ #[test]
+ fn store_offset_wire_request_round_trips_with_quorum_ack() {
+ let stream_id = Identifier::numeric(1).expect("valid stream id");
+ let topic_id = Identifier::named("orders").expect("valid topic id");
+ let command = StoreConsumerOffset {
+ consumer: Consumer::new(Identifier::named("c1").expect("valid id")),
+ partition_id: Some(1),
+ offset: 42,
+ };
+ let request = store_offset_wire_request(&stream_id, &topic_id, &command).expect("maps");
+ let bytes = request.to_bytes();
+ assert_eq!(*bytes.last().expect("non-empty"), AckLevel::Quorum.as_u8());
+ let (decoded, consumed) =
+ StoreConsumerOffset2Request::decode(&bytes).expect("decodes as the server does");
+ assert_eq!(consumed, bytes.len());
+ assert_eq!(decoded, request);
+ assert_eq!(decoded.consumer.kind, 1);
+ assert_eq!(decoded.partition_id, Some(1));
+ assert_eq!(decoded.offset, 42);
+ assert_eq!(decoded.ack, AckLevel::Quorum);
+ }
+
+ #[test]
+ fn store_offset_wire_request_passes_omitted_partition_through() {
+ let stream_id = Identifier::numeric(1).expect("valid stream id");
+ let topic_id = Identifier::numeric(2).expect("valid topic id");
+ let command = StoreConsumerOffset {
+ consumer: Consumer::new(Identifier::numeric(7).expect("valid id")),
+ partition_id: None,
+ offset: u64::MAX,
+ };
+ let request = store_offset_wire_request(&stream_id, &topic_id, &command).expect("maps");
+ let bytes = request.to_bytes();
+ let (decoded, consumed) =
+ StoreConsumerOffset2Request::decode(&bytes).expect("decodes as the server does");
+ assert_eq!(consumed, bytes.len());
+ assert_eq!(decoded.partition_id, None);
+ assert_eq!(decoded.offset, u64::MAX);
+ assert_eq!(decoded.ack, AckLevel::Quorum);
+ }
+
+ #[test]
+ fn delete_offset_wire_request_round_trips_partition_variants() {
+ let stream_id = Identifier::named("stream-1").expect("valid stream id");
+ let topic_id = Identifier::numeric(2).expect("valid topic id");
+ for partition_id in [Some(1), None] {
+ let request = delete_offset_wire_request(
+ &stream_id,
+ &topic_id,
+ &Consumer::new(Identifier::named("c1").expect("valid id")),
+ partition_id,
+ )
+ .expect("maps");
+ let bytes = request.to_bytes();
+ assert_eq!(*bytes.last().expect("non-empty"), AckLevel::Quorum.as_u8());
+ let (decoded, consumed) =
+ DeleteConsumerOffset2Request::decode(&bytes).expect("decodes as the server does");
+ assert_eq!(consumed, bytes.len());
+ assert_eq!(decoded, request);
+ assert_eq!(decoded.consumer.kind, 1);
+ assert_eq!(decoded.partition_id, partition_id);
+ assert_eq!(decoded.ack, AckLevel::Quorum);
+ }
+ }
+
+ #[test]
+ fn delete_offset_query_parses_optional_partition_id() {
+ let uri: Uri = "/x?partition_id=3".parse().expect("valid uri");
+ let Query(query) = Query::<DeleteConsumerOffset>::try_from_uri(&uri).expect("parses");
+ assert_eq!(query.partition_id, Some(3));
+ let bare: Uri = "/x".parse().expect("valid uri");
+ let Query(query) = Query::<DeleteConsumerOffset>::try_from_uri(&bare).expect("parses");
+ assert_eq!(query.partition_id, None);
+ }
+
// -- polled messages body decode --
/// Wrap one stored `SendMessages2` batch (`[256B header][blob]`) as the
@@ -2720,13 +2948,29 @@ mod tests {
assert!(classify_partition_reply(&reply).is_ok());
}
+ /// The 204 path of the offset write routes: a committed
+ /// `StoreConsumerOffset2` reply (op >= 1) classifies as success.
+ #[test]
+ fn committed_offset_write_reply_classifies_as_success() {
+ let prepare = PrepareHeader {
+ command: Command2::Prepare,
+ operation: Operation::StoreConsumerOffset2,
+ client: 42,
+ op: 3,
+ request: 2,
+ ..Default::default()
+ };
+ let reply = frozen(consensus::build_reply_message(&prepare, &Bytes::new()));
+ assert!(classify_partition_reply(&reply).is_ok());
+ }
+
#[test]
fn gate_failure_empty_reply_classifies_as_not_found() {
let request = build_request_message(Operation::SendMessages, 42, 7, 1, &[]);
let reply = frozen(build_empty_reply(request.header(), 42, 0, 9));
assert!(matches!(
classify_partition_reply(&reply),
- Err(ProduceError::NotFound)
+ Err(PartitionWriteError::NotFound)
));
}
@@ -2735,7 +2979,7 @@ mod tests {
let request = build_request_message(Operation::SendMessages, 42, 7, 1, &[]);
assert!(matches!(
classify_partition_reply(&request.into_generic().into_frozen()),
- Err(ProduceError::Rejected(IggyError::InvalidCommand))
+ Err(PartitionWriteError::Rejected(IggyError::InvalidCommand))
));
}
}
(iggy) 13/19: feat(server-ng): add in-process reply slots for HTTP data plane
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 547e760d815ab436e013e1634a628f0514e0ed76
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 22:51:49 2026 +0200
feat(server-ng): add in-process reply slots for HTTP data plane
An HTTP caller holds no consensus socket: partition replies fire
post-commit through send_to_client, which until now could only
resolve to a per-peer wire queue, and the partition routing gates
lived inline in handle_client_request where a transport-less caller
could not reach them.
Teach the connection registry an in-process entry (no reader/writer
tasks) with a request-keyed reply slot API: install_reply_slot hands
back a oneshot receiver plus an RAII guard that removes the slot on
drop, so a timed-out caller never leaks its waiter. Duplicate request
ids are rejected instead of silently replacing a live waiter, and
entry teardown cancels every pending receiver. Extract the partition
tail of handle_client_request into dispatch_partition_request so an
already-authenticated in-process caller routes partition ops through
the same namespace and routability gates; TCP behavior is unchanged.
Drop the partition plane's speculative take_reply_sender call site:
in-process partition replies correlate by request id at the bus, not
by pipeline subscriber, so it would never gain a caller.
---
core/message_bus/src/lib.rs | 53 +++-
.../src/lifecycle/connection_registry.rs | 287 ++++++++++++++++++++-
core/message_bus/src/lifecycle/mod.rs | 2 +-
core/partitions/src/iggy_partition.rs | 10 +-
core/server-ng/src/dispatch.rs | 185 +++++++------
5 files changed, 444 insertions(+), 93 deletions(-)
diff --git a/core/message_bus/src/lib.rs b/core/message_bus/src/lib.rs
index ea50e4cd1..9679c6906 100644
--- a/core/message_bus/src/lib.rs
+++ b/core/message_bus/src/lib.rs
@@ -95,7 +95,8 @@ pub use installer::conn_info::{
};
pub use lifecycle::{
BusMessage, BusReceiver, BusSender, ConnectionRegistry, DrainOutcome, FusedShutdown,
- ReplicaRegistry, ReplyRoute, Shutdown, ShutdownToken,
+ InstanceToken, ReplicaRegistry, ReplyRoute, ReplySlotError, ReplySlotGuard, Shutdown,
+ ShutdownToken,
};
pub use transports::tls::TlsServerCredentials;
@@ -1395,6 +1396,56 @@ mod tests {
assert!(matches!(err, SendError::ClientNotFound(_)));
}
+ #[allow(clippy::cast_possible_truncation)]
+ fn reply_message(request: u64) -> Message<ReplyHeader> {
+ Message::<ReplyHeader>::new(HEADER_SIZE).transmute_header(|_, h: &mut ReplyHeader| {
+ h.command = Command2::Reply;
+ h.size = HEADER_SIZE as u32;
+ h.request = request;
+ })
+ }
+
+ /// Full in-process seam: entry + slot installed, `send_to_client`
+ /// resolves the request id from the reply header and fires the oneshot.
+ #[compio::test]
+ #[allow(clippy::future_not_send)]
+ async fn send_to_client_in_process_fires_matching_reply_slot() {
+ let bus = IggyMessageBus::new(0);
+ let client_id = 1u128;
+ bus.clients()
+ .insert_in_process(client_id)
+ .expect("fresh key");
+ let (_guard, reply_rx) = bus
+ .clients()
+ .install_reply_slot(client_id, 42)
+ .expect("slot installs");
+
+ bus.send_to_client(client_id, reply_message(42).into_generic().into_frozen())
+ .await
+ .expect("in-process delivery");
+
+ let received = reply_rx.await.expect("reply delivered");
+ assert_eq!(reply_request_id(&received), 42);
+ }
+
+ /// Reply whose request id has no waiter (timed-out caller already
+ /// removed its slot): shed as `ClientNotFound`, nothing panics.
+ #[compio::test]
+ #[allow(clippy::future_not_send)]
+ async fn send_to_client_in_process_sheds_reply_without_waiter() {
+ let bus = IggyMessageBus::new(0);
+ let client_id = 1u128;
+ bus.clients()
+ .insert_in_process(client_id)
+ .expect("fresh key");
+
+ let err = bus
+ .send_to_client(client_id, reply_message(42).into_generic().into_frozen())
+ .await
+ .unwrap_err();
+ assert!(matches!(err, SendError::ClientNotFound(_)));
+ }
+
#[compio::test]
#[allow(clippy::future_not_send)]
async fn send_to_client_returns_route_missing_when_remote_and_no_forward_fn() {
diff --git a/core/message_bus/src/lifecycle/connection_registry.rs b/core/message_bus/src/lifecycle/connection_registry.rs
index 6a01110b8..5858ab906 100644
--- a/core/message_bus/src/lifecycle/connection_registry.rs
+++ b/core/message_bus/src/lifecycle/connection_registry.rs
@@ -41,6 +41,7 @@ use std::collections::hash_map::Entry as HmEntry;
use std::fmt::Debug;
use std::hash::Hash;
use std::time::{Duration, Instant};
+use thiserror::Error;
use tracing::{debug, warn};
/// Opaque per-insert generation token.
@@ -139,7 +140,6 @@ pub struct DrainOutcome {
#[derive(Debug)]
enum ReplyTarget {
Socket(BusSender),
- #[allow(dead_code)] // wired in P2 (HTTP data plane)
InProcess {
by_request: HashMap<u64, oneshot::Sender<BusMessage>>,
},
@@ -170,6 +170,68 @@ pub enum ReplyRoute {
NoSlot(BusMessage),
}
+/// Rejection reasons for [`ConnectionRegistry::install_reply_slot`].
+#[derive(Debug, Error, PartialEq, Eq)]
+pub enum ReplySlotError {
+ /// No registry entry exists for the key: the in-process entry was
+ /// never installed or has already been torn down.
+ #[error("no registry entry for this key")]
+ EntryNotFound,
+ /// The entry is a socket target. Socket replies go through the
+ /// per-peer queue; a reply slot on such an entry could never fire.
+ #[error("registry entry is a socket target")]
+ NotInProcess,
+ /// A waiter is already registered under this request id. Replacing it
+ /// would drop the first waiter's sender and could deliver its reply
+ /// to the wrong caller, so the second install is rejected instead.
+ #[error("a reply slot for this request id is already installed")]
+ DuplicateRequest,
+}
+
+/// Cancel-safety handle for an installed in-process reply slot.
+///
+/// Dropping the guard removes the slot unless the reply already fired
+/// ([`ConnectionRegistry::fire_in_process`] consumes the slot on delivery,
+/// making the drop a no-op) or the whole entry was replaced (fenced by the
+/// entry's [`InstanceToken`], mirroring `remove_if_token_matches`). A
+/// caller that times out or is cancelled therefore never leaks its oneshot
+/// sender.
+///
+/// Contract: a request id must not be reused under the same key while a
+/// previous guard for that id is still live; the stale guard's drop would
+/// remove the new slot. Callers mint monotonically increasing request ids,
+/// which rules this out.
+#[must_use = "dropping the guard removes the reply slot"]
+#[derive(Debug)]
+pub struct ReplySlotGuard<'a, K>
+where
+ K: Eq + Hash + Copy + Debug + 'static,
+{
+ registry: &'a ConnectionRegistry<K>,
+ key: K,
+ request: u64,
+ token: InstanceToken,
+}
+
+impl<K> Drop for ReplySlotGuard<'_, K>
+where
+ K: Eq + Hash + Copy + Debug + 'static,
+{
+ fn drop(&mut self) {
+ let mut entries = self.registry.entries.borrow_mut();
+ let Some(entry) = entries.get_mut(&self.key) else {
+ return;
+ };
+ if entry.token != self.token {
+ return;
+ }
+ let ReplyTarget::InProcess { by_request } = &mut entry.reply else {
+ return;
+ };
+ by_request.remove(&self.request);
+ }
+}
+
#[derive(Debug)]
struct Entry {
reply: ReplyTarget,
@@ -313,6 +375,49 @@ where
}
}
+ /// Register a oneshot waiter for the reply to `request` on `key`'s
+ /// in-process entry.
+ ///
+ /// The returned receiver resolves when `send_to_client` routes a reply
+ /// carrying this request id (via [`Self::fire_in_process`]), or with
+ /// `Canceled` when the entry is torn down first. The guard removes the
+ /// slot on drop; see [`ReplySlotGuard`] for the exact semantics.
+ ///
+ /// # Errors
+ ///
+ /// [`ReplySlotError::EntryNotFound`] when `key` has no entry,
+ /// [`ReplySlotError::NotInProcess`] when the entry is a socket target,
+ /// [`ReplySlotError::DuplicateRequest`] when a waiter is already
+ /// registered under `request`.
+ pub fn install_reply_slot(
+ &self,
+ key: K,
+ request: u64,
+ ) -> Result<(ReplySlotGuard<'_, K>, oneshot::Receiver<BusMessage>), ReplySlotError> {
+ let mut entries = self.entries.borrow_mut();
+ let Some(entry) = entries.get_mut(&key) else {
+ return Err(ReplySlotError::EntryNotFound);
+ };
+ let token = entry.token;
+ let ReplyTarget::InProcess { by_request } = &mut entry.reply else {
+ return Err(ReplySlotError::NotInProcess);
+ };
+ let HmEntry::Vacant(slot) = by_request.entry(request) else {
+ return Err(ReplySlotError::DuplicateRequest);
+ };
+ let (reply_tx, reply_rx) = oneshot::channel();
+ slot.insert(reply_tx);
+ Ok((
+ ReplySlotGuard {
+ registry: self,
+ key,
+ request,
+ token,
+ },
+ reply_rx,
+ ))
+ }
+
/// Register a new connection. Stores the producer side of the per-peer
/// queue alongside the writer + reader task handles so graceful shutdown
/// can drain everything.
@@ -365,6 +470,38 @@ where
Ok(token)
}
+ /// Register an in-process reply target for `key`.
+ ///
+ /// In-process peers have no socket and no reader / writer tasks:
+ /// replies routed to `key` resolve request-keyed oneshots installed
+ /// via [`Self::install_reply_slot`]. Teardown goes through
+ /// [`Self::remove`] / [`Self::remove_if_token_matches`]; dropping the
+ /// entry drops every pending sender, so all outstanding receivers
+ /// resolve `Canceled`.
+ ///
+ /// Returns `None` when `key` is already registered (socket or
+ /// in-process); the caller must tear the existing entry down first.
+ pub fn insert_in_process(&self, key: K) -> Option<InstanceToken> {
+ let mut entries = self.entries.borrow_mut();
+ let HmEntry::Vacant(slot) = entries.entry(key) else {
+ return None;
+ };
+ let token = self.mint_token();
+ // Nothing ever waits on this shutdown (no reader to wake); it
+ // exists only because every entry owns one.
+ let (conn_shutdown, _) = super::Shutdown::new();
+ slot.insert(Entry {
+ reply: ReplyTarget::InProcess {
+ by_request: HashMap::new(),
+ },
+ writer_handle: None,
+ reader_handle: None,
+ token,
+ _conn_shutdown: conn_shutdown,
+ });
+ Some(token)
+ }
+
/// Unregister a connection without awaiting its tasks.
///
/// Returns `true` if an entry was removed. Dropping the entry drops the
@@ -1206,4 +1343,152 @@ mod tests {
let returned = outcome.expect_err("missing slot should return msg");
assert_eq!(returned.len(), original_len);
}
+
+ /// End-to-end registry seam: install entry + slot, route, fire, await.
+ #[compio::test]
+ async fn in_process_slot_receives_fired_reply() {
+ let reg: ConnectionRegistry<u8> = ConnectionRegistry::new();
+ reg.insert_in_process(1u8).expect("fresh key");
+
+ let (guard, reply_rx) = reg.install_reply_slot(1u8, 42).expect("slot installs");
+ let msg = match reg.try_send_or_return(1u8, make_bus_msg()) {
+ ReplyRoute::InProcess(msg) => msg,
+ other => panic!("expected InProcess route, got {other:?}"),
+ };
+ reg.fire_in_process(1u8, 42, msg).expect("waiter present");
+
+ let received = reply_rx.await.expect("reply delivered");
+ assert_eq!(received.len(), HEADER_SIZE);
+ drop(guard);
+ assert!(reg.contains(1u8), "guard drop must not remove the entry");
+ }
+
+ #[compio::test]
+ async fn insert_in_process_duplicate_key_returns_none() {
+ let reg: ConnectionRegistry<u8> = ConnectionRegistry::new();
+ assert!(reg.insert_in_process(1u8).is_some());
+ assert!(reg.insert_in_process(1u8).is_none());
+ assert_eq!(reg.len(), 1);
+ }
+
+ #[compio::test]
+ async fn install_reply_slot_rejects_missing_and_socket_entries() {
+ let reg: ConnectionRegistry<u8> = ConnectionRegistry::new();
+ assert_eq!(
+ reg.install_reply_slot(1u8, 1).unwrap_err(),
+ ReplySlotError::EntryNotFound
+ );
+
+ let (_shutdown, token) = Shutdown::new();
+ let (tx, rx) = async_channel::bounded(8);
+ reg.insert(
+ 1u8,
+ tx,
+ spawn_dummy_writer(rx),
+ spawn_dummy_reader(token),
+ dummy_conn_shutdown(),
+ )
+ .expect("socket insert ok");
+ assert_eq!(
+ reg.install_reply_slot(1u8, 1).unwrap_err(),
+ ReplySlotError::NotInProcess
+ );
+ }
+
+ /// Second install under a live request id must be rejected, and the
+ /// first waiter must stay wired (the reply still reaches it).
+ #[compio::test]
+ async fn install_reply_slot_rejects_duplicate_request() {
+ let reg: ConnectionRegistry<u8> = ConnectionRegistry::new();
+ reg.insert_in_process(1u8).expect("fresh key");
+
+ let (_guard, reply_rx) = reg.install_reply_slot(1u8, 42).expect("first install");
+ assert_eq!(
+ reg.install_reply_slot(1u8, 42).unwrap_err(),
+ ReplySlotError::DuplicateRequest
+ );
+
+ reg.fire_in_process(1u8, 42, make_bus_msg())
+ .expect("first waiter still registered");
+ assert_eq!(reply_rx.await.expect("reply delivered").len(), HEADER_SIZE);
+ }
+
+ /// Reply for a request id nobody waits on: handed back, no panic, and
+ /// the unrelated slot stays registered.
+ #[compio::test]
+ async fn fire_in_process_unknown_request_returns_msg() {
+ let reg: ConnectionRegistry<u8> = ConnectionRegistry::new();
+ reg.insert_in_process(1u8).expect("fresh key");
+ let (_guard, reply_rx) = reg.install_reply_slot(1u8, 42).expect("slot installs");
+
+ let msg = make_bus_msg();
+ let original_len = msg.len();
+ let returned = reg
+ .fire_in_process(1u8, 7, msg)
+ .expect_err("no waiter for request 7");
+ assert_eq!(returned.len(), original_len);
+
+ reg.fire_in_process(1u8, 42, make_bus_msg())
+ .expect("slot 42 untouched");
+ assert_eq!(reply_rx.await.expect("reply delivered").len(), HEADER_SIZE);
+ }
+
+ /// Timeout / cancellation path: dropping the guard removes the slot,
+ /// cancels the receiver, and a later reply is shed as no-waiter.
+ #[compio::test]
+ async fn dropped_guard_removes_slot_and_cancels_receiver() {
+ let reg: ConnectionRegistry<u8> = ConnectionRegistry::new();
+ reg.insert_in_process(1u8).expect("fresh key");
+
+ let (guard, reply_rx) = reg.install_reply_slot(1u8, 42).expect("slot installs");
+ drop(guard);
+
+ assert!(reply_rx.await.is_err(), "sender dropped with the slot");
+ assert!(
+ reg.fire_in_process(1u8, 42, make_bus_msg()).is_err(),
+ "late reply must be handed back, not delivered"
+ );
+ let (_guard, _reply_rx) = reg
+ .install_reply_slot(1u8, 42)
+ .expect("request id reusable after guard drop");
+ }
+
+ /// After the reply fired the slot is gone; the guard's drop is a no-op
+ /// and the request id becomes reusable.
+ #[compio::test]
+ async fn guard_drop_is_noop_after_reply_fired() {
+ let reg: ConnectionRegistry<u8> = ConnectionRegistry::new();
+ reg.insert_in_process(1u8).expect("fresh key");
+
+ let (guard, reply_rx) = reg.install_reply_slot(1u8, 42).expect("slot installs");
+ reg.fire_in_process(1u8, 42, make_bus_msg())
+ .expect("waiter present");
+ drop(guard);
+
+ assert_eq!(reply_rx.await.expect("reply delivered").len(), HEADER_SIZE);
+ let (_guard, _reply_rx) = reg
+ .install_reply_slot(1u8, 42)
+ .expect("request id reusable after fire + drop");
+ }
+
+ /// Entry teardown with a pending waiter: the receiver resolves
+ /// `Canceled` instead of hanging, and the stale guard's later drop
+ /// must not disturb a reinstalled entry's slot (token fence).
+ #[compio::test]
+ async fn entry_teardown_cancels_pending_receiver_and_fences_stale_guard() {
+ let reg: ConnectionRegistry<u8> = ConnectionRegistry::new();
+ reg.insert_in_process(1u8).expect("fresh key");
+ let (stale_guard, reply_rx) = reg.install_reply_slot(1u8, 42).expect("slot installs");
+
+ assert!(reg.remove(1u8));
+ assert!(reply_rx.await.is_err(), "teardown cancels pending waiter");
+
+ reg.insert_in_process(1u8)
+ .expect("reinstall after teardown");
+ let (_guard, reply_rx) = reg.install_reply_slot(1u8, 42).expect("slot installs");
+ drop(stale_guard);
+ reg.fire_in_process(1u8, 42, make_bus_msg())
+ .expect("stale guard must not evict the new slot");
+ assert_eq!(reply_rx.await.expect("reply delivered").len(), HEADER_SIZE);
+ }
}
diff --git a/core/message_bus/src/lifecycle/mod.rs b/core/message_bus/src/lifecycle/mod.rs
index e80bec1f7..5081a516a 100644
--- a/core/message_bus/src/lifecycle/mod.rs
+++ b/core/message_bus/src/lifecycle/mod.rs
@@ -26,6 +26,6 @@ pub mod shutdown;
pub use connection_registry::{
BusMessage, BusReceiver, BusSender, ConnectionRegistry, DrainOutcome, InstanceToken,
- RejectedRegistration, ReplicaRegistry, ReplyRoute,
+ RejectedRegistration, ReplicaRegistry, ReplyRoute, ReplySlotError, ReplySlotGuard,
};
pub use shutdown::{FusedShutdown, Shutdown, ShutdownToken};
diff --git a/core/partitions/src/iggy_partition.rs b/core/partitions/src/iggy_partition.rs
index c396eb948..dc7fbe113 100644
--- a/core/partitions/src/iggy_partition.rs
+++ b/core/partitions/src/iggy_partition.rs
@@ -1762,7 +1762,7 @@ where
let committed_visible_offsets = self.resolve_committed_visible_offsets(&drained).await;
let mut messages_committed = false;
- for mut entry in drained {
+ for entry in drained {
let prepare_header = entry.header;
if !self
.commit_partition_entry(
@@ -1814,14 +1814,6 @@ where
// commit. Session lifecycle is metadata-only.
let reply = build_reply_message(&prepare_header, &bytes::Bytes::new());
- // TODO: no production caller yet. Partition has no in-process
- // subscriber (only metadata uses pipeline_message_with_subscriber);
- // wired for forward-compat. Fired AFTER local commit (slot-first
- // ordering analog). Dropped receiver ignored.
- if let Some(sender) = entry.take_reply_sender() {
- let _ = sender.send(reply.clone());
- }
-
if send_client_replies {
let reply_buffers = reply.into_generic().into_frozen();
emit_sim_event(SimEventKind::ClientReplyEmitted, &event);
diff --git a/core/server-ng/src/dispatch.rs b/core/server-ng/src/dispatch.rs
index 3bf5fa33b..f942ebefb 100644
--- a/core/server-ng/src/dispatch.rs
+++ b/core/server-ng/src/dispatch.rs
@@ -548,90 +548,17 @@ async fn handle_client_request(
return;
}
- // Partition data-plane op (SendMessages / consumer-offset writes): the
- // op belongs to the partition's own consensus group, not the metadata
- // group. Route it through the shard mesh by namespace; the owning
- // shard's partitions plane runs at-least-once consensus and replies
- // directly via `send_to_client`. `header.client` therefore stays the
- // TRANSPORT id (home-shard routing bits), not the VSR session id --
- // partition ops are sessionless ("session lifecycle is metadata-only").
if header.operation.is_partition() {
- // The consumer-group offset fence keys on the bound VSR client id (the
- // member id), not the transport id stamped into the partition-op
- // header. `bound` is Some here (unbound transports returned above).
- let vsr_client_id = bound.map_or(0, |(client_id, _)| client_id);
- let namespace = match resolve_partition_request_namespace(
+ // `bound` is Some here (unbound transports returned above).
+ let (vsr_client_id, bound_session) = bound.unwrap_or((0, 0));
+ dispatch_partition_request(
shard,
- header.operation,
- request_body(&request),
+ request,
vsr_client_id,
- ) {
- Ok(namespace) => namespace,
- Err(error) => {
- // A partition op against a stream/topic that no longer
- // resolves (e.g. a consumer's trailing auto-commit racing a
- // `delete_stream`). A silent drop wedges the SDK connection
- // forever; reply empty so the client fails fast instead.
- warn!(
- transport_client_id,
- error = %error,
- operation = ?header.operation,
- "partition request with unresolved namespace; replying empty"
- );
- send_empty_partition_reply(shard, transport_client_id, &header).await;
- return;
- }
- };
- // Convergence wait: a CreateTopic commit returns to the client
- // before the per-shard reconcilers seed routing rows and
- // materialise the partition (next wake/periodic tick). The SDK
- // does not replay sends, so an immediately-following partition op
- // would be dropped as unroutable. Absorb that window here with a
- // bounded wait; steady-state sends (row present, partition probed
- // once) skip it entirely.
- if !wait_for_partition_routable(shard, IggyNamespace::from_raw(namespace)).await {
- warn!(
- transport_client_id,
- namespace,
- operation = ?header.operation,
- "partition request not routable within budget; replying empty"
- );
- send_empty_partition_reply(shard, transport_client_id, &header).await;
- return;
- }
- // A group consumer-offset op carries the group NAME on the wire; the
- // partition plane keys the offset by the group's monotonic id (the same
- // key the poll path auto-commits under and the read path resolves), so
- // rewrite the consumer id before replication -- the apply layer has no
- // metadata access to resolve it.
- let request = match maybe_rewrite_consumer_offset_request(shard, request) {
- Ok(rewritten) => rewritten,
- Err(error) => {
- warn!(
- transport_client_id,
- error = %error,
- operation = ?header.operation,
- "failed to rewrite consumer-offset request; replying empty"
- );
- send_empty_partition_reply(shard, transport_client_id, &header).await;
- return;
- }
- };
- let request = request.transmute_header(|header, new_header: &mut RequestHeader| {
- *new_header = header;
- new_header.namespace = namespace;
- new_header.client = transport_client_id;
- // Header validation requires `session > 0 && request > 0` for
- // non-register ops. The partition plane itself is sessionless
- // (at-least-once, no `ClientTable` dedup), so the bound VSR
- // session merely satisfies validation, and a zero request id
- // (the SDK does not number data-plane ops) is normalized.
- if let Some((_, bound_session)) = bound {
- new_header.session = bound_session;
- }
- new_header.request = new_header.request.max(1);
- });
- shard.dispatch(request.into_generic());
+ bound_session,
+ transport_client_id,
+ )
+ .await;
return;
}
@@ -732,6 +659,102 @@ async fn handle_client_request(
}
}
+/// Route a partition data-plane op (`SendMessages` / consumer-offset writes)
+/// through the shard mesh by namespace: the op belongs to the partition's
+/// own consensus group, not the metadata group. The owning shard's
+/// partitions plane runs at-least-once consensus and replies directly via
+/// `send_to_client`. `header.client` therefore stays the TRANSPORT id
+/// (home-shard routing bits), not the VSR session id -- partition ops are
+/// sessionless ("session lifecycle is metadata-only").
+///
+/// Callers must have authenticated the transport already: `vsr_client_id` /
+/// `bound_session` come from its bound VSR session. Every failure before
+/// dispatch replies empty so the client fails fast instead of wedging on a
+/// silent drop.
+///
+/// `vsr_client_id` keys the consumer-group offset fence (the member id),
+/// not the transport id stamped into the partition-op header.
+#[allow(clippy::future_not_send)]
+pub(crate) async fn dispatch_partition_request(
+ shard: &Rc<ServerNgShard>,
+ request: Message<RequestHeader>,
+ vsr_client_id: u128,
+ bound_session: u64,
+ transport_client_id: u128,
+) {
+ let header = *request.header();
+ let namespace = match resolve_partition_request_namespace(
+ shard,
+ header.operation,
+ request_body(&request),
+ vsr_client_id,
+ ) {
+ Ok(namespace) => namespace,
+ Err(error) => {
+ // A partition op against a stream/topic that no longer
+ // resolves (e.g. a consumer's trailing auto-commit racing a
+ // `delete_stream`). A silent drop wedges the SDK connection
+ // forever; reply empty so the client fails fast instead.
+ warn!(
+ transport_client_id,
+ error = %error,
+ operation = ?header.operation,
+ "partition request with unresolved namespace; replying empty"
+ );
+ send_empty_partition_reply(shard, transport_client_id, &header).await;
+ return;
+ }
+ };
+ // Convergence wait: a CreateTopic commit returns to the client
+ // before the per-shard reconcilers seed routing rows and
+ // materialise the partition (next wake/periodic tick). The SDK
+ // does not replay sends, so an immediately-following partition op
+ // would be dropped as unroutable. Absorb that window here with a
+ // bounded wait; steady-state sends (row present, partition probed
+ // once) skip it entirely.
+ if !wait_for_partition_routable(shard, IggyNamespace::from_raw(namespace)).await {
+ warn!(
+ transport_client_id,
+ namespace,
+ operation = ?header.operation,
+ "partition request not routable within budget; replying empty"
+ );
+ send_empty_partition_reply(shard, transport_client_id, &header).await;
+ return;
+ }
+ // A group consumer-offset op carries the group NAME on the wire; the
+ // partition plane keys the offset by the group's monotonic id (the same
+ // key the poll path auto-commits under and the read path resolves), so
+ // rewrite the consumer id before replication -- the apply layer has no
+ // metadata access to resolve it.
+ let request = match maybe_rewrite_consumer_offset_request(shard, request) {
+ Ok(rewritten) => rewritten,
+ Err(error) => {
+ warn!(
+ transport_client_id,
+ error = %error,
+ operation = ?header.operation,
+ "failed to rewrite consumer-offset request; replying empty"
+ );
+ send_empty_partition_reply(shard, transport_client_id, &header).await;
+ return;
+ }
+ };
+ let request = request.transmute_header(|header, new_header: &mut RequestHeader| {
+ *new_header = header;
+ new_header.namespace = namespace;
+ new_header.client = transport_client_id;
+ // Header validation requires `session > 0 && request > 0` for
+ // non-register ops. The partition plane itself is sessionless
+ // (at-least-once, no `ClientTable` dedup), so the bound VSR
+ // session merely satisfies validation, and a zero request id
+ // (the SDK does not number data-plane ops) is normalized.
+ new_header.session = bound_session;
+ new_header.request = new_header.request.max(1);
+ });
+ shard.dispatch(request.into_generic());
+}
+
#[allow(clippy::future_not_send)]
async fn handle_non_replicated_request(
shard: &Rc<ServerNgShard>,
(iggy) 08/19: feat(server-ng): add HTTP personal-access-token create/delete routes
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 0633cdb606bc66ec2e00067e77a349fbea95cc75
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 16:09:59 2026 +0200
feat(server-ng): add HTTP personal-access-token create/delete routes
The shard-0 HTTP listener had no way to mint or revoke personal
access tokens, leaving REST callers short of legacy parity.
A raw PAT is non-deterministic and secret, so it must never enter
consensus. rewrite_pat_request_for_user mints it on shard 0 and
replicates only its hash, leaving a successful committed reply body
empty; build_raw_pat_reply then splices the raw secret back in
locally after the commit. Reaching that committed reply Message
means submit_write's core (gate + lock + rewrites + submit) is
extracted into submit_committed, which returns the Message and
request header; submit_write now wraps it and decodes the body, so
every existing stream/topic/user caller is behavior-unchanged.
A committed create can still carry a business rejection (duplicate
name), so create_pat honors the reply's result code through the
shared committed_payload classifier before splicing the secret:
only a genuine success gets a token, a rejection renders the legacy
error. Both new routes inherit the root-only admin gate through the
shared core.
---
core/server-ng/src/http.rs | 193 ++++++++++++++++++++++++++++++++++++---------
core/server-ng/src/pat.rs | 21 ++++-
2 files changed, 172 insertions(+), 42 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index 719219c2f..ad3f3968f 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -37,6 +37,9 @@ use configs::http::HttpJwtConfig;
use iggy_binary_protocol::consensus::{
Command2, EvictionHeader, EvictionReason, HEADER_SIZE, result_code, result_section_len,
};
+use iggy_binary_protocol::requests::personal_access_tokens::{
+ CreatePersonalAccessTokenRequest, DeletePersonalAccessTokenRequest,
+};
use iggy_binary_protocol::requests::streams::{
CreateStreamRequest, DeleteStreamRequest, PurgeStreamRequest, UpdateStreamRequest,
};
@@ -47,6 +50,7 @@ use iggy_binary_protocol::requests::users::{
ChangePasswordRequest, CreateUserRequest, DeleteUserRequest, UpdatePermissionsRequest,
UpdateUserRequest,
};
+use iggy_binary_protocol::responses::personal_access_tokens::RawPersonalAccessTokenResponse;
use iggy_binary_protocol::responses::streams::get_stream::GetStreamResponse;
use iggy_binary_protocol::responses::topics::get_topic::GetTopicResponse;
use iggy_binary_protocol::responses::users::get_user::UserDetailsResponse;
@@ -55,6 +59,7 @@ use iggy_binary_protocol::{
GenericHeader, Operation, RequestHeader, WireDecode, WireEncode, WireName,
};
use iggy_common::change_password::ChangePassword;
+use iggy_common::create_personal_access_token::CreatePersonalAccessToken;
use iggy_common::create_stream::CreateStream;
use iggy_common::create_topic::CreateTopic;
use iggy_common::create_user::CreateUser;
@@ -67,8 +72,8 @@ use iggy_common::update_topic::UpdateTopic;
use iggy_common::update_user::UpdateUser;
use iggy_common::wire_conversions::{identifier_to_wire, permissions_to_wire};
use iggy_common::{
- Identifier, IdentityInfo, IggyError, IggyTimestamp, StreamDetails, TokenInfo, TopicDetails,
- UserInfoDetails, Validatable,
+ Identifier, IdentityInfo, IggyError, IggyTimestamp, RawPersonalAccessToken, StreamDetails,
+ TokenInfo, TopicDetails, UserInfoDetails, Validatable,
};
use message_bus::client_listener;
use secrecy::ExposeSecret;
@@ -84,6 +89,8 @@ use crate::dispatch::{submit_client_request_on_owner, submit_register_on_owner};
use crate::http::extractor::Authenticated;
use crate::http::jwt::JwtManager;
use crate::login_register::LoginRegisterError;
+use crate::pat::rewrite_pat_request_for_user;
+use crate::responses::build_raw_pat_reply;
use crate::server_error::ServerNgError;
use crate::users::maybe_rewrite_user_password_request;
@@ -382,6 +389,8 @@ fn router(state: HttpState) -> Router {
"/streams/{stream_id}/topics/{topic_id}/purge",
delete(purge_topic),
)
+ .route("/personal-access-tokens", post(create_pat))
+ .route("/personal-access-tokens/{name}", delete(delete_pat))
.with_state(state)
}
@@ -770,30 +779,101 @@ async fn update_permissions(
Ok(StatusCode::NO_CONTENT)
}
-/// Run one authenticated control-plane write end to end and return the committed
-/// reply's typed payload. Shared by every write route: `create_stream` decodes
-/// the payload into an entity, the update/delete/purge routes ignore it (it is
-/// empty) and answer 204.
+/// `POST /personal-access-tokens`: mint a personal access token for the caller
+/// and return its one-time raw secret as the same `{"token": ...}` JSON the
+/// legacy server returns, with HTTP 200.
///
-/// Serializes this session's writes behind its gate and holds it across the
-/// submit so request ids reach the primary strictly in order. The gate advances
-/// only on a committed `Reply`: a transient failure or an eviction leaves the id
-/// free for the caller's retry, which the depth-1 dedup requires (the next
-/// accepted request must be `committed + 1`, so a consumed-but-uncommitted id
-/// would wedge the session on `RequestGap`).
+/// The raw token is non-deterministic and secret, so it must never enter
+/// consensus: [`rewrite_pat_request_for_user`] (invoked inside
+/// [`submit_committed`]) mints it on shard 0 and replicates only its hash, so a
+/// successful committed reply body is empty. [`build_raw_pat_reply`] then splices
+/// the raw secret back into that reply locally, using the confirmed commit
+/// position. The token is surfaced only after the write commits; a malformed
+/// splice fails closed rather than emitting a blank token.
///
-/// Reply discrimination mirrors the SDK's `split_metadata_result`: the body
-/// leads with a result section (`Some(0)` is success followed by the typed
-/// payload; a nonzero first result is a committed business rejection carrying an
-/// `IggyError` code). An eviction frame carries no result section and means the
-/// session is dead (re-authenticate); a missing or short result section is a
-/// malformed committed reply, mapped to an error rather than a false success.
-async fn submit_write(
+/// A committed create can still carry a business rejection (duplicate name,
+/// invalid expiry), so the result code is honored via [`committed_payload`] -
+/// exactly as the mechanical routes do - BEFORE the secret is spliced. Only a
+/// genuine success gets a token; a rejection renders the legacy error instead of
+/// a bogus 200 + token.
+async fn create_pat(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Json(command): Json<CreatePersonalAccessToken>,
+) -> Result<Json<RawPersonalAccessToken>, WriteError> {
+ // Rejects an empty/oversized token name before any consensus work.
+ command.validate().map_err(WriteError::Rejected)?;
+ let request = CreatePersonalAccessTokenRequest {
+ name: WireName::new(&command.name)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidPersonalAccessTokenName))?,
+ expiry: command.expiry.into(),
+ };
+ let body = request.to_bytes();
+ let (request_header, committed, raw_token) = SendWrapper::new(submit_committed(
+ &state,
+ &identity.session,
+ Operation::CreatePersonalAccessToken,
+ &body,
+ ))
+ .await?;
+ // Reject a committed business error before splicing the secret; the success
+ // payload is empty, so the returned slice is discarded.
+ committed_payload(&committed)?;
+ let reply =
+ build_raw_pat_reply(&request_header, committed, raw_token).map_err(WriteError::Rejected)?;
+ Ok(Json(RawPersonalAccessToken {
+ token: decode_raw_pat_token(&reply)?,
+ }))
+}
+
+/// `DELETE /personal-access-tokens/{name}`: delete one of the caller's tokens by
+/// name. Returns 204 on commit. Inherits the root-only gate via [`submit_write`].
+async fn delete_pat(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path(name): Path<String>,
+) -> Result<StatusCode, WriteError> {
+ let request = DeletePersonalAccessTokenRequest {
+ name: WireName::new(&name)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidPersonalAccessTokenName))?,
+ };
+ let body = request.to_bytes();
+ SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::DeletePersonalAccessToken,
+ &body,
+ ))
+ .await?;
+ Ok(StatusCode::NO_CONTENT)
+}
+
+/// Run one authenticated control-plane write to commit and hand back the
+/// committed reply `Message`, the request header, and any raw PAT token minted
+/// along the way. Shared core of every HTTP write: [`submit_write`] decodes the
+/// reply body for the stream/topic/user routes, while [`create_pat`] needs the
+/// raw `Message` + request header to substitute the one-time token.
+///
+/// Enforces the root-only admin gate, then serializes this session's writes
+/// behind its gate and holds it across the submit so request ids reach the
+/// primary strictly in order. The gate advances only on a committed `Reply`: a
+/// transient failure or an eviction leaves the id free for the caller's retry,
+/// which the depth-1 dedup requires (the next accepted request must be
+/// `committed + 1`, so a consumed-but-uncommitted id would wedge the session on
+/// `RequestGap`). An eviction means the session is dead, mapped to 401.
+///
+/// Both shard-0 request rewrites run here before consensus, mirroring the TCP
+/// dispatch path: the PAT rewrite mints a raw token and replicates only its hash
+/// (`CreatePersonalAccessToken`), and the user-password rewrite hashes plaintext
+/// (`CreateUser` / `ChangePassword`). Both are no-ops for every other operation,
+/// so plaintext secrets never enter consensus on any write. On either rewrite's
+/// decode failure the gate is released without advancing, leaving the id free.
+async fn submit_committed(
state: &HttpInner,
session: &HttpSession,
operation: Operation,
body: &[u8],
-) -> Result<Bytes, WriteError> {
+) -> Result<(RequestHeader, Message<GenericHeader>, Option<String>), WriteError> {
// Interim authorization: until server-ng has an RBAC permissioner, every
// control-plane write is root-only. A non-root credential is authenticated
// but unprivileged, rejected before any consensus work is spent.
@@ -808,11 +888,10 @@ async fn submit_write(
*next_request_id,
body,
);
- // Hash raw passwords on shard 0 before replication (CreateUser /
- // ChangePassword); a no-op for every other operation. Mirrors the TCP
- // dispatch path so plaintext never enters consensus. On decode failure the
- // gate is released without advancing, leaving the request id free to retry.
+ let (message, raw_token) =
+ rewrite_pat_request_for_user(session.user_id, message).map_err(WriteError::Rejected)?;
let message = maybe_rewrite_user_password_request(message).map_err(WriteError::Rejected)?;
+ let request_header = *message.header();
let reply = submit_client_request_on_owner(&state.shard, message).await;
let Some(reply) = reply else {
return Err(WriteError::Unavailable);
@@ -822,26 +901,52 @@ async fn submit_write(
Command2::Reply => {
*next_request_id += 1;
drop(next_request_id);
- let size = reply.header().size as usize;
- let reply_body = reply.as_slice().get(HEADER_SIZE..size).unwrap_or_default();
- match result_code(reply_body) {
- Some(0) => {
- let payload_start = result_section_len(reply_body)
- .ok_or(WriteError::Rejected(IggyError::InvalidCommand))?;
- let payload = reply_body
- .get(payload_start..)
- .ok_or(WriteError::Rejected(IggyError::InvalidCommand))?;
- Ok(Bytes::copy_from_slice(payload))
- }
- Some(code) => Err(WriteError::Rejected(IggyError::from_code(code))),
- None => Err(WriteError::Rejected(IggyError::InvalidCommand)),
- }
+ Ok((request_header, reply, raw_token))
}
Command2::Eviction => Err(WriteError::Rejected(eviction_error(&reply))),
_ => Err(WriteError::Rejected(IggyError::InvalidCommand)),
}
}
+/// Run one authenticated control-plane write end to end and return the committed
+/// reply's typed payload. Wraps [`submit_committed`] and decodes the reply body
+/// via [`committed_payload`]: `create_stream` decodes the payload into an entity,
+/// the update/delete/purge routes ignore it (it is empty) and answer 204.
+async fn submit_write(
+ state: &HttpInner,
+ session: &HttpSession,
+ operation: Operation,
+ body: &[u8],
+) -> Result<Bytes, WriteError> {
+ let (_request_header, reply, _raw_token) =
+ submit_committed(state, session, operation, body).await?;
+ Ok(Bytes::copy_from_slice(committed_payload(&reply)?))
+}
+
+/// Classify a committed reply's leading result section and return the typed
+/// payload slice on success. Mirrors the SDK's `split_metadata_result`:
+/// `Some(0)` is success and the payload follows the result section; a nonzero
+/// first result is a committed business rejection carrying an `IggyError` code
+/// (e.g. a duplicate token name); a missing or short section is a malformed
+/// committed reply, mapped to an error rather than a false success. Shared by
+/// [`submit_write`] and [`create_pat`] so a committed rejection can never render
+/// as a 2xx.
+fn committed_payload(reply: &Message<GenericHeader>) -> Result<&[u8], WriteError> {
+ let size = reply.header().size as usize;
+ let reply_body = reply.as_slice().get(HEADER_SIZE..size).unwrap_or_default();
+ match result_code(reply_body) {
+ Some(0) => {
+ let payload_start = result_section_len(reply_body)
+ .ok_or(WriteError::Rejected(IggyError::InvalidCommand))?;
+ reply_body
+ .get(payload_start..)
+ .ok_or(WriteError::Rejected(IggyError::InvalidCommand))
+ }
+ Some(code) => Err(WriteError::Rejected(IggyError::from_code(code))),
+ None => Err(WriteError::Rejected(IggyError::InvalidCommand)),
+ }
+}
+
/// Build a `Message<RequestHeader>` for a control-plane write by filling a zeroed
/// `#[repr(C)]` header, mirroring `wire::rewrite_request_body` and the partition
/// reconciler's prepare builder. `body` is the already-encoded wire request,
@@ -896,6 +1001,18 @@ fn decode_user_details(payload: &[u8]) -> Result<UserInfoDetails, WriteError> {
UserInfoDetails::try_from(response).map_err(WriteError::Rejected)
}
+/// Extract the raw one-time token from a [`build_raw_pat_reply`] output. That
+/// reply's body is a bare `RawPersonalAccessTokenResponse` (no leading result
+/// section, unlike a committed metadata reply), so decode it straight past the
+/// header.
+fn decode_raw_pat_token(reply: &Message<GenericHeader>) -> Result<String, WriteError> {
+ let size = reply.header().size as usize;
+ let body = reply.as_slice().get(HEADER_SIZE..size).unwrap_or_default();
+ let response = RawPersonalAccessTokenResponse::decode_from(body)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidCommand))?;
+ Ok(response.token.as_str().to_string())
+}
+
/// Map an eviction frame to the same typed [`IggyError`] the SDK's
/// `decode_eviction` produces, so an HTTP caller sees the identical status a TCP
/// caller would (session-terminal reasons render as 401 -> re-authenticate).
diff --git a/core/server-ng/src/pat.rs b/core/server-ng/src/pat.rs
index 0a8199fa9..69bd83566 100644
--- a/core/server-ng/src/pat.rs
+++ b/core/server-ng/src/pat.rs
@@ -42,18 +42,31 @@ pub(crate) fn maybe_rewrite_pat_request(
transport_client_id: u128,
request: Message<RequestHeader>,
) -> Result<(Message<RequestHeader>, Option<String>), IggyError> {
- let operation = request.header().operation;
- let user_id = match operation {
+ let user_id = match request.header().operation {
Operation::CreatePersonalAccessToken | Operation::DeletePersonalAccessToken => sessions
.borrow()
.get_user_id(transport_client_id)
.ok_or(IggyError::Unauthenticated)?,
_ => return Ok((request, None)),
};
+ rewrite_pat_request_for_user(user_id, request)
+}
+/// PAT rewrite for a caller that already holds the authenticated `user_id`.
+///
+/// The HTTP listener authenticates against its own session table rather than
+/// the transport `SessionManager`, so it resolves the id itself and calls this
+/// directly. Mints the raw token + hash for `CreatePersonalAccessToken` and
+/// hands the raw token back; every other operation passes through unchanged
+/// with `None` (the HTTP write core routes all of its ops here, so the non-PAT
+/// arm is a no-op, not `unreachable`).
+pub(crate) fn rewrite_pat_request_for_user(
+ user_id: u32,
+ request: Message<RequestHeader>,
+) -> Result<(Message<RequestHeader>, Option<String>), IggyError> {
let body = request_body(&request);
let mut raw_token = None;
- let rewritten = match operation {
+ let rewritten = match request.header().operation {
Operation::CreatePersonalAccessToken => {
let wire = WireCreatePersonalAccessTokenRequest::decode_from(body)
.map_err(|_| IggyError::InvalidCommand)?;
@@ -84,7 +97,7 @@ pub(crate) fn maybe_rewrite_pat_request(
}
.to_bytes()
}
- _ => unreachable!(),
+ _ => return Ok((request, None)),
};
Ok((rewrite_request_body(&request, &rewritten)?, raw_token))
(iggy) 05/19: feat(server-ng): add HTTP create-topic route
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 466077bb47cfb34a1755d853afe8f47812a67553
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 14:21:43 2026 +0200
feat(server-ng): add HTTP create-topic route
server-ng exposed topic update/delete/purge over HTTP but
not creation, leaving a management-parity hole.
A client CreateTopic is already converted to
CreateTopicWithAssignments at prepare time, where the
metadata owner allocates the consensus group ids. Reuse
that path: the handler submits a plain CreateTopic and
decodes the committed GetTopicResponse into TopicDetails,
so no metadata or consensus code changes. Allocation stays
on the primary's serialized prepare step, keeping ids
collision-free without a new state-machine handler.
---
core/server-ng/src/http.rs | 54 ++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 52 insertions(+), 2 deletions(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index f2f52868b..f42a129ce 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -41,21 +41,24 @@ use iggy_binary_protocol::requests::streams::{
CreateStreamRequest, DeleteStreamRequest, PurgeStreamRequest, UpdateStreamRequest,
};
use iggy_binary_protocol::requests::topics::{
- DeleteTopicRequest, PurgeTopicRequest, UpdateTopicRequest,
+ CreateTopicRequest, DeleteTopicRequest, PurgeTopicRequest, UpdateTopicRequest,
};
use iggy_binary_protocol::responses::streams::get_stream::GetStreamResponse;
+use iggy_binary_protocol::responses::topics::get_topic::GetTopicResponse;
use iggy_binary_protocol::version::IGGY_PROTOCOL_VERSION;
use iggy_binary_protocol::{
GenericHeader, Operation, RequestHeader, WireDecode, WireEncode, WireName,
};
use iggy_common::create_stream::CreateStream;
+use iggy_common::create_topic::CreateTopic;
use iggy_common::login_user::LoginUser;
use iggy_common::login_with_personal_access_token::LoginWithPersonalAccessToken;
use iggy_common::update_stream::UpdateStream;
use iggy_common::update_topic::UpdateTopic;
use iggy_common::wire_conversions::identifier_to_wire;
use iggy_common::{
- Identifier, IdentityInfo, IggyError, IggyTimestamp, StreamDetails, TokenInfo, Validatable,
+ Identifier, IdentityInfo, IggyError, IggyTimestamp, StreamDetails, TokenInfo, TopicDetails,
+ Validatable,
};
use message_bus::client_listener;
use secrecy::ExposeSecret;
@@ -356,6 +359,7 @@ fn router(state: HttpState) -> Router {
put(update_stream).delete(delete_stream),
)
.route("/streams/{stream_id}/purge", delete(purge_stream))
+ .route("/streams/{stream_id}/topics", post(create_topic))
.route(
"/streams/{stream_id}/topics/{topic_id}",
put(update_topic).delete(delete_topic),
@@ -500,6 +504,43 @@ async fn purge_stream(
Ok(StatusCode::NO_CONTENT)
}
+/// `POST /streams/{stream_id}/topics`: create a topic under a stream and render
+/// the committed reply as the same `TopicDetails` JSON the legacy server returns.
+///
+/// The stream comes from the path; the JSON body carries the remaining fields.
+/// The submitted op is a plain `CreateTopic`; the metadata owner allocates the
+/// consensus group ids and rewrites it to `CreateTopicWithAssignments` before
+/// replication, so this handler stays a pure submit-and-decode.
+async fn create_topic(
+ State(state): State<HttpState>,
+ identity: Authenticated,
+ Path(stream_id): Path<String>,
+ Json(command): Json<CreateTopic>,
+) -> Result<Json<TopicDetails>, WriteError> {
+ let stream_id = Identifier::from_str_value(&stream_id).map_err(WriteError::Rejected)?;
+ // Rejects empty/oversized name, partitions_count > MAX, replication_factor == Some(0).
+ command.validate().map_err(WriteError::Rejected)?;
+ let request = CreateTopicRequest {
+ stream_id: identifier_to_wire(&stream_id).map_err(WriteError::Rejected)?,
+ partitions_count: command.partitions_count,
+ compression_algorithm: command.compression_algorithm.as_code(),
+ message_expiry: command.message_expiry.into(),
+ max_topic_size: command.max_topic_size.into(),
+ replication_factor: command.replication_factor.unwrap_or(0),
+ name: WireName::new(command.name)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidTopicName))?,
+ };
+ let body = request.to_bytes();
+ let payload = SendWrapper::new(submit_write(
+ &state,
+ &identity.session,
+ Operation::CreateTopic,
+ &body,
+ ))
+ .await?;
+ Ok(Json(decode_topic_details(&payload)?))
+}
+
/// `PUT /streams/{stream_id}/topics/{topic_id}`: update a topic. Returns 204.
async fn update_topic(
State(state): State<HttpState>,
@@ -676,6 +717,15 @@ fn decode_stream_details(payload: &[u8]) -> Result<StreamDetails, WriteError> {
StreamDetails::try_from(response).map_err(WriteError::Rejected)
}
+/// Decode the `GetTopicResponse` payload of a committed create-topic reply into
+/// `TopicDetails`. `payload` is the slice past the result section that
+/// [`submit_write`] already validated as a success.
+fn decode_topic_details(payload: &[u8]) -> Result<TopicDetails, WriteError> {
+ let response = GetTopicResponse::decode_from(payload)
+ .map_err(|_| WriteError::Rejected(IggyError::InvalidCommand))?;
+ TopicDetails::try_from(response).map_err(WriteError::Rejected)
+}
+
/// Map an eviction frame to the same typed [`IggyError`] the SDK's
/// `decode_eviction` produces, so an HTTP caller sees the identical status a TCP
/// caller would (session-terminal reasons render as 401 -> re-authenticate).
(iggy) 06/19: feat(server-ng): gate HTTP write routes to root user
Posted by hg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch http-listener
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit f5f43521028098fc1b8d7ac58b18e46a1e46f8d8
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Wed Jul 1 15:08:27 2026 +0200
feat(server-ng): gate HTTP write routes to root user
server-ng has no RBAC permissioner yet, so any authenticated token could
create, update, delete, or purge streams and topics. That is a privilege
escalation: a valid but non-root credential must not mutate management
state.
Add an interim authorization gate inside submit_write, the single
chokepoint every HTTP control-plane write funnels through, rejecting any
non-root user with Unauthorized (403) before any consensus work is spent.
Root (user_id 0) is the sole admin marker until per-op RBAC (F8) replaces
this blanket gate.
---
core/server-ng/src/http.rs | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/core/server-ng/src/http.rs b/core/server-ng/src/http.rs
index f42a129ce..be7dcd9b5 100644
--- a/core/server-ng/src/http.rs
+++ b/core/server-ng/src/http.rs
@@ -51,6 +51,7 @@ use iggy_binary_protocol::{
};
use iggy_common::create_stream::CreateStream;
use iggy_common::create_topic::CreateTopic;
+use iggy_common::defaults::DEFAULT_ROOT_USER_ID;
use iggy_common::login_user::LoginUser;
use iggy_common::login_with_personal_access_token::LoginWithPersonalAccessToken;
use iggy_common::update_stream::UpdateStream;
@@ -103,7 +104,6 @@ struct HttpSession {
session: u64,
/// User the credential authenticated as. Consumed by the write path for
/// authorization.
- #[expect(dead_code, reason = "read once authorization is wired for writes")]
user_id: u32,
/// Credential expiry in unix seconds (`u64::MAX` = never). Drives lazy
/// eviction of stale table entries.
@@ -644,6 +644,12 @@ async fn submit_write(
operation: Operation,
body: &[u8],
) -> Result<Bytes, WriteError> {
+ // Interim authorization: until server-ng has an RBAC permissioner, every
+ // control-plane write is root-only. A non-root credential is authenticated
+ // but unprivileged, rejected before any consensus work is spent.
+ if session.user_id != DEFAULT_ROOT_USER_ID {
+ return Err(WriteError::Rejected(IggyError::Unauthorized));
+ }
let mut next_request_id = session.gate.lock().await;
let message = build_request_message(
operation,