You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by hg...@apache.org on 2026/10/09 14:53:34 UTC
(iggy) 01/01: feat(partitions)!: fence ownership and history changes
This is an automated email from the ASF dual-hosted git repository.
hubcio pushed a commit to branch feat/ownership-and-history
in repository https://gitbox.apache.org/repos/asf/iggy.git
commit 219b99f70c4f7e73856676fcb52605e08af997c5
Author: Hubert Gruszecki <h....@gmail.com>
AuthorDate: Mon Oct 5 18:30:40 2026 +0200
feat(partitions)!: fence ownership and history changes
Queued requests, detached polls and retries can outlive their
consumer-group assignment or partition incarnation. Changing metadata
alone does not fence those operations at partition apply or recovery.
Order owner installs, owner retirement and delete fences in partition
logs. Require applied partition evidence before completing handoffs
or destructive metadata lifecycles. Retain owner and fence evidence
and lifecycle intents through checkpoints, restart and state
transfer. A committed delete fence closes the partition incarnation
for good. Admission drains admitted work before the fence, and the
partition refuses later requests and repaired prepares.
Reclaim deleted owner rows using replicated metadata evidence and
check the partition-set revision before finalizing sessions. Preserve
stable assignments and add an immutable fixed/mutable partition-resize
policy. Re-ack matching committed prepares without reapplying them.
Carry captured context through Rust and foreign SDK operations and
retain request context across uncertain retries. Add position-based
offset commits and update local stored progress after successful
offset-store calls.
Repair can journal batches out of operation order. Lower batches
then look persisted after a higher batch reaches disk. Restore
journal order before materialization and wait at operation gaps, so
that materialization does not drop those batches.
Protect acknowledged prepares above the receiver's known commit
when installing transferred state. Gate acknowledgements on the
contiguous WAL admission frontier, and keep recovery for
unacknowledged bodies above WAL gaps. Accept complete local repair
prefixes and cancel obsolete transfer re-arms after catching up.
Fail boot on incomplete segment-directory listings before recovery
can remove unseen committed logs. Use the configured wedge timeout
for file jobs, where zero disables it. Stop the node when a
timed-out job belongs to a tombstoned partition.
Consolidate superblock writes and request queue conditions while
preserving CREATE backoff behavior and lazy offset checks. Remove
unreachable dispatcher branches and redundant sampler/pipeline state.
BREAKING CHANGE: affects storage, replica-to-replica communication,
and SDK-to-server communication.
Storage: require the IGGY-OWNERSHIP-HISTORY-4 marker, metadata
snapshots v11 and ICO1 partition checkpoint/state-transfer data v7.
Reject incompatible data directories before mutation. This change
provides no in-place migration.
Replica-to-replica: prepare headers carry the partition incarnation,
offset prepares carry owner generations, and replicated control
operations and payloads change. Mixed-version replicas are
unsupported.
SDK-to-server: binary request/reply context fields and poll, routing
and topic response layouts change. The partition context is 24 bytes:
incarnation, owner generation and metadata op. Context-bearing SDK
APIs also change. Upgrade servers and compatible SDKs together.
---
Cargo.lock | 1 +
bdd/go/tests/tcp_test/consumers_steps.go | 17 +-
bdd/go/tests/tcp_test/messages_feature_send.go | 4 +-
core/bench/src/actors/consumer/client/low_level.rs | 5 +-
core/binary_protocol/Cargo.toml | 1 +
core/binary_protocol/src/codes.rs | 2 +
core/binary_protocol/src/consensus/header.rs | 111 +-
core/binary_protocol/src/consensus/operation.rs | 32 +-
core/binary_protocol/src/consensus/request.rs | 4 +-
core/binary_protocol/src/dispatch.rs | 8 +-
core/binary_protocol/src/lib.rs | 1 +
core/binary_protocol/src/primitives/mod.rs | 1 +
.../src/primitives/partition_history.rs | 123 +
.../src/requests/consumer_offsets/mod.rs | 19 +
.../src/requests/messages/get_send_context.rs | 90 +
core/binary_protocol/src/requests/messages/mod.rs | 2 +
.../partitions/install_consumer_group_owner.rs | 138 +
.../binary_protocol/src/requests/partitions/mod.rs | 6 +
.../partitions/retire_consumer_group_owners.rs | 151 +
.../partitions/transition_partition_history.rs | 107 +
.../src/requests/system/finalize_session.rs | 96 +
core/binary_protocol/src/requests/system/mod.rs | 2 +
.../src/responses/messages/poll_messages.rs | 19 +-
.../src/responses/messages/poll_routing.rs | 14 +-
.../src/responses/topics/get_topic.rs | 16 +-
core/common/Cargo.toml | 2 +-
core/common/src/consumer_group_client_state.rs | 217 +-
core/common/src/error/iggy_error.rs | 14 +
.../http/consumer_offsets/store_consumer_offset.rs | 4 +-
.../src/traits/binary_impls/consumer_offsets.rs | 30 +-
core/common/src/traits/binary_impls/messages.rs | 81 +-
core/common/src/traits/binary_impls/partitions.rs | 2 +
core/common/src/traits/binary_impls/streams.rs | 1 +
core/common/src/traits/binary_impls/topics.rs | 6 +-
core/common/src/traits/binary_transport.rs | 35 +-
core/common/src/traits/consumer_offset_client.rs | 15 +-
core/common/src/traits/message_client.rs | 4 +
core/common/src/traits/partition_client.rs | 7 +
.../src/types/consumer/consumer_offset_info.rs | 12 +
core/common/src/types/message/polled_messages.rs | 45 +-
core/common/src/types/message/polling_strategy.rs | 21 +-
core/common/src/types/options/mod.rs | 129 +-
.../options/partition_resize_policy.rs} | 31 +-
core/common/src/types/partition/mod.rs | 6 +
core/common/src/wire_conversions.rs | 1 +
core/configs/src/common/validators.rs | 42 +-
core/configs/src/server_config/cluster.rs | 3 +-
core/consensus/src/impls.rs | 71 +-
core/consensus/src/lib.rs | 5 -
core/consensus/src/observability.rs | 4 +
core/integration/src/harness/context.rs | 16 +-
core/integration/src/harness/helpers.rs | 59 +-
core/integration/src/harness/mod.rs | 1 +
.../tests/cluster/client_table_adversarial.rs | 13 +-
.../tests/cluster/client_table_restart.rs | 40 +-
.../tests/cluster/consumer_offset_quota.rs | 11 +-
.../tests/cluster/fast_primary_rejoin.rs | 1 +
core/integration/tests/cluster/partition_dedup.rs | 209 +-
.../tests/cluster/partition_primary_routing.rs | 93 +-
.../tests/connectors/fixtures/influxdb/sink_v3.rs | 11 +-
.../tests/data_integrity/storage_compat.rs | 3 +
.../verify_consumer_group_partition_assignment.rs | 172 +-
.../tests/sdk/consumer_group_membership.rs | 145 +-
core/integration/tests/sdk/options.rs | 103 +
core/integration/tests/server/cg.rs | 12 +-
.../tests/server/consumer_offset_quota_vsr.rs | 1 +
core/integration/tests/server/http_vsr.rs | 3 +
core/integration/tests/server/raw_tcp.rs | 18 +
.../server/scenarios/authentication_scenario.rs | 14 +-
.../scenarios/invalid_consumer_offset_scenario.rs | 15 +-
.../server/scenarios/log_rotation_scenario.rs | 5 +
core/integration/tests/server/scenarios/mod.rs | 11 +
.../tests/server/scenarios/permissions_scenario.rs | 5 +
.../stale_client_consumer_group_scenario.rs | 34 +-
.../tests/server/scenarios/system_scenario.rs | 10 +-
core/metadata/src/impls/metadata.rs | 387 ++-
core/metadata/src/impls/recovery.rs | 104 +-
core/metadata/src/stm/authz.rs | 17 +-
core/metadata/src/stm/consumer_group.rs | 1160 ++++---
core/metadata/src/stm/lifecycle.rs | 1175 +++++++
core/metadata/src/stm/mod.rs | 40 +-
core/metadata/src/stm/result.rs | 33 +-
core/metadata/src/stm/snapshot.rs | 44 +-
core/metadata/src/stm/stream.rs | 530 +++-
core/metadata/src/stm/user.rs | 4 +-
core/partitions/src/iggy_partition.rs | 3345 +++++++++++++++++---
core/partitions/src/iggy_partitions.rs | 17 +-
core/partitions/src/journal.rs | 178 +-
core/partitions/src/lib.rs | 5 +
core/partitions/src/poll_plan.rs | 10 +
core/partitions/src/state_transfer.rs | 439 ++-
core/partitions/src/types.rs | 8 +-
core/partitions/tests/consumer_offset_history.rs | 478 +++
.../binary_consumer_offset_client.rs | 38 +-
core/sdk/src/clients/binary_consumer_offset.rs | 16 +-
core/sdk/src/clients/consumer.rs | 630 +++-
core/sdk/src/clients/producer.rs | 29 +-
core/sdk/src/consumer_ext/consumer_message_ext.rs | 7 +-
core/sdk/src/http/consumer_offsets.rs | 41 +-
core/sdk/src/http/http_client.rs | 27 +-
core/sdk/src/poll_routing.rs | 411 ++-
core/sdk/src/prelude.rs | 32 +-
core/sdk/src/quic/quic_client.rs | 80 +-
core/sdk/src/tcp/tcp_client.rs | 236 +-
core/sdk/src/vsr.rs | 190 +-
core/sdk/src/websocket/websocket_client.rs | 64 +-
core/server/README.md | 10 +-
core/server/config.toml | 9 +-
core/server/src/boot/mod.rs | 33 +-
core/server/src/boot/recovery.rs | 8 +-
core/server/src/consumer_group.rs | 760 +----
core/server/src/consumer_group/liveness.rs | 39 +-
core/server/src/dispatch/authz.rs | 7 +-
core/server/src/dispatch/failure.rs | 1 +
core/server/src/dispatch/mod.rs | 8 +-
core/server/src/dispatch/partition.rs | 115 +-
core/server/src/dispatch/reads.rs | 97 +-
core/server/src/dispatch/submit.rs | 15 +-
core/server/src/http/handlers.rs | 42 +-
core/server/src/http/submit.rs | 39 +-
core/server/src/http/wire.rs | 8 +
core/server/src/lib.rs | 1 +
core/server/src/partition_helpers.rs | 5 +-
core/server/src/partition_reconciler.rs | 568 +++-
core/server/src/reply_frame.rs | 55 +-
core/server/src/responses.rs | 30 +-
core/server/src/server_error.rs | 2 +-
core/server/src/sysinfo_printer.rs | 12 +-
core/server/src/sysinfo_probe.rs | 25 +-
core/server_common/src/consensus_message.rs | 11 +-
core/shard/src/lib.rs | 183 +-
core/shard/src/partition_io.rs | 191 +-
core/shard/src/poll.rs | 24 +-
core/shard/src/poll/completion.rs | 6 +-
core/shard/src/poll/completion_tests.rs | 140 +-
core/shard/src/poll/test_support.rs | 46 +
core/shard/src/poll/timeout_tests.rs | 49 +-
core/shard/src/router.rs | 17 +-
core/simulator/src/client.rs | 33 +-
core/simulator/src/lib.rs | 479 ++-
core/simulator/src/replica.rs | 33 +-
core/simulator/src/storage/bookmark_recovery.rs | 63 +-
core/simulator/src/workload/mod.rs | 69 +-
.../src/workload/ops/create_partitions.rs | 1 +
.../src/workload/ops/delete_partitions.rs | 1 +
core/simulator/src/workload/ops/update_topic.rs | 2 +-
core/simulator/src/workload/oracle.rs | 38 +-
.../apache/iggy/examples/ConsumerOffsetTest.java | 11 +-
foreign/cpp/include/iggy.hpp | 3 +
foreign/cpp/src/type_conversion.rs | 1 +
.../Iggy_SDK/Consumers/IggyConsumer.Rented.cs | 1 +
foreign/csharp/Iggy_SDK/Consumers/IggyConsumer.cs | 15 +-
.../Iggy_SDK/Contracts/Auth/GlobalPermissions.cs | 7 +-
.../Iggy_SDK/Contracts/Http/StoreOffsetRequest.cs | 1 +
.../StoreOffsetRequest.cs => PartitionContext.cs} | 25 +-
.../csharp/Iggy_SDK/Contracts/PartitionResponse.cs | 3 +
.../csharp/Iggy_SDK/Contracts/PolledMessages.cs | 3 +
.../Iggy_SDK/Contracts/PolledMessagesRental.cs | 3 +
.../csharp/Iggy_SDK/Contracts/StreamPermissions.cs | 7 +-
.../csharp/Iggy_SDK/Contracts/TopicPermissions.cs | 7 +-
foreign/csharp/Iggy_SDK/IggyClient/IIggyOffset.cs | 16 +
.../Implementations/HttpMessageStream.cs | 19 +-
.../TcpMessageStream.PollRouting.cs | 141 +-
.../Implementations/TcpMessageStream.Vsr.cs | 9 +-
.../IggyClient/Implementations/TcpMessageStream.cs | 59 +-
foreign/csharp/Iggy_SDK/Mappers/BinaryMapper.cs | 35 +-
foreign/csharp/Iggy_SDK/Utils/CommandCodes.cs | 1 +
foreign/csharp/Iggy_SDK/Vsr/VsrConnection.cs | 7 +-
foreign/csharp/Iggy_SDK/Vsr/VsrError.cs | 2 +
foreign/csharp/Iggy_SDK/Vsr/VsrHeader.cs | 15 +-
foreign/csharp/Iggy_SDK/Vsr/VsrOperation.cs | 15 +-
.../ClientTests/HttpConsumerOffsetTests.cs} | 27 +-
.../ClientTests/HttpTopicOptionsTests.cs | 2 +-
.../MessageBatchGoldenVectorTests.cs | 11 +-
.../Iggy_SDK_Tests/MapperTests/BinaryMapper.cs | 45 +-
.../csharp/Iggy_SDK_Tests/Utils/BinaryFactory.cs | 17 +
.../Iggy_SDK_Tests/VsrTests/GroupPollingTests.cs | 2 +-
foreign/csharp/Iggy_SDK_Tests/VsrTests/MockNode.cs | 23 +
.../VsrTests/PartitionContextTests.cs | 340 ++
.../Iggy_SDK_Tests/VsrTests/PrimaryPollingTests.cs | 10 +-
.../Iggy_SDK_Tests/VsrTests/VsrHeaderTests.cs | 14 +
.../VsrTests/VsrProtocolDriftTests.cs | 3 +
foreign/go/README.md | 8 +-
.../binary_response_deserializer.go | 31 +-
.../binary_response_deserializer_test.go | 1 +
.../vsr_response_deserializer_test.go | 5 +-
foreign/go/client/tcp/tcp_core.go | 32 +-
foreign/go/client/tcp/tcp_core_review_test.go | 2 +-
foreign/go/client/tcp/tcp_core_test.go | 83 +-
foreign/go/client/tcp/tcp_group_polling_test.go | 22 +-
foreign/go/client/tcp/tcp_messaging.go | 75 +-
foreign/go/client/tcp/tcp_offset_management.go | 15 +
foreign/go/client/tcp/tcp_partition_management.go | 2 +-
foreign/go/client/tcp/tcp_poll_routing.go | 25 +-
foreign/go/client/tcp/tcp_poll_routing_test.go | 91 +-
foreign/go/client/tcp/tcp_testing_test.go | 24 +
foreign/go/client/tcp/tcp_topic_cache.go | 37 +-
foreign/go/client/tcp/tcp_topic_cache_test.go | 7 +-
foreign/go/contracts/client.go | 23 +-
foreign/go/contracts/messages.go | 1 +
foreign/go/contracts/offset.go | 8 +
foreign/go/contracts/partitions.go | 31 +-
foreign/go/errors/errors.yaml | 12 +
foreign/go/errors/errors_gen.go | 51 +
foreign/go/internal/command/code.go | 1 +
foreign/go/internal/vsr/header.go | 13 +-
foreign/go/internal/vsr/operation.go | 18 +-
foreign/go/internal/vsr/protocol_parity_test.go | 8 +
.../iggy/client/async/ConsumerOffsetsClient.java | 22 +
.../iggy/client/async/tcp/AsyncTcpConnection.java | 81 +-
.../client/async/tcp/ConsumerOffsetsTcpClient.java | 60 +-
.../iggy/client/async/tcp/MessagesTcpClient.java | 47 +-
.../iggy/client/async/tcp/PartitionContexts.java | 154 +
.../apache/iggy/client/async/tcp/PollRouter.java | 27 +-
.../iggy/client/async/tcp/vsr/VsrHeaders.java | 9 +-
.../iggy/client/async/tcp/vsr/VsrOperation.java | 8 +
.../client/async/tcp/vsr/VsrRequestEncoder.java | 12 +
.../client/blocking/ConsumerOffsetsClient.java | 14 +
.../blocking/http/ConsumerOffsetsHttpClient.java | 30 +-
.../blocking/tcp/ConsumerOffsetsTcpClient.java | 12 +
.../org/apache/iggy/exception/IggyErrorCode.java | 4 +
.../org/apache/iggy/message/PolledMessages.java | 5 +-
.../java/org/apache/iggy/partition/Partition.java | 3 +-
.../{Partition.java => PartitionContext.java} | 13 +-
.../org/apache/iggy/serde/BytesDeserializer.java | 21 +-
.../java/org/apache/iggy/serde/CommandCode.java | 6 +-
.../AsyncIggyTcpClientPartitionContextTest.java | 303 ++
.../AsyncIggyTcpClientTransientFailoverTest.java | 144 +-
.../async/tcp/vsr/VsrRequestEncoderTest.java | 25 +
.../client/blocking/http/ObjectMapperTest.java | 30 +
.../blocking/tcp/ConsumerGroupsTcpClientTest.java | 30 +-
.../apache/iggy/exception/IggyErrorCodeTest.java | 3 +
.../apache/iggy/serde/BytesDeserializerTest.java | 1 +
.../iggy/serde/MessagesBatchWireFormatTest.java | 8 +-
.../org/apache/iggy/topic/TopicDetailsTest.java | 4 +-
foreign/node/README.md | 7 +-
foreign/node/src/client/client.socket.test.ts | 133 +-
foreign/node/src/client/client.socket.ts | 125 +-
foreign/node/src/index.ts | 1 +
foreign/node/src/wire/command.code.ts | 1 +
foreign/node/src/wire/error.code.test.ts | 12 +
foreign/node/src/wire/error.code.ts | 4 +
foreign/node/src/wire/identifier.utils.ts | 4 +
.../node/src/wire/message/message-batch.test.ts | 16 +-
.../src/wire/message/poll-messages.command.test.ts | 5 +-
.../node/src/wire/message/poll-messages.command.ts | 8 +-
foreign/node/src/wire/message/poll.utils.ts | 8 +-
.../src/wire/message/send-messages.command.test.ts | 86 +-
.../node/src/wire/message/send-messages.command.ts | 77 +-
foreign/node/src/wire/topic/topic.utils.ts | 6 +-
foreign/node/src/wire/vsr/header.test.ts | 6 +-
foreign/node/src/wire/vsr/header.ts | 38 +-
foreign/node/src/wire/vsr/index.ts | 5 +-
foreign/node/src/wire/vsr/operation.test.ts | 4 +-
foreign/node/src/wire/vsr/operation.ts | 11 +-
foreign/python/apache_iggy.pyi | 12 +-
foreign/python/src/client.rs | 3 +
foreign/python/src/consumer.rs | 10 +-
foreign/python/tests/test_consumer_group.py | 2 +-
.../swift/Sources/Iggy/Errors/IggyErrorCode.swift | 6 +
.../swift/Sources/Iggy/Models/IggyMessage.swift | 4 +-
foreign/swift/Sources/Iggy/Models/Resources.swift | 34 +-
.../swift/Sources/Iggy/Models/TopicOptions.swift | 4 +-
foreign/swift/Tests/IggyTests/ErrorCodeTests.swift | 1 +
foreign/swift/Tests/IggyTests/Fixtures/golden.json | 10 +-
.../swift/Tests/IggyTests/ModelVectorTests.swift | 14 +
foreign/swift/Tools/golden-vectors/src/main.rs | 16 +
gateways/kafka/src/bridge/error.rs | 24 +-
gateways/kafka/src/bridge/iggy_bridge/fetch.rs | 1 +
gateways/kafka/src/protocol/handlers/fetch.rs | 14 +-
gateways/kafka/tests/fetch_real_bridge_tests.rs | 8 +
271 files changed, 15604 insertions(+), 3531 deletions(-)
diff --git a/Cargo.lock b/Cargo.lock
index e19738894..46329d4cb 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -7091,6 +7091,7 @@ dependencies = [
"bytes",
"enumset",
"secrecy",
+ "serde",
"thiserror 2.0.20",
"twox-hash",
]
diff --git a/bdd/go/tests/tcp_test/consumers_steps.go b/bdd/go/tests/tcp_test/consumers_steps.go
index f00d885be..91638e199 100644
--- a/bdd/go/tests/tcp_test/consumers_steps.go
+++ b/bdd/go/tests/tcp_test/consumers_steps.go
@@ -20,6 +20,7 @@ package tcp_test
import (
"context"
"fmt"
+ "time"
iggcon "github.com/apache/iggy/foreign/go/contracts"
ierror "github.com/apache/iggy/foreign/go/errors"
@@ -131,7 +132,21 @@ func itShouldSuccessfullyJoinConsumer(streamId uint32, topicId uint32, groupId u
streamIdentifier, _ := iggcon.NewIdentifier(streamId)
topicIdentifier, _ := iggcon.NewIdentifier(topicId)
groupIdentifier, _ := iggcon.NewIdentifier(groupId)
- consumer, err := client.GetConsumerGroup(context.Background(), streamIdentifier, topicIdentifier, groupIdentifier)
+ const assignmentTimeout = 5 * time.Second
+ const assignmentPollInterval = 10 * time.Millisecond
+ ctx, cancel := context.WithTimeout(context.Background(), assignmentTimeout)
+ defer cancel()
+ ticker := time.NewTicker(assignmentPollInterval)
+ defer ticker.Stop()
+ consumer, err := client.GetConsumerGroup(ctx, streamIdentifier, topicIdentifier, groupIdentifier)
+ for err == nil && consumer != nil && (len(consumer.Members) != 1 || consumer.Members[0].PartitionsCount != 2) {
+ select {
+ case <-ctx.Done():
+ err = ctx.Err()
+ case <-ticker.C:
+ consumer, err = client.GetConsumerGroup(ctx, streamIdentifier, topicIdentifier, groupIdentifier)
+ }
+ }
ginkgo.It("should join consumer with id "+string(rune(groupId)), func() {
gomega.Expect(consumer).NotTo(gomega.BeNil())
diff --git a/bdd/go/tests/tcp_test/messages_feature_send.go b/bdd/go/tests/tcp_test/messages_feature_send.go
index 60f3c393b..29dd0cd14 100644
--- a/bdd/go/tests/tcp_test/messages_feature_send.go
+++ b/bdd/go/tests/tcp_test/messages_feature_send.go
@@ -93,9 +93,7 @@ var _ = ginkgo.Describe("SEND MESSAGES:", func() {
iggcon.PartitionId(createRandomUInt32()),
messages,
)
- // The send is routed by its packed namespace, so the shard that
- // owns no such partition reports a missing resource.
- itShouldReturnSpecificError(err, ierror.ErrResourceNotFound)
+ itShouldReturnSpecificError(err, ierror.ErrPartitionNotFound)
})
ginkgo.Context("and tries to send messages to valid topic but with 0 messages in payload", func() {
diff --git a/core/bench/src/actors/consumer/client/low_level.rs b/core/bench/src/actors/consumer/client/low_level.rs
index 964231f62..d0e187a0c 100644
--- a/core/bench/src/actors/consumer/client/low_level.rs
+++ b/core/bench/src/actors/consumer/client/low_level.rs
@@ -88,7 +88,10 @@ impl ConsumerClient for LowLevelConsumerClient {
let polled = match polled {
Ok(p) => p,
Err(e) => {
- if matches!(e, IggyError::TopicIdNotFound(_, _)) {
+ if matches!(
+ e,
+ IggyError::TopicIdNotFound(_, _) | IggyError::TransientNotAccepted
+ ) {
return Ok(None);
}
return Err(e);
diff --git a/core/binary_protocol/Cargo.toml b/core/binary_protocol/Cargo.toml
index 946500b06..9c5e86db8 100644
--- a/core/binary_protocol/Cargo.toml
+++ b/core/binary_protocol/Cargo.toml
@@ -34,6 +34,7 @@ bytemuck = { workspace = true }
bytes = { workspace = true }
enumset = { workspace = true }
secrecy = { workspace = true }
+serde = { workspace = true }
thiserror = { workspace = true }
twox-hash = { workspace = true }
diff --git a/core/binary_protocol/src/codes.rs b/core/binary_protocol/src/codes.rs
index 22a713e49..a4d125062 100644
--- a/core/binary_protocol/src/codes.rs
+++ b/core/binary_protocol/src/codes.rs
@@ -57,6 +57,7 @@ pub const SEND_MESSAGES_CODE: u32 = 101;
// 102 was FLUSH_UNSAVED_BUFFER. Never reuse it: older clients still send it.
pub const GET_POLL_ROUTING_CODE: u32 = 103;
pub const POLL_MESSAGES_ON_PRIMARY_CODE: u32 = 104;
+pub const GET_SEND_CONTEXT_CODE: u32 = 105;
// -- Consumer Offsets --
pub const GET_CONSUMER_OFFSET_CODE: u32 = 120;
@@ -143,6 +144,7 @@ mod tests {
POLL_MESSAGES_CODE,
GET_POLL_ROUTING_CODE,
POLL_MESSAGES_ON_PRIMARY_CODE,
+ GET_SEND_CONTEXT_CODE,
SEND_MESSAGES_CODE,
GET_CONSUMER_OFFSET_CODE,
STORE_CONSUMER_OFFSET_CODE,
diff --git a/core/binary_protocol/src/consensus/header.rs b/core/binary_protocol/src/consensus/header.rs
index ac0bfa4c8..f69930962 100644
--- a/core/binary_protocol/src/consensus/header.rs
+++ b/core/binary_protocol/src/consensus/header.rs
@@ -318,7 +318,14 @@ pub struct RequestHeader {
/// On `Register` it carries the freshly authenticated user.
/// The submitter's wire value is never trusted.
pub user_id: u32,
- pub reserved: [u8; 60],
+ /// Little-endian command code in `reserved[0..4]` for `NonReplicated`.
+ pub reserved: [u8; 4],
+ /// Captured with the request and retained across retries and route refreshes.
+ pub partition_incarnation: u64,
+ pub owner_generation: u64,
+ /// Minimum committed metadata operation needed to interpret this request.
+ pub minimum_metadata_op: u64,
+ pub reserved_tail: [u8; 32],
}
const _: () = {
assert!(size_of::<RequestHeader>() == HEADER_SIZE);
@@ -329,7 +336,10 @@ const _: () = {
assert!(
offset_of!(RequestHeader, user_id) == offset_of!(RequestHeader, session) + size_of::<u64>()
);
- assert!(offset_of!(RequestHeader, reserved) + size_of::<[u8; 60]>() == HEADER_SIZE);
+ assert!(offset_of!(RequestHeader, partition_incarnation) == 200);
+ assert!(offset_of!(RequestHeader, owner_generation) == 208);
+ assert!(offset_of!(RequestHeader, minimum_metadata_op) == 216);
+ assert!(offset_of!(RequestHeader, reserved_tail) + size_of::<[u8; 32]>() == HEADER_SIZE);
};
/// A [`RequestHeader`] AFTER the receiving node resolved its target.
@@ -342,7 +352,7 @@ const _: () = {
/// so this is where the derivation result lives for the internal hop.
///
/// Layout: identical to [`RequestHeader`] with server routing fields claiming
-/// bytes 240..256; the leading 44 reserved bytes keep their client-wire meaning.
+/// bytes 240..256; captured incarnation and ownership retain their client-wire offsets.
#[repr(C)]
#[derive(Debug, Clone, Copy, CheckedBitPattern, NoUninit)]
pub struct RoutedRequestHeader {
@@ -364,10 +374,14 @@ pub struct RoutedRequestHeader {
pub operation_padding: [u8; 7],
pub session: u64,
pub user_id: u32,
- /// Same offset and meaning as the leading 44 bytes of
- /// `RequestHeader::reserved` -- this region CARRIES DATA (the
- /// non-replicated op code range), so `group` must not displace it.
- pub reserved: [u8; 44],
+ /// Little-endian command code in `reserved[0..4]` for `NonReplicated`.
+ pub reserved: [u8; 4],
+ /// Captured with the request and retained across retries and route refreshes.
+ pub partition_incarnation: u64,
+ pub owner_generation: u64,
+ /// Minimum committed metadata operation needed to interpret this request.
+ pub minimum_metadata_op: u64,
+ pub reserved_tail: [u8; 16],
/// Server-observed metadata frontier at partition admission.
pub metadata_watermark: u64,
/// The resolved consensus group id (see `binary_protocol::namespace`),
@@ -383,6 +397,18 @@ const _: () = {
assert!(offset_of!(RoutedRequestHeader, session) == offset_of!(RequestHeader, session));
assert!(offset_of!(RoutedRequestHeader, user_id) == offset_of!(RequestHeader, user_id));
assert!(offset_of!(RoutedRequestHeader, reserved) == offset_of!(RequestHeader, reserved));
+ assert!(
+ offset_of!(RoutedRequestHeader, partition_incarnation)
+ == offset_of!(RequestHeader, partition_incarnation)
+ );
+ assert!(
+ offset_of!(RoutedRequestHeader, owner_generation)
+ == offset_of!(RequestHeader, owner_generation)
+ );
+ assert!(
+ offset_of!(RoutedRequestHeader, minimum_metadata_op)
+ == offset_of!(RequestHeader, minimum_metadata_op)
+ );
assert!(offset_of!(RoutedRequestHeader, group) + size_of::<u64>() == HEADER_SIZE);
};
@@ -406,7 +432,11 @@ impl Default for RoutedRequestHeader {
operation_padding: [0; 7],
session: 0,
user_id: 0,
- reserved: [0; 44],
+ reserved: [0; 4],
+ partition_incarnation: 0,
+ owner_generation: 0,
+ minimum_metadata_op: 0,
+ reserved_tail: [0; 16],
metadata_watermark: 0,
group: 0,
}
@@ -433,7 +463,11 @@ impl Default for RequestHeader {
operation_padding: [0; 7],
session: 0,
user_id: 0,
- reserved: [0; 60],
+ reserved: [0; 4],
+ partition_incarnation: 0,
+ owner_generation: 0,
+ minimum_metadata_op: 0,
+ reserved_tail: [0; 32],
}
}
}
@@ -605,7 +639,10 @@ pub struct ReplyHeader {
/// Carved from `reserved` like `user_id` in `RequestHeader` / `PrepareHeader`;
/// no existing field offset moves and `validate` does not inspect it.
pub status: u32,
- pub reserved: [u8; 36],
+ pub reserved: [u8; 4],
+ pub partition_incarnation: u64,
+ pub owner_generation: u64,
+ pub reserved_tail: [u8; 16],
}
const _: () = {
assert!(size_of::<ReplyHeader>() == HEADER_SIZE);
@@ -613,7 +650,13 @@ const _: () = {
offset_of!(ReplyHeader, request_checksum)
== offset_of!(ReplyHeader, reserved_frame) + size_of::<[u8; 66]>()
);
- assert!(offset_of!(ReplyHeader, reserved) + size_of::<[u8; 36]>() == HEADER_SIZE);
+ assert!(
+ offset_of!(ReplyHeader, reserved) + size_of::<[u8; 4]>()
+ == offset_of!(ReplyHeader, partition_incarnation)
+ );
+ assert!(offset_of!(ReplyHeader, partition_incarnation) == 224);
+ assert!(offset_of!(ReplyHeader, owner_generation) == 232);
+ assert!(offset_of!(ReplyHeader, reserved_tail) + size_of::<[u8; 16]>() == HEADER_SIZE);
};
impl Default for ReplyHeader {
@@ -638,7 +681,10 @@ impl Default for ReplyHeader {
operation: Operation::Reserved,
operation_padding: [0; 7],
status: 0,
- reserved: [0; 36],
+ reserved: [0; 4],
+ partition_incarnation: 0,
+ owner_generation: 0,
+ reserved_tail: [0; 16],
}
}
}
@@ -668,14 +714,16 @@ impl ReplyHeader {
timestamp: prepare_header.timestamp,
request: prepare_header.request,
operation: prepare_header.operation,
+ partition_incarnation: prepare_header.partition_incarnation,
..Self::default()
}
}
/// The base of a reply that answers `request_header` without a prepare
/// (rejections, denials, non-replicated reads): `cluster`, `view`,
- /// `release`, `replica`, `request_checksum`, `timestamp`, `request` and
- /// `operation` echo the request, `command` is `Reply`, `size` is the
+ /// `release`, `replica`, `request_checksum`, `timestamp`, `request`,
+ /// `operation`, `partition_incarnation` and `owner_generation` echo the
+ /// request, `command` is `Reply`, `size` is the
/// caller's frame length, and every other field is zero.
///
/// `client`, `op`, `commit` and `status` stay zero. Callers stamp two
@@ -696,6 +744,8 @@ impl ReplyHeader {
timestamp: request_header.timestamp,
request: request_header.request,
operation: request_header.operation,
+ partition_incarnation: request_header.partition_incarnation,
+ owner_generation: request_header.owner_generation,
..Self::default()
}
}
@@ -1012,7 +1062,8 @@ pub struct PrepareHeader {
pub user_id: u32,
/// Immutable capacity nominated by the first committed prepare in this group.
pub retry_capacity: u32,
- pub reserved: [u8; 16],
+ pub partition_incarnation: u64,
+ pub reserved: [u8; 8],
/// Metadata Register commit identifying the authenticated session.
/// Zero is reserved for registration and server-originated operations.
pub session: u64,
@@ -1027,7 +1078,8 @@ const _: () = {
offset_of!(PrepareHeader, user_id) == offset_of!(PrepareHeader, group) + size_of::<u64>()
);
assert!(offset_of!(PrepareHeader, retry_capacity) == 228);
- assert!(offset_of!(PrepareHeader, reserved) + size_of::<[u8; 16]>() == 248);
+ assert!(offset_of!(PrepareHeader, partition_incarnation) == 232);
+ assert!(offset_of!(PrepareHeader, reserved) == 240);
assert!(offset_of!(PrepareHeader, session) == 248);
assert!(offset_of!(PrepareHeader, session) + size_of::<u64>() == HEADER_SIZE);
};
@@ -1056,7 +1108,8 @@ impl Default for PrepareHeader {
group: 0,
user_id: 0,
retry_capacity: 0,
- reserved: [0; 16],
+ partition_incarnation: 0,
+ reserved: [0; 8],
session: 0,
}
}
@@ -3137,10 +3190,8 @@ mod tests {
);
}
- // `group` claims the TAIL of the client header's reserved area; the
- // leading 52 reserved bytes carry data (the non-replicated op code lives
- // in `reserved[0..4]`), so a reshuffle of the carve must trip here rather
- // than silently move the code range.
+ // Routing must preserve the client prefix, including the command code
+ // in `reserved[0..4]` and the captured partition context.
#[test]
fn routed_request_group_claims_reserved_tail() {
use std::mem::offset_of;
@@ -3242,7 +3293,8 @@ mod tests {
group: 17,
user_id: 18,
retry_capacity: 31,
- reserved: [19; 16],
+ partition_incarnation: 19,
+ reserved: [23; 8],
session: 37,
};
let reply = ReplyHeader::from_prepare(&prepare, 20);
@@ -3265,7 +3317,9 @@ mod tests {
assert_eq!(reply.context, 0);
assert_eq!(reply.operation_padding, [0; 7]);
assert_eq!(reply.status, 0);
- assert_eq!(reply.reserved, [0; 36]);
+ assert_eq!(reply.reserved, [0; 4]);
+ assert_eq!(reply.reserved_tail, [0; 16]);
+ assert_eq!(reply.partition_incarnation, 19);
}
#[test]
@@ -3288,7 +3342,11 @@ mod tests {
operation_padding: [13; 7],
session: 14,
user_id: 15,
- reserved: [16; 44],
+ reserved: [16; 4],
+ partition_incarnation: 19,
+ owner_generation: 29,
+ minimum_metadata_op: 31,
+ reserved_tail: [0; 16],
metadata_watermark: 17,
group: 17,
};
@@ -3313,7 +3371,10 @@ mod tests {
assert_eq!(reply.reserved_frame, [0; 66]);
assert_eq!(reply.context, 0);
assert_eq!(reply.operation_padding, [0; 7]);
- assert_eq!(reply.reserved, [0; 36]);
+ assert_eq!(reply.reserved, [0; 4]);
+ assert_eq!(reply.reserved_tail, [0; 16]);
+ assert_eq!(reply.partition_incarnation, 19);
+ assert_eq!(reply.owner_generation, 29);
}
// Wire-discriminant pin: any change breaks SDK decoders.
diff --git a/core/binary_protocol/src/consensus/operation.rs b/core/binary_protocol/src/consensus/operation.rs
index 81d14a944..7961e2167 100644
--- a/core/binary_protocol/src/consensus/operation.rs
+++ b/core/binary_protocol/src/consensus/operation.rs
@@ -63,6 +63,7 @@ pub enum Operation {
/// the same offset deterministically. No client wire code.
TruncatePartition = 68,
FinalizeSession = 70,
+ CompleteLifecycle = 71,
// Metadata operations (shard 0)
CreateStream = 128,
@@ -98,6 +99,9 @@ pub enum Operation {
DeleteConsumerOffset = 162,
// 163 is reserved; 164 and 165 are retired offset operations.
RetireSession = 166,
+ InstallConsumerGroupOwner = 167,
+ TransitionPartitionHistory = 168,
+ RetireConsumerGroupOwners = 169,
}
impl Operation {
@@ -133,8 +137,28 @@ impl Operation {
#[must_use]
#[inline]
pub const fn is_internal(&self) -> bool {
+ // The SDK protocol checks parse this list, so it names every operation.
((*self as u8) >= Self::INTERNAL_START && (*self as u8) < Self::METADATA_START)
- || matches!(self, Self::RetireSession)
+ || matches!(
+ self,
+ Self::RetireSession
+ | Self::InstallConsumerGroupOwner
+ | Self::TransitionPartitionHistory
+ | Self::RetireConsumerGroupOwners
+ )
+ }
+
+ /// Partition fences that metadata orders. Each one changes consumer-group
+ /// ownership or the partition history, so it applies alone.
+ #[must_use]
+ #[inline]
+ pub const fn is_partition_lifecycle(&self) -> bool {
+ matches!(
+ self,
+ Self::InstallConsumerGroupOwner
+ | Self::TransitionPartitionHistory
+ | Self::RetireConsumerGroupOwners
+ )
}
/// Metadata / control-plane operations handled by shard 0.
@@ -249,7 +273,11 @@ impl Operation {
| Self::CompleteConsumerGroupRevocation
| Self::TruncatePartition
| Self::FinalizeSession
- | Self::RetireSession => None,
+ | Self::CompleteLifecycle
+ | Self::RetireSession
+ | Self::InstallConsumerGroupOwner
+ | Self::TransitionPartitionHistory
+ | Self::RetireConsumerGroupOwners => None,
Self::CreateStream
| Self::UpdateStream
| Self::DeleteStream
diff --git a/core/binary_protocol/src/consensus/request.rs b/core/binary_protocol/src/consensus/request.rs
index 0e2d2f8ae..16747cad2 100644
--- a/core/binary_protocol/src/consensus/request.rs
+++ b/core/binary_protocol/src/consensus/request.rs
@@ -75,7 +75,7 @@ impl RequestHeader {
} else {
u128::from(XxHash3_64::oneshot(payload))
};
- let mut reserved = [0; 60];
+ let mut reserved = [0; 4];
if operation == Operation::NonReplicated {
reserved[NON_REPLICATED_CODE_RANGE].copy_from_slice(&code.to_le_bytes());
}
@@ -123,7 +123,7 @@ mod tests {
header.request_checksum,
u128::from(XxHash3_64::oneshot(&payload))
);
- assert_eq!(header.reserved, [0; 60]);
+ assert_eq!(header.reserved, [0; 4]);
}
#[test]
diff --git a/core/binary_protocol/src/dispatch.rs b/core/binary_protocol/src/dispatch.rs
index e92ae9032..68d4677ac 100644
--- a/core/binary_protocol/src/dispatch.rs
+++ b/core/binary_protocol/src/dispatch.rs
@@ -186,6 +186,7 @@ pub const COMMAND_TABLE: &[CommandMeta] = &[
CommandMeta::non_replicated(GET_POLL_ROUTING_CODE, "message.poll_routing"),
CommandMeta::non_replicated(POLL_MESSAGES_ON_PRIMARY_CODE, "message.poll_primary"),
CommandMeta::non_replicated(GET_CONSUMER_OFFSET_ROUTING_CODE, "consumer_offset.routing"),
+ CommandMeta::non_replicated(GET_SEND_CONTEXT_CODE, "message.send_context"),
];
/// Lookup command metadata by command code.
@@ -249,6 +250,7 @@ pub const fn lookup_command(code: u32) -> Option<&'static CommandMeta> {
GET_POLL_ROUTING_CODE => 49,
POLL_MESSAGES_ON_PRIMARY_CODE => 50,
GET_CONSUMER_OFFSET_ROUTING_CODE => 51,
+ GET_SEND_CONTEXT_CODE => 52,
_ => return None,
};
Some(&COMMAND_TABLE[idx])
@@ -295,7 +297,11 @@ pub const fn lookup_by_operation(op: Operation) -> Option<&'static CommandMeta>
| Operation::Logout
| Operation::NonReplicated
| Operation::FinalizeSession
- | Operation::RetireSession => return None,
+ | Operation::RetireSession
+ | Operation::CompleteLifecycle
+ | Operation::TransitionPartitionHistory
+ | Operation::InstallConsumerGroupOwner
+ | Operation::RetireConsumerGroupOwners => return None,
};
Some(&COMMAND_TABLE[idx])
}
diff --git a/core/binary_protocol/src/lib.rs b/core/binary_protocol/src/lib.rs
index 0f4c1755f..968d990d3 100644
--- a/core/binary_protocol/src/lib.rs
+++ b/core/binary_protocol/src/lib.rs
@@ -92,6 +92,7 @@ pub use primitives::consumer::{
pub use primitives::identifier::{MAX_WIRE_NAME_LENGTH, WireIdentifier, WireName};
pub use primitives::options::{MAX_OPTIONS, MAX_OPTIONS_BYTES, WireOptions, validate_options};
pub use primitives::partition_assignment::CreatedPartitionAssignment;
+pub use primitives::partition_history::{ConsumerGroupOwner, PartitionContext};
pub use primitives::partitioning::{MAX_MESSAGES_KEY_LENGTH, WirePartitioning};
pub use primitives::permissions::{
WireGlobalPermissions, WirePermissions, WireStreamPermissions, WireTopicPermissions,
diff --git a/core/binary_protocol/src/primitives/mod.rs b/core/binary_protocol/src/primitives/mod.rs
index c72528925..07f107578 100644
--- a/core/binary_protocol/src/primitives/mod.rs
+++ b/core/binary_protocol/src/primitives/mod.rs
@@ -22,6 +22,7 @@ pub mod consumer;
pub mod identifier;
pub mod options;
pub mod partition_assignment;
+pub mod partition_history;
pub mod partitioning;
pub mod permissions;
pub mod polling_strategy;
diff --git a/core/binary_protocol/src/primitives/partition_history.rs b/core/binary_protocol/src/primitives/partition_history.rs
new file mode 100644
index 000000000..2f17af71a
--- /dev/null
+++ b/core/binary_protocol/src/primitives/partition_history.rs
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::{BufMut, BytesMut};
+use serde::{Deserialize, Serialize};
+
+use crate::WireError;
+use crate::codec::{WireDecode, WireEncode, read_u64_le, read_u128_le};
+
+/// Captured before an operation is sent. Retries and delayed offset writes retain
+/// this value; discovering a newer incarnation or owner does not authorize old work.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
+pub struct PartitionContext {
+ /// Metadata creation revision of the partition. It changes when the numeric
+ /// partition ID is reused, so offsets and retry results never cross incarnations.
+ pub incarnation: u64,
+ /// Installed consumer-group owner generation; zero for operations without a group.
+ pub owner_generation: u64,
+ /// Applied metadata frontier, sent as the request's `minimum_metadata_op`.
+ pub metadata_op: u64,
+}
+
+impl PartitionContext {
+ pub const ENCODED_SIZE: usize = 3 * size_of::<u64>();
+
+ #[must_use]
+ pub fn to_le_bytes(self) -> [u8; Self::ENCODED_SIZE] {
+ let mut bytes = [0; Self::ENCODED_SIZE];
+ bytes[..8].copy_from_slice(&self.incarnation.to_le_bytes());
+ bytes[8..16].copy_from_slice(&self.owner_generation.to_le_bytes());
+ bytes[16..].copy_from_slice(&self.metadata_op.to_le_bytes());
+ bytes
+ }
+
+ pub const fn stamp(self, header: &mut crate::RequestHeader) {
+ header.partition_incarnation = self.incarnation;
+ header.owner_generation = self.owner_generation;
+ header.minimum_metadata_op = self.metadata_op;
+ }
+}
+
+impl WireEncode for PartitionContext {
+ fn encoded_size(&self) -> usize {
+ Self::ENCODED_SIZE
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ buf.put_slice(&self.to_le_bytes());
+ }
+}
+
+impl WireDecode for PartitionContext {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ Ok((
+ Self {
+ incarnation: read_u64_le(buf, 0)?,
+ owner_generation: read_u64_le(buf, size_of::<u64>())?,
+ metadata_op: read_u64_le(buf, 2 * size_of::<u64>())?,
+ },
+ Self::ENCODED_SIZE,
+ ))
+ }
+}
+
+/// Exact offset authority installed by a partition log.
+///
+/// `generation` changes only when this partition's assignment changes.
+/// Zero client and session together install an unowned fence without discarding its generation.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)]
+pub struct ConsumerGroupOwner {
+ pub client_id: u128,
+ pub session: u64,
+ pub generation: u64,
+}
+
+impl ConsumerGroupOwner {
+ #[must_use]
+ pub const fn is_unassigned(self) -> bool {
+ self.client_id == 0 && self.session == 0
+ }
+}
+
+impl WireEncode for ConsumerGroupOwner {
+ fn encoded_size(&self) -> usize {
+ size_of::<u128>() + 2 * size_of::<u64>()
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ buf.put_u128_le(self.client_id);
+ buf.put_u64_le(self.session);
+ buf.put_u64_le(self.generation);
+ }
+}
+
+impl WireDecode for ConsumerGroupOwner {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let owner = Self {
+ client_id: read_u128_le(buf, 0)?,
+ session: read_u64_le(buf, size_of::<u128>())?,
+ generation: read_u64_le(buf, size_of::<u128>() + size_of::<u64>())?,
+ };
+ if (owner.client_id == 0) != (owner.session == 0) {
+ return Err(WireError::Validation(
+ "incomplete consumer owner identity".into(),
+ ));
+ }
+ Ok((owner, owner.encoded_size()))
+ }
+}
diff --git a/core/binary_protocol/src/requests/consumer_offsets/mod.rs b/core/binary_protocol/src/requests/consumer_offsets/mod.rs
index cda0a9ceb..424f16d76 100644
--- a/core/binary_protocol/src/requests/consumer_offsets/mod.rs
+++ b/core/binary_protocol/src/requests/consumer_offsets/mod.rs
@@ -23,6 +23,25 @@ pub use delete_consumer_offset::DeleteConsumerOffsetRequest;
pub use get_consumer_offset::GetConsumerOffsetRequest;
pub use store_consumer_offset::StoreConsumerOffsetRequest;
+/// Server projection appends this immutable owner token to replicated offset bodies.
+pub const PREPARED_OWNER_GENERATION_SIZE: usize = size_of::<u64>();
+
+/// Split a partition-log body from its owner generation without accepting a truncated token.
+///
+/// # Errors
+/// Returns an error when the prepared offset context is incomplete.
+pub fn split_prepared_offset(body: &[u8]) -> Result<(&[u8], u64), crate::WireError> {
+ let offset = body
+ .len()
+ .checked_sub(PREPARED_OWNER_GENERATION_SIZE)
+ .ok_or(crate::WireError::UnexpectedEof {
+ offset: 0,
+ need: PREPARED_OWNER_GENERATION_SIZE,
+ have: body.len(),
+ })?;
+ Ok((&body[..offset], crate::codec::read_u64_le(body, offset)?))
+}
+
#[cfg(test)]
mod tests {
use super::*;
diff --git a/core/binary_protocol/src/requests/messages/get_send_context.rs b/core/binary_protocol/src/requests/messages/get_send_context.rs
new file mode 100644
index 000000000..04bf2a264
--- /dev/null
+++ b/core/binary_protocol/src/requests/messages/get_send_context.rs
@@ -0,0 +1,90 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::{BufMut, BytesMut};
+
+use crate::codec::{WireDecode, WireEncode, read_u32_le};
+use crate::{WireError, WireIdentifier};
+
+/// Partition context discovery authorized by send permission, without exposing topic details.
+/// The response is a `PartitionContext` captured before a new send is admitted.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct GetSendContextRequest {
+ pub stream_id: WireIdentifier,
+ pub topic_id: WireIdentifier,
+ pub partition_id: u32,
+}
+
+impl WireEncode for GetSendContextRequest {
+ fn encoded_size(&self) -> usize {
+ self.stream_id.encoded_size() + self.topic_id.encoded_size() + size_of::<u32>()
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ self.stream_id.encode(buf);
+ self.topic_id.encode(buf);
+ buf.put_u32_le(self.partition_id);
+ }
+}
+
+impl WireDecode for GetSendContextRequest {
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let (stream_id, stream_size) = WireIdentifier::decode(buf)?;
+ let (topic_id, topic_size) = WireIdentifier::decode(&buf[stream_size..])?;
+ let position = stream_size + topic_size;
+ let partition_id = read_u32_le(buf, position)?;
+ Ok((
+ Self {
+ stream_id,
+ topic_id,
+ partition_id,
+ },
+ position + size_of::<u32>(),
+ ))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn given_send_context_request_when_encoded_should_preserve_named_and_numeric_targets() {
+ for (stream_id, topic_id) in [
+ (WireIdentifier::numeric(3), WireIdentifier::numeric(7)),
+ (
+ WireIdentifier::named("events").unwrap(),
+ WireIdentifier::named("orders").unwrap(),
+ ),
+ ] {
+ let request = GetSendContextRequest {
+ stream_id,
+ topic_id,
+ partition_id: 2,
+ };
+ let bytes = request.to_bytes();
+ assert_eq!(bytes.len(), request.encoded_size());
+ assert_eq!(GetSendContextRequest::decode_from(&bytes).unwrap(), request);
+ for length in 0..bytes.len() {
+ assert!(
+ GetSendContextRequest::decode_from(&bytes[..length]).is_err(),
+ "accepted a request truncated at {length} bytes"
+ );
+ }
+ }
+ }
+}
diff --git a/core/binary_protocol/src/requests/messages/mod.rs b/core/binary_protocol/src/requests/messages/mod.rs
index 58112850a..60ebe70a7 100644
--- a/core/binary_protocol/src/requests/messages/mod.rs
+++ b/core/binary_protocol/src/requests/messages/mod.rs
@@ -15,8 +15,10 @@
// specific language governing permissions and limitations
// under the License.
+pub mod get_send_context;
pub mod poll_messages;
pub mod send_messages;
+pub use get_send_context::GetSendContextRequest;
pub use poll_messages::PollMessagesRequest;
pub use send_messages::{RawMessage, SendMessagesEncoder, SendMessagesHeader};
diff --git a/core/binary_protocol/src/requests/partitions/install_consumer_group_owner.rs b/core/binary_protocol/src/requests/partitions/install_consumer_group_owner.rs
new file mode 100644
index 000000000..bd206eb38
--- /dev/null
+++ b/core/binary_protocol/src/requests/partitions/install_consumer_group_owner.rs
@@ -0,0 +1,138 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::{BufMut, BytesMut};
+use serde::{Deserialize, Serialize};
+
+use crate::WireError;
+use crate::codec::{WireDecode, WireEncode, read_u64_le};
+use crate::primitives::partition_history::ConsumerGroupOwner;
+
+/// Server-only partition barrier. Metadata membership alone cannot authorize offsets.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+pub struct InstallConsumerGroupOwnerRequest {
+ pub incarnation: u64,
+ pub group_id: u64,
+ pub owner: ConsumerGroupOwner,
+ pub metadata_op: u64,
+}
+
+impl InstallConsumerGroupOwnerRequest {
+ pub const ENCODED_SIZE: usize = 5 * size_of::<u64>() + size_of::<u128>();
+}
+
+impl WireEncode for InstallConsumerGroupOwnerRequest {
+ fn encoded_size(&self) -> usize {
+ Self::ENCODED_SIZE
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ buf.put_u64_le(self.incarnation);
+ buf.put_u64_le(self.group_id);
+ self.owner.encode(buf);
+ buf.put_u64_le(self.metadata_op);
+ }
+}
+
+impl WireDecode for InstallConsumerGroupOwnerRequest {
+ fn decode_from(buf: &[u8]) -> Result<Self, WireError> {
+ let (request, consumed) = Self::decode(buf)?;
+ if consumed != buf.len() {
+ return Err(WireError::Validation(
+ "trailing ownership installation bytes".into(),
+ ));
+ }
+ Ok(request)
+ }
+
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let incarnation = read_u64_le(buf, 0)?;
+ let group_id = read_u64_le(buf, size_of::<u64>())?;
+ let mut pos = 2 * size_of::<u64>();
+ let (owner, consumed) = ConsumerGroupOwner::decode(&buf[pos..])?;
+ pos += consumed;
+ let metadata_op = read_u64_le(buf, pos)?;
+ pos += size_of::<u64>();
+ if owner.generation == 0 || metadata_op == 0 {
+ return Err(WireError::Validation(
+ "incomplete ownership transition identity".into(),
+ ));
+ }
+ Ok((
+ Self {
+ incarnation,
+ group_id,
+ owner,
+ metadata_op,
+ },
+ pos,
+ ))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn ownership_installation_pins_fields_and_rejects_incomplete_frames() {
+ let request = InstallConsumerGroupOwnerRequest {
+ incarnation: 17,
+ group_id: 3,
+ owner: ConsumerGroupOwner {
+ client_id: 4,
+ session: 5,
+ generation: 6,
+ },
+ metadata_op: 7,
+ };
+ let encoded = request.to_bytes();
+ assert_eq!(
+ encoded.len(),
+ InstallConsumerGroupOwnerRequest::ENCODED_SIZE
+ );
+ assert_eq!(&encoded[0..8], &17_u64.to_le_bytes());
+ assert_eq!(&encoded[8..16], &3_u64.to_le_bytes());
+ assert_eq!(&encoded[16..32], &4_u128.to_le_bytes());
+ assert_eq!(&encoded[32..40], &5_u64.to_le_bytes());
+ assert_eq!(&encoded[40..48], &6_u64.to_le_bytes());
+ assert_eq!(&encoded[48..56], &7_u64.to_le_bytes());
+ assert_eq!(
+ InstallConsumerGroupOwnerRequest::decode_from(&encoded).unwrap(),
+ request
+ );
+ for length in 0..encoded.len() {
+ assert!(InstallConsumerGroupOwnerRequest::decode_from(&encoded[..length]).is_err());
+ }
+ for field in [16..32, 32..40, 40..48, 48..56] {
+ let mut invalid = encoded.to_vec();
+ invalid[field].fill(0);
+ assert!(InstallConsumerGroupOwnerRequest::decode_from(&invalid).is_err());
+ }
+ let mut no_owner = encoded.to_vec();
+ no_owner[16..40].fill(0);
+ assert!(
+ InstallConsumerGroupOwnerRequest::decode_from(&no_owner)
+ .unwrap()
+ .owner
+ .is_unassigned()
+ );
+ let mut trailing = encoded.to_vec();
+ trailing.push(0);
+ assert!(InstallConsumerGroupOwnerRequest::decode_from(&trailing).is_err());
+ }
+}
diff --git a/core/binary_protocol/src/requests/partitions/mod.rs b/core/binary_protocol/src/requests/partitions/mod.rs
index 77c6c4d1e..85bb37f06 100644
--- a/core/binary_protocol/src/requests/partitions/mod.rs
+++ b/core/binary_protocol/src/requests/partitions/mod.rs
@@ -18,8 +18,14 @@
pub mod create_partitions;
pub mod create_partitions_with_assignments;
pub mod delete_partitions;
+pub mod install_consumer_group_owner;
+pub mod retire_consumer_group_owners;
+pub mod transition_partition_history;
pub use crate::primitives::partition_assignment::CreatedPartitionAssignment;
pub use create_partitions::CreatePartitionsRequest;
pub use create_partitions_with_assignments::CreatePartitionsWithAssignmentsRequest;
pub use delete_partitions::DeletePartitionsRequest;
+pub use install_consumer_group_owner::InstallConsumerGroupOwnerRequest;
+pub use retire_consumer_group_owners::RetireConsumerGroupOwnersRequest;
+pub use transition_partition_history::TransitionPartitionHistoryRequest;
diff --git a/core/binary_protocol/src/requests/partitions/retire_consumer_group_owners.rs b/core/binary_protocol/src/requests/partitions/retire_consumer_group_owners.rs
new file mode 100644
index 000000000..2108f6695
--- /dev/null
+++ b/core/binary_protocol/src/requests/partitions/retire_consumer_group_owners.rs
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::{BufMut, BytesMut};
+
+use crate::WireError;
+use crate::codec::{WireDecode, WireEncode, read_u32_le, read_u64_le};
+
+pub const CONSUMER_GROUP_OWNERS_MAX: u32 = 1 << 20;
+
+/// Committed metadata proves IDs below the allocation boundary absent from the
+/// live catalog permanently deleted. Only the newest catalog must be retained.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RetireConsumerGroupOwnersRequest {
+ pub incarnation: u64,
+ pub metadata_op: u64,
+ pub next_group_id: u64,
+ pub live_group_ids: Vec<u64>,
+}
+
+impl RetireConsumerGroupOwnersRequest {
+ pub const PREFIX_SIZE: usize = 3 * size_of::<u64>() + size_of::<u32>();
+
+ #[must_use]
+ pub fn retires(&self, group_id: u64) -> bool {
+ group_id < self.next_group_id && self.live_group_ids.binary_search(&group_id).is_err()
+ }
+}
+
+impl WireEncode for RetireConsumerGroupOwnersRequest {
+ fn encoded_size(&self) -> usize {
+ Self::PREFIX_SIZE + self.live_group_ids.len() * size_of::<u64>()
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ buf.put_u64_le(self.incarnation);
+ buf.put_u64_le(self.metadata_op);
+ buf.put_u64_le(self.next_group_id);
+ buf.put_u32_le(
+ u32::try_from(self.live_group_ids.len()).expect("bounded live group catalog"),
+ );
+ for group_id in &self.live_group_ids {
+ buf.put_u64_le(*group_id);
+ }
+ }
+}
+
+impl WireDecode for RetireConsumerGroupOwnersRequest {
+ fn decode_from(buf: &[u8]) -> Result<Self, WireError> {
+ let (request, consumed) = Self::decode(buf)?;
+ if consumed != buf.len() {
+ return Err(WireError::Validation(
+ "trailing owner retirement bytes".into(),
+ ));
+ }
+ Ok(request)
+ }
+
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let incarnation = read_u64_le(buf, 0)?;
+ let metadata_op = read_u64_le(buf, size_of::<u64>())?;
+ let mut position = 2 * size_of::<u64>();
+ let next_group_id = read_u64_le(buf, position)?;
+ position += size_of::<u64>();
+ let count = read_u32_le(buf, position)?;
+ position += size_of::<u32>();
+ if metadata_op == 0 || count > CONSUMER_GROUP_OWNERS_MAX {
+ return Err(WireError::Validation(
+ "invalid owner retirement catalog".into(),
+ ));
+ }
+ let end = position + count as usize * size_of::<u64>();
+ if end > buf.len() {
+ return Err(WireError::Validation(
+ "truncated owner retirement catalog".into(),
+ ));
+ }
+ let mut live_group_ids = Vec::with_capacity(count as usize);
+ for _ in 0..count {
+ let group_id = read_u64_le(buf, position)?;
+ position += size_of::<u64>();
+ if group_id >= next_group_id
+ || live_group_ids
+ .last()
+ .is_some_and(|previous| *previous >= group_id)
+ {
+ return Err(WireError::Validation(
+ "unordered owner retirement catalog".into(),
+ ));
+ }
+ live_group_ids.push(group_id);
+ }
+ Ok((
+ Self {
+ incarnation,
+ metadata_op,
+ next_group_id,
+ live_group_ids,
+ },
+ position,
+ ))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn given_retirement_catalog_when_decoded_should_preserve_live_and_future_groups() {
+ let request = RetireConsumerGroupOwnersRequest {
+ incarnation: 7,
+ metadata_op: 11,
+ next_group_id: 6,
+ live_group_ids: vec![1, 4],
+ };
+ let encoded = request.to_bytes();
+ assert_eq!(
+ RetireConsumerGroupOwnersRequest::decode_from(&encoded).unwrap(),
+ request
+ );
+ assert!(request.retires(0));
+ assert!(request.retires(5));
+ assert!(!request.retires(1));
+ assert!(!request.retires(6));
+ for length in 0..encoded.len() {
+ assert!(RetireConsumerGroupOwnersRequest::decode_from(&encoded[..length]).is_err());
+ }
+ let mut invalid = request;
+ invalid.live_group_ids = vec![4, 1];
+ assert!(RetireConsumerGroupOwnersRequest::decode_from(&invalid.to_bytes()).is_err());
+ invalid.live_group_ids = vec![1, 1];
+ assert!(RetireConsumerGroupOwnersRequest::decode_from(&invalid.to_bytes()).is_err());
+ invalid.live_group_ids = vec![6];
+ assert!(RetireConsumerGroupOwnersRequest::decode_from(&invalid.to_bytes()).is_err());
+ }
+}
diff --git a/core/binary_protocol/src/requests/partitions/transition_partition_history.rs b/core/binary_protocol/src/requests/partitions/transition_partition_history.rs
new file mode 100644
index 000000000..a37b4114c
--- /dev/null
+++ b/core/binary_protocol/src/requests/partitions/transition_partition_history.rs
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::{BufMut, BytesMut};
+use serde::{Deserialize, Serialize};
+
+use crate::WireError;
+use crate::codec::{WireDecode, WireEncode, read_u64_le};
+
+/// Terminal fence of a partition incarnation that metadata is deleting. Once
+/// committed, the partition refuses every later request of that incarnation.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+pub struct TransitionPartitionHistoryRequest {
+ pub incarnation: u64,
+ pub metadata_op: u64,
+}
+
+impl TransitionPartitionHistoryRequest {
+ pub const ENCODED_SIZE: usize = 2 * size_of::<u64>();
+}
+
+impl WireEncode for TransitionPartitionHistoryRequest {
+ fn encoded_size(&self) -> usize {
+ Self::ENCODED_SIZE
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ buf.put_u64_le(self.incarnation);
+ buf.put_u64_le(self.metadata_op);
+ }
+}
+
+impl WireDecode for TransitionPartitionHistoryRequest {
+ fn decode_from(buf: &[u8]) -> Result<Self, WireError> {
+ let (request, consumed) = Self::decode(buf)?;
+ if consumed != buf.len() {
+ return Err(WireError::Validation(
+ "trailing history transition bytes".into(),
+ ));
+ }
+ Ok(request)
+ }
+
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let incarnation = read_u64_le(buf, 0)?;
+ let metadata_op = read_u64_le(buf, size_of::<u64>())?;
+ if metadata_op == 0 {
+ return Err(WireError::Validation(
+ "missing history transition identity".into(),
+ ));
+ }
+ Ok((
+ Self {
+ incarnation,
+ metadata_op,
+ },
+ Self::ENCODED_SIZE,
+ ))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn history_fence_round_trip_and_malformed_frames() {
+ let request = TransitionPartitionHistoryRequest {
+ incarnation: 17,
+ metadata_op: 52,
+ };
+ let encoded = request.to_bytes();
+ assert_eq!(
+ encoded.len(),
+ TransitionPartitionHistoryRequest::ENCODED_SIZE
+ );
+ assert_eq!(&encoded[0..8], &17_u64.to_le_bytes());
+ assert_eq!(&encoded[8..16], &52_u64.to_le_bytes());
+ assert_eq!(
+ TransitionPartitionHistoryRequest::decode_from(&encoded).unwrap(),
+ request
+ );
+ for length in 0..encoded.len() {
+ assert!(TransitionPartitionHistoryRequest::decode_from(&encoded[..length]).is_err());
+ }
+ let mut trailing = encoded.to_vec();
+ trailing.push(0);
+ assert!(TransitionPartitionHistoryRequest::decode_from(&trailing).is_err());
+ let mut no_metadata_op = encoded.to_vec();
+ no_metadata_op[8..16].fill(0);
+ assert!(TransitionPartitionHistoryRequest::decode_from(&no_metadata_op).is_err());
+ }
+}
diff --git a/core/binary_protocol/src/requests/system/finalize_session.rs b/core/binary_protocol/src/requests/system/finalize_session.rs
new file mode 100644
index 000000000..8d84d25e7
--- /dev/null
+++ b/core/binary_protocol/src/requests/system/finalize_session.rs
@@ -0,0 +1,96 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use bytes::{BufMut, BytesMut};
+
+use crate::WireError;
+use crate::codec::{WireDecode, WireEncode, read_u64_le};
+use crate::requests::system::SessionIdentity;
+
+/// Retirement evidence covers only the partition set at `namespace_revision`.
+/// Ordered metadata apply must recheck that revision before reclaiming the session.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct FinalizeSessionRequest {
+ pub identity: SessionIdentity,
+ pub namespace_revision: u64,
+}
+
+impl WireEncode for FinalizeSessionRequest {
+ fn encoded_size(&self) -> usize {
+ self.identity.encoded_size() + size_of::<u64>()
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ self.identity.encode(buf);
+ buf.put_u64_le(self.namespace_revision);
+ }
+}
+
+impl WireDecode for FinalizeSessionRequest {
+ fn decode_from(buf: &[u8]) -> Result<Self, WireError> {
+ let (request, consumed) = Self::decode(buf)?;
+ if consumed != buf.len() {
+ return Err(WireError::Validation(
+ "trailing session finalization bytes".into(),
+ ));
+ }
+ Ok(request)
+ }
+
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let (identity, consumed) = SessionIdentity::decode(buf)?;
+ let namespace_revision = read_u64_le(buf, consumed)?;
+ Ok((
+ Self {
+ identity,
+ namespace_revision,
+ },
+ consumed + size_of::<u64>(),
+ ))
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn given_finalization_evidence_when_decoding_should_require_the_namespace_revision() {
+ let request = FinalizeSessionRequest {
+ identity: SessionIdentity {
+ client_id: u128::MAX,
+ session: 7,
+ metadata_watermark: 101,
+ },
+ namespace_revision: 2,
+ };
+ let bytes = request.to_bytes();
+ assert_eq!(
+ FinalizeSessionRequest::decode_from(&bytes).unwrap(),
+ request
+ );
+ for end in 0..bytes.len() {
+ assert!(
+ FinalizeSessionRequest::decode_from(&bytes[..end]).is_err(),
+ "accepted truncation at {end}"
+ );
+ }
+ let mut trailing = bytes.to_vec();
+ trailing.push(0);
+ assert!(FinalizeSessionRequest::decode_from(&trailing).is_err());
+ }
+}
diff --git a/core/binary_protocol/src/requests/system/mod.rs b/core/binary_protocol/src/requests/system/mod.rs
index 5d457658f..9be7a86c2 100644
--- a/core/binary_protocol/src/requests/system/mod.rs
+++ b/core/binary_protocol/src/requests/system/mod.rs
@@ -17,6 +17,7 @@
pub mod bind_session;
pub mod describe_options;
+pub mod finalize_session;
pub mod get_client;
pub mod get_clients;
pub mod get_cluster_metadata;
@@ -30,6 +31,7 @@ pub use bind_session::{BindSessionRequest, SessionIdentity};
pub use describe_options::{
DescribeOptionsRequest, OPTIONS_SCOPE_STREAM, OPTIONS_SCOPE_TOPIC, OPTIONS_SCOPE_USER,
};
+pub use finalize_session::FinalizeSessionRequest;
pub use get_client::GetClientRequest;
pub use get_clients::GetClientsRequest;
pub use get_cluster_metadata::GetClusterMetadataRequest;
diff --git a/core/binary_protocol/src/responses/messages/poll_messages.rs b/core/binary_protocol/src/responses/messages/poll_messages.rs
index ae7dce74f..f6f3fca1d 100644
--- a/core/binary_protocol/src/responses/messages/poll_messages.rs
+++ b/core/binary_protocol/src/responses/messages/poll_messages.rs
@@ -17,7 +17,7 @@
//! Decoder for the `PollMessages` response.
//!
-//! The body after the 16-byte prefix is a stream of canonical batch records
+//! The body after the 40-byte prefix is a stream of canonical batch records
//! (see [`crate::batch`]) served as stored: each record's header carries the
//! stamped `base_offset` / `base_timestamp`, and each frame's deltas resolve
//! against them. A record may be a server-sliced view of a larger stored
@@ -27,19 +27,21 @@
use crate::batch::{BatchHeader, BatchIntegrity, BatchIterator, decode_batch_slice_with};
use crate::codec::{WireDecode, WireEncode, read_u32_le, read_u64_le};
use crate::error::WireError;
+use crate::primitives::partition_history::PartitionContext;
use bytes::{BufMut, BytesMut};
-/// Size of the `PollMessages` response header: `partition_id(4) + current_offset(8) + count(4)`.
-const POLL_RESPONSE_HEADER_SIZE: usize = 16;
+/// Size of the `PollMessages` response header: `partition_id(4) + current_offset(8) + count(4) + context(24)`.
+pub const POLL_RESPONSE_HEADER_SIZE: usize = 16 + PartitionContext::ENCODED_SIZE;
-/// The 16-byte metadata prefix of a `PollMessages` response.
+/// The 40-byte metadata prefix of a `PollMessages` response.
///
-/// Layout: `partition_id(4) + current_offset(8) + messages_count(4)`.
+/// Layout: `partition_id(4) + current_offset(8) + messages_count(4) + context(24)`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PollMessagesResponseHeader {
pub partition_id: u32,
pub current_offset: u64,
pub messages_count: u32,
+ pub context: PartitionContext,
}
impl WireEncode for PollMessagesResponseHeader {
@@ -51,6 +53,7 @@ impl WireEncode for PollMessagesResponseHeader {
buf.put_u32_le(self.partition_id);
buf.put_u64_le(self.current_offset);
buf.put_u32_le(self.messages_count);
+ self.context.encode(buf);
}
}
@@ -59,11 +62,13 @@ impl WireDecode for PollMessagesResponseHeader {
let partition_id = read_u32_le(buf, 0)?;
let current_offset = read_u64_le(buf, 4)?;
let messages_count = read_u32_le(buf, 12)?;
+ let (context, _) = PartitionContext::decode(&buf[16..])?;
Ok((
Self {
partition_id,
current_offset,
messages_count,
+ context,
},
POLL_RESPONSE_HEADER_SIZE,
))
@@ -176,7 +181,7 @@ pub struct PollMessagesResponse<'a> {
impl<'a> PollMessagesResponse<'a> {
/// Decode from a response payload buffer. Borrows the buffer.
///
- /// Reads the 16-byte header then creates an iterator over the batch
+ /// Reads the 40-byte header then creates an iterator over the batch
/// records that follow. Records are validated lazily during iteration.
///
/// # Errors
@@ -235,6 +240,7 @@ mod tests {
body.extend_from_slice(&7u32.to_le_bytes());
body.extend_from_slice(¤t_offset.to_le_bytes());
body.extend_from_slice(&count.to_le_bytes());
+ body.extend_from_slice(&PartitionContext::default().to_le_bytes());
for batch in batches {
body.extend_from_slice(batch);
}
@@ -247,6 +253,7 @@ mod tests {
partition_id: 3,
current_offset: 42,
messages_count: 7,
+ context: PartitionContext::default(),
};
let bytes = header.to_bytes();
let (decoded, consumed) = PollMessagesResponseHeader::decode(&bytes).unwrap();
diff --git a/core/binary_protocol/src/responses/messages/poll_routing.rs b/core/binary_protocol/src/responses/messages/poll_routing.rs
index d8ef8e61a..1248c41d1 100644
--- a/core/binary_protocol/src/responses/messages/poll_routing.rs
+++ b/core/binary_protocol/src/responses/messages/poll_routing.rs
@@ -17,6 +17,7 @@
use crate::WireError;
use crate::codec::{WireDecode, WireEncode};
+use crate::primitives::partition_history::PartitionContext;
use crate::requests::system::SessionIdentity;
use crate::responses::system::get_cluster_metadata::ClusterNodeResponse;
use bytes::BytesMut;
@@ -29,16 +30,20 @@ use bytes::BytesMut;
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PollRoutingResponse {
pub consumer_session: SessionIdentity,
+ pub context: PartitionContext,
pub primary: ClusterNodeResponse,
}
impl WireEncode for PollRoutingResponse {
fn encoded_size(&self) -> usize {
- self.consumer_session.encoded_size() + self.primary.encoded_size()
+ self.consumer_session.encoded_size()
+ + self.context.encoded_size()
+ + self.primary.encoded_size()
}
fn encode(&self, buf: &mut BytesMut) {
self.consumer_session.encode(buf);
+ self.context.encode(buf);
self.primary.encode(buf);
}
}
@@ -46,13 +51,15 @@ impl WireEncode for PollRoutingResponse {
impl WireDecode for PollRoutingResponse {
fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
let (consumer_session, consumed) = SessionIdentity::decode(buf)?;
- let (primary, node_size) = ClusterNodeResponse::decode(&buf[consumed..])?;
+ let (context, context_size) = PartitionContext::decode(&buf[consumed..])?;
+ let (primary, node_size) = ClusterNodeResponse::decode(&buf[consumed + context_size..])?;
Ok((
Self {
consumer_session,
+ context,
primary,
},
- consumed + node_size,
+ consumed + context_size + node_size,
))
}
}
@@ -69,6 +76,7 @@ mod tests {
session: 7,
metadata_watermark: 11,
},
+ context: PartitionContext::default(),
primary: ClusterNodeResponse {
name: "node-1".to_owned(),
ip: "::1".to_owned(),
diff --git a/core/binary_protocol/src/responses/topics/get_topic.rs b/core/binary_protocol/src/responses/topics/get_topic.rs
index 793800a3c..3ed40c84a 100644
--- a/core/binary_protocol/src/responses/topics/get_topic.rs
+++ b/core/binary_protocol/src/responses/topics/get_topic.rs
@@ -17,14 +17,15 @@
use crate::WireError;
use crate::codec::{WireDecode, WireEncode, read_u32_le, read_u64_le};
+use crate::primitives::partition_history::PartitionContext;
use crate::responses::streams::get_stream::TopicHeader;
use bytes::{BufMut, BytesMut};
/// Partition details within a `GetTopic` response.
///
-/// Wire format (40 bytes fixed):
+/// Wire format (64 bytes fixed):
/// ```text
-/// [id:4][created_at:8][segments_count:4][current_offset:8][size_bytes:8][messages_count:8]
+/// [id:4][created_at:8][segments_count:4][current_offset:8][size_bytes:8][messages_count:8][context:24]
/// ```
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct PartitionResponse {
@@ -34,10 +35,11 @@ pub struct PartitionResponse {
pub current_offset: u64,
pub size_bytes: u64,
pub messages_count: u64,
+ pub context: PartitionContext,
}
impl PartitionResponse {
- const FIXED_SIZE: usize = 4 + 8 + 4 + 8 + 8 + 8; // 40
+ const FIXED_SIZE: usize = 4 + 8 + 4 + 8 + 8 + 8 + PartitionContext::ENCODED_SIZE;
}
impl WireEncode for PartitionResponse {
@@ -52,6 +54,7 @@ impl WireEncode for PartitionResponse {
buf.put_u64_le(self.current_offset);
buf.put_u64_le(self.size_bytes);
buf.put_u64_le(self.messages_count);
+ self.context.encode(buf);
}
}
@@ -63,6 +66,7 @@ impl WireDecode for PartitionResponse {
let current_offset = read_u64_le(buf, 16)?;
let size_bytes = read_u64_le(buf, 24)?;
let messages_count = read_u64_le(buf, 32)?;
+ let (context, _) = PartitionContext::decode(&buf[40..])?;
Ok((
Self {
@@ -72,6 +76,7 @@ impl WireDecode for PartitionResponse {
current_offset,
size_bytes,
messages_count,
+ context,
},
Self::FIXED_SIZE,
))
@@ -159,6 +164,11 @@ mod tests {
current_offset: 99,
size_bytes: 1024,
messages_count: 100,
+ context: PartitionContext {
+ incarnation: 23,
+ owner_generation: 0,
+ metadata_op: 101,
+ },
}
}
diff --git a/core/common/Cargo.toml b/core/common/Cargo.toml
index bd02da7a3..3f2d0ff68 100644
--- a/core/common/Cargo.toml
+++ b/core/common/Cargo.toml
@@ -49,7 +49,7 @@ ring = { workspace = true }
secrecy = { workspace = true }
serde = { workspace = true }
serde_json = { workspace = true }
-serde_with = { workspace = true, features = ["base64"] }
+serde_with = { workspace = true, features = ["base64", "json"] }
strum = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true }
diff --git a/core/common/src/consumer_group_client_state.rs b/core/common/src/consumer_group_client_state.rs
index 0686bc742..1d991b05c 100644
--- a/core/common/src/consumer_group_client_state.rs
+++ b/core/common/src/consumer_group_client_state.rs
@@ -27,6 +27,7 @@
//! re-sync that resets the cursor.
use crate::Identifier;
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
use std::collections::HashMap;
use std::sync::Mutex;
@@ -37,14 +38,19 @@ struct GroupAssignment {
cursor: usize,
}
-/// Per-transport cache of consumer-group assignments + partition counts used to
-/// resolve partitioning client-side. Keys are `stream|topic[|group]` strings
-/// built from the request identifiers.
+#[derive(Debug, Default)]
+struct TopicPartitions {
+ count: Option<u32>,
+ contexts: HashMap<u32, PartitionContext>,
+ cursor: usize,
+}
+
+/// Per-transport cache of group assignments, producer cursors, partition counts
+/// and send contexts. Topic lookups borrow identifiers without formatting keys.
#[derive(Debug, Default)]
pub struct ConsumerGroupClientState {
assignments: Mutex<HashMap<String, GroupAssignment>>,
- balanced_cursors: Mutex<HashMap<String, usize>>,
- partition_counts: Mutex<HashMap<String, u32>>,
+ topic_partitions: Mutex<HashMap<Identifier, HashMap<Identifier, TopicPartitions>>>,
/// Identifiers of the joined groups, so the heartbeat can rebuild a sync
/// request without re-deriving them from the cache key. Keyed by the same
/// `stream|topic|group` string as `assignments`.
@@ -111,34 +117,131 @@ impl ConsumerGroupClientState {
/// The next `Balanced` produce partition for a topic, advancing the cursor.
#[must_use]
#[allow(clippy::cast_possible_truncation)]
- pub fn next_balanced_partition(&self, key: &str, partition_count: u32) -> u32 {
+ pub fn next_balanced_partition(
+ &self,
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ partition_count: u32,
+ ) -> u32 {
if partition_count == 0 {
return 0;
}
let mut map = self
- .balanced_cursors
+ .topic_partitions
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner);
- let cursor = map.entry(key.to_owned()).or_default();
- let partition = (*cursor % partition_count as usize) as u32;
- *cursor = cursor.wrapping_add(1);
- partition
+ if let Some(topic) = map
+ .get_mut(stream_id)
+ .and_then(|topics| topics.get_mut(topic_id))
+ {
+ let partition = (topic.cursor % partition_count as usize) as u32;
+ topic.cursor = topic.cursor.wrapping_add(1);
+ return partition;
+ }
+ map.entry(stream_id.clone()).or_default().insert(
+ topic_id.clone(),
+ TopicPartitions {
+ cursor: 1,
+ ..Default::default()
+ },
+ );
+ 0
}
#[must_use]
- pub fn partition_count(&self, key: &str) -> Option<u32> {
- self.partition_counts
+ pub fn partition_count(&self, stream_id: &Identifier, topic_id: &Identifier) -> Option<u32> {
+ self.topic_partitions
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
- .get(key)
+ .get(stream_id)?
+ .get(topic_id)?
+ .count
+ }
+
+ pub fn set_topic_partitions(
+ &self,
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ partitions: &[crate::Partition],
+ ) {
+ let Ok(count) = u32::try_from(partitions.len()) else {
+ return;
+ };
+ let mut map = self
+ .topic_partitions
+ .lock()
+ .unwrap_or_else(std::sync::PoisonError::into_inner);
+ let topic = map
+ .entry(stream_id.clone())
+ .or_default()
+ .entry(topic_id.clone())
+ .or_default();
+ topic.count = Some(count);
+ topic.contexts = partitions
+ .iter()
+ .map(|partition| (partition.id, partition.context))
+ .collect();
+ }
+
+ #[must_use]
+ pub fn partition_context(
+ &self,
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ partition_id: u32,
+ ) -> Option<PartitionContext> {
+ self.topic_partitions
+ .lock()
+ .unwrap_or_else(std::sync::PoisonError::into_inner)
+ .get(stream_id)?
+ .get(topic_id)?
+ .contexts
+ .get(&partition_id)
.copied()
}
- pub fn set_partition_count(&self, key: String, partition_count: u32) {
- self.partition_counts
+ pub fn set_partition_context(
+ &self,
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ partition_id: u32,
+ context: PartitionContext,
+ ) {
+ self.topic_partitions
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
- .insert(key, partition_count);
+ .entry(stream_id.clone())
+ .or_default()
+ .entry(topic_id.clone())
+ .or_default()
+ .contexts
+ .insert(partition_id, context);
+ }
+
+ pub fn invalidate_topic(&self, stream_id: &Identifier, topic_id: &Identifier) {
+ if let Some(topic) = self
+ .topic_partitions
+ .lock()
+ .unwrap_or_else(std::sync::PoisonError::into_inner)
+ .get_mut(stream_id)
+ .and_then(|topics| topics.get_mut(topic_id))
+ {
+ topic.count = None;
+ topic.contexts.clear();
+ }
+ }
+
+ /// Names and numeric identifiers can cache the same partition under different keys.
+ /// Already captured request contexts are independent of this discovery cache.
+ pub fn invalidate_topic_discovery(&self) {
+ let mut map = self
+ .topic_partitions
+ .lock()
+ .unwrap_or_else(std::sync::PoisonError::into_inner);
+ for topic in map.values_mut().flat_map(HashMap::values_mut) {
+ topic.count = None;
+ topic.contexts.clear();
+ }
}
/// Record a joined group's identifiers so the heartbeat can re-sync it.
@@ -208,6 +311,35 @@ impl ConsumerGroupClientState {
mod tests {
use super::*;
+ fn partitions(count: u32) -> Vec<crate::Partition> {
+ (0..count)
+ .map(|id| crate::Partition {
+ id,
+ created_at: Default::default(),
+ segments_count: 0,
+ current_offset: 0,
+ size: Default::default(),
+ messages_count: 0,
+ context: PartitionContext::default(),
+ })
+ .collect()
+ }
+
+ #[test]
+ fn topic_identifiers_do_not_alias_and_discovery_refresh_preserves_the_cursor() {
+ let state = ConsumerGroupClientState::new();
+ let stream = Identifier::numeric(1).unwrap();
+ let numeric = Identifier::numeric(2).unwrap();
+ let named = Identifier::named("2").unwrap();
+ state.set_topic_partitions(&stream, &numeric, &partitions(3));
+ assert_eq!(state.partition_count(&stream, &named), None);
+ assert_eq!(state.next_balanced_partition(&stream, &numeric, 3), 0);
+ state.invalidate_topic_discovery();
+ assert_eq!(state.partition_count(&stream, &numeric), None);
+ state.set_topic_partitions(&stream, &numeric, &partitions(3));
+ assert_eq!(state.next_balanced_partition(&stream, &numeric, 3), 1);
+ }
+
#[test]
fn group_partition_round_robins_then_wraps() {
let state = ConsumerGroupClientState::new();
@@ -233,11 +365,36 @@ mod tests {
fn balanced_round_robins() {
let state = ConsumerGroupClientState::new();
let picks: Vec<u32> = (0..4)
- .map(|_| state.next_balanced_partition("s|t", 3))
+ .map(|_| {
+ state.next_balanced_partition(
+ &Identifier::named("s").unwrap(),
+ &Identifier::named("t").unwrap(),
+ 3,
+ )
+ })
.collect();
assert_eq!(picks, vec![0, 1, 2, 0]);
}
+ #[test]
+ fn send_context_discovery_does_not_invent_a_partition_count() {
+ let stream = Identifier::named("s").unwrap();
+ let topic = Identifier::named("t").unwrap();
+ let state = ConsumerGroupClientState::new();
+ let context = PartitionContext::default();
+ state.set_partition_context(&stream, &topic, 2, context);
+ assert_eq!(state.partition_context(&stream, &topic, 2), Some(context));
+ assert_eq!(state.partition_count(&stream, &topic), None);
+
+ state.set_topic_partitions(&stream, &topic, &partitions(3));
+ assert_eq!(state.partition_count(&stream, &topic), Some(3));
+ assert_eq!(state.partition_context(&stream, &topic, 2), Some(context));
+
+ state.invalidate_topic(&stream, &topic);
+ assert_eq!(state.partition_context(&stream, &topic, 2), None);
+ assert_eq!(state.partition_count(&stream, &topic), None);
+ }
+
#[test]
fn missing_assignment_yields_none() {
let state = ConsumerGroupClientState::new();
@@ -262,12 +419,21 @@ mod tests {
#[test]
fn session_reset_drops_membership_and_keeps_topic_state() {
+ let stream = Identifier::named("s").unwrap();
+ let topic = Identifier::named("t").unwrap();
let state = ConsumerGroupClientState::new();
let id = Identifier::named("g").unwrap();
state.register_group("s|t|g".to_owned(), id.clone(), id.clone(), id);
state.set_assignment("s|t|g".to_owned(), 1, vec![0, 1]);
- assert_eq!(state.next_balanced_partition("s|t", 3), 0);
- state.set_partition_count("s|t".to_owned(), 3);
+ assert_eq!(
+ state.next_balanced_partition(
+ &Identifier::named("s").unwrap(),
+ &Identifier::named("t").unwrap(),
+ 3
+ ),
+ 0
+ );
+ state.set_topic_partitions(&stream, &topic, &partitions(3));
state.clear_session_scoped();
@@ -276,7 +442,14 @@ mod tests {
assert!(!state.has_assignment("s|t|g"));
// Topic state outlives the session: the produce cursor carries on and
// the partition count is still cached.
- assert_eq!(state.next_balanced_partition("s|t", 3), 1);
- assert_eq!(state.partition_count("s|t"), Some(3));
+ assert_eq!(
+ state.next_balanced_partition(
+ &Identifier::named("s").unwrap(),
+ &Identifier::named("t").unwrap(),
+ 3
+ ),
+ 1
+ );
+ assert_eq!(state.partition_count(&stream, &topic), Some(3));
}
}
diff --git a/core/common/src/error/iggy_error.rs b/core/common/src/error/iggy_error.rs
index 73350926b..76541fad3 100644
--- a/core/common/src/error/iggy_error.rs
+++ b/core/common/src/error/iggy_error.rs
@@ -178,6 +178,10 @@ pub enum IggyError {
"Request is below the deduplication window; outcome unknown, resending may duplicate the write"
)]
RequestTooOld = 85,
+ #[error("The partition history is no longer available; this request cannot execute again")]
+ HistoryUnavailable = 87,
+ #[error("A committed lifecycle operation prevents this mutation. Retry as a new request.")]
+ LifecycleBusy = 88,
#[error("Client with ID: {0} was not found.")]
ClientNotFound(u32) = 100,
#[error("Invalid client ID")]
@@ -292,6 +296,8 @@ pub enum IggyError {
TooManyTopics = 2021,
#[error("Partitions limit reached, raise [metadata] partitions_max")]
PartitionsLimitReached = 2022,
+ #[error("Partition resizing is disabled for this fixed topic")]
+ PartitionResizeDisabled = 2023,
#[error("Cannot create partition with ID: {0} for stream with ID: {1} and topic with ID: {2}")]
CannotCreatePartition(usize, usize, usize) = 3000,
#[error(
@@ -653,6 +659,14 @@ mod tests {
assert_eq!(IggyError::from_code_as_string(2022), error.as_string());
}
+ #[test]
+ fn partition_resize_disabled_round_trips_by_code() {
+ let error = IggyError::PartitionResizeDisabled;
+ assert_eq!(error.as_code(), 2023);
+ assert_eq!(IggyError::from_code(2023), error);
+ assert_eq!(IggyError::from_code_as_string(2023), error.as_string());
+ }
+
#[test]
fn too_many_consumer_offsets_round_trips_by_code() {
let error = IggyError::TooManyConsumerOffsets;
diff --git a/core/common/src/http/consumer_offsets/store_consumer_offset.rs b/core/common/src/http/consumer_offsets/store_consumer_offset.rs
index 903ed82c3..f664c7bd1 100644
--- a/core/common/src/http/consumer_offsets/store_consumer_offset.rs
+++ b/core/common/src/http/consumer_offsets/store_consumer_offset.rs
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-use crate::Consumer;
+use crate::{Consumer, PartitionContext};
use serde::{Deserialize, Serialize};
#[derive(Debug, Serialize, Deserialize)]
@@ -24,4 +24,6 @@ pub struct StoreConsumerOffset {
pub consumer: Consumer,
pub partition_id: Option<u32>,
pub offset: u64,
+ #[serde(default)]
+ pub context: Option<PartitionContext>,
}
diff --git a/core/common/src/traits/binary_impls/consumer_offsets.rs b/core/common/src/traits/binary_impls/consumer_offsets.rs
index 8ec0b8143..fb9034c70 100644
--- a/core/common/src/traits/binary_impls/consumer_offsets.rs
+++ b/core/common/src/traits/binary_impls/consumer_offsets.rs
@@ -18,7 +18,8 @@
use crate::traits::binary_auth::fail_if_not_authenticated;
use crate::wire_conversions::{consumer_to_wire, identifier_to_wire};
use crate::{
- BinaryClient, Consumer, ConsumerOffsetClient, ConsumerOffsetInfo, Identifier, IggyError,
+ BinaryClient, Consumer, ConsumerOffsetClient, ConsumerOffsetInfo, ConsumerPosition, Identifier,
+ IggyError,
};
use iggy_binary_protocol::AckLevel;
use iggy_binary_protocol::codec::WireEncode;
@@ -32,6 +33,31 @@ use iggy_binary_protocol::responses::consumer_offsets::get_consumer_offset::Cons
#[async_trait::async_trait]
impl<B: BinaryClient> ConsumerOffsetClient for B {
+ async fn store_consumer_position(
+ &self,
+ consumer: &Consumer,
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ position: ConsumerPosition,
+ ) -> Result<(), IggyError> {
+ fail_if_not_authenticated(self).await?;
+ self.send_offset_write_with_response(
+ STORE_CONSUMER_OFFSET_CODE,
+ StoreConsumerOffsetRequest {
+ consumer: consumer_to_wire(consumer)?,
+ stream_id: identifier_to_wire(stream_id)?,
+ topic_id: identifier_to_wire(topic_id)?,
+ partition_id: Some(position.partition_id),
+ offset: position.offset,
+ ack: AckLevel::Quorum,
+ }
+ .to_bytes(),
+ Some(position.context),
+ )
+ .await?;
+ Ok(())
+ }
+
async fn store_consumer_offset(
&self,
consumer: &Consumer,
@@ -55,6 +81,7 @@ impl<B: BinaryClient> ConsumerOffsetClient for B {
ack: AckLevel::Quorum,
}
.to_bytes(),
+ None,
)
.await?;
Ok(())
@@ -111,6 +138,7 @@ impl<B: BinaryClient> ConsumerOffsetClient for B {
ack: AckLevel::Quorum,
}
.to_bytes(),
+ None,
)
.await?;
Ok(())
diff --git a/core/common/src/traits/binary_impls/messages.rs b/core/common/src/traits/binary_impls/messages.rs
index 1c42efe74..046992131 100644
--- a/core/common/src/traits/binary_impls/messages.rs
+++ b/core/common/src/traits/binary_impls/messages.rs
@@ -28,11 +28,12 @@ use crate::{ConsumerKind, PartitioningKind, TopicClient, calculate_32};
use bytes::BytesMut;
use iggy_binary_protocol::codec::WireDecode;
use iggy_binary_protocol::codec::WireEncode;
-use iggy_binary_protocol::codes::SEND_MESSAGES_CODE;
use iggy_binary_protocol::codes::SYNC_CONSUMER_GROUP_CODE;
+use iggy_binary_protocol::codes::{GET_SEND_CONTEXT_CODE, SEND_MESSAGES_CODE};
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
use iggy_binary_protocol::requests::consumer_groups::SyncConsumerGroupRequest;
use iggy_binary_protocol::requests::messages::{
- PollMessagesRequest, RawMessage, SendMessagesEncoder,
+ GetSendContextRequest, PollMessagesRequest, RawMessage, SendMessagesEncoder,
};
use iggy_binary_protocol::responses::consumer_groups::SyncConsumerGroupResponse;
@@ -44,10 +45,6 @@ fn group_cache_key(stream_id: &Identifier, topic_id: &Identifier, group_id: &Ide
format!("{stream_id}|{topic_id}|{group_id}")
}
-fn topic_cache_key(stream_id: &Identifier, topic_id: &Identifier) -> String {
- format!("{stream_id}|{topic_id}")
-}
-
/// Sync the requesting member's assignment from the coordinator into the
/// transport cache. An empty reply means the client is not a member.
async fn sync_group_assignment<B: BinaryClient>(
@@ -112,16 +109,15 @@ async fn topic_partition_count<B: BinaryClient>(
stream_id: &Identifier,
topic_id: &Identifier,
) -> Result<u32, IggyError> {
- let key = topic_cache_key(stream_id, topic_id);
- if let Some(count) = client.consumer_group_state().partition_count(&key) {
+ if let Some(count) = client
+ .consumer_group_state()
+ .partition_count(stream_id, topic_id)
+ {
return Ok(count);
}
let details = TopicClient::get_topic(client, stream_id, topic_id)
.await?
.ok_or_else(|| IggyError::TopicIdNotFound(topic_id.clone(), stream_id.clone()))?;
- client
- .consumer_group_state()
- .set_partition_count(key, details.partitions_count);
Ok(details.partitions_count)
}
@@ -142,10 +138,9 @@ async fn resolve_partitioning<B: BinaryClient>(
stream_id.clone(),
));
}
- let key = topic_cache_key(stream_id, topic_id);
let partition = client
.consumer_group_state()
- .next_balanced_partition(&key, count);
+ .next_balanced_partition(stream_id, topic_id, count);
Ok(Partitioning::partition_id(partition))
}
PartitioningKind::MessagesKey => {
@@ -209,7 +204,10 @@ async fn poll_group_messages<B: BinaryClient>(
count,
auto_commit,
};
- match client.send_poll_with_response(&request).await {
+ match client
+ .send_poll_with_response(&request, strategy.context)
+ .await
+ {
Ok(response) => {
let polled = PolledMessages::from_bytes(response)?;
// The coordinator can't yet signal a generation fence as a typed
@@ -225,9 +223,12 @@ async fn poll_group_messages<B: BinaryClient>(
}
return Ok(polled);
}
- Err(IggyError::ConsumerGroupPartitionNotOwned(..)) => {
+ Err(error @ IggyError::ConsumerGroupPartitionNotOwned(..)) => {
client.consumer_group_state().invalidate_assignment(&key);
sync_group_assignment(client, stream_id, topic_id, &consumer.id).await?;
+ if strategy.context.is_some() {
+ return Err(error);
+ }
}
Err(error) => return Err(error),
}
@@ -331,7 +332,7 @@ impl<B: BinaryClient> MessageClient for B {
count,
auto_commit,
};
- let response = self.send_poll_with_response(&req).await?;
+ let response = self.send_poll_with_response(&req, strategy.context).await?;
PolledMessages::from_bytes(response)
}
@@ -357,6 +358,15 @@ impl<B: BinaryClient> MessageClient for B {
let wire_stream_id = identifier_to_wire(stream_id)?;
let wire_topic_id = identifier_to_wire(topic_id)?;
let wire_partitioning = partitioning_to_wire(partitioning)?;
+ let partition_id = u32::from_le_bytes(
+ partitioning
+ .value
+ .as_slice()
+ .try_into()
+ .map_err(|_| IggyError::InvalidCommand)?,
+ );
+ let state = self.consumer_group_state();
+ let mut context = state.partition_context(stream_id, topic_id, partition_id);
// The producer owns message ids now that batches ride the wire
// verbatim: a zero id is minted here, before the frame checksum
// covers it.
@@ -394,10 +404,41 @@ impl<B: BinaryClient> MessageClient for B {
}
_ => IggyError::InvalidCommand,
})?;
- let response = self
- .send_raw_with_response(SEND_MESSAGES_CODE, buf.freeze())
- .await?;
- Ok(committed_send_confirmations(&response))
+ let payload = buf.freeze();
+ let mut refresh_available = true;
+ loop {
+ let captured = if let Some(context) = context {
+ context
+ } else {
+ let request = GetSendContextRequest {
+ stream_id: wire_stream_id.clone(),
+ topic_id: wire_topic_id.clone(),
+ partition_id,
+ };
+ let response = self
+ .send_raw_with_response(GET_SEND_CONTEXT_CODE, request.to_bytes())
+ .await?;
+ let context = super::decode_response::<PartitionContext>(&response)?;
+ state.set_partition_context(stream_id, topic_id, partition_id, context);
+ context
+ };
+ match self
+ .send_raw_with_context(SEND_MESSAGES_CODE, payload.clone(), captured)
+ .await
+ {
+ Ok(response) => return Ok(committed_send_confirmations(&response)),
+ Err(IggyError::HistoryUnavailable) => {
+ state.invalidate_topic(stream_id, topic_id);
+ if refresh_available {
+ refresh_available = false;
+ context = None;
+ continue;
+ }
+ return Err(IggyError::HistoryUnavailable);
+ }
+ Err(error) => return Err(error),
+ }
+ }
}
}
diff --git a/core/common/src/traits/binary_impls/partitions.rs b/core/common/src/traits/binary_impls/partitions.rs
index dedf63b48..e2c04df15 100644
--- a/core/common/src/traits/binary_impls/partitions.rs
+++ b/core/common/src/traits/binary_impls/partitions.rs
@@ -45,6 +45,7 @@ impl<B: BinaryClient> PartitionClient for B {
.to_bytes(),
)
.await?;
+ self.consumer_group_state().invalidate_topic_discovery();
Ok(())
}
@@ -67,6 +68,7 @@ impl<B: BinaryClient> PartitionClient for B {
.to_bytes(),
)
.await?;
+ self.consumer_group_state().invalidate_topic_discovery();
Ok(())
}
}
diff --git a/core/common/src/traits/binary_impls/streams.rs b/core/common/src/traits/binary_impls/streams.rs
index 2667341d9..6b8133f27 100644
--- a/core/common/src/traits/binary_impls/streams.rs
+++ b/core/common/src/traits/binary_impls/streams.rs
@@ -109,6 +109,7 @@ impl<B: BinaryClient> StreamClient for B {
DeleteStreamRequest { stream_id: wire_id }.to_bytes(),
)
.await?;
+ self.consumer_group_state().invalidate_topic_discovery();
Ok(())
}
}
diff --git a/core/common/src/traits/binary_impls/topics.rs b/core/common/src/traits/binary_impls/topics.rs
index 7ce2f88f4..868a3119d 100644
--- a/core/common/src/traits/binary_impls/topics.rs
+++ b/core/common/src/traits/binary_impls/topics.rs
@@ -56,7 +56,10 @@ impl<B: BinaryClient> TopicClient for B {
return Ok(None);
}
let wire_resp = super::decode_response::<GetTopicResponse>(&response)?;
- Ok(Some(TopicDetails::try_from(wire_resp)?))
+ let details = TopicDetails::try_from(wire_resp)?;
+ self.consumer_group_state()
+ .set_topic_partitions(stream_id, topic_id, &details.partitions);
+ Ok(Some(details))
}
async fn get_topics(&self, stream_id: &Identifier) -> Result<Vec<Topic>, IggyError> {
@@ -145,6 +148,7 @@ impl<B: BinaryClient> TopicClient for B {
.to_bytes(),
)
.await?;
+ self.consumer_group_state().invalidate_topic_discovery();
Ok(())
}
}
diff --git a/core/common/src/traits/binary_transport.rs b/core/common/src/traits/binary_transport.rs
index cfbf3cfee..629c0d80c 100644
--- a/core/common/src/traits/binary_transport.rs
+++ b/core/common/src/traits/binary_transport.rs
@@ -17,12 +17,12 @@
use crate::{
ClientState, Credentials, DiagnosticEvent, Identifier, IggyError, NonZeroIggyDuration,
+ PartitionContext,
};
use async_trait::async_trait;
use bytes::Bytes;
use iggy_binary_protocol::WireDecode;
use iggy_binary_protocol::WireEncode;
-use iggy_binary_protocol::codes::POLL_MESSAGES_CODE;
use iggy_binary_protocol::requests::messages::PollMessagesRequest;
use iggy_binary_protocol::requests::system::{BindSessionRequest, SessionIdentity};
use iggy_binary_protocol::requests::users::login_register::BindSecret;
@@ -36,30 +36,43 @@ pub trait BinaryTransport {
/// Sets the state of the client.
async fn set_state(&self, state: ClientState);
async fn publish_event(&self, event: DiagnosticEvent);
- async fn send_raw_with_response(&self, code: u32, payload: Bytes) -> Result<Bytes, IggyError>;
+ async fn send_raw_with_response(&self, code: u32, payload: Bytes) -> Result<Bytes, IggyError>
+ where
+ Self: Sync,
+ {
+ self.send_raw_with_context(code, payload, PartitionContext::default())
+ .await
+ }
+ /// Send with a captured partition incarnation and owner. Every retry must retain it.
+ /// A history refusal after an uncertain attempt must return
+ /// `TransientNotCommitted`, so callers cannot refresh and repeat that write.
+ async fn send_raw_with_context(
+ &self,
+ code: u32,
+ payload: Bytes,
+ context: PartitionContext,
+ ) -> Result<Bytes, IggyError>;
/// Route a store or delete offset request while retaining the membership connection.
+ /// Every retry keeps a captured `context`. Without one, the request takes the context
+ /// its route reports.
async fn send_offset_write_with_response(
&self,
code: u32,
payload: Bytes,
+ context: Option<PartitionContext>,
) -> Result<Bytes, IggyError>
where
- Self: Sync,
- {
- self.send_raw_with_response(code, payload).await
- }
+ Self: Sync;
+
/// Transports may route an auto-commit poll without moving the connection
/// that owns consumer-group membership.
async fn send_poll_with_response(
&self,
request: &PollMessagesRequest,
+ context: Option<PartitionContext>,
) -> Result<Bytes, IggyError>
where
- Self: Sync,
- {
- self.send_raw_with_response(POLL_MESSAGES_CODE, request.to_bytes())
- .await
- }
+ Self: Sync;
fn get_heartbeat_interval(&self) -> NonZeroIggyDuration;
/// Per-transport consumer-group + partitioning cache used to resolve
diff --git a/core/common/src/traits/consumer_offset_client.rs b/core/common/src/traits/consumer_offset_client.rs
index dc16d7a38..3a756924b 100644
--- a/core/common/src/traits/consumer_offset_client.rs
+++ b/core/common/src/traits/consumer_offset_client.rs
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-use crate::{Consumer, ConsumerOffsetInfo, Identifier, IggyError};
+use crate::{Consumer, ConsumerOffsetInfo, ConsumerPosition, Identifier, IggyError};
use async_trait::async_trait;
/// This trait defines the methods to interact with the consumer offset module.
@@ -25,6 +25,19 @@ use async_trait::async_trait;
/// [`IggyError::FeatureUnavailable`] there.
#[async_trait]
pub trait ConsumerOffsetClient {
+ /// Commit a captured position without substituting a newer incarnation or owner.
+ /// Authentication and permission to poll messages are required.
+ /// Returns [`IggyError::HistoryUnavailable`] (87) for a deleted or replaced incarnation,
+ /// [`IggyError::ConsumerGroupPartitionNotOwned`] (5009) for a lost owner,
+ /// or [`IggyError::TooManyConsumerOffsets`] (3024) at the new-key limit.
+ async fn store_consumer_position(
+ &self,
+ consumer: &Consumer,
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ position: ConsumerPosition,
+ ) -> Result<(), IggyError>;
+
/// Store the consumer offset for a specific consumer or consumer group for the given stream and topic by unique IDs or names.
///
/// Authentication is required, and the permission to poll the messages.
diff --git a/core/common/src/traits/message_client.rs b/core/common/src/traits/message_client.rs
index 9e18bab63..0f2105fa3 100644
--- a/core/common/src/traits/message_client.rs
+++ b/core/common/src/traits/message_client.rs
@@ -128,6 +128,10 @@ pub trait MessageClient {
/// implies uniqueness. Confirmation follows VSR quorum commit. Persisted
/// message durability also requires recoverable stable-storage copies on
/// the quorum.
+ ///
+ /// Binary clients refresh the partition context once after a definitive
+ /// `HistoryUnavailable` (87) refusal. A refusal after an uncertain send
+ /// keeps its original context and returns `TransientNotCommitted` (57).
async fn send_messages(
&self,
stream_id: &Identifier,
diff --git a/core/common/src/traits/partition_client.rs b/core/common/src/traits/partition_client.rs
index 650ca76af..cc04b6b82 100644
--- a/core/common/src/traits/partition_client.rs
+++ b/core/common/src/traits/partition_client.rs
@@ -27,6 +27,11 @@ pub trait PartitionClient {
///
/// Authentication is required, and the permission to manage the partitions.
///
+ /// A topic created with `PartitionResizePolicy::Fixed` rejects resizing
+ /// with `IggyError::PartitionResizeDisabled`. HTTP reports `HttpResponseError`.
+ /// Mutable topics do not guarantee per-key ordering across layouts or move
+ /// existing messages to new partitions.
+ ///
/// If the new partitions would take the node past `[metadata]
/// partitions_max`, a binary transport returns
/// `IggyError::PartitionsLimitReached`. `HttpClient` returns
@@ -42,6 +47,8 @@ pub trait PartitionClient {
/// For example, given a topic with 5 partitions, if you delete 2 partitions, the topic will have 3 partitions left (from 1 to 3).
///
/// Authentication is required, and the permission to manage the partitions.
+ /// Topics created with `PartitionResizePolicy::Fixed` reject this operation
+ /// with `IggyError::PartitionResizeDisabled`. HTTP reports `HttpResponseError`.
async fn delete_partitions(
&self,
stream_id: &Identifier,
diff --git a/core/common/src/types/consumer/consumer_offset_info.rs b/core/common/src/types/consumer/consumer_offset_info.rs
index 57d2676bd..b8b4cf591 100644
--- a/core/common/src/types/consumer/consumer_offset_info.rs
+++ b/core/common/src/types/consumer/consumer_offset_info.rs
@@ -15,6 +15,7 @@
// specific language governing permissions and limitations
// under the License.
+use crate::PartitionContext;
use serde::{Deserialize, Serialize};
/// `ConsumerOffsetInfo` represents the information about a consumer offset.
@@ -31,3 +32,14 @@ pub struct ConsumerOffsetInfo {
/// The stored offset by the consumer in the partition.
pub stored_offset: u64,
}
+
+/// A checkpoint retains the partition incarnation and owner that delivered its messages.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+pub struct ConsumerPosition {
+ /// Partition from which the message was consumed.
+ pub partition_id: u32,
+ /// Inclusive message offset to commit.
+ pub offset: u64,
+ /// Partition incarnation and owner captured by the poll; retain both for a delayed commit.
+ pub context: PartitionContext,
+}
diff --git a/core/common/src/types/message/polled_messages.rs b/core/common/src/types/message/polled_messages.rs
index d974fa479..d1ad9911b 100644
--- a/core/common/src/types/message/polled_messages.rs
+++ b/core/common/src/types/message/polled_messages.rs
@@ -17,7 +17,10 @@
use crate::{IggyMessage, IggyMessageHeader, error::IggyError};
use bytes::Bytes;
+use iggy_binary_protocol::WireDecode;
use iggy_binary_protocol::batch::{BATCH_HEADER_SIZE, BatchHeader, BatchMessageHeader};
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
+use iggy_binary_protocol::responses::messages::PollMessagesResponseHeader;
use serde::{Deserialize, Serialize};
use tracing::error;
@@ -29,6 +32,8 @@ use tracing::error;
/// - `messages`: the collection of messages.
#[derive(Debug, Serialize, Deserialize)]
pub struct PolledMessages {
+ /// Partition incarnation and owner captured by the accepted poll.
+ pub context: PartitionContext,
/// The identifier of the partition. An empty reply can carry a sentinel instead of a real
/// id: [`NO_ASSIGNED_PARTITION`](crate::NO_ASSIGNED_PARTITION) for a consumer-group member
/// that holds no partitions, or
@@ -46,6 +51,7 @@ pub struct PolledMessages {
impl PolledMessages {
pub fn empty() -> Self {
Self {
+ context: PartitionContext::default(),
partition_id: 0,
current_offset: 0,
count: 0,
@@ -55,7 +61,7 @@ impl PolledMessages {
}
impl PolledMessages {
- /// Decode a `PollMessages` response body: the 16-byte prefix followed by
+ /// Decode a `PollMessages` response body: the 40-byte prefix followed by
/// the served batch records (`[256B batch header][frames]`, deltas
/// resolved against the stamped bases).
///
@@ -63,28 +69,18 @@ impl PolledMessages {
/// [`IggyError::InvalidNumberEncoding`] on a short prefix;
/// [`IggyError::InvalidMessagePayloadLength`] on a malformed record.
pub fn from_bytes(bytes: Bytes) -> Result<Self, IggyError> {
- if bytes.len() < 16 {
- return Err(IggyError::InvalidNumberEncoding);
- }
- let partition_id = u32::from_le_bytes(
- bytes[0..4]
- .try_into()
- .map_err(|_| IggyError::InvalidNumberEncoding)?,
- );
- let current_offset = u64::from_le_bytes(
- bytes[4..12]
- .try_into()
- .map_err(|_| IggyError::InvalidNumberEncoding)?,
- );
- let count = u32::from_le_bytes(
- bytes[12..16]
- .try_into()
- .map_err(|_| IggyError::InvalidNumberEncoding)?,
- );
-
- let messages = messages_from_batches(bytes.slice(16..), count)?;
+ let (header, consumed) = PollMessagesResponseHeader::decode(&bytes)
+ .map_err(|_| IggyError::InvalidNumberEncoding)?;
+ let PollMessagesResponseHeader {
+ partition_id,
+ current_offset,
+ messages_count: count,
+ context,
+ } = header;
+ let messages = messages_from_batches(bytes.slice(consumed..), count)?;
Ok(Self {
+ context,
partition_id,
current_offset,
count,
@@ -97,7 +93,9 @@ impl PolledMessages {
/// values. Payload and user-header `Bytes` are zero-copy slices of the
/// response buffer.
fn messages_from_batches(buffer: Bytes, count: u32) -> Result<Vec<IggyMessage>, IggyError> {
- let mut messages = Vec::with_capacity(count as usize);
+ let mut messages = Vec::with_capacity(
+ (count as usize).min(buffer.len() / iggy_binary_protocol::batch::BATCH_MESSAGE_HEADER_SIZE),
+ );
let mut position = 0usize;
while position < buffer.len() {
let batch = BatchHeader::decode(&buffer[position..]).map_err(|decode_error| {
@@ -150,5 +148,8 @@ fn messages_from_batches(buffer: Bytes, count: u32) -> Result<Vec<IggyMessage>,
position = batch_end;
}
+ if messages.len() != count as usize {
+ return Err(IggyError::InvalidMessagePayloadLength);
+ }
Ok(messages)
}
diff --git a/core/common/src/types/message/polling_strategy.rs b/core/common/src/types/message/polling_strategy.rs
index 2939dfeaf..63ed838f9 100644
--- a/core/common/src/types/message/polling_strategy.rs
+++ b/core/common/src/types/message/polling_strategy.rs
@@ -17,8 +17,9 @@
use crate::types::message::polling_kind::PollingKind;
use crate::utils::timestamp::IggyTimestamp;
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
use serde::{Deserialize, Serialize};
-use serde_with::{DisplayFromStr, serde_as};
+use serde_with::{DisplayFromStr, json::JsonString, serde_as};
use std::fmt::Display;
/// Default value for the polling strategy.
@@ -42,11 +43,17 @@ pub struct PollingStrategy {
#[serde_as(as = "DisplayFromStr")]
#[serde(default = "PollingStrategy::default_value")]
pub value: u64,
+ /// Partition incarnation and owner captured with a continuation offset. A stale
+ /// context rejects the poll instead of interpreting that offset in another incarnation.
+ #[serde_as(as = "Option<JsonString>")]
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub context: Option<PartitionContext>,
}
impl Default for PollingStrategy {
fn default() -> Self {
Self {
+ context: None,
kind: PollingKind::Offset,
value: 0,
}
@@ -63,14 +70,23 @@ impl PollingStrategy {
/// Poll messages from the specified offset.
pub fn offset(value: u64) -> Self {
Self {
+ context: None,
kind: PollingKind::Offset,
value,
}
}
+ /// Continue at an offset under the incarnation and owner that produced it.
+ #[must_use]
+ pub fn with_context(mut self, context: PartitionContext) -> Self {
+ self.context = Some(context);
+ self
+ }
+
/// Poll messages from the specified timestamp.
pub fn timestamp(value: IggyTimestamp) -> Self {
Self {
+ context: None,
kind: PollingKind::Timestamp,
value: value.into(),
}
@@ -79,6 +95,7 @@ impl PollingStrategy {
/// Poll messages from the first message in the partition.
pub fn first() -> Self {
Self {
+ context: None,
kind: PollingKind::First,
value: 0,
}
@@ -87,6 +104,7 @@ impl PollingStrategy {
/// Poll messages from the last message in the partition.
pub fn last() -> Self {
Self {
+ context: None,
kind: PollingKind::Last,
value: 0,
}
@@ -100,6 +118,7 @@ impl PollingStrategy {
/// [poll recovery contract](crate::MessageClient::poll_messages) before retrying.
pub fn next() -> Self {
Self {
+ context: None,
kind: PollingKind::Next,
value: 0,
}
diff --git a/core/common/src/types/options/mod.rs b/core/common/src/types/options/mod.rs
index 717f6b0c4..d64a49de6 100644
--- a/core/common/src/types/options/mod.rs
+++ b/core/common/src/types/options/mod.rs
@@ -54,8 +54,10 @@
//! rather than a description to interpret.
mod durability;
+mod partition_resize_policy;
pub use durability::Durability;
+pub use partition_resize_policy::PartitionResizePolicy;
use std::collections::BTreeMap;
use std::str::FromStr;
@@ -213,6 +215,8 @@ pub mod topic_option_keys {
/// Message completion policy: `String`, either `replicated` or `persisted`.
pub const DURABILITY: &str = "durability";
pub const CONSUMER_OFFSET_DURABILITY: &str = "consumer_offset_durability";
+ /// Immutable partition-set policy: `String`, either `mutable` or `fixed`.
+ pub const PARTITION_RESIZE_POLICY: &str = "partition_resize_policy";
/// Flush the journal once it holds this many messages: `Uint32`.
/// Must be non-zero.
pub const MESSAGES_REQUIRED_TO_SAVE: &str = "messages_required_to_save";
@@ -446,6 +450,7 @@ pub const TOPIC_OPTION_KEYS: &[&str] = &[
topic_option_keys::SEGMENT_SIZE,
topic_option_keys::DURABILITY,
topic_option_keys::CONSUMER_OFFSET_DURABILITY,
+ topic_option_keys::PARTITION_RESIZE_POLICY,
topic_option_keys::MESSAGES_REQUIRED_TO_SAVE,
topic_option_keys::SIZE_OF_MESSAGES_REQUIRED_TO_SAVE,
topic_option_keys::PREALLOCATE_SEGMENTS,
@@ -715,6 +720,8 @@ pub struct TopicCreateOptions {
/// Message completion policy, independent of consumer-offset durability.
pub durability: Durability,
pub consumer_offset_durability: Durability,
+ /// Immutable after creation. `None` resolves to [`PartitionResizePolicy::Mutable`].
+ pub partition_resize_policy: Option<PartitionResizePolicy>,
/// Per-topic message-count flush threshold. `None` resolves to 1024.
/// `0` is rejected.
pub messages_required_to_save: Option<u32>,
@@ -820,10 +827,13 @@ impl TopicCreateOptions {
parsed.segment_size = (size != 0).then_some(IggyByteSize::from(size));
}
topic_option_keys::DURABILITY => {
- parsed.durability = Self::durability_value(entry, key)?;
+ parsed.durability = Self::string_value(entry, key)?;
}
topic_option_keys::CONSUMER_OFFSET_DURABILITY => {
- parsed.consumer_offset_durability = Self::durability_value(entry, key)?;
+ parsed.consumer_offset_durability = Self::string_value(entry, key)?;
+ }
+ topic_option_keys::PARTITION_RESIZE_POLICY => {
+ parsed.partition_resize_policy = Some(Self::string_value(entry, key)?);
}
topic_option_keys::MESSAGES_REQUIRED_TO_SAVE => {
let messages = parse_u32(entry, key)?;
@@ -867,6 +877,9 @@ impl TopicCreateOptions {
segment_size: self.segment_size.or(defaults.segment_size),
durability: self.durability,
consumer_offset_durability: self.consumer_offset_durability,
+ partition_resize_policy: self
+ .partition_resize_policy
+ .or(defaults.partition_resize_policy),
messages_required_to_save: self
.messages_required_to_save
.or(defaults.messages_required_to_save),
@@ -901,6 +914,13 @@ impl TopicCreateOptions {
/// See `raw_options_map`.
pub fn to_option_map(&self) -> Result<ResourceOptions, IggyError> {
let mut options = raw_options_map(&self.raw)?;
+ if let Some(policy) = self.partition_resize_policy {
+ options.insert(
+ HeaderKey::from_str(topic_option_keys::PARTITION_RESIZE_POLICY)
+ .expect("catalog key is a valid header key"),
+ OptionValue::explicit(HeaderValue::from_str(policy.as_ref())?),
+ );
+ }
if let Some(compression_algorithm) = self.compression_algorithm {
options.insert(
HeaderKey::from_str(topic_option_keys::COMPRESSION_ALGORITHM)
@@ -991,6 +1011,12 @@ impl TopicCreateOptions {
// Typed fields are inserted over the raw entries, matching the
// collision rule `to_wire` applies.
let mut options = self.raw.clone();
+ if let Some(policy) = self.partition_resize_policy {
+ options.insert(
+ topic_option_keys::PARTITION_RESIZE_POLICY.to_owned(),
+ policy.to_string(),
+ );
+ }
if let Some(segment_size) = self.segment_size {
options.insert(
topic_option_keys::SEGMENT_SIZE.to_owned(),
@@ -1048,6 +1074,15 @@ impl TopicCreateOptions {
supplied: &WireOptions,
) -> Result<WireOptions, IggyError> {
let mut derived = ResourceOptions::new();
+ if self.partition_resize_policy.is_none() {
+ derived.insert(
+ HeaderKey::from_str(topic_option_keys::PARTITION_RESIZE_POLICY)
+ .expect("catalog key is a valid header key"),
+ OptionValue::derived(HeaderValue::from_str(
+ PartitionResizePolicy::default().as_ref(),
+ )?),
+ );
+ }
if self.compression_algorithm.is_none() {
derived.insert(
HeaderKey::from_str(topic_option_keys::COMPRESSION_ALGORITHM)
@@ -1156,10 +1191,10 @@ impl TopicCreateOptions {
}
parsed
}
- fn durability_value(
+ fn string_value<T: FromStr>(
entry: &WireUserHeaderEntry<'_>,
key: &str,
- ) -> Result<Durability, IggyError> {
+ ) -> Result<T, IggyError> {
if entry.value_kind.0 != HeaderKind::String.as_code() {
return Err(IggyError::InvalidOptionValue(key.to_owned()));
}
@@ -1281,12 +1316,98 @@ mod tests {
preallocate_segments: Some(false),
partitions_count: None,
consumer_offset_durability: Durability::Replicated,
+ partition_resize_policy: Some(PartitionResizePolicy::Fixed),
raw: BTreeMap::new(),
};
let parsed = TopicCreateOptions::parse(&options.to_wire().unwrap()).unwrap();
assert_eq!(parsed, options);
}
+ #[test]
+ fn given_omitted_resize_policy_when_resolving_defaults_should_persist_mutable_as_derived() {
+ let options = TopicCreateOptions::default();
+ let supplied = options.to_wire().unwrap();
+ let derived = options
+ .derived_block(
+ CompressionAlgorithm::default(),
+ IggyExpiry::default(),
+ MaxTopicSize::default(),
+ TopicRuntimeDefaults::default(),
+ &supplied,
+ )
+ .unwrap();
+ let stored = crate::wire_conversions::resource_options_from_wire(&derived, false).unwrap();
+ let policy =
+ &stored[&HeaderKey::from_str(topic_option_keys::PARTITION_RESIZE_POLICY).unwrap()];
+ assert_eq!(policy.value.as_bytes(), b"mutable");
+ assert!(!policy.explicit);
+ assert_eq!(
+ TopicCreateOptions::from_resource_options(&stored).partition_resize_policy,
+ Some(PartitionResizePolicy::Mutable),
+ );
+ assert!(
+ !supplied
+ .into_iter()
+ .any(|entry| entry.key == topic_option_keys::PARTITION_RESIZE_POLICY.as_bytes())
+ );
+ }
+
+ #[test]
+ fn given_raw_resize_policy_when_encoding_should_match_binary_and_http_values() {
+ let mut options = TopicCreateOptions {
+ raw: BTreeMap::from([(
+ topic_option_keys::PARTITION_RESIZE_POLICY.to_owned(),
+ "fixed".to_owned(),
+ )]),
+ ..TopicCreateOptions::default()
+ };
+ assert_eq!(
+ TopicCreateOptions::parse(&options.to_wire().unwrap())
+ .unwrap()
+ .partition_resize_policy,
+ Some(PartitionResizePolicy::Fixed),
+ );
+ assert_eq!(
+ options.to_string_options().unwrap()[topic_option_keys::PARTITION_RESIZE_POLICY],
+ "fixed"
+ );
+
+ options.partition_resize_policy = Some(PartitionResizePolicy::Mutable);
+ assert_eq!(
+ TopicCreateOptions::parse(&options.to_wire().unwrap())
+ .unwrap()
+ .partition_resize_policy,
+ Some(PartitionResizePolicy::Mutable),
+ );
+ assert_eq!(
+ options.to_string_options().unwrap()[topic_option_keys::PARTITION_RESIZE_POLICY],
+ "mutable"
+ );
+ }
+
+ #[test]
+ fn given_invalid_resize_policy_when_parsing_should_reject_value_and_kind() {
+ for value in [
+ HeaderValue::from_str("unknown").unwrap(),
+ HeaderValue::from(1_u32),
+ HeaderValue::from(true),
+ ] {
+ let options = ResourceOptions::from([(
+ HeaderKey::from_str(topic_option_keys::PARTITION_RESIZE_POLICY).unwrap(),
+ OptionValue::explicit(value),
+ )]);
+ let wire =
+ crate::wire_conversions::resource_options_to_wire(&options, OptionsProvenance::All)
+ .unwrap();
+ assert_eq!(
+ TopicCreateOptions::parse(&wire),
+ Err(IggyError::InvalidOptionValue(
+ topic_option_keys::PARTITION_RESIZE_POLICY.to_owned()
+ )),
+ );
+ }
+ }
+
#[test]
fn partitions_count_never_enters_the_options_block() {
// It is a fixed field of the CreateTopic command, so encoding it as a
diff --git a/core/common/src/http/consumer_offsets/store_consumer_offset.rs b/core/common/src/types/options/partition_resize_policy.rs
similarity index 56%
copy from core/common/src/http/consumer_offsets/store_consumer_offset.rs
copy to core/common/src/types/options/partition_resize_policy.rs
index 903ed82c3..420f26aff 100644
--- a/core/common/src/http/consumer_offsets/store_consumer_offset.rs
+++ b/core/common/src/types/options/partition_resize_policy.rs
@@ -15,13 +15,30 @@
// specific language governing permissions and limitations
// under the License.
-use crate::Consumer;
use serde::{Deserialize, Serialize};
+use strum::{AsRefStr, Display, EnumString};
-#[derive(Debug, Serialize, Deserialize)]
-pub struct StoreConsumerOffset {
- #[serde(flatten)]
- pub consumer: Consumer,
- pub partition_id: Option<u32>,
- pub offset: u64,
+/// Immutable topic policy for adding and deleting partitions.
+/// Retention and deletion of the topic remain destructive under either policy.
+#[derive(
+ Debug,
+ Clone,
+ Copy,
+ Default,
+ PartialEq,
+ Eq,
+ Serialize,
+ Deserialize,
+ Display,
+ EnumString,
+ AsRefStr,
+)]
+#[serde(rename_all = "lowercase")]
+#[strum(serialize_all = "lowercase", ascii_case_insensitive)]
+pub enum PartitionResizePolicy {
+ /// Resize is allowed without a bounded cutover or per-key ordering across layouts.
+ #[default]
+ Mutable,
+ /// The partition set is fixed at topic creation.
+ Fixed,
}
diff --git a/core/common/src/types/partition/mod.rs b/core/common/src/types/partition/mod.rs
index fdd7161ab..b00e77aa0 100644
--- a/core/common/src/types/partition/mod.rs
+++ b/core/common/src/types/partition/mod.rs
@@ -19,6 +19,8 @@ use crate::utils::byte_size::IggyByteSize;
use crate::utils::timestamp::IggyTimestamp;
use serde::{Deserialize, Serialize};
+pub use iggy_binary_protocol::primitives::partition_history::PartitionContext;
+
/// `Partition` represents the information about a partition.
/// It consists of the following fields:
/// - `id`: unique identifier of the partition.
@@ -27,6 +29,7 @@ use serde::{Deserialize, Serialize};
/// - `current_offset`: the current offset of the partition.
/// - `size_bytes`: the size of the partition in bytes.
/// - `messages_count`: the number of messages in the partition.
+/// - `context`: the partition incarnation and the metadata operation the details were read at.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct Partition {
/// Unique identifier of the partition.
@@ -41,4 +44,7 @@ pub struct Partition {
pub size: IggyByteSize,
/// The number of messages in the partition.
pub messages_count: u64,
+ /// The partition incarnation and the metadata operation the details were read at. The
+ /// owner generation is zero because the details belong to no consumer group.
+ pub context: PartitionContext,
}
diff --git a/core/common/src/wire_conversions.rs b/core/common/src/wire_conversions.rs
index ea6782b2d..139768a20 100644
--- a/core/common/src/wire_conversions.rs
+++ b/core/common/src/wire_conversions.rs
@@ -160,6 +160,7 @@ impl From<PartitionResponse> for Partition {
current_offset: w.current_offset,
size: IggyByteSize::from(w.size_bytes),
messages_count: w.messages_count,
+ context: w.context,
}
}
}
diff --git a/core/configs/src/common/validators.rs b/core/configs/src/common/validators.rs
index e2d2fce3c..d649a7715 100644
--- a/core/configs/src/common/validators.rs
+++ b/core/configs/src/common/validators.rs
@@ -309,6 +309,30 @@ mod logging_tests {
use iggy_common::{IggyByteSize, IggyDuration};
use std::time::Duration;
+ #[test]
+ fn given_print_interval_samples_when_validating_should_accept_zero_or_at_least_one_second() {
+ let one_second = Duration::from_secs(1);
+ let samples = [
+ Duration::ZERO,
+ Duration::from_nanos(1),
+ Duration::from_micros(1),
+ Duration::from_millis(1),
+ Duration::from_millis(500),
+ Duration::from_nanos(999_999_999),
+ one_second,
+ Duration::from_nanos(1_000_000_001),
+ Duration::from_secs(10),
+ ];
+ for interval in samples {
+ let accepted = with_sysinfo_print_interval(interval).validate().is_ok();
+ assert_eq!(
+ accepted,
+ interval.is_zero() || interval >= one_second,
+ "sysinfo_print_interval {interval:?}"
+ );
+ }
+ }
+
fn with_sysinfo_print_interval(interval: Duration) -> LoggingConfig {
LoggingConfig {
sysinfo_print_interval: IggyDuration::new(interval),
@@ -316,24 +340,6 @@ mod logging_tests {
}
}
- #[test]
- fn sub_second_sysinfo_print_interval_is_rejected() {
- let config = with_sysinfo_print_interval(Duration::from_millis(999));
- assert!(config.validate().is_err());
- }
-
- #[test]
- fn one_second_sysinfo_print_interval_is_accepted() {
- let config = with_sysinfo_print_interval(Duration::from_secs(1));
- assert!(config.validate().is_ok());
- }
-
- #[test]
- fn disabled_sysinfo_print_interval_is_accepted() {
- let config = with_sysinfo_print_interval(Duration::ZERO);
- assert!(config.validate().is_ok());
- }
-
#[test]
fn level_that_is_not_a_level_is_rejected() {
let config = LoggingConfig {
diff --git a/core/configs/src/server_config/cluster.rs b/core/configs/src/server_config/cluster.rs
index 140847951..b0c787b53 100644
--- a/core/configs/src/server_config/cluster.rs
+++ b/core/configs/src/server_config/cluster.rs
@@ -308,7 +308,8 @@ pub struct ClusterConfig {
/// Applies per plane: the metadata superblock, and each PARTITION's own. While
/// wedged the group is already fenced quorum-invisible and peers elect around
/// it; this converts the log-only limp into a distinct exit status a
- /// supervisor can act on. Zero (and the `0` /
+ /// supervisor can act on. It is also the deadline of each running
+ /// partition file job (`sharding.partition_io_capacity`). Zero (and the `0` /
/// `disabled` / `unlimited` sentinels, which all parse to zero) disables
/// the fail-stop; nonzero values below
/// `MIN_SUPERBLOCK_WEDGED_FATAL_TIMEOUT` are rejected at boot.
diff --git a/core/consensus/src/impls.rs b/core/consensus/src/impls.rs
index 5dc4d84a5..cf8b881b9 100644
--- a/core/consensus/src/impls.rs
+++ b/core/consensus/src/impls.rs
@@ -27,9 +27,9 @@ use crate::{
use bit_set::BitSet;
use clock::{Clock, IggySystemClock};
use iggy_binary_protocol::{
- Command, ConsensusHeader, DoViewChangeHeader, GenericHeader, PrepareHeader, PrepareOkHeader,
- ReplyHeader, RequestStartViewHeader, RoutedRequestHeader, StartViewChangeHeader,
- StartViewHeader, frame_body,
+ Command, ConsensusHeader, DoViewChangeHeader, GenericHeader, Operation, PrepareHeader,
+ PrepareOkHeader, ReplyHeader, RequestStartViewHeader, RoutedRequestHeader,
+ StartViewChangeHeader, StartViewHeader, frame_body,
};
use iggy_common::IggyTimestamp;
use iggy_common::calculate_checksum;
@@ -989,10 +989,6 @@ impl Pipeline for LocalPipeline {
Self::pop_request(self)
}
- fn request_head(&self) -> Option<&Self::Request> {
- Self::request_head(self)
- }
-
fn request_queue_len(&self) -> usize {
Self::request_queue_len(self)
}
@@ -4324,6 +4320,34 @@ where
type Consensus = VsrConsensus<B, P>;
fn project(self, consensus: &Self::Consensus) -> Message<PrepareHeader> {
+ let message = if matches!(
+ self.header().operation,
+ Operation::StoreConsumerOffset | Operation::DeleteConsumerOffset
+ ) {
+ let header = *self.header();
+ let payload_end = header.size as usize;
+ let total_size = payload_end
+ + iggy_binary_protocol::requests::consumer_offsets::PREPARED_OWNER_GENERATION_SIZE;
+ let mut prepared = if message_bus::is_auto_commit_client(header.client)
+ && self.as_slice().len() == total_size
+ {
+ self
+ } else {
+ let mut prepared = Self::new(total_size);
+ prepared.as_mut_slice()[size_of::<RoutedRequestHeader>()..payload_end]
+ .copy_from_slice(self.body());
+ prepared
+ };
+ prepared.as_mut_slice()[payload_end..total_size]
+ .copy_from_slice(&header.owner_generation.to_le_bytes());
+ prepared.transmute_header(|_, expanded: &mut RoutedRequestHeader| {
+ *expanded = header;
+ expanded.size =
+ u32::try_from(total_size).expect("admitted offset prepare fits u32");
+ })
+ } else {
+ self
+ };
let op = consensus.sequencer.current_sequence() + 1;
// Primary stamps the injected clock once at prepare-build (wall time
// in production, virtual under the simulator); the value is
@@ -4342,13 +4366,13 @@ where
// chain are sealed too, for both planes, by `seal_prepare_checksum` below;
// they exclude `view`, which is what lets a restamp leave them valid.
//
- // Metadata plane only. A partition produce prepare already carries a verified
+ // A partition produce prepare already carries a verified
// `batch_checksum` over the same bytes, so a second full-payload pass is pure
// cost on the produce path, and it would describe the WRONG bytes:
// `stamp_prepare_for_persistence` rewrites the command header INSIDE this
// sealed region before the entry is journaled. Leaving those prepares at `0`
- // is the designed "nothing to verify" sentinel: partition prepares
- // leave message-body integrity to the batch checksums.
+ // is the designed "nothing to verify" sentinel for produce. Offset and
+ // control bodies include their fencing identities in this checksum.
//
// TODO(consensus): a partition prepare's `checksum` covers its header alone,
// so two at one op with matching header fields are indistinguishable however
@@ -4368,16 +4392,16 @@ where
//
// Bounded by `size`, the range every verifier re-reads; the prepare
// inherits it verbatim below.
- let checksum_body = if consensus.group == METADATA_GROUP {
+ let checksum_body = if message.header().operation == Operation::SendMessages {
+ 0
+ } else {
u128::from(calculate_checksum(frame_body(
- self.as_slice(),
- self.header().size,
+ message.as_slice(),
+ message.header().size,
)))
- } else {
- 0
};
- let prepared = self.transmute_header(|old, new| {
+ let prepared = message.transmute_header(|old, new| {
*new = PrepareHeader {
cluster: consensus.cluster,
size: old.size,
@@ -4405,6 +4429,7 @@ where
// ops (and the authenticated user on Register), so the in-apply
// RBAC gate resolves the same identity on every backup.
user_id: old.user_id,
+ partition_incarnation: old.partition_incarnation,
..Default::default()
}
});
@@ -5153,15 +5178,15 @@ mod timestamp_clamp_tests {
}
#[test]
- fn given_partition_namespace_when_projecting_should_leave_the_body_unsealed() {
- // The body seal is metadata-only. A partition produce prepare already carries a
+ fn given_partition_operation_when_projecting_should_seal_controls_but_not_batches() {
+ // A partition produce prepare already carries a
// verified `batch_checksum` over the same bytes, so a second full-payload hash
// is pure cost on the produce path, and it would describe bytes that never reach
// the journal: `stamp_prepare_for_persistence` rewrites the command header
// inside the sealed region. `0` is the designed "nothing to verify" sentinel, so
// a durable partition journal skips verification rather than reading every entry
// as corrupt.
- let seal = |namespace: u64| -> u128 {
+ let seal = |namespace: u64, operation: Operation| -> u128 {
// Fixed clock: `project` stamps the prepare timestamp, and Miri covers this
// crate, where a real clock read is an unsupported syscall.
let consensus = VsrConsensus::with_clock(
@@ -5184,18 +5209,20 @@ mod timestamp_clamp_tests {
header.command = Command::Request;
header.client = 1;
header.request = 1;
- header.operation = iggy_binary_protocol::Operation::SendMessages;
+ header.operation = operation;
header.size = u32::try_from(header_size + body.len()).expect("fits u32");
msg.project(&consensus).header().checksum_body
};
assert_ne!(
- seal(METADATA_GROUP),
+ seal(METADATA_GROUP, Operation::CreateStream),
0,
"a metadata prepare must be sealed: the WAL scan verifies it after a crash"
);
+ assert_ne!(seal(1, Operation::InstallConsumerGroupOwner), 0);
+ assert_ne!(seal(1, Operation::TransitionPartitionHistory), 0);
assert_eq!(
- seal(1),
+ seal(1, Operation::SendMessages),
0,
"a partition prepare must be left unsealed, since its sealed region is \
rewritten before it is journaled"
diff --git a/core/consensus/src/lib.rs b/core/consensus/src/lib.rs
index c952d399d..1de069145 100644
--- a/core/consensus/src/lib.rs
+++ b/core/consensus/src/lib.rs
@@ -107,11 +107,6 @@ pub trait Pipeline {
fn pop_request(&mut self) -> Option<Self::Request> {
None
}
-
- /// Inspect the original queue head without transferring reply ownership.
- fn request_head(&self) -> Option<&Self::Request> {
- None
- }
}
pub type RequestMessage<C> = <C as Consensus>::Message<<C as Consensus>::RoutedRequestHeader>;
diff --git a/core/consensus/src/observability.rs b/core/consensus/src/observability.rs
index ffbf1f289..d85a4da4e 100644
--- a/core/consensus/src/observability.rs
+++ b/core/consensus/src/observability.rs
@@ -703,6 +703,10 @@ pub const fn operation_as_str(operation: Operation) -> &'static str {
Operation::TruncatePartition => "truncate_partition",
Operation::FinalizeSession => "finalize_session",
Operation::RetireSession => "retire_session",
+ Operation::CompleteLifecycle => "complete_lifecycle",
+ Operation::TransitionPartitionHistory => "transition_partition_history",
+ Operation::RetireConsumerGroupOwners => "retire_consumer_group_owners",
+ Operation::InstallConsumerGroupOwner => "install_consumer_group_owner",
Operation::CreateStream => "create_stream",
Operation::UpdateStream => "update_stream",
Operation::DeleteStream => "delete_stream",
diff --git a/core/integration/src/harness/context.rs b/core/integration/src/harness/context.rs
index 57e409039..63e823a1a 100644
--- a/core/integration/src/harness/context.rs
+++ b/core/integration/src/harness/context.rs
@@ -25,6 +25,7 @@ use std::thread;
use uuid::Uuid;
const TEST_CLEANUP_DISABLED_ENV_VAR: &str = "IGGY_TEST_CLEANUP_DISABLED";
+const TEST_DIRECTORY_NAME_MAX_BYTES: usize = 255;
/// Global registry mapping test names to their log directories.
static TEST_DIRECTORIES: LazyLock<RwLock<HashMap<String, PathBuf>>> =
@@ -70,7 +71,9 @@ impl TestContext {
pub fn new(test_name: Option<String>, cleanup: bool) -> Result<Self, TestBinaryError> {
let test_name = test_name.unwrap_or_else(Self::derive_test_name);
let uuid_suffix = Uuid::new_v4().to_string()[..8].to_string();
- let dir_name = format!("{}_{}", sanitize_path(&test_name), uuid_suffix);
+ let mut prefix = sanitize_path(&test_name);
+ prefix.truncate(TEST_DIRECTORY_NAME_MAX_BYTES - uuid_suffix.len() - 1);
+ let dir_name = format!("{prefix}_{uuid_suffix}");
let base_dir = (*TEST_LOGS_DIR).join(dir_name);
@@ -193,4 +196,15 @@ mod tests {
);
assert!(ctx.mcp_stdout_path(1).ends_with("mcp_1_stdout.log"));
}
+
+ #[test]
+ fn given_long_parameterized_name_when_creating_context_should_preserve_its_identity() {
+ let name = "consumer_λ::".repeat(TEST_DIRECTORY_NAME_MAX_BYTES);
+ let mut first = TestContext::new(Some(name.clone()), true).unwrap();
+ let second = TestContext::new(Some(name.clone()), true).unwrap();
+ first.ensure_created().unwrap();
+ assert_eq!(first.test_name(), name);
+ assert_ne!(first.base_dir(), second.base_dir());
+ assert_eq!(get_test_directory(&name).unwrap(), first.base_dir());
+ }
}
diff --git a/core/integration/src/harness/helpers.rs b/core/integration/src/harness/helpers.rs
index c07f681a8..ec2aef4ce 100644
--- a/core/integration/src/harness/helpers.rs
+++ b/core/integration/src/harness/helpers.rs
@@ -16,11 +16,66 @@
// under the License.
use iggy::prelude::{
- DEFAULT_ROOT_PASSWORD, DEFAULT_ROOT_USERNAME, GlobalPermissions, Identifier, IdentityInfo,
- Permissions, StreamClient, UserClient, UserStatus,
+ ConsumerGroupClient, ConsumerGroupDetails, DEFAULT_ROOT_PASSWORD, DEFAULT_ROOT_USERNAME,
+ GlobalPermissions, Identifier, IdentityInfo, Permissions, StreamClient, UserClient, UserStatus,
};
+use std::collections::HashSet;
+use std::time::Duration;
+use tokio::time::{Instant, sleep};
pub const USER_PASSWORD: &str = "secret";
+const ASSIGNMENT_POLL_INTERVAL: Duration = Duration::from_millis(100);
+
+/// Wait for the expected assignment distribution, checking uniqueness during convergence.
+pub async fn wait_for_consumer_group_assignment(
+ client: &impl ConsumerGroupClient,
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ group_id: &Identifier,
+ members_count: u32,
+ budget: Duration,
+) -> ConsumerGroupDetails {
+ let deadline = Instant::now() + budget;
+ loop {
+ let group = client
+ .get_consumer_group(stream_id, topic_id, group_id)
+ .await
+ .expect("read consumer group")
+ .expect("consumer group exists");
+ let mut assigned = HashSet::new();
+ for member in &group.members {
+ for partition in &member.partitions {
+ assert!(
+ assigned.insert(*partition),
+ "duplicate partition owner: {group:?}"
+ );
+ }
+ }
+ let fewest = group
+ .members
+ .iter()
+ .map(|member| member.partitions_count)
+ .min()
+ .unwrap_or(0);
+ let most = group
+ .members
+ .iter()
+ .map(|member| member.partitions_count)
+ .max()
+ .unwrap_or(0);
+ if group.members_count == members_count
+ && (members_count == 0 || assigned.len() == group.partitions_count as usize)
+ && most - fewest <= 1
+ {
+ return group;
+ }
+ assert!(
+ Instant::now() < deadline,
+ "ownership did not converge to {members_count} members within {budget:?}: {group:?}"
+ );
+ sleep(ASSIGNMENT_POLL_INTERVAL).await;
+ }
+}
/// Login as root user.
pub async fn login_root(client: &impl UserClient) -> Result<IdentityInfo, iggy_common::IggyError> {
diff --git a/core/integration/src/harness/mod.rs b/core/integration/src/harness/mod.rs
index 17edb6741..bfb83efcc 100644
--- a/core/integration/src/harness/mod.rs
+++ b/core/integration/src/harness/mod.rs
@@ -67,6 +67,7 @@ pub use traits::{IggyServerDependent, Restartable, TestBinary};
pub use helpers::{
USER_PASSWORD, assert_clean_system, create_user, delete_user, login_root, login_user,
+ wait_for_consumer_group_assignment,
};
pub use fixture::TestFixture;
diff --git a/core/integration/tests/cluster/client_table_adversarial.rs b/core/integration/tests/cluster/client_table_adversarial.rs
index ee6fbf44a..fc956f170 100644
--- a/core/integration/tests/cluster/client_table_adversarial.rs
+++ b/core/integration/tests/cluster/client_table_adversarial.rs
@@ -210,6 +210,13 @@ async fn given_partition_batches_spent_request_ids_when_a_metadata_request_is_re
)
.await
.expect("create topic");
+ let context = crate::server::raw_tcp::partition_context(
+ &setup,
+ &Identifier::named(GAP_STREAM).unwrap(),
+ &Identifier::named(GAP_TOPIC).unwrap(),
+ 0,
+ )
+ .await;
drop(setup);
let addr = tcp_addr(harness);
@@ -217,7 +224,7 @@ async fn given_partition_batches_spent_request_ids_when_a_metadata_request_is_re
for request in 1..=GAP_BATCHES {
let batch = send_messages_payload(u128::from(request));
- commit_batch(&mut stream, CLIENT_A, session, request, &batch).await;
+ commit_batch(&mut stream, CLIENT_A, session, request, &batch, context).await;
}
let payload = create_stream_payload("adv-m-after-gap");
@@ -405,8 +412,9 @@ async fn commit_batch(
session: u64,
request: u64,
body: &Bytes,
+ context: PartitionContext,
) {
- let header = RequestHeader {
+ let mut header = RequestHeader {
command: Command::Request,
operation: Operation::SendMessages,
size: u32::try_from(HEADER_SIZE + body.len()).unwrap(),
@@ -415,6 +423,7 @@ async fn commit_batch(
request,
..Default::default()
};
+ context.stamp(&mut header);
let deadline = Instant::now() + COMMIT_BUDGET;
loop {
match exchange(stream, &header, body).await {
diff --git a/core/integration/tests/cluster/client_table_restart.rs b/core/integration/tests/cluster/client_table_restart.rs
index c5339b219..8a3dcc1bf 100644
--- a/core/integration/tests/cluster/client_table_restart.rs
+++ b/core/integration/tests/cluster/client_table_restart.rs
@@ -53,7 +53,7 @@ use crate::server::raw_tcp::TEST_BIND_SECRET;
use bytes::Bytes;
use iggy::prelude::*;
use iggy_binary_protocol::codec::{WireDecode, WireEncode};
-use iggy_binary_protocol::codes::{PING_CODE, POLL_MESSAGES_CODE};
+use iggy_binary_protocol::codes::{GET_POLL_ROUTING_CODE, PING_CODE, POLL_MESSAGES_CODE};
use iggy_binary_protocol::consensus::{
Command, Operation, ReplyHeader, RequestHeader, read_size_field, result_code,
result_section_len,
@@ -66,7 +66,7 @@ use iggy_binary_protocol::requests::streams::CreateStreamRequest;
use iggy_binary_protocol::requests::system::{BindSessionRequest, SessionIdentity};
use iggy_binary_protocol::requests::users::LoginRegisterRequest;
use iggy_binary_protocol::requests::users::login_register::BindSecret;
-use iggy_binary_protocol::responses::messages::PollMessagesResponse;
+use iggy_binary_protocol::responses::messages::{PollMessagesResponse, PollRoutingResponse};
use iggy_binary_protocol::responses::users::LoginRegisterResponse;
use iggy_binary_protocol::{
ClientVersionInfo, HEADER_SIZE, IGGY_PROTOCOL_VERSION, WireIdentifier, WireName, WireOptions,
@@ -733,7 +733,15 @@ async fn given_full_registry_when_new_clients_arrive_should_preserve_live_member
exchange(&mut member, &header, &join).await.verdict(),
Verdict::Success(_)
));
- let original = liveness_group(&observer).await;
+ let original = integration::harness::wait_for_consumer_group_assignment(
+ &observer,
+ &stream,
+ &topic,
+ &group,
+ 1,
+ COMMIT_BUDGET,
+ )
+ .await;
let mut poll_request = PollMessagesRequest {
consumer: WireConsumer::consumer_group(WireIdentifier::named(LIVENESS_GROUP).unwrap()),
stream_id: WireIdentifier::named(LIVENESS_STREAM).unwrap(),
@@ -744,8 +752,19 @@ async fn given_full_registry_when_new_clients_arrive_should_preserve_live_member
auto_commit: true,
};
let first_poll = poll_request.to_bytes();
+ let mut routing_header = request_header(Operation::NonReplicated, session, 0, first_poll.len());
+ routing_header.reserved[..size_of::<u32>()]
+ .copy_from_slice(&GET_POLL_ROUTING_CODE.to_le_bytes());
+ let Exchange::Reply {
+ status: 0, body, ..
+ } = exchange(&mut member, &routing_header, &first_poll).await
+ else {
+ panic!("the installed owner must discover its poll context");
+ };
+ let context = PollRoutingResponse::decode_from(&body).unwrap().context;
let mut first_header = request_header(Operation::NonReplicated, session, 0, first_poll.len());
first_header.reserved[..size_of::<u32>()].copy_from_slice(&POLL_MESSAGES_CODE.to_le_bytes());
+ context.stamp(&mut first_header);
let deadline = Instant::now() + COMMIT_BUDGET;
loop {
let Exchange::Reply {
@@ -801,6 +820,7 @@ async fn given_full_registry_when_new_clients_arrive_should_preserve_live_member
let poll = poll_request.to_bytes();
let mut header = request_header(Operation::NonReplicated, session, 0, poll.len());
header.reserved[..size_of::<u32>()].copy_from_slice(&POLL_MESSAGES_CODE.to_le_bytes());
+ context.stamp(&mut header);
let deadline = Instant::now() + LIVENESS_OBSERVATION;
while Instant::now() < deadline {
assert!(
@@ -1146,11 +1166,15 @@ async fn bind_group_member_on_backup(
let (backup_connection, new_session) = register(harness.node(backup).tcp_addr().unwrap()).await;
assert_eq!(new_session, session);
drop(original);
- let members = observer
- .get_consumer_group(&stream, &topic, &group)
- .await
- .unwrap()
- .unwrap();
+ let members = integration::harness::wait_for_consumer_group_assignment(
+ observer,
+ &stream,
+ &topic,
+ &group,
+ 1,
+ COMMIT_BUDGET,
+ )
+ .await;
assert_eq!(members.members_count, 1);
(backup_connection, session, primary, backup)
}
diff --git a/core/integration/tests/cluster/consumer_offset_quota.rs b/core/integration/tests/cluster/consumer_offset_quota.rs
index e71e5360b..bb47b51ab 100644
--- a/core/integration/tests/cluster/consumer_offset_quota.rs
+++ b/core/integration/tests/cluster/consumer_offset_quota.rs
@@ -85,6 +85,7 @@ async fn given_replicated_partition_when_no_ack_offsets_mutate_should_converge_a
let address = harness.node(leader).tcp_addr().expect("leader TCP address");
let mut raw = raw_tcp::connect_to(address).await;
let session = raw_tcp::register_root(&mut raw, RAW_CLIENT_ID).await;
+ let context = raw_tcp::partition_context(&client, &stream, &topic, PARTITION_ID).await;
let wire_stream = WireIdentifier::Numeric(stream_details.id);
let wire_topic = WireIdentifier::Numeric(topic_details.id);
let raw_consumer_id = 41;
@@ -98,13 +99,14 @@ async fn given_replicated_partition_when_no_ack_offsets_mutate_should_converge_a
ack: AckLevel::NoAck,
}
.to_bytes();
- let store_header = raw_tcp::request_header(
+ let mut store_header = raw_tcp::request_header(
Operation::StoreConsumerOffset,
RAW_CLIENT_ID,
session,
1,
store.len(),
);
+ context.stamp(&mut store_header);
let (reply, _) = raw_tcp::exchange(&mut raw, &store_header, &store).await;
assert_eq!(raw_tcp::reply_status(&reply), 0);
wait_for_file_state(
@@ -124,13 +126,14 @@ async fn given_replicated_partition_when_no_ack_offsets_mutate_should_converge_a
ack: AckLevel::NoAck,
}
.to_bytes();
- let delete_header = raw_tcp::request_header(
+ let mut delete_header = raw_tcp::request_header(
Operation::DeleteConsumerOffset,
RAW_CLIENT_ID,
session,
2,
delete.len(),
);
+ context.stamp(&mut delete_header);
let (reply, _) = raw_tcp::exchange(&mut raw, &delete_header, &delete).await;
assert_eq!(raw_tcp::reply_status(&reply), 0);
wait_for_file_state(
@@ -200,6 +203,10 @@ async fn given_replicated_partition_when_no_ack_offsets_mutate_should_converge_a
.join_consumer_group(&stream, &topic, &group)
.await
.expect("join group");
+ integration::harness::wait_for_consumer_group_assignment(
+ &client, &stream, &topic, &group, 1, WAIT,
+ )
+ .await;
client
.store_consumer_offset(
&Consumer::group(group.clone()),
diff --git a/core/integration/tests/cluster/fast_primary_rejoin.rs b/core/integration/tests/cluster/fast_primary_rejoin.rs
index 32a518d2b..bbc5645ec 100644
--- a/core/integration/tests/cluster/fast_primary_rejoin.rs
+++ b/core/integration/tests/cluster/fast_primary_rejoin.rs
@@ -456,6 +456,7 @@ async fn given_http_writes_on_a_rejoined_backup_when_the_primary_moved_should_fo
let offsets_path = format!("/streams/{STREAM_NAME}/topics/{TOPIC_NAME}/consumer-offsets");
let consumer_id = Identifier::numeric(1).expect("valid consumer id");
let store = StoreConsumerOffset {
+ context: None,
consumer: Consumer::new(consumer_id),
partition_id: Some(PARTITION_ID),
offset: 0,
diff --git a/core/integration/tests/cluster/partition_dedup.rs b/core/integration/tests/cluster/partition_dedup.rs
index c8a564822..84a644afd 100644
--- a/core/integration/tests/cluster/partition_dedup.rs
+++ b/core/integration/tests/cluster/partition_dedup.rs
@@ -27,7 +27,7 @@
//! which is precisely the input under test. The SDK is still used for setup and
//! for reading the log back, where it is the more honest observer.
-use crate::server::raw_tcp::TEST_BIND_SECRET;
+use crate::server::raw_tcp::{self, TEST_BIND_SECRET};
use bytes::{Bytes, BytesMut};
use consensus::client_table::COMMITTED_WINDOW_BITS;
use futures::future::join_all;
@@ -83,13 +83,13 @@ async fn given_committed_send_when_replayed_should_absorb_without_a_second_copy(
.root_client_for_node(0)
.await
.expect("connect a root client");
- seed_topic(&client).await;
+ let context = seed_topic(&client).await;
let addr = harness.node(0).tcp_addr().expect("node tcp address");
let (mut stream, session) = register(addr).await;
let body = send_messages_body(b"only-once");
- let header = request_header(Operation::SendMessages, session, 1, body.len());
+ let header = request_header(context, Operation::SendMessages, session, 1, body.len());
let original = receipt_until_committed(&mut stream, &header, &body).await;
let replayed = receipt_until_committed(&mut stream, &header, &body).await;
@@ -99,7 +99,13 @@ async fn given_committed_send_when_replayed_should_absorb_without_a_second_copy(
);
let changed_body = send_messages_body(b"changed-payload-must-not-append");
- let changed_header = request_header(Operation::SendMessages, session, 1, changed_body.len());
+ let changed_header = request_header(
+ context,
+ Operation::SendMessages,
+ session,
+ 1,
+ changed_body.len(),
+ );
let replayed = receipt_until_committed(&mut stream, &changed_header, &changed_body).await;
assert_eq!(
replayed, original,
@@ -211,11 +217,11 @@ async fn given_lost_send_reply_when_whole_cluster_restarts_should_return_the_ori
async fn verify_lost_reply_restart(harness: &mut TestHarness) {
let observer = harness.root_client_for_node(0).await.unwrap();
- seed_topic(&observer).await;
+ let context = seed_topic(&observer).await;
let addr = harness.node(0).tcp_addr().unwrap();
let (mut original, session) = register(addr).await;
let body = send_messages_body(b"lost-reply-durable");
- let header = request_header(Operation::SendMessages, session, 1, body.len());
+ let header = request_header(context, Operation::SendMessages, session, 1, body.len());
original
.write_all(bytemuck::bytes_of(&header))
.await
@@ -250,7 +256,13 @@ async fn verify_lost_reply_restart(harness: &mut TestHarness) {
"restart must return the original append receipt"
);
let changed_body = send_messages_body(b"changed-after-recovery");
- let changed_header = request_header(Operation::SendMessages, session, 1, changed_body.len());
+ let changed_header = request_header(
+ context,
+ Operation::SendMessages,
+ session,
+ 1,
+ changed_body.len(),
+ );
let receipt = replay_on_nodes(harness, &nodes, session, &changed_header, &changed_body).await;
assert_eq!(
receipt, expected,
@@ -263,7 +275,13 @@ async fn verify_lost_reply_restart(harness: &mut TestHarness) {
"lost reply retry must not append again"
);
let next_body = send_messages_body(b"after-recovery");
- let next_header = request_header(Operation::SendMessages, session, 2, next_body.len());
+ let next_header = request_header(
+ context,
+ Operation::SendMessages,
+ session,
+ 2,
+ next_body.len(),
+ );
let _ = replay_on_nodes(harness, &nodes, session, &next_header, &next_body).await;
assert_eq!(
poll_all(&observer).await,
@@ -284,7 +302,7 @@ async fn given_reclaimed_wal_when_recovered_quorum_replays_should_return_the_ori
const WAL_CHECKPOINT_OFFSET: usize = 48;
const WAL_HEAD_OFFSET: usize = 72;
let observer = harness.root_client_for_node(0).await.unwrap();
- seed_topic(&observer).await;
+ let context = seed_topic(&observer).await;
let (mut connection, session) = register(harness.node(0).tcp_addr().unwrap()).await;
let mut last = None;
for request in 1..=CHECKPOINT_OP {
@@ -298,7 +316,13 @@ async fn given_reclaimed_wal_when_recovered_quorum_replays_should_return_the_ori
})
.collect::<Vec<_>>();
let body = encode_send_messages(&messages);
- let header = request_header(Operation::SendMessages, session, request, body.len());
+ let header = request_header(
+ context,
+ Operation::SendMessages,
+ session,
+ request,
+ body.len(),
+ );
let receipt = receipt_until_committed(&mut connection, &header, &body).await;
last = Some((header, body, receipt));
}
@@ -359,6 +383,7 @@ async fn given_reclaimed_wal_when_recovered_quorum_replays_should_return_the_ori
);
let changed_body = send_messages_body(b"changed-after-wal-reclamation");
let changed_header = request_header(
+ context,
Operation::SendMessages,
session,
CHECKPOINT_OP,
@@ -414,14 +439,20 @@ async fn given_committed_send_when_next_request_id_arrives_should_admit_it(
.root_client_for_node(0)
.await
.expect("connect a root client");
- seed_topic(&client).await;
+ let context = seed_topic(&client).await;
let addr = harness.node(0).tcp_addr().expect("node tcp address");
let (mut stream, session) = register(addr).await;
for request in 1..=3u64 {
let body = send_messages_body(format!("message-{request}").as_bytes());
- let header = request_header(Operation::SendMessages, session, request, body.len());
+ let header = request_header(
+ context,
+ Operation::SendMessages,
+ session,
+ request,
+ body.len(),
+ );
let status = exchange_until_committed(&mut stream, &header, &body).await;
assert_eq!(status, 0, "request {request} must commit");
}
@@ -438,14 +469,20 @@ async fn given_gapped_request_id_when_sent_should_commit(harness: &mut TestHarne
.root_client_for_node(0)
.await
.expect("connect a root client");
- seed_topic(&client).await;
+ let context = seed_topic(&client).await;
let addr = harness.node(0).tcp_addr().expect("node tcp address");
let (mut stream, session) = register(addr).await;
for request in [1u64, 9, 40] {
let body = send_messages_body(format!("gap-{request}").as_bytes());
- let header = request_header(Operation::SendMessages, session, request, body.len());
+ let header = request_header(
+ context,
+ Operation::SendMessages,
+ session,
+ request,
+ body.len(),
+ );
let status = exchange_until_committed(&mut stream, &header, &body).await;
assert_eq!(status, 0, "gapped request {request} must commit");
}
@@ -462,7 +499,7 @@ async fn given_committed_consumer_offset_when_replayed_should_absorb(harness: &m
.root_client_for_node(0)
.await
.expect("connect a root client");
- seed_topic(&client).await;
+ let context = seed_topic(&client).await;
let addr = harness.node(0).tcp_addr().expect("node tcp address");
let (mut stream, session) = register(addr).await;
@@ -481,7 +518,8 @@ async fn given_committed_consumer_offset_when_replayed_should_absorb(harness: &m
payload: b"seed-next",
},
]);
- let produce_header = request_header(Operation::SendMessages, session, 1, produce.len());
+ let produce_header =
+ request_header(context, Operation::SendMessages, session, 1, produce.len());
assert_eq!(
exchange_until_committed(&mut stream, &produce_header, &produce).await,
0,
@@ -489,7 +527,13 @@ async fn given_committed_consumer_offset_when_replayed_should_absorb(harness: &m
);
let body = store_offset_body(0);
- let header = request_header(Operation::StoreConsumerOffset, session, 2, body.len());
+ let header = request_header(
+ context,
+ Operation::StoreConsumerOffset,
+ session,
+ 2,
+ body.len(),
+ );
let original = receipt_until_committed(&mut stream, &header, &body).await;
let replayed = receipt_until_committed(&mut stream, &header, &body).await;
@@ -499,6 +543,7 @@ async fn given_committed_consumer_offset_when_replayed_should_absorb(harness: &m
);
let changed_body = store_offset_body(1);
let changed_header = request_header(
+ context,
Operation::StoreConsumerOffset,
session,
2,
@@ -523,7 +568,13 @@ async fn given_committed_consumer_offset_when_replayed_should_absorb(harness: &m
// The next id still gets through: the watermark must not wedge the client.
let next = store_offset_body(0);
- let next_header = request_header(Operation::StoreConsumerOffset, session, 3, next.len());
+ let next_header = request_header(
+ context,
+ Operation::StoreConsumerOffset,
+ session,
+ 3,
+ next.len(),
+ );
assert_eq!(
exchange_until_committed(&mut stream, &next_header, &next).await,
0,
@@ -561,7 +612,7 @@ async fn given_more_writers_than_prepare_slots_when_all_send_at_once_should_answ
.root_client_for_node(0)
.await
.expect("connect a root client");
- seed_topic(&client).await;
+ let context = seed_topic(&client).await;
let addr = harness.node(0).tcp_addr().expect("node tcp address");
// Register every connection first so the writes race each other, not the
@@ -577,8 +628,14 @@ async fn given_more_writers_than_prepare_slots_when_all_send_at_once_should_answ
.iter_mut()
.map(|(client_id, stream, session)| async move {
let body = send_messages_body(format!("writer-{client_id:x}").as_bytes());
- let header =
- request_header_for(*client_id, Operation::SendMessages, *session, 1, body.len());
+ let header = request_header_for(
+ context,
+ *client_id,
+ Operation::SendMessages,
+ *session,
+ 1,
+ body.len(),
+ );
exchange_with_budget(stream, &header, &body, COMMIT_BUDGET).await
});
let statuses = join_all(sends).await;
@@ -667,13 +724,21 @@ async fn given_transferred_dedup_slice_when_old_request_replays_should_absorb(
.root_client_for_node(0)
.await
.expect("connect a root client");
- seed_topic(&client).await;
+ let context = seed_topic(&client).await;
let addr = harness.node(0).tcp_addr().expect("node 0 tcp address");
let (mut stream, session) = register(addr).await;
- raw_produce(&mut stream, session, 1..=PRE_STOP_SENDS, COMMIT_BUDGET).await;
+ raw_produce(
+ context,
+ &mut stream,
+ session,
+ 1..=PRE_STOP_SENDS,
+ COMMIT_BUDGET,
+ )
+ .await;
sleep(Duration::from_secs(1)).await;
harness.stop_node(2).expect("stop node 2");
raw_produce(
+ context,
&mut stream,
session,
(PRE_STOP_SENDS + 1)..=TESTED_CLIENT_SENDS,
@@ -684,6 +749,7 @@ async fn given_transferred_dedup_slice_when_old_request_replays_should_absorb(
let (mut filler, filler_session) =
register_client_with_budget(addr, FILLER_CLIENT_ID, COMMIT_BUDGET).await;
raw_produce_for(
+ context,
&mut filler,
FILLER_CLIENT_ID,
filler_session,
@@ -717,19 +783,33 @@ async fn given_transferred_dedup_slice_when_old_request_replays_should_absorb(
// transfer). Sends reconnect + re-register on eviction; dedup keys on the
// client id and must hold across a re-register.
let addr = harness.node(2).tcp_addr().expect("node 2 tcp address");
- let fresh = send_reconnecting(addr, PROBE_CLIENT_ID, 1, FINAL_COMMIT_BUDGET).await;
+ let fresh = send_reconnecting(context, addr, PROBE_CLIENT_ID, 1, FINAL_COMMIT_BUDGET).await;
assert_eq!(
fresh, 0,
"the probe client's send must commit on the new primary"
);
- let replayed = send_reconnecting(addr, CLIENT_ID, REPLAYED_REQUEST, FINAL_COMMIT_BUDGET).await;
+ let replayed = send_reconnecting(
+ context,
+ addr,
+ CLIENT_ID,
+ REPLAYED_REQUEST,
+ FINAL_COMMIT_BUDGET,
+ )
+ .await;
assert_eq!(
replayed, 0,
"a replay of a transferred watermark is absorbed as a success"
);
- let aged_out = send_reconnecting(addr, CLIENT_ID, AGED_OUT_REQUEST, FINAL_COMMIT_BUDGET).await;
+ let aged_out = send_reconnecting(
+ context,
+ addr,
+ CLIENT_ID,
+ AGED_OUT_REQUEST,
+ FINAL_COMMIT_BUDGET,
+ )
+ .await;
assert_eq!(
aged_out,
IggyError::RequestTooOld.as_code(),
@@ -753,7 +833,13 @@ async fn given_transferred_dedup_slice_when_old_request_replays_should_absorb(
/// One send under `request`, surviving evictions: reconnect, re-register, and
/// retry the identical frame until it answers or the budget runs out.
-async fn send_reconnecting(addr: SocketAddr, client: u128, request: u64, budget: Duration) -> u32 {
+async fn send_reconnecting(
+ context: PartitionContext,
+ addr: SocketAddr,
+ client: u128,
+ request: u64,
+ budget: Duration,
+) -> u32 {
let deadline = Instant::now() + budget;
let body = send_messages_body(format!("send-{request}").as_bytes());
loop {
@@ -764,6 +850,7 @@ async fn send_reconnecting(addr: SocketAddr, client: u128, request: u64, budget:
);
let (mut stream, session) = register_client_with_budget(addr, client, remaining).await;
let header = request_header_for(
+ context,
client,
Operation::SendMessages,
session,
@@ -781,15 +868,17 @@ async fn send_reconnecting(addr: SocketAddr, client: u128, request: u64, budget:
/// Produce one single-message batch per request id over the lockstep raw
/// connection, waiting out each commit.
async fn raw_produce(
+ context: PartitionContext,
stream: &mut TcpStream,
session: u64,
requests: std::ops::RangeInclusive<u64>,
budget: Duration,
) {
- raw_produce_for(stream, CLIENT_ID, session, requests, budget).await;
+ raw_produce_for(context, stream, CLIENT_ID, session, requests, budget).await;
}
async fn raw_produce_for(
+ context: PartitionContext,
stream: &mut TcpStream,
client: u128,
session: u64,
@@ -799,6 +888,7 @@ async fn raw_produce_for(
for request in requests {
let body = send_messages_body(format!("send-{request}").as_bytes());
let header = request_header_for(
+ context,
client,
Operation::SendMessages,
session,
@@ -825,7 +915,7 @@ async fn await_marker(harness: &TestHarness, node: usize, marker: &str) {
}
}
-async fn seed_topic(client: &IggyClient) {
+async fn seed_topic(client: &IggyClient) -> PartitionContext {
client
.create_stream(STREAM_NAME)
.await
@@ -849,6 +939,13 @@ async fn seed_topic(client: &IggyClient) {
)
.await
.expect("create topic");
+ raw_tcp::partition_context(
+ client,
+ &stream_id,
+ &Identifier::named(TOPIC_NAME).unwrap(),
+ PARTITION_ID,
+ )
+ .await
}
async fn poll_all(client: &IggyClient) -> u32 {
@@ -858,20 +955,27 @@ async fn poll_all(client: &IggyClient) -> u32 {
async fn poll_up_to(client: &IggyClient, max: u32) -> u32 {
let stream_id = Identifier::named(STREAM_NAME).expect("stream identifier");
let topic_id = Identifier::named(TOPIC_NAME).expect("topic identifier");
- client
- .poll_messages(
- &stream_id,
- &topic_id,
- Some(PARTITION_ID),
- &Consumer::new(Identifier::numeric(1).expect("consumer identifier")),
- &PollingStrategy::offset(0),
- max,
- false,
- )
- .await
- .expect("poll messages")
- .messages
- .len() as u32
+ let deadline = Instant::now() + COMMIT_BUDGET;
+ loop {
+ match client
+ .poll_messages(
+ &stream_id,
+ &topic_id,
+ Some(PARTITION_ID),
+ &Consumer::new(Identifier::numeric(1).expect("consumer identifier")),
+ &PollingStrategy::offset(0),
+ max,
+ false,
+ )
+ .await
+ {
+ Ok(polled) => return polled.messages.len() as u32,
+ Err(IggyError::TransientNotAccepted) if Instant::now() < deadline => {
+ sleep(RETRY_PAUSE).await;
+ }
+ Err(error) => panic!("poll messages after recovery: {error:?}"),
+ }
+ }
}
/// Full `SendMessages` body: metadata prefix, batch header, one message.
@@ -899,22 +1003,24 @@ fn encode_send_messages(messages: &[RawMessage<'_>]) -> Bytes {
}
fn request_header(
+ context: PartitionContext,
operation: Operation,
session: u64,
request: u64,
body_len: usize,
) -> RequestHeader {
- request_header_for(CLIENT_ID, operation, session, request, body_len)
+ request_header_for(context, CLIENT_ID, operation, session, request, body_len)
}
fn request_header_for(
+ context: PartitionContext,
client: u128,
operation: Operation,
session: u64,
request: u64,
body_len: usize,
) -> RequestHeader {
- RequestHeader {
+ let mut header = RequestHeader {
command: Command::Request,
operation,
size: u32::try_from(HEADER_SIZE + body_len).unwrap(),
@@ -922,7 +1028,9 @@ fn request_header_for(
session,
request,
..Default::default()
- }
+ };
+ context.stamp(&mut header);
+ header
}
/// Exchange until the server stops answering transiently, returning the reply
@@ -1131,7 +1239,14 @@ async fn login_on(stream: &mut TcpStream, client: u128) -> Option<u64> {
bind_secret: BindSecret::new(Box::new(TEST_BIND_SECRET)),
}
.to_bytes();
- let header = request_header_for(client, Operation::Register, 0, 0, body.len());
+ let header = request_header_for(
+ PartitionContext::default(),
+ client,
+ Operation::Register,
+ 0,
+ 0,
+ body.len(),
+ );
stream.write_all(bytemuck::bytes_of(&header)).await.unwrap();
stream.write_all(&body).await.unwrap();
diff --git a/core/integration/tests/cluster/partition_primary_routing.rs b/core/integration/tests/cluster/partition_primary_routing.rs
index e6495707a..861d9d457 100644
--- a/core/integration/tests/cluster/partition_primary_routing.rs
+++ b/core/integration/tests/cluster/partition_primary_routing.rs
@@ -189,21 +189,30 @@ async fn given_metadata_view_moved_when_producing_to_a_fresh_topic_should_reach_
);
}
-#[iggy_harness(cluster_nodes = 3, server(metadata.journal_slots = "256"))]
+#[iggy_harness(cluster_nodes = 3, server(
+ metadata.journal_slots = "256",
+ consumer_group.rebalancing_timeout = "1s",
+))]
async fn given_different_metadata_and_partition_primaries_when_group_auto_commits_should_return_messages(
harness: &mut TestHarness,
) {
assert_group_auto_commit_routing(harness, TransportProtocol::Tcp).await;
}
-#[iggy_harness(cluster_nodes = 3, server(metadata.journal_slots = "256"))]
+#[iggy_harness(cluster_nodes = 3, server(
+ metadata.journal_slots = "256",
+ consumer_group.rebalancing_timeout = "1s",
+))]
async fn given_different_metadata_and_partition_primaries_when_quic_group_auto_commits_should_return_messages(
harness: &mut TestHarness,
) {
assert_group_auto_commit_routing(harness, TransportProtocol::Quic).await;
}
-#[iggy_harness(cluster_nodes = 3, server(metadata.journal_slots = "256"))]
+#[iggy_harness(cluster_nodes = 3, server(
+ metadata.journal_slots = "256",
+ consumer_group.rebalancing_timeout = "1s",
+))]
async fn given_different_metadata_and_partition_primaries_when_websocket_group_auto_commits_should_return_messages(
harness: &mut TestHarness,
) {
@@ -325,6 +334,15 @@ async fn assert_group_offset_routing(
.join_consumer_group(&stream, &topic, &group)
.await
.unwrap();
+ integration::harness::wait_for_consumer_group_assignment(
+ member,
+ &stream,
+ &topic,
+ &group,
+ 1,
+ PRECONDITION_BUDGET,
+ )
+ .await;
let before = member.get_me().await.unwrap();
for offset in 0..GROUP_PAYLOADS.len() as u64 {
timeout(
@@ -566,6 +584,15 @@ async fn assert_group_auto_commit_routing(harness: &mut TestHarness, transport:
.join_consumer_group(&stream, &topic, &group)
.await
.unwrap();
+ integration::harness::wait_for_consumer_group_assignment(
+ &member,
+ &stream,
+ &topic,
+ &group,
+ 1,
+ PRECONDITION_BUDGET,
+ )
+ .await;
let membership = member
.get_consumer_group(&stream, &topic, &group)
.await
@@ -868,7 +895,7 @@ async fn assert_data_session_fences(
data.bind_vsr_session(binding.identity.session)
.await
.unwrap();
- data.send_raw_with_response(POLL_MESSAGES_ON_PRIMARY_CODE, poll.clone())
+ data.send_raw_with_context(POLL_MESSAGES_ON_PRIMARY_CODE, poll.clone(), route.context)
.await
.unwrap();
@@ -898,7 +925,7 @@ async fn assert_data_session_fences(
.await
.expect("the data node must observe the member leaving");
let response = data
- .send_raw_with_response(POLL_MESSAGES_ON_PRIMARY_CODE, poll.clone())
+ .send_raw_with_context(POLL_MESSAGES_ON_PRIMARY_CODE, poll.clone(), route.context)
.await
.unwrap();
let polled = PolledMessages::from_bytes(response).unwrap();
@@ -918,8 +945,12 @@ async fn assert_data_session_fences(
.to_bytes();
assert!(
matches!(
- data.send_raw_with_response(STORE_CONSUMER_OFFSET_CODE, offset_write.clone())
- .await,
+ data.send_raw_with_context(
+ STORE_CONSUMER_OFFSET_CODE,
+ offset_write.clone(),
+ route.context
+ )
+ .await,
Err(IggyError::ConsumerGroupPartitionNotOwned(..))
),
"a departed member must not commit through its attachment"
@@ -929,23 +960,35 @@ async fn assert_data_session_fences(
.join_consumer_group(stream, topic, &consumer.id)
.await
.unwrap();
- member
- .poll_messages(
- stream,
- topic,
- None,
- consumer,
- &PollingStrategy::next(),
- 1,
- true,
- )
- .await
- .unwrap();
- let response = member
- .send_binary_request(GET_POLL_ROUTING_CODE, poll.clone())
- .await
- .unwrap();
- let route = PollRoutingResponse::decode_from(&response).unwrap();
+ integration::harness::wait_for_consumer_group_assignment(
+ member,
+ stream,
+ topic,
+ &consumer.id,
+ 1,
+ PRECONDITION_BUDGET,
+ )
+ .await;
+ // The rejoining session can still appear as the draining old owner.
+ let replacement_route = timeout(PRECONDITION_BUDGET, async {
+ loop {
+ match member
+ .send_binary_request(GET_POLL_ROUTING_CODE, poll.clone())
+ .await
+ {
+ Ok(response) => break PollRoutingResponse::decode_from(&response).unwrap(),
+ Err(
+ IggyError::ConsumerGroupPartitionNotOwned(..) | IggyError::TransientNotAccepted,
+ ) => {}
+ Err(error) => panic!("route discovery failed after rejoining: {error:?}"),
+ }
+ sleep(PRECONDITION_POLL).await;
+ }
+ })
+ .await
+ .expect("rejoining must install a pollable owner within the routing budget");
+ assert!(replacement_route.context.owner_generation > route.context.owner_generation);
+ let route = replacement_route;
binding.identity = route.consumer_session;
data.send_raw_with_response(BIND_SESSION_CODE, binding.to_bytes())
.await
@@ -954,7 +997,7 @@ async fn assert_data_session_fences(
timeout(PRECONDITION_BUDGET, async {
loop {
let result = data
- .send_raw_with_response(POLL_MESSAGES_ON_PRIMARY_CODE, poll.clone())
+ .send_raw_with_context(POLL_MESSAGES_ON_PRIMARY_CODE, poll.clone(), route.context)
.await;
if matches!(result, Err(IggyError::Unauthenticated)) {
break;
diff --git a/core/integration/tests/connectors/fixtures/influxdb/sink_v3.rs b/core/integration/tests/connectors/fixtures/influxdb/sink_v3.rs
index 1261d1b16..b6af63e3c 100644
--- a/core/integration/tests/connectors/fixtures/influxdb/sink_v3.rs
+++ b/core/integration/tests/connectors/fixtures/influxdb/sink_v3.rs
@@ -30,7 +30,8 @@ use std::time::Duration;
use tokio::time::sleep;
use tracing::info;
-const POLL_ATTEMPTS_V3: usize = 100;
+// Each serial batch can wait for InfluxDB's default one-second WAL flush.
+const POLL_ATTEMPTS_V3: usize = 300;
const POLL_INTERVAL_MS_V3: u64 = 50;
pub struct InfluxDb3SinkFixture {
@@ -57,18 +58,22 @@ impl InfluxDb3SinkFixture {
) -> Result<usize, TestBinaryError> {
let sql = format!("SELECT * FROM \"{measurement}\"");
info!("V3 wait_for_points SQL: {sql}");
+ let mut observed = 0;
for _ in 0..POLL_ATTEMPTS_V3 {
match self.query_count(&sql).await {
Ok(n) if n >= expected => {
info!("Found {n} rows in InfluxDB 3 (expected {expected})");
return Ok(n);
}
- Ok(_) | Err(_) => {}
+ Ok(count) => observed = count,
+ Err(_) => {}
}
sleep(Duration::from_millis(POLL_INTERVAL_MS_V3)).await;
}
Err(TestBinaryError::InvalidState {
- message: format!("Expected at least {expected} rows after {POLL_ATTEMPTS_V3} attempts"),
+ message: format!(
+ "Expected at least {expected} rows after {POLL_ATTEMPTS_V3} attempts, observed {observed}"
+ ),
})
}
diff --git a/core/integration/tests/data_integrity/storage_compat.rs b/core/integration/tests/data_integrity/storage_compat.rs
index d1a92a5a4..6935a1b10 100644
--- a/core/integration/tests/data_integrity/storage_compat.rs
+++ b/core/integration/tests/data_integrity/storage_compat.rs
@@ -1262,6 +1262,9 @@ async fn assert_topic_recovered(
segment_size: seed.segment_size.and(recovered.segment_size),
durability: recovered.durability,
consumer_offset_durability: recovered.consumer_offset_durability,
+ partition_resize_policy: seed
+ .partition_resize_policy
+ .and(recovered.partition_resize_policy),
messages_required_to_save: seed
.messages_required_to_save
.and(recovered.messages_required_to_save),
diff --git a/core/integration/tests/data_integrity/verify_consumer_group_partition_assignment.rs b/core/integration/tests/data_integrity/verify_consumer_group_partition_assignment.rs
index 3364cbf5f..346ed6dee 100644
--- a/core/integration/tests/data_integrity/verify_consumer_group_partition_assignment.rs
+++ b/core/integration/tests/data_integrity/verify_consumer_group_partition_assignment.rs
@@ -33,6 +33,7 @@
//! memberships. Their stale clients send no further activity.
use iggy::prelude::*;
+use integration::harness::wait_for_consumer_group_assignment;
use integration::iggy_harness;
use std::collections::HashSet;
use std::str::FromStr;
@@ -45,8 +46,10 @@ const TOPIC_NAME: &str = "cg-partition-test-topic";
const CONSUMER_GROUP_NAME: &str = "cg-partition-test-group";
const PARTITIONS_COUNT: u32 = 3;
/// Bounds [`await_members_count`], including logical-session lease expiry.
+/// Specs that prove a drain or commit completes a handoff keep the default
+/// `consumer_group.rebalancing_timeout`, which outlasts this bound, so the
+/// deadline cannot complete the handoff for them.
const MEMBERS_CONVERGENCE_TIMEOUT: Duration = Duration::from_secs(15);
-const MEMBERS_RETRY_INTERVAL: Duration = Duration::from_millis(100);
/// Slices the slab-reuse wait so the surviving consumer can prove liveness
/// inside the server's staleness window. Product of the two is the 3s the
/// spec waits for the freed slab to become reusable.
@@ -78,6 +81,7 @@ async fn create_tcp_client(server_addr: &str) -> IggyClient {
}
#[iggy_harness(server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "2s",
consumer_group.heartbeat_interval = "500ms",
@@ -155,7 +159,7 @@ async fn should_not_duplicate_partition_assignments_after_stale_client_cleanup(
}
// 4. Verify initial state: 3 members, each with 1 unique partition
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3, "Expected 3 members before kill");
assert_unique_partition_assignments(&cg);
@@ -202,7 +206,7 @@ async fn should_not_duplicate_partition_assignments_after_stale_client_cleanup(
}
// 10. Verify exactly 3 members with unique partition assignments
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(
cg.members_count, 3,
"Expected 3 members after new clients join, got {}. Members: {:?}",
@@ -259,6 +263,7 @@ async fn should_not_duplicate_partition_assignments_after_stale_client_cleanup(
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -281,7 +286,7 @@ async fn should_not_reshuffle_partitions_when_new_member_joins(harness: &TestHar
}
// 3. Record the current partition assignments
- let cg_before = get_consumer_group(&root_client).await;
+ let cg_before = await_members_count(&root_client, 2).await;
assert_eq!(cg_before.members_count, 2);
let assignments_before: Vec<(u32, Vec<u32>)> = cg_before
@@ -300,7 +305,7 @@ async fn should_not_reshuffle_partitions_when_new_member_joins(harness: &TestHar
// 5. Verify: members that had exactly 1 partition still have it (stable assignment).
// Members that were over-assigned may have given up excess partitions - that's expected.
- let cg_after = get_consumer_group(&root_client).await;
+ let cg_after = await_members_count(&root_client, 3).await;
assert_eq!(cg_after.members_count, 3);
for (old_id, old_partitions) in &assignments_before {
@@ -335,6 +340,7 @@ async fn should_not_reshuffle_partitions_when_new_member_joins(harness: &TestHar
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -415,6 +421,7 @@ async fn should_skip_revoked_partitions_in_round_robin(harness: &TestHarness) {
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -547,7 +554,7 @@ async fn should_not_lose_messages_with_concurrent_polls_during_partition_add(
let _ = poll_task_2.await;
// 4. Verify: no duplicates, all partitions assigned
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -573,6 +580,7 @@ async fn should_not_lose_messages_with_concurrent_polls_during_partition_add(
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.interval = "60s",
consumer_group.heartbeat_interval = "500ms",
consumer_group.session_timeout = "8s",
@@ -666,7 +674,7 @@ async fn should_handle_partition_add_then_consumer_disconnect_then_new_join(harn
sleep(Duration::from_millis(500)).await;
// 5. Verify: both members have partitions, no duplicates
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -720,6 +728,7 @@ async fn should_handle_partition_add_then_consumer_disconnect_then_new_join(harn
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -832,7 +841,7 @@ async fn should_handle_partition_delete_while_multiple_consumers_polling(harness
}
// 4. Verify: 3 members, 3 remaining partitions, no duplicates
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -857,6 +866,7 @@ async fn should_handle_partition_delete_while_multiple_consumers_polling(harness
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -924,7 +934,7 @@ async fn should_reach_even_distribution_after_multiple_joins(harness: &TestHarne
join_cg(&client2).await;
sleep(Duration::from_millis(200)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -939,7 +949,7 @@ async fn should_reach_even_distribution_after_multiple_joins(harness: &TestHarne
join_cg(&client3).await;
sleep(Duration::from_millis(200)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -962,7 +972,7 @@ async fn should_reach_even_distribution_after_multiple_joins(harness: &TestHarne
join_cg(&client4).await;
sleep(Duration::from_millis(200)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 4).await;
assert_eq!(cg.members_count, 4);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -988,7 +998,7 @@ async fn should_reach_even_distribution_after_multiple_joins(harness: &TestHarne
}
sleep(Duration::from_millis(200)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 6).await;
assert_eq!(cg.members_count, 6);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -1039,6 +1049,7 @@ async fn should_reach_even_distribution_after_multiple_joins(harness: &TestHarne
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -1060,7 +1071,7 @@ async fn should_split_evenly_when_consumer_joins_after_partitions_added(harness:
}
sleep(Duration::from_millis(200)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
assert_eq!(total, 3);
@@ -1103,7 +1114,7 @@ async fn should_split_evenly_when_consumer_joins_after_partitions_added(harness:
sleep(Duration::from_millis(500)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -1158,6 +1169,7 @@ async fn should_split_evenly_when_consumer_joins_after_partitions_added(harness:
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -1209,7 +1221,7 @@ async fn should_not_duplicate_messages_when_partitions_added_during_polling(harn
sleep(Duration::from_millis(500)).await;
// 4. After rebalance, verify no partition is assigned to both consumers
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -1257,6 +1269,7 @@ async fn should_not_duplicate_messages_when_partitions_added_during_polling(harn
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -1344,7 +1357,7 @@ async fn should_handle_delete_partitions_with_uncommitted_work(harness: &TestHar
sleep(Duration::from_millis(500)).await;
// 4. After rebalance: 3 remaining partitions, no duplicates, even split
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -1389,6 +1402,7 @@ async fn should_handle_delete_partitions_with_uncommitted_work(harness: &TestHar
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -1470,7 +1484,7 @@ async fn should_handle_rapid_partition_changes_with_active_consumers(harness: &T
sleep(Duration::from_millis(500)).await;
// 3. 9 partitions, 3 consumers → 3 each
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -1525,6 +1539,7 @@ async fn should_handle_rapid_partition_changes_with_active_consumers(harness: &T
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -1548,7 +1563,7 @@ async fn should_rebalance_after_adding_partitions(harness: &TestHarness) {
sleep(Duration::from_millis(200)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
for member in &cg.members {
@@ -1572,7 +1587,7 @@ async fn should_rebalance_after_adding_partitions(harness: &TestHarness) {
sleep(Duration::from_millis(500)).await;
// 3. After rebalance, each consumer should have 2 partitions (6/3)
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -1597,6 +1612,7 @@ async fn should_rebalance_after_adding_partitions(harness: &TestHarness) {
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -1642,7 +1658,7 @@ async fn should_rebalance_after_deleting_partitions(harness: &TestHarness) {
sleep(Duration::from_millis(200)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
assert_eq!(total, 6);
@@ -1660,7 +1676,7 @@ async fn should_rebalance_after_deleting_partitions(harness: &TestHarness) {
sleep(Duration::from_millis(500)).await;
// 3. After rebalance, each consumer should have 1 partition (3/3)
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -1685,6 +1701,7 @@ async fn should_rebalance_after_deleting_partitions(harness: &TestHarness) {
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -1742,7 +1759,7 @@ async fn should_handle_partition_add_during_pending_revocation(harness: &TestHar
// 4. After full rebalance triggered by partition change,
// all 6 partitions should be distributed, no duplicates
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
@@ -1783,7 +1800,7 @@ async fn should_timeout_revocation(harness: &TestHarness) {
.unwrap();
join_cg(&client1).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 1).await;
assert_eq!(cg.members_count, 1);
assert_eq!(cg.members[0].partitions_count, PARTITIONS_COUNT);
@@ -1827,7 +1844,7 @@ async fn should_timeout_revocation(harness: &TestHarness) {
sleep(Duration::from_secs(6)).await;
// 6. The periodic checker should have force-completed the revocation.
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
@@ -1873,6 +1890,7 @@ async fn should_timeout_revocation(harness: &TestHarness) {
}
#[iggy_harness(server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "2s",
consumer_group.heartbeat_interval = "500ms",
@@ -1972,7 +1990,7 @@ async fn should_not_duplicate_after_reconnect_without_heartbeat(harness: &TestHa
join_cg(client).await;
}
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
@@ -2015,6 +2033,7 @@ async fn should_not_duplicate_after_reconnect_without_heartbeat(harness: &TestHa
#[iggy_harness(
test_client_transport = [Tcp, WebSocket, Quic],
server(
+ consumer_group.rebalancing_timeout = "1s",
consumer_group.heartbeat_interval = "500ms",
consumer_group.session_timeout = "8s"
)
@@ -2085,7 +2104,7 @@ async fn should_not_duplicate_partition_assignments_after_client_reconnect(harne
.unwrap();
}
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
@@ -2139,7 +2158,7 @@ async fn should_not_duplicate_partition_assignments_after_client_reconnect(harne
.unwrap();
}
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
@@ -2175,26 +2194,16 @@ async fn should_not_duplicate_partition_assignments_after_client_reconnect(harne
.unwrap();
}
-/// Poll the group until it reports `expected` members, then return it.
-///
-/// Closing a binding preserves its logical session until lease expiry.
-/// Wait for the committed expiry and membership update before checking ownership.
async fn await_members_count(client: &IggyClient, expected: u32) -> ConsumerGroupDetails {
- let deadline = tokio::time::Instant::now() + MEMBERS_CONVERGENCE_TIMEOUT;
- loop {
- let group = get_consumer_group(client).await;
- if group.members_count == expected {
- return group;
- }
- assert!(
- tokio::time::Instant::now() < deadline,
- "expected {expected} members within {MEMBERS_CONVERGENCE_TIMEOUT:?}, \
- last saw {}. Members: {:?}",
- group.members_count,
- group.members
- );
- sleep(MEMBERS_RETRY_INTERVAL).await;
- }
+ wait_for_consumer_group_assignment(
+ client,
+ &Identifier::named(STREAM_NAME).unwrap(),
+ &Identifier::named(TOPIC_NAME).unwrap(),
+ &Identifier::named(CONSUMER_GROUP_NAME).unwrap(),
+ expected,
+ MEMBERS_CONVERGENCE_TIMEOUT,
+ )
+ .await
}
async fn get_consumer_group(client: &IggyClient) -> ConsumerGroupDetails {
@@ -2280,6 +2289,9 @@ async fn join_cg(client: &IggyClient) {
)
.await
.unwrap();
+ if get_consumer_group(client).await.members_count == 1 {
+ await_members_count(client, 1).await;
+ }
}
fn assert_balanced_partition_distribution(cg: &ConsumerGroupDetails, expected_total: u32) {
@@ -2335,7 +2347,7 @@ async fn should_not_return_same_message_to_two_consumers_during_rebalance(harnes
.unwrap();
join_cg(&client1).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 1).await;
assert_eq!(cg.members_count, 1);
assert_eq!(cg.members[0].partitions_count, PARTITIONS_COUNT);
@@ -2413,7 +2425,7 @@ async fn should_not_return_same_message_to_two_consumers_during_rebalance(harnes
sleep(Duration::from_millis(100)).await;
// 7. After commit, the partition should have transferred. Verify final state.
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
@@ -2473,7 +2485,7 @@ async fn should_complete_revocation_on_auto_commit(harness: &TestHarness) {
sleep(Duration::from_millis(100)).await;
// 3. Verify partitions are distributed
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
@@ -2529,9 +2541,9 @@ async fn should_transfer_never_polled_partitions_immediately(harness: &TestHarne
.unwrap();
join_cg(&client3).await;
- // 4. All 3 members should have exactly 1 partition each - immediately,
- // no waiting for commits
- let cg = get_consumer_group(&root_client).await;
+ // 4. All 3 members should have exactly 1 partition each, no waiting for
+ // commits or for the rebalancing deadline
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(cg.members_count, 3);
assert_unique_partition_assignments(&cg);
@@ -2551,6 +2563,7 @@ async fn should_transfer_never_polled_partitions_immediately(harness: &TestHarne
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.interval = "60s",
consumer_group.heartbeat_interval = "500ms",
consumer_group.session_timeout = "8s",
@@ -2641,6 +2654,7 @@ async fn should_rebalance_when_member_with_pending_revocation_leaves(harness: &T
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -2725,8 +2739,7 @@ async fn should_not_produce_duplicate_messages_with_sequential_consumer_joins(
.await
.unwrap();
- // 6. Small delay for revocation completions
- sleep(Duration::from_millis(200)).await;
+ await_members_count(&root_client, 3).await;
// 7. All 3 consumers poll messages with manual ack - collect ALL messages
let mut all_messages: Vec<(u32, u64)> = Vec::new();
@@ -2883,7 +2896,7 @@ async fn should_wait_for_manual_commit_before_completing_revocation(harness: &Te
sleep(Duration::from_millis(200)).await;
// 5. Now consumer2 should have partitions
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
assert_unique_partition_assignments(&cg);
@@ -2904,6 +2917,7 @@ async fn should_wait_for_manual_commit_before_completing_revocation(harness: &Te
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
consumer_group.heartbeat_interval = "500ms",
@@ -2997,6 +3011,7 @@ async fn should_redistribute_when_revocation_target_leaves(harness: &TestHarness
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -3036,7 +3051,7 @@ async fn should_distribute_partitions_evenly_with_concurrent_joins(harness: &Tes
sleep(Duration::from_millis(200)).await;
// Verify: 3 members, each with exactly 1 partition
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_eq!(
cg.members_count, 3,
"Expected 3 members after concurrent joins, got {}. Members: {:?}",
@@ -3128,7 +3143,7 @@ async fn should_not_assign_partition_to_wrong_member_after_slab_reuse(harness: &
.unwrap();
}
- // 2. Consumer2 joins — triggers pending revocation targeting consumer2's slab
+ // Consumer2 joins while consumer1 still has uncommitted offsets.
let client2 = harness.new_client().await.unwrap();
client2
.login_user(DEFAULT_ROOT_USERNAME, DEFAULT_ROOT_PASSWORD)
@@ -3141,22 +3156,14 @@ async fn should_not_assign_partition_to_wrong_member_after_slab_reuse(harness: &
let cg = get_consumer_group(&root_client).await;
assert_eq!(cg.members_count, 2);
- // 3. Consumer2 (revocation target) disconnects — its slab is freed
+ // The next connection may reuse the departed transport's slab.
drop(client2);
- // Consumer1 must stay alive across the wait. This spec asks the server for
- // `heartbeat.interval = 2s`, so its verifier evicts any consumer-group
- // member idle past 1.2 intervals, and harness clients never ping on their
- // own (the SDK pinger is spawned by `IggyClient::connect`, which the
- // builder does not call). Silence here evicted consumer1 mid-wait and its
- // offset store below came back `StaleClient`. Any request refreshes
- // liveness; a ping is the cheapest. Consumer2 stays silent by construction
- // - it is already dropped, and the socket close frees its slab.
+ // Keep consumer1 alive without draining its offsets during transport cleanup.
for _ in 0..CONSUMER1_KEEPALIVE_PINGS {
sleep(CONSUMER1_KEEPALIVE_INTERVAL).await;
client1.ping().await.unwrap();
}
- // 4. Consumer3 joins — may reuse consumer2's old slab
let client3 = harness.new_client().await.unwrap();
client3
.login_user(DEFAULT_ROOT_USERNAME, DEFAULT_ROOT_PASSWORD)
@@ -3164,8 +3171,7 @@ async fn should_not_assign_partition_to_wrong_member_after_slab_reuse(harness: &
.unwrap();
join_cg(&client3).await;
- // 5. Consumer1 commits all offsets — revocation completion should detect
- // slab reuse via target_member_id validation and trigger full rebalance
+ // Complete the handoff before its deadline, after a new transport can reuse the slab.
for partition_id in 0..PARTITIONS_COUNT {
client1
.store_consumer_offset(
@@ -3179,10 +3185,7 @@ async fn should_not_assign_partition_to_wrong_member_after_slab_reuse(harness: &
.unwrap();
}
- sleep(Duration::from_millis(500)).await;
-
- // 6. Verify no partition duplication and all partitions assigned
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 3).await;
assert_unique_partition_assignments(&cg);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
assert_eq!(
@@ -3199,6 +3202,7 @@ async fn should_not_assign_partition_to_wrong_member_after_slab_reuse(harness: &
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
consumer_group.heartbeat_interval = "500ms",
@@ -3314,6 +3318,7 @@ async fn should_not_complete_other_members_revocations_on_leave(harness: &TestHa
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -3359,7 +3364,7 @@ async fn should_distribute_16_partitions_evenly_across_16_consumers(harness: &Te
sleep(Duration::from_millis(500)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 16).await;
assert_eq!(cg.members_count, 16);
assert_unique_partition_assignments(&cg);
assert_balanced_partition_distribution(&cg, 16);
@@ -3405,6 +3410,7 @@ async fn should_distribute_16_partitions_evenly_across_16_consumers(harness: &Te
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -3449,7 +3455,7 @@ async fn should_distribute_excess_evenly_when_multiple_idle_members_join(harness
sleep(Duration::from_millis(500)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 4).await;
assert_eq!(cg.members_count, 4);
assert_unique_partition_assignments(&cg);
assert_balanced_partition_distribution(&cg, 12);
@@ -3470,6 +3476,7 @@ async fn should_distribute_excess_evenly_when_multiple_idle_members_join(harness
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -3498,7 +3505,7 @@ async fn should_distribute_remainder_fairly_with_uneven_ratio(harness: &TestHarn
sleep(Duration::from_millis(500)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 4).await;
assert_eq!(cg.members_count, 4);
assert_unique_partition_assignments(&cg);
assert_balanced_partition_distribution(&cg, 10);
@@ -3533,6 +3540,7 @@ async fn should_distribute_remainder_fairly_with_uneven_ratio(harness: &TestHarn
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -3553,7 +3561,7 @@ async fn should_collect_excess_from_multiple_overassigned_members(harness: &Test
sleep(Duration::from_millis(200)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 2).await;
assert_eq!(cg.members_count, 2);
let total: u32 = cg.members.iter().map(|m| m.partitions_count).sum();
assert_eq!(total, 16);
@@ -3571,7 +3579,7 @@ async fn should_collect_excess_from_multiple_overassigned_members(harness: &Test
sleep(Duration::from_millis(500)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 8).await;
assert_eq!(cg.members_count, 8);
assert_unique_partition_assignments(&cg);
assert_balanced_partition_distribution(&cg, 16);
@@ -3592,6 +3600,7 @@ async fn should_collect_excess_from_multiple_overassigned_members(harness: &Test
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
))]
@@ -3623,7 +3632,7 @@ async fn should_not_starve_any_member_in_large_scale_rebalance(harness: &TestHar
sleep(Duration::from_millis(500)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 8).await;
assert_eq!(cg.members_count, 8);
assert_unique_partition_assignments(&cg);
assert_balanced_partition_distribution(&cg, 24);
@@ -3645,6 +3654,7 @@ async fn should_not_starve_any_member_in_large_scale_rebalance(harness: &TestHar
}
#[iggy_harness(test_client_transport = [Tcp, WebSocket, Quic], server(
+ consumer_group.rebalancing_timeout = "1s",
heartbeat.enabled = true,
heartbeat.interval = "60s",
consumer_group.heartbeat_interval = "500ms",
@@ -3670,7 +3680,7 @@ async fn should_maintain_balance_after_member_churn(harness: &TestHarness) {
sleep(Duration::from_millis(300)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 4).await;
assert_eq!(cg.members_count, 4);
assert_unique_partition_assignments(&cg);
assert_balanced_partition_distribution(&cg, 16);
@@ -3697,7 +3707,7 @@ async fn should_maintain_balance_after_member_churn(harness: &TestHarness) {
sleep(Duration::from_millis(500)).await;
- let cg = get_consumer_group(&root_client).await;
+ let cg = await_members_count(&root_client, 4).await;
assert_eq!(cg.members_count, 4);
assert_unique_partition_assignments(&cg);
assert_balanced_partition_distribution(&cg, 16);
diff --git a/core/integration/tests/sdk/consumer_group_membership.rs b/core/integration/tests/sdk/consumer_group_membership.rs
index a8e7f59e7..79f2218b7 100644
--- a/core/integration/tests/sdk/consumer_group_membership.rs
+++ b/core/integration/tests/sdk/consumer_group_membership.rs
@@ -23,8 +23,9 @@ use futures::StreamExt;
use iggy::prelude::locking::IggyRwLockFn;
use iggy::prelude::*;
use iggy_common::{BinaryTransport, ConsumerGroupClientState};
+use integration::harness::wait_for_consumer_group_assignment;
use integration::iggy_harness;
-use tokio::time::timeout;
+use tokio::time::{sleep, timeout};
const STREAM_NAME: &str = "cg-membership-stream";
const TOPIC_NAME: &str = "cg-membership-topic";
@@ -35,6 +36,7 @@ const CONSUMER_GROUP_NAME: &str = "cg-membership-group";
const PARK_TIMEOUT: Duration = Duration::from_secs(2);
// Generous bound: on regression the poll hangs until this elapses.
const RESOLVE_TIMEOUT: Duration = Duration::from_secs(10);
+const ASSIGNMENT_POLL_INTERVAL: Duration = Duration::from_millis(20);
// Legacy wire error codes for the join/leave failure ladder. Pinned as literals
// (not derived from `IggyError`) so renumbering the wire contract fails here
@@ -111,6 +113,7 @@ async fn given_group_member_holds_no_partitions_when_group_deleted_should_surfac
.build();
consumer.init().await.unwrap();
+ wait_for_assignment(&admin, 2).await;
let group = admin
.get_consumer_group(&stream_id, &topic_id, &group_id)
.await
@@ -209,6 +212,7 @@ async fn given_member_holds_no_partitions_when_polled_should_report_no_assigned_
.await
.unwrap();
+ wait_for_assignment(&first_member, 2).await;
let consumer = Consumer::group(group_id.clone());
let owner_poll = first_member
.poll_messages(
@@ -456,6 +460,7 @@ async fn given_group_member_when_session_reset_should_forget_membership(harness:
let consumer = Consumer::group(group_id.clone());
// The first group poll syncs the assignment, which is what registers the
// membership in the transport cache.
+ wait_for_assignment(&client, 1).await;
client
.poll_messages(
&stream_id,
@@ -517,6 +522,144 @@ async fn given_group_member_when_session_reset_should_forget_membership(harness:
);
}
+#[iggy_harness(cluster_nodes = 3)]
+async fn given_weak_offset_durability_when_owner_changes_should_preserve_progress_after_crash(
+ harness: &mut TestHarness,
+) {
+ const MESSAGE_COUNT: u32 = 3;
+ const STORED_OFFSET: u64 = 1;
+ let predecessor = harness.tcp_root_client().await.unwrap();
+ let successor = harness.tcp_root_client().await.unwrap();
+ let stream_id = Identifier::named(STREAM_NAME).unwrap();
+ let topic_id = Identifier::named(TOPIC_NAME).unwrap();
+ let group_id = Identifier::named(CONSUMER_GROUP_NAME).unwrap();
+ let consumer = Consumer::group(group_id.clone());
+ predecessor.create_stream(STREAM_NAME).await.unwrap();
+ predecessor
+ .create_topic(
+ &stream_id,
+ TOPIC_NAME,
+ &TopicCreateOptions {
+ partitions_count: Some(1),
+ durability: Durability::Replicated,
+ consumer_offset_durability: Durability::Replicated,
+ ..TopicCreateOptions::default()
+ },
+ )
+ .await
+ .unwrap();
+ predecessor
+ .create_consumer_group(&stream_id, &topic_id, CONSUMER_GROUP_NAME)
+ .await
+ .unwrap();
+ predecessor
+ .join_consumer_group(&stream_id, &topic_id, &group_id)
+ .await
+ .unwrap();
+ wait_for_assignment(&predecessor, 1).await;
+ let mut messages: Vec<_> = (0..MESSAGE_COUNT)
+ .map(|offset| IggyMessage::from_str(&format!("record-{offset}")).unwrap())
+ .collect();
+ predecessor
+ .send_messages(
+ &stream_id,
+ &topic_id,
+ &Partitioning::partition_id(0),
+ &mut messages,
+ )
+ .await
+ .unwrap();
+ let polled = predecessor
+ .poll_messages(
+ &stream_id,
+ &topic_id,
+ Some(0),
+ &consumer,
+ &PollingStrategy::first(),
+ 2,
+ false,
+ )
+ .await
+ .unwrap();
+ assert_eq!(polled.messages.len(), 2);
+ let position = ConsumerPosition {
+ partition_id: 0,
+ offset: STORED_OFFSET,
+ context: polled.context,
+ };
+ predecessor
+ .store_consumer_position(&consumer, &stream_id, &topic_id, position)
+ .await
+ .unwrap();
+
+ let primary = integration::harness::disk::leader_node_index(harness).await;
+ harness
+ .kill_node((primary + 1) % harness.cluster_size())
+ .unwrap();
+ predecessor
+ .leave_consumer_group(&stream_id, &topic_id, &group_id)
+ .await
+ .unwrap();
+ successor
+ .join_consumer_group(&stream_id, &topic_id, &group_id)
+ .await
+ .unwrap();
+ wait_for_assignment(&successor, 1).await;
+ let resumed = successor
+ .poll_messages(
+ &stream_id,
+ &topic_id,
+ Some(0),
+ &consumer,
+ &PollingStrategy::next(),
+ MESSAGE_COUNT,
+ false,
+ )
+ .await
+ .unwrap();
+ assert_eq!(resumed.messages.len(), 1);
+ assert_eq!(resumed.messages[0].header.offset, STORED_OFFSET + 1);
+ assert!(resumed.context.owner_generation > position.context.owner_generation);
+ assert_eq!(resumed.context.incarnation, position.context.incarnation);
+ assert!(matches!(
+ successor
+ .store_consumer_position(&consumer, &stream_id, &topic_id, position)
+ .await,
+ Err(IggyError::ConsumerGroupPartitionNotOwned(..))
+ ));
+
+ harness.kill_cluster().unwrap();
+ harness.restart_cluster().await.unwrap();
+ let recovered = harness.tcp_root_client().await.unwrap();
+ timeout(RESOLVE_TIMEOUT, async {
+ loop {
+ let offset = recovered
+ .get_consumer_offset(&consumer, &stream_id, &topic_id, Some(0))
+ .await
+ .unwrap();
+ if let Some(offset) = offset {
+ assert_eq!(offset.stored_offset, STORED_OFFSET);
+ break;
+ }
+ sleep(ASSIGNMENT_POLL_INTERVAL).await;
+ }
+ })
+ .await
+ .expect("the installed owner's committed offset must survive whole-cluster crash");
+}
+
+async fn wait_for_assignment(client: &IggyClient, members_count: u32) {
+ wait_for_consumer_group_assignment(
+ client,
+ &Identifier::named(STREAM_NAME).unwrap(),
+ &Identifier::named(TOPIC_NAME).unwrap(),
+ &Identifier::named(CONSUMER_GROUP_NAME).unwrap(),
+ members_count,
+ RESOLVE_TIMEOUT,
+ )
+ .await;
+}
+
/// The consumer-group cache lives on the transport `IggyClient` wraps.
async fn consumer_group_state(client: &IggyClient) -> Arc<ConsumerGroupClientState> {
match &*client.client().read().await {
diff --git a/core/integration/tests/sdk/options.rs b/core/integration/tests/sdk/options.rs
index 2fba9262a..fb8027ae7 100644
--- a/core/integration/tests/sdk/options.rs
+++ b/core/integration/tests/sdk/options.rs
@@ -524,3 +524,106 @@ async fn given_rename_only_when_updating_topic_should_leave_settings_alone(harne
"a key the update did not carry keeps its value"
);
}
+
+#[iggy_harness(test_client_transport = [Tcp, Http, Quic, WebSocket])]
+async fn given_resize_policy_when_resizing_should_enforce_the_immutable_partition_set(
+ harness: &TestHarness,
+) {
+ let client = harness.root_client().await.unwrap();
+ client.create_stream("resize-stream").await.unwrap();
+ let stream = Identifier::named("resize-stream").unwrap();
+ let policy_key = HeaderKey::from_str(topic_option_keys::PARTITION_RESIZE_POLICY).unwrap();
+
+ for (name, policy) in [
+ ("default", None),
+ ("mutable", Some(PartitionResizePolicy::Mutable)),
+ ("fixed", Some(PartitionResizePolicy::Fixed)),
+ ] {
+ let created = client
+ .create_topic(
+ &stream,
+ name,
+ &TopicCreateOptions {
+ partitions_count: Some(2),
+ partition_resize_policy: policy,
+ ..TopicCreateOptions::default()
+ },
+ )
+ .await
+ .unwrap();
+ let effective = policy.unwrap_or_default();
+ assert_eq!(
+ created.options[&policy_key].value.to_string_value(),
+ effective.to_string()
+ );
+ assert_eq!(created.options[&policy_key].explicit, policy.is_some());
+ let topic = Identifier::named(name).unwrap();
+
+ let added = client.create_partitions(&stream, &topic, 1).await;
+ if effective == PartitionResizePolicy::Fixed {
+ assert_policy_error(added.unwrap_err(), IggyError::PartitionResizeDisabled);
+ } else {
+ added.unwrap();
+ }
+ let details = client.get_topic(&stream, &topic).await.unwrap().unwrap();
+ assert_eq!(
+ details.partitions.len(),
+ if effective == PartitionResizePolicy::Fixed {
+ 2
+ } else {
+ 3
+ }
+ );
+
+ let deleted = client.delete_partitions(&stream, &topic, 1).await;
+ if effective == PartitionResizePolicy::Fixed {
+ assert_policy_error(deleted.unwrap_err(), IggyError::PartitionResizeDisabled);
+ } else {
+ deleted.unwrap();
+ }
+ let details = client.get_topic(&stream, &topic).await.unwrap().unwrap();
+ assert_eq!(details.partitions.len(), 2);
+
+ let replacement = if effective == PartitionResizePolicy::Fixed {
+ "mutable"
+ } else {
+ "fixed"
+ };
+ let updated = client
+ .update_topic(
+ &stream,
+ &topic,
+ "renamed",
+ &TopicUpdateOptions {
+ raw: BTreeMap::from([(
+ topic_option_keys::PARTITION_RESIZE_POLICY.to_owned(),
+ replacement.to_owned(),
+ )]),
+ ..TopicUpdateOptions::default()
+ },
+ )
+ .await;
+ assert_policy_error(
+ updated.unwrap_err(),
+ IggyError::UnsupportedOptionKey(String::new()),
+ );
+ let details = client.get_topic(&stream, &topic).await.unwrap().unwrap();
+ assert_eq!(details.name, name);
+ assert_eq!(
+ details.options[&policy_key].value.to_string_value(),
+ effective.to_string()
+ );
+ client.delete_topic(&stream, &topic).await.unwrap();
+ }
+}
+
+fn assert_policy_error(error: IggyError, expected: IggyError) {
+ if let IggyError::HttpResponseError(status, body) = error {
+ assert_eq!(status, reqwest::StatusCode::BAD_REQUEST.as_u16());
+ let body: serde_json::Value = serde_json::from_str(&body).unwrap();
+ assert_eq!(body["id"], expected.as_code());
+ assert_eq!(body["code"], expected.as_string());
+ } else {
+ assert_eq!(error.as_code(), expected.as_code(), "{error:?}");
+ }
+}
diff --git a/core/integration/tests/server/cg.rs b/core/integration/tests/server/cg.rs
index f01cd4d66..69a061006 100644
--- a/core/integration/tests/server/cg.rs
+++ b/core/integration/tests/server/cg.rs
@@ -35,7 +35,11 @@ use integration::iggy_harness;
#[iggy_harness(
test_client_transport = [Tcp, WebSocket, Quic],
- server(heartbeat.enabled = true, heartbeat.interval = "60s")
+ server(
+ heartbeat.enabled = true,
+ heartbeat.interval = "60s",
+ consumer_group.rebalancing_timeout = "1s"
+ )
)]
async fn join(harness: &TestHarness) {
consumer_group_join_scenario::run(harness).await;
@@ -51,7 +55,11 @@ async fn single_client(harness: &TestHarness) {
#[iggy_harness(
test_client_transport = [Tcp, WebSocket, Quic],
- server(heartbeat.enabled = true, heartbeat.interval = "60s")
+ server(
+ heartbeat.enabled = true,
+ heartbeat.interval = "60s",
+ consumer_group.rebalancing_timeout = "1s"
+ )
)]
async fn multiple_clients(harness: &TestHarness) {
consumer_group_with_multiple_clients_polling_messages_scenario::run(harness).await;
diff --git a/core/integration/tests/server/consumer_offset_quota_vsr.rs b/core/integration/tests/server/consumer_offset_quota_vsr.rs
index ca9faaf02..080a63299 100644
--- a/core/integration/tests/server/consumer_offset_quota_vsr.rs
+++ b/core/integration/tests/server/consumer_offset_quota_vsr.rs
@@ -327,6 +327,7 @@ async fn given_full_consumer_offset_table_when_creating_another_should_reject_wi
)))
.bearer_auth(&http.token)
.json(&StoreConsumerOffset {
+ context: None,
consumer: Consumer::new(Identifier::numeric(6).expect("consumer identifier")),
partition_id: Some(PARTITION_ID),
offset: 0,
diff --git a/core/integration/tests/server/http_vsr.rs b/core/integration/tests/server/http_vsr.rs
index a900469e7..50645348e 100644
--- a/core/integration/tests/server/http_vsr.rs
+++ b/core/integration/tests/server/http_vsr.rs
@@ -482,6 +482,7 @@ async fn given_consumer_offset_when_stored_and_deleted_should_round_trip(harness
let offsets_path = "/streams/http-offsets/topics/offsets/consumer-offsets";
let store = StoreConsumerOffset {
+ context: None,
consumer: Consumer::new(Identifier::numeric(CONSUMER_ID).expect("consumer id")),
partition_id: Some(PARTITION_ID),
offset: 1,
@@ -598,6 +599,7 @@ async fn given_missing_consumer_offset_when_deleting_should_reject_404_fast(harn
"produce must commit"
);
let store = StoreConsumerOffset {
+ context: None,
consumer: Consumer::new(Identifier::numeric(CONSUMER_ID).expect("consumer id")),
partition_id: Some(PARTITION_ID),
offset: 0,
@@ -706,6 +708,7 @@ async fn given_out_of_range_consumer_offset_when_storing_should_reject_400_fast(
// Offset 5 runs past the committed tail (offset 1), so it is out of range.
let offsets_path = "/streams/http-store-range/topics/offsets/consumer-offsets";
let store = StoreConsumerOffset {
+ context: None,
consumer: Consumer::new(Identifier::numeric(CONSUMER_ID).expect("consumer id")),
partition_id: Some(PARTITION_ID),
offset: 5,
diff --git a/core/integration/tests/server/raw_tcp.rs b/core/integration/tests/server/raw_tcp.rs
index 8dc70c4c4..b65a3fd4c 100644
--- a/core/integration/tests/server/raw_tcp.rs
+++ b/core/integration/tests/server/raw_tcp.rs
@@ -66,6 +66,24 @@ pub(crate) async fn connect_to(addr: SocketAddr) -> TcpStream {
TcpStream::connect(addr).await.unwrap()
}
+pub(crate) async fn partition_context(
+ client: &IggyClient,
+ stream_id: &Identifier,
+ topic_id: &Identifier,
+ partition_id: u32,
+) -> PartitionContext {
+ client
+ .get_topic(stream_id, topic_id)
+ .await
+ .expect("discover partition context")
+ .expect("topic exists")
+ .partitions
+ .into_iter()
+ .find(|partition| partition.id == partition_id)
+ .expect("partition exists")
+ .context
+}
+
pub(crate) fn request_header(
operation: Operation,
client: u128,
diff --git a/core/integration/tests/server/scenarios/authentication_scenario.rs b/core/integration/tests/server/scenarios/authentication_scenario.rs
index 0f296407d..7e05aa8e1 100644
--- a/core/integration/tests/server/scenarios/authentication_scenario.rs
+++ b/core/integration/tests/server/scenarios/authentication_scenario.rs
@@ -33,7 +33,7 @@ use iggy_binary_protocol::WireEncode;
use iggy_binary_protocol::codes::*;
use iggy_binary_protocol::dispatch::COMMAND_TABLE;
use iggy_binary_protocol::requests::consumer_offsets::GetConsumerOffsetRequest;
-use iggy_binary_protocol::requests::messages::PollMessagesRequest;
+use iggy_binary_protocol::requests::messages::{GetSendContextRequest, PollMessagesRequest};
use iggy_binary_protocol::requests::system::{BindSessionRequest, SessionIdentity};
use iggy_binary_protocol::requests::users::login_register::BindSecret;
use iggy_binary_protocol::{ClientVersionInfo, IGGY_PROTOCOL_VERSION, WireName};
@@ -277,6 +277,18 @@ async fn test_all_commands_require_auth(client: &IggyClient, binding_identity: S
}
// Messages
+ GET_SEND_CONTEXT_CODE => client
+ .send_binary_request(
+ code,
+ GetSendContextRequest {
+ stream_id: identifier_to_wire(&ctx.stream_id).unwrap(),
+ topic_id: identifier_to_wire(&ctx.topic_id).unwrap(),
+ partition_id: 0,
+ }
+ .to_bytes(),
+ )
+ .await
+ .map(|_| ()),
SEND_MESSAGES_CODE => {
let mut msgs = vec![
IggyMessage::builder()
diff --git a/core/integration/tests/server/scenarios/invalid_consumer_offset_scenario.rs b/core/integration/tests/server/scenarios/invalid_consumer_offset_scenario.rs
index 92e82fc9e..731c34c40 100644
--- a/core/integration/tests/server/scenarios/invalid_consumer_offset_scenario.rs
+++ b/core/integration/tests/server/scenarios/invalid_consumer_offset_scenario.rs
@@ -18,7 +18,7 @@
use bytes::Bytes;
use iggy::prelude::*;
use iggy_common::IggyError;
-use integration::harness::{TestHarness, assert_clean_system};
+use integration::harness::{TestHarness, assert_clean_system, wait_for_consumer_group_assignment};
const STREAM_NAME: &str = "test-stream-offsets";
const TOPIC_NAME: &str = "test-topic-offsets";
@@ -114,7 +114,18 @@ async fn initialize(client: &IggyClient, stream: &Identifier, topic: &Identifier
)
.await;
match join_result {
- Ok(_) => true,
+ Ok(_) => {
+ wait_for_consumer_group_assignment(
+ client,
+ stream,
+ topic,
+ &Identifier::named(CONSUMER_GROUP_NAME).unwrap(),
+ 1,
+ super::CONVERGENCE_TIMEOUT,
+ )
+ .await;
+ true
+ }
Err(e) => {
assert_eq!(e.as_code(), IggyError::FeatureUnavailable.as_code());
false
diff --git a/core/integration/tests/server/scenarios/log_rotation_scenario.rs b/core/integration/tests/server/scenarios/log_rotation_scenario.rs
index d752f412f..8827a1736 100644
--- a/core/integration/tests/server/scenarios/log_rotation_scenario.rs
+++ b/core/integration/tests/server/scenarios/log_rotation_scenario.rs
@@ -108,6 +108,11 @@ fn build_server_config(log_config: &LogRotationTestConfig) -> TestServerConfig {
"IGGY_LOGGING_RETENTION".to_string(),
format!("{}", log_config.retention),
);
+ // The log generator performs hundreds of durable partition deletions.
+ extra_envs.insert(
+ "IGGY_SHARDING_RECONCILE_PERIODIC_INTERVAL".to_string(),
+ "100ms".to_string(),
+ );
TestServerConfig::builder().extra_envs(extra_envs).build()
}
diff --git a/core/integration/tests/server/scenarios/mod.rs b/core/integration/tests/server/scenarios/mod.rs
index cea6fa6e1..0a28726bc 100644
--- a/core/integration/tests/server/scenarios/mod.rs
+++ b/core/integration/tests/server/scenarios/mod.rs
@@ -265,6 +265,17 @@ async fn join_consumer_group(client: &IggyClient) {
)
.await
.unwrap();
+
+ let members_count = get_consumer_group(client).await.members_count;
+ integration::harness::wait_for_consumer_group_assignment(
+ client,
+ &Identifier::named(STREAM_NAME).unwrap(),
+ &Identifier::named(TOPIC_NAME).unwrap(),
+ &Identifier::named(CONSUMER_GROUP_NAME).unwrap(),
+ members_count,
+ CONVERGENCE_TIMEOUT,
+ )
+ .await;
}
async fn leave_consumer_group(client: &IggyClient) {
diff --git a/core/integration/tests/server/scenarios/permissions_scenario.rs b/core/integration/tests/server/scenarios/permissions_scenario.rs
index fa5075f2c..000b2aefd 100644
--- a/core/integration/tests/server/scenarios/permissions_scenario.rs
+++ b/core/integration/tests/server/scenarios/permissions_scenario.rs
@@ -748,6 +748,11 @@ async fn test_message_permissions(harness: &TestHarness, root_client: &IggyClien
.await
.expect("send_messages: send_messages should work");
+ assert_unauthorized(
+ client.get_topic(&stream_id, &topic_id).await,
+ "send_messages: get_topic",
+ );
+
// Cannot poll messages
assert_unauthorized(
client
diff --git a/core/integration/tests/server/scenarios/stale_client_consumer_group_scenario.rs b/core/integration/tests/server/scenarios/stale_client_consumer_group_scenario.rs
index b9cf68c46..9b83c0601 100644
--- a/core/integration/tests/server/scenarios/stale_client_consumer_group_scenario.rs
+++ b/core/integration/tests/server/scenarios/stale_client_consumer_group_scenario.rs
@@ -130,11 +130,15 @@ async fn given_live_member_when_backup_restarts_should_preserve_membership(
.join_consumer_group(&stream, &topic, &group_id)
.await
.unwrap();
- let before = consumer_client
- .get_consumer_group(&stream, &topic, &group_id)
- .await
- .unwrap()
- .unwrap();
+ let before = integration::harness::wait_for_consumer_group_assignment(
+ &consumer_client,
+ &stream,
+ &topic,
+ &group_id,
+ 1,
+ STALE_EVICTION_TIMEOUT,
+ )
+ .await;
assert_eq!(before.members_count, 1);
harness.kill_node(2).unwrap();
@@ -171,6 +175,15 @@ async fn verify_replacement_after_restart(
.join_consumer_group(&stream, &topic, &group_id)
.await
.unwrap();
+ integration::harness::wait_for_consumer_group_assignment(
+ &original,
+ &stream,
+ &topic,
+ &group_id,
+ 1,
+ STALE_EVICTION_TIMEOUT,
+ )
+ .await;
let before = original
.poll_messages(
&stream,
@@ -184,6 +197,17 @@ async fn verify_replacement_after_restart(
.await
.unwrap();
assert_eq!(before.messages.len(), 1);
+ // A poll reply does not acknowledge durable AUTO completion.
+ original
+ .store_consumer_offset(
+ &consumer,
+ &stream,
+ &topic,
+ Some(before.partition_id),
+ before.messages[0].header.offset,
+ )
+ .await
+ .expect("persist the restart cursor before stopping the original member");
match restart {
Restart::Client => {}
Restart::ServerShutdown => harness.stop_node(0).unwrap(),
diff --git a/core/integration/tests/server/scenarios/system_scenario.rs b/core/integration/tests/server/scenarios/system_scenario.rs
index 5c4418780..1bf1ed635 100644
--- a/core/integration/tests/server/scenarios/system_scenario.rs
+++ b/core/integration/tests/server/scenarios/system_scenario.rs
@@ -502,7 +502,15 @@ pub async fn run(harness: &TestHarness) {
match result {
Ok(_) => {
- let consumer_group = get_consumer_group(&client).await;
+ let consumer_group = integration::harness::wait_for_consumer_group_assignment(
+ &client,
+ &Identifier::named(STREAM_NAME).unwrap(),
+ &Identifier::named(TOPIC_NAME).unwrap(),
+ &Identifier::named(CONSUMER_GROUP_NAME).unwrap(),
+ 1,
+ TOPIC_CONVERGENCE_TIMEOUT,
+ )
+ .await;
assert_eq!(consumer_group.id, consumer_group_id);
assert_eq!(consumer_group.partitions_count, PARTITIONS_COUNT);
assert_eq!(consumer_group.name, CONSUMER_GROUP_NAME);
diff --git a/core/metadata/src/impls/metadata.rs b/core/metadata/src/impls/metadata.rs
index 600bb2b9b..48fbd15ed 100644
--- a/core/metadata/src/impls/metadata.rs
+++ b/core/metadata/src/impls/metadata.rs
@@ -44,7 +44,7 @@ use iggy_binary_protocol::WireIdentifier;
use iggy_binary_protocol::primitives::partition_assignment::CreatedPartitionAssignment;
use iggy_binary_protocol::requests::partitions::CreatePartitionsRequest as WireCreatePartitionsRequest;
use iggy_binary_protocol::requests::partitions::CreatePartitionsWithAssignmentsRequest as PersistedCreatePartitionsRequest;
-use iggy_binary_protocol::requests::system::SessionIdentity;
+use iggy_binary_protocol::requests::system::{FinalizeSessionRequest, SessionIdentity};
use iggy_binary_protocol::requests::topics::CreateTopicRequest as WireCreateTopicRequest;
use iggy_binary_protocol::requests::topics::CreateTopicWithAssignmentsRequest as PersistedCreateTopicRequest;
use iggy_binary_protocol::{
@@ -795,9 +795,12 @@ where
.borrow()
.get_epoch(header.client)
.unwrap_or(header.op);
- mux_stm
- .streams()
- .refresh_consumer_group_session(header.client, epoch);
+ mux_stm.streams().refresh_consumer_group_session(
+ header.client,
+ epoch,
+ header.op,
+ iggy_common::IggyTimestamp::from(header.timestamp),
+ );
return build_reply_message(&header, &bytes::Bytes::new());
}
if header.operation == Operation::Logout {
@@ -823,19 +826,23 @@ where
mux_stm.streams().remove_consumer_group_member(
header.client,
iggy_common::IggyTimestamp::from(header.timestamp),
+ header.op,
);
}
return build_reply_message(&header, &bytes::Bytes::new());
}
if header.operation == Operation::FinalizeSession {
- let identity = SessionIdentity::decode_from(prepare.body()).unwrap_or_else(|_| {
+ let request = FinalizeSessionRequest::decode_from(prepare.body()).unwrap_or_else(|_| {
fatal(
FatalReason::UnreconcilableLogFrontier,
"invalid session finalization",
)
});
- if table_mutations_allowed {
- client_table.borrow_mut().finalize_session(identity);
+ if table_mutations_allowed
+ && mux_stm.streams().read(|inner| inner.namespace_revision)
+ == request.namespace_revision
+ {
+ client_table.borrow_mut().finalize_session(request.identity);
}
return build_reply_message(&header, &bytes::Bytes::new());
}
@@ -864,7 +871,10 @@ where
if table_mutations_allowed
&& header.client != RESERVED_CLIENT_ID
&& !message_bus::is_auto_commit_client(header.client)
- && header.operation != Operation::CompleteConsumerGroupRevocation
+ && !matches!(
+ header.operation,
+ Operation::CompleteConsumerGroupRevocation | Operation::CompleteLifecycle
+ )
{
let outcome =
client_table
@@ -897,6 +907,7 @@ fn apply_user_revocation<M: StreamsFrontend>(
mux_stm.streams().remove_consumer_group_member(
identity.client_id,
iggy_common::IggyTimestamp::from(header.timestamp),
+ header.op,
);
}
}
@@ -1368,7 +1379,12 @@ where
message.body(),
request_checksum,
);
- apply_preflight_consensus_plane(consensus, outcome, client_id).await
+ apply_preflight_consensus_plane(
+ consensus,
+ self.fence_lifecycle_preflight(outcome, &message),
+ client_id,
+ )
+ .await
}
} else {
let outcome = request_preflight(
@@ -1379,7 +1395,12 @@ where
request,
operation,
);
- apply_preflight_consensus_plane(consensus, outcome, client_id).await
+ apply_preflight_consensus_plane(
+ consensus,
+ self.fence_lifecycle_preflight(outcome, &message),
+ client_id,
+ )
+ .await
};
if !dispatch {
return;
@@ -2428,6 +2449,29 @@ where
})
}
+ fn fence_lifecycle_preflight(
+ &self,
+ outcome: PreflightOutcome,
+ request: &Message<RoutedRequestHeader>,
+ ) -> PreflightOutcome {
+ if let PreflightOutcome::Replay(reply) = &outcome
+ && let Some(bytes) = reply.as_slice().get(..size_of::<ReplyHeader>())
+ && let Ok(header) = bytemuck::checked::try_from_bytes::<ReplyHeader>(bytes)
+ && self.mux_stm.streams().lifecycle_pending(header.op)
+ {
+ return PreflightOutcome::NotReady;
+ }
+ if matches!(outcome, PreflightOutcome::Dispatch)
+ && self
+ .mux_stm
+ .streams()
+ .request_lifecycle_blocked(request.header().operation, request.body())
+ {
+ return PreflightOutcome::Reject(IggyError::TransientNotAccepted.as_code());
+ }
+ outcome
+ }
+
/// Turn a non-`Dispatch` [`PreflightOutcome`] into the answer the home
/// shard writes to the originating socket, or `None` to dispatch.
///
@@ -2752,6 +2796,7 @@ where
pub async fn submit_session_finalization(
&self,
identity: SessionIdentity,
+ namespace_revision: u64,
) -> Result<Option<u64>, MetadataSubmitError> {
let consensus = self
.consensus
@@ -2760,6 +2805,14 @@ where
if !self.is_caught_up_primary() {
return Err(MetadataSubmitError::NotCaughtUp);
}
+ if self
+ .mux_stm
+ .streams()
+ .read(|inner| inner.namespace_revision)
+ != namespace_revision
+ {
+ return Ok(None);
+ }
if !self
.client_table
.borrow()
@@ -2784,7 +2837,11 @@ where
consensus,
&header,
Operation::FinalizeSession,
- &identity.to_bytes(),
+ &FinalizeSessionRequest {
+ identity,
+ namespace_revision,
+ }
+ .to_bytes(),
),
capacity,
);
@@ -2819,26 +2876,67 @@ where
&self,
stream_id: u32,
topic_id: u32,
- group_id: u64,
- source_client_id: u128,
partition_id: u32,
+ installation: iggy_binary_protocol::requests::partitions::InstallConsumerGroupOwnerRequest,
+ partition_op: u64,
) -> Result<u64, MetadataSubmitError> {
- const INTERNAL_REQUEST_ID: u64 = u64::MAX;
- // Reserved internal client id, distinct per (group, partition) target.
- // The high 64 bits are all-ones -- never coordinator-minted (those carry
- // a small home-shard number in the top bits) -- and the low 64 bits pack
- // (group_id, partition_id). Distinct ids matter: the pipeline dedups by
- // client id, so a shared id would cap internal completions at one
- // in-flight cluster-wide and drain a wide rebalance one per consensus
- // round-trip. Per-target ids let completions for different partitions
- // pipeline concurrently while still deduping a retry of the same target.
- let internal_client_id: u128 =
- (u128::from(u64::MAX) << 64) | (u128::from(group_id) << 32) | u128::from(partition_id);
+ let request = CompleteConsumerGroupRevocationRequest {
+ stream_id: WireIdentifier::numeric(stream_id),
+ topic_id: WireIdentifier::numeric(topic_id),
+ partition_id,
+ installation,
+ partition_op,
+ };
+ self.submit_partition_completion(
+ Operation::CompleteConsumerGroupRevocation,
+ server_common::sharding::IggyNamespace::new(
+ stream_id as usize,
+ topic_id as usize,
+ partition_id as usize,
+ )
+ .inner(),
+ &request.to_bytes(),
+ )
+ .await
+ }
+ /// Commit durable partition completion on the metadata primary.
+ ///
+ /// # Errors
+ /// Returns the same leadership, pipeline and cancellation errors as
+ /// [`Self::submit_complete_revocation_in_process`].
+ #[allow(clippy::future_not_send)]
+ pub async fn submit_complete_lifecycle_in_process(
+ &self,
+ request: crate::stm::lifecycle::CompleteLifecycleRequest,
+ ) -> Result<u64, MetadataSubmitError> {
+ self.submit_partition_completion(
+ Operation::CompleteLifecycle,
+ server_common::sharding::IggyNamespace::new(
+ request.stream_id as usize,
+ request.topic_id as usize,
+ request.partition_id as usize,
+ )
+ .inner(),
+ &request.to_bytes(),
+ )
+ .await
+ }
+
+ #[allow(clippy::future_not_send)]
+ async fn submit_partition_completion(
+ &self,
+ operation: Operation,
+ namespace: u64,
+ body: &[u8],
+ ) -> Result<u64, MetadataSubmitError> {
+ // One internal client per partition serializes completions across groups.
+ const INTERNAL_REQUEST_ID: u64 = u64::MAX;
+ let internal_client_id = (u128::from(u64::MAX) << 64) | u128::from(namespace);
let consensus = self
.consensus
.as_ref()
- .expect("submit_complete_revocation_in_process: consensus only exists on shard 0");
+ .expect("submit_partition_completion: consensus only exists on shard 0");
// Deliberately bounce-based (no request-queue absorption, unlike the
// client submit paths above): the caller is the partition
@@ -2861,19 +2959,12 @@ where
return Err(MetadataSubmitError::PipelineFull);
}
- let request = CompleteConsumerGroupRevocationRequest {
- stream_id: WireIdentifier::numeric(stream_id),
- topic_id: WireIdentifier::numeric(topic_id),
- group_id,
- source_client_id,
- partition_id,
- };
- let body = request.to_bytes();
- let message = build_complete_revocation_request_message(
+ let message = build_partition_completion_request_message(
consensus,
+ operation,
internal_client_id,
INTERNAL_REQUEST_ID,
- &body,
+ body,
);
let prepare = consensus::seal_prepare_capacity(
message.project(consensus),
@@ -3050,7 +3141,11 @@ where
request,
operation,
);
- if let Some(answer) = Self::answer_preflight(consensus, &request_header, outcome) {
+ if let Some(answer) = Self::answer_preflight(
+ consensus,
+ &request_header,
+ self.fence_lifecycle_preflight(outcome, &message),
+ ) {
return answer;
}
@@ -3376,6 +3471,15 @@ where
// (slot-first ordering, see take_reply_sender). Dropped
// receiver: ignored. Still inside the sync region, so an
// in-process awaiter is woken atomically with its commit.
+ let reply = if self.mux_stm.streams().lifecycle_pending(prepare_header.op) {
+ let pending =
+ crate::stm::result::ApplyReply::err(IggyError::TransientNotCommitted.as_code());
+ build_reply_message_with(&prepare_header, pending.reply_body_len(), |dst| {
+ pending.write_reply_body(dst);
+ })
+ } else {
+ reply
+ };
let had_in_process_subscriber = entry.has_reply_sender();
if let Some(sender) = entry.take_reply_sender() {
let _ = sender.send(reply.clone());
@@ -3510,7 +3614,12 @@ where
req.message.body(),
request_checksum,
);
- apply_preflight_consensus_plane(consensus, outcome, client_id).await
+ apply_preflight_consensus_plane(
+ consensus,
+ self.fence_lifecycle_preflight(outcome, &req.message),
+ client_id,
+ )
+ .await
}
} else {
let outcome = request_preflight(
@@ -3521,7 +3630,12 @@ where
request,
operation,
);
- apply_preflight_consensus_plane(consensus, outcome, client_id).await
+ apply_preflight_consensus_plane(
+ consensus,
+ self.fence_lifecycle_preflight(outcome, &req.message),
+ client_id,
+ )
+ .await
};
if !dispatch {
continue;
@@ -4322,8 +4436,9 @@ where
msg
}
-fn build_complete_revocation_request_message<B, P>(
+fn build_partition_completion_request_message<B, P>(
consensus: &VsrConsensus<B, P>,
+ operation: Operation,
client_id: u128,
request: u64,
body: &[u8],
@@ -4343,7 +4458,7 @@ where
.expect("zeroed bytes are a valid RoutedRequestHeader");
*header = RoutedRequestHeader {
command: Command::Request,
- operation: Operation::CompleteConsumerGroupRevocation,
+ operation,
size: u32::try_from(total).expect("request size fits u32"),
cluster: consensus.cluster(),
view: consensus.view(),
@@ -4634,6 +4749,7 @@ fn validate_partitions_limit(
mod tests {
use super::*;
use crate::stm::StateHandler;
+ use crate::stm::authz::GatedApply;
use crate::stm::consumer_group::JoinConsumerGroupRequest;
use crate::stm::stream::{Streams, StreamsInner};
use crate::stm::user::{Users, UsersInner};
@@ -6757,7 +6873,7 @@ mod tests {
.unwrap()
.epoch;
let consensus = metadata.consensus.as_ref().unwrap();
- consensus.advance_commit_max(session + 1);
+ consensus.advance_commit_max(consensus.commit_min() + 1);
assert_eq!(
metadata
.submit_expired_logout_in_process(CLIENT, Some(session))
@@ -6784,12 +6900,13 @@ mod tests {
.await
.unwrap()
.epoch;
+ let logout_op = metadata.consensus.as_ref().unwrap().commit_min() + 1;
assert_eq!(
metadata
.submit_expired_logout_in_process(CLIENT, Some(session))
.await
.unwrap(),
- Some(session + 1)
+ Some(logout_op)
);
assert_eq!(
metadata
@@ -6806,13 +6923,15 @@ mod tests {
async fn expired_logout_removes_a_member_without_a_client_table_entry() {
const CLIENT: u128 = 1;
let (_dir, metadata) = metadata_with_group_member(CLIENT).await;
+ let session = metadata.client_table.borrow().get_epoch(CLIENT).unwrap();
+ let logout_op = metadata.consensus.as_ref().unwrap().commit_min() + 1;
assert_eq!(
metadata
- .submit_expired_logout_in_process(CLIENT, Some(1))
+ .submit_expired_logout_in_process(CLIENT, Some(session))
.await
.unwrap(),
- Some(2)
+ Some(logout_op)
);
assert_eq!(
metadata
@@ -6830,7 +6949,7 @@ mod tests {
[]
);
assert_eq!(metadata.mux_stm.streams().consumer_group_count(), 1);
- assert_eq!(metadata.consensus.as_ref().unwrap().commit_min(), 2);
+ assert_eq!(metadata.consensus.as_ref().unwrap().commit_min(), logout_op);
}
#[compio::test]
@@ -6839,6 +6958,7 @@ mod tests {
const WATERMARK: u64 = 5;
let (_dir, metadata) = metadata_with_group_member(CLIENT).await;
let session = metadata.client_table.borrow().get_epoch(CLIENT).unwrap();
+ let committed_op = metadata.consensus.as_ref().unwrap().commit_min();
let reply = committed_reply(CLIENT, WATERMARK, Operation::CreateStream, 0);
assert_eq!(
metadata
@@ -6869,7 +6989,11 @@ mod tests {
.len(),
1
);
- assert_eq!(metadata.consensus.as_ref().unwrap().commit_min(), session);
+ assert_eq!(
+ metadata.consensus.as_ref().unwrap().commit_min(),
+ committed_op,
+ "a refused logout must not commit a metadata operation"
+ );
}
}
@@ -6893,6 +7017,124 @@ mod tests {
assert_eq!(metadata.applied_frontier.recovered_revision(), revision);
}
+ #[compio::test]
+ async fn given_group_logout_when_finalizing_session_should_preserve_retirement_coverage() {
+ const CLIENT: u128 = 1;
+ let (_dir, metadata) = metadata_with_group_member(CLIENT).await;
+ let namespace_revision = metadata.mux_stm.namespace_revision();
+ let session = metadata.client_table.borrow().get_epoch(CLIENT).unwrap();
+ metadata
+ .submit_logout_in_process(CLIENT, session, 1)
+ .await
+ .unwrap();
+ let identity = metadata
+ .client_table
+ .borrow()
+ .ended_sessions()
+ .next()
+ .unwrap();
+ assert_eq!(metadata.mux_stm.namespace_revision(), namespace_revision);
+ assert_ne!(
+ metadata.mux_stm.streams().read(|inner| inner.revision),
+ namespace_revision
+ );
+
+ assert!(
+ metadata
+ .submit_session_finalization(identity, namespace_revision)
+ .await
+ .unwrap()
+ .is_some(),
+ "membership changes must preserve retirement coverage"
+ );
+ assert_eq!(
+ metadata.client_table.borrow().ended_sessions().next(),
+ None,
+ "successful finalization must release the ended session's slot"
+ );
+ }
+
+ #[compio::test]
+ async fn given_retirement_reports_when_namespace_changes_before_apply_should_keep_the_session()
+ {
+ const CLIENT: u128 = 1;
+ let (_dir, metadata) = metadata_with_group_member(CLIENT).await;
+ let session = metadata.client_table.borrow().get_epoch(CLIENT).unwrap();
+ metadata
+ .submit_logout_in_process(CLIENT, session, 1)
+ .await
+ .unwrap();
+ let identity = metadata
+ .client_table
+ .borrow()
+ .ended_sessions()
+ .next()
+ .unwrap();
+ let revision = metadata.mux_stm.namespace_revision();
+ let consensus = metadata.consensus.as_ref().unwrap();
+ let create = build_prepare_message(
+ consensus,
+ &RoutedRequestHeader::default(),
+ Operation::CreatePartitionsWithAssignments,
+ &PersistedCreatePartitionsRequest {
+ created_view: 0,
+ request: WireCreatePartitionsRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ partitions_count: 1,
+ },
+ partitions: vec![CreatedPartitionAssignment {
+ partition_id: 1,
+ consensus_group_id: 2,
+ }],
+ }
+ .to_bytes(),
+ );
+ assert_eq!(
+ gated_apply(metadata.mux_stm.as_ref(), create).unwrap().code,
+ 0
+ );
+ let current_revision = metadata.mux_stm.namespace_revision();
+ assert_ne!(current_revision, revision);
+ assert_eq!(
+ metadata
+ .submit_session_finalization(identity, revision)
+ .await
+ .unwrap(),
+ None
+ );
+
+ for (reported_revision, expected_session) in
+ [(revision, Some(identity)), (current_revision, None)]
+ {
+ let prepare = consensus::seal_prepare_capacity(
+ build_prepare_message(
+ consensus,
+ &RoutedRequestHeader::default(),
+ Operation::FinalizeSession,
+ &FinalizeSessionRequest {
+ identity,
+ namespace_revision: reported_revision,
+ }
+ .to_bytes(),
+ ),
+ metadata.client_table_capacity(),
+ );
+ apply_committed_prepare(
+ metadata.mux_stm.as_ref(),
+ &metadata.client_table,
+ true,
+ |_| {},
+ prepare,
+ );
+ assert_eq!(
+ metadata.client_table.borrow().ended_sessions().next(),
+ expected_session,
+ "finalization must use the namespace revision at ordered apply"
+ );
+ }
+ }
+
#[compio::test]
async fn stale_logout_preserves_a_newer_session_and_its_membership() {
const CLIENT: u128 = 1;
@@ -6904,6 +7146,7 @@ mod tests {
.await
.unwrap()
.epoch;
+ let committed_op = metadata.consensus.as_ref().unwrap().commit_min();
metadata
.submit_logout_in_process(CLIENT, OLD_SESSION, EXPIRED_SESSION_REQUEST_ID)
@@ -6933,7 +7176,8 @@ mod tests {
);
assert_eq!(
metadata.consensus.as_ref().unwrap().commit_min(),
- new_session
+ committed_op,
+ "an old session must not advance the committed frontier"
);
}
@@ -7093,7 +7337,11 @@ mod tests {
consensus,
&request,
Operation::FinalizeSession,
- &ended[0].to_bytes(),
+ &FinalizeSessionRequest {
+ identity: ended[0],
+ namespace_revision: metadata.mux_stm.namespace_revision(),
+ }
+ .to_bytes(),
),
metadata.client_table_capacity(),
));
@@ -7281,18 +7529,6 @@ mod tests {
&mut inner,
timestamp,
);
- let _ = StateHandler::apply(
- &JoinConsumerGroupRequest {
- stream_id: WireIdentifier::numeric(0),
- topic_id: WireIdentifier::numeric(0),
- group_id: WireIdentifier::numeric(0),
- client_id,
- in_flight: Vec::new(),
- session: None,
- },
- &mut inner,
- timestamp,
- );
let dir = tempfile::tempdir().unwrap();
let mut metadata = metadata_plane();
metadata.journal = Some(
@@ -7305,6 +7541,37 @@ mod tests {
.submit_register_in_process(client_id, USER, [0x5a; 32])
.await
.unwrap();
+ metadata.mux_stm.users().ensure_root_user("root", "hash");
+ let consensus = metadata.consensus.as_ref().unwrap();
+ let join = consensus::seal_prepare_capacity(
+ build_prepare_message(
+ consensus,
+ &RoutedRequestHeader::default(),
+ Operation::JoinConsumerGroup,
+ &JoinConsumerGroupRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ group_id: WireIdentifier::numeric(0),
+ client_id,
+ session: metadata.client_table.borrow().get_epoch(client_id).unwrap(),
+ }
+ .to_bytes(),
+ ),
+ metadata.client_table_capacity(),
+ );
+ let join_op = join.header().op;
+ consensus.pipeline_message(PlaneKind::Metadata, &join);
+ metadata
+ .journal
+ .as_ref()
+ .unwrap()
+ .handle()
+ .append(join)
+ .await
+ .unwrap();
+ consensus.advance_commit_max(join_op);
+ metadata.resume_stranded_commits().await;
+ assert_eq!(consensus.commit_min(), join_op);
assert_eq!(
metadata
.mux_stm
diff --git a/core/metadata/src/impls/recovery.rs b/core/metadata/src/impls/recovery.rs
index cebed0189..5530f80c7 100644
--- a/core/metadata/src/impls/recovery.rs
+++ b/core/metadata/src/impls/recovery.rs
@@ -25,6 +25,7 @@ use consensus::{
};
use iggy_binary_protocol::WireDecode;
use iggy_binary_protocol::consensus::{CHECKSUM_UNSEALED, Operation, PrepareHeader};
+use iggy_binary_protocol::requests::system::FinalizeSessionRequest;
use iggy_common::IggyError;
use journal::Journal as _;
use journal::prepare_journal::{JournalError, PrepareJournal};
@@ -660,13 +661,12 @@ where
"committed finalization is missing from WAL",
))
})?;
- let identity = iggy_binary_protocol::requests::system::SessionIdentity::decode_from(
- prepare.body(),
- )
- .map_err(|_| {
+ let request = FinalizeSessionRequest::decode_from(prepare.body()).map_err(|_| {
RecoveryError::RetryProtection(consensus::ClientTableWireError::InvalidReply)
})?;
- client_table.finalize_session(identity);
+ if mux_stm.namespace_revision() == request.namespace_revision {
+ client_table.finalize_session(request.identity);
+ }
last_applied_op = Some(header.op);
continue;
}
@@ -694,7 +694,10 @@ where
}
// Every client commit must restore its original receipt before admission.
if header.client != consensus::client_table::RESERVED_CLIENT_ID
- && header.operation != Operation::CompleteConsumerGroupRevocation
+ && !matches!(
+ header.operation,
+ Operation::CompleteConsumerGroupRevocation | Operation::CompleteLifecycle
+ )
{
let cached = build_reply_message_with(header, reply.reply_body_len(), |dst| {
reply.write_reply_body(dst);
@@ -854,8 +857,8 @@ fn verify_checkpoint_pairing(
mod tests {
use super::*;
use crate::impls::metadata::{StreamsFrontend, checkpoint_checksum};
- use crate::stm::snapshot::SNAPSHOT_FORMAT_VERSION;
- use crate::stm::stream::Streams;
+ use crate::stm::snapshot::{SNAPSHOT_FORMAT_VERSION, Snapshotable};
+ use crate::stm::stream::{Streams, StreamsSnapshot};
use crate::stm::user::Users;
use consensus::CLIENTS_TABLE_MAX;
use iggy_binary_protocol::consensus::{Command, Operation};
@@ -1467,6 +1470,91 @@ mod tests {
);
}
+ #[compio::test]
+ async fn given_retirement_evidence_when_replaying_after_namespace_change_should_keep_the_session()
+ {
+ const CLIENT: u128 = 7;
+ const USER: u32 = 0;
+ const SNAPSHOT_OP: u64 = 2;
+ const CURRENT_REVISION: u64 = 2;
+ type SessionStm = MuxStateMachine<iggy_common::variadic!(Users, Streams)>;
+
+ for reported_revision in [CURRENT_REVISION - 1, CURRENT_REVISION] {
+ let dir = tempdir().unwrap();
+ let metadata_dir = dir.path().join("metadata");
+ std::fs::create_dir_all(&metadata_dir).unwrap();
+ let mut table = ClientTable::new(CLIENTS_TABLE_MAX);
+ let register = make_client_prepare(1, Operation::Register, CLIENT, USER, 0);
+ table
+ .commit_register(
+ CLIENT,
+ USER,
+ [0x5a; 32],
+ build_reply_message(register.header(), &bytes::Bytes::new()),
+ )
+ .unwrap();
+ let logout = make_client_prepare(SNAPSHOT_OP, Operation::Logout, CLIENT, USER, 1);
+ table
+ .commit_logout(
+ CLIENT,
+ USER,
+ 1,
+ build_reply_message(logout.header(), &bytes::Bytes::new()),
+ )
+ .unwrap();
+ let identity = table.ended_sessions().next().unwrap();
+ let mut snapshot = protected_snapshot(SNAPSHOT_OP);
+ snapshot.snapshot_mut().client_table = Some(table.to_snapshot());
+ snapshot.snapshot_mut().users = Some(Users::default().to_snapshot());
+ snapshot.snapshot_mut().streams = Some(StreamsSnapshot {
+ lifecycle_intents: std::collections::BTreeMap::default(),
+ items: Vec::new(),
+ revision: CURRENT_REVISION,
+ namespace_revision: CURRENT_REVISION,
+ });
+ snapshot
+ .persist(&metadata_dir.join("snapshot.bin"))
+ .await
+ .unwrap();
+
+ let body = FinalizeSessionRequest {
+ identity,
+ namespace_revision: reported_revision,
+ }
+ .to_bytes();
+ let mut prepare = make_prepare(SNAPSHOT_OP + 1, body.len());
+ prepare.as_mut_slice()[HEADER_SIZE..].copy_from_slice(&body);
+ let prepare = prepare.transmute_header(|old, header: &mut PrepareHeader| {
+ *header = old;
+ header.operation = Operation::FinalizeSession;
+ header.checksum_body = u128::from(iggy_common::calculate_checksum(&body));
+ });
+ {
+ let journal = PrepareJournal::open(&metadata_dir.join("journal.wal"), SNAPSHOT_OP)
+ .await
+ .unwrap();
+ journal.append(prepare).await.unwrap();
+ journal.storage_ref().fsync().await.unwrap();
+ }
+ let recovered = recover::<SessionStm>(
+ dir.path(),
+ SOLO,
+ journal::prepare_journal::DEFAULT_SLOT_COUNT,
+ CLIENTS_TABLE_MAX,
+ |_| {},
+ |_, _| {},
+ )
+ .await
+ .unwrap();
+ assert_eq!(recovered.last_applied_op, Some(SNAPSHOT_OP + 1));
+ assert_eq!(
+ recovered.client_table.ended_sessions().next(),
+ (reported_revision != CURRENT_REVISION).then_some(identity),
+ "WAL replay must preserve protection when the report covered an older partition set"
+ );
+ }
+ }
+
// The IGGY-137 restart contract: a rebooted node must remember where
// each client left off. Replay re-mints the epoch, restores the
// watermark, and re-caches the reply so a retry of the last committed
diff --git a/core/metadata/src/stm/authz.rs b/core/metadata/src/stm/authz.rs
index 2508c3483..69c60bc00 100644
--- a/core/metadata/src/stm/authz.rs
+++ b/core/metadata/src/stm/authz.rs
@@ -95,12 +95,19 @@ pub trait GatedApply:
/// # Errors
/// Propagates the underlying [`StateMachine::update`] error.
fn gated_update(&self, prepare: Message<PrepareHeader>) -> Result<ApplyReply, IggyError>;
+
+ /// Partition-set revision used to validate retirement coverage during WAL replay.
+ fn namespace_revision(&self) -> u64;
}
impl GatedApply for MuxStateMachine<variadic!(Users, Streams)> {
fn gated_update(&self, prepare: Message<PrepareHeader>) -> Result<ApplyReply, IggyError> {
gated_apply(self, prepare)
}
+
+ fn namespace_revision(&self) -> u64 {
+ self.streams().read(|inner| inner.namespace_revision)
+ }
}
/// The root user is always the first user created (slab id 0) and holds every
@@ -321,7 +328,11 @@ pub(crate) fn authorize(
| Operation::DeletePersonalAccessToken
| Operation::SendMessages
| Operation::StoreConsumerOffset
- | Operation::DeleteConsumerOffset => None,
+ | Operation::DeleteConsumerOffset
+ | Operation::CompleteLifecycle
+ | Operation::TransitionPartitionHistory
+ | Operation::InstallConsumerGroupOwner
+ | Operation::RetireConsumerGroupOwners => None,
}
}
@@ -420,6 +431,10 @@ impl GatedApply for MuxStateMachine<()> {
fn gated_update(&self, prepare: Message<PrepareHeader>) -> Result<ApplyReply, IggyError> {
self.update(prepare)
}
+
+ fn namespace_revision(&self) -> u64 {
+ 0
+ }
}
#[cfg(test)]
diff --git a/core/metadata/src/stm/consumer_group.rs b/core/metadata/src/stm/consumer_group.rs
index 3b3923ffd..bc4b54ecd 100644
--- a/core/metadata/src/stm/consumer_group.rs
+++ b/core/metadata/src/stm/consumer_group.rs
@@ -28,51 +28,65 @@
use crate::stm::StateHandler;
use crate::stm::id_slab::IdSlab;
+use crate::stm::lifecycle::LifecycleAction;
use crate::stm::result::{
ApplyReply, CreateConsumerGroupResult, DeleteConsumerGroupResult, JoinConsumerGroupResult,
LeaveConsumerGroupResult,
};
-use crate::stm::stream::StreamsInner;
+use crate::stm::stream::{Partition, StreamsInner};
use bytes::Bytes;
use bytes::{BufMut, BytesMut};
use iggy_binary_protocol::WireIdentifier;
-use iggy_binary_protocol::codec::{
- WireDecode, WireEncode, bounded_capacity, read_u32_le, read_u64_le, read_u128_le,
-};
+use iggy_binary_protocol::codec::{WireDecode, WireEncode, read_u32_le, read_u64_le, read_u128_le};
+use iggy_binary_protocol::primitives::partition_history::ConsumerGroupOwner;
use iggy_binary_protocol::requests::consumer_groups::{
CreateConsumerGroupRequest, DeleteConsumerGroupRequest,
};
+use iggy_binary_protocol::requests::partitions::InstallConsumerGroupOwnerRequest;
use iggy_binary_protocol::responses::consumer_groups::consumer_group_response::ConsumerGroupResponse;
use iggy_binary_protocol::responses::consumer_groups::get_consumer_group::ConsumerGroupDetailsResponse;
use iggy_common::IggyTimestamp;
use serde::{Deserialize, Serialize};
+use std::collections::BTreeMap;
use std::sync::Arc;
-/// A partition being cooperatively handed off from this member to `target`.
-///
-/// The source keeps polling/committing it until the handoff completes (the
-/// consumer drains what it polled, or the revocation times out), so an
-/// in-flight batch is never lost mid-rebalance.
-#[derive(Debug, Clone)]
-pub struct PendingRevocation {
- pub partition_id: usize,
- /// Slab key of the member the partition moves to once completed.
- pub target_member: usize,
- /// Monotonic micros when the revocation was created, for timeout.
+/// A desired owner remains inactive until its exact partition installation is durable.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+pub struct PendingOwnership {
+ pub owner: ConsumerGroupOwner,
+ pub metadata_op: u64,
+ pub created_at: u64,
+ pub skip_drain: bool,
+}
+
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub struct ConsumerGroupAssignment {
+ pub incarnation: u64,
+ pub owner: Option<ConsumerGroupOwner>,
+ pub pending: Option<PendingOwnership>,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConsumerGroupOwnershipTransition {
+ pub stream_id: u32,
+ pub topic_id: u32,
+ pub partition_id: u32,
+ pub source: Option<ConsumerGroupOwner>,
+ pub installation: InstallConsumerGroupOwnerRequest,
pub created_at: u64,
+ pub skip_drain: bool,
}
#[derive(Debug, Clone)]
pub struct ConsumerGroupMember {
pub id: usize,
pub client_id: u128,
- /// Retained independently of client-table capacity eviction. None for legacy snapshots.
+ /// Retained independently of client-table capacity eviction.
pub session: Option<u64>,
pub partitions: Vec<usize>,
- /// Partitions marked for cooperative handoff away from this member but not
- /// yet moved. The member still owns them (poll + commit) until completion.
- pub pending_revocations: Vec<PendingRevocation>,
+ /// Derived from the authoritative assignments. Polls stop while offsets drain.
+ pub pending_revocations: Vec<usize>,
}
impl ConsumerGroupMember {
@@ -106,11 +120,7 @@ impl ConsumerGroupMember {
/// the member's partition count), so the linear scan beats a set.
#[must_use]
pub fn is_pollable(&self, partition_id: usize) -> bool {
- self.partitions.contains(&partition_id)
- && !self
- .pending_revocations
- .iter()
- .any(|revocation| revocation.partition_id == partition_id)
+ self.partitions.contains(&partition_id) && !self.pending_revocations.contains(&partition_id)
}
}
@@ -125,6 +135,7 @@ pub struct ConsumerGroup {
pub generation: u64,
pub name: Arc<str>,
pub members: IdSlab<ConsumerGroupMember>,
+ pub assignments: BTreeMap<usize, ConsumerGroupAssignment>,
}
impl ConsumerGroup {
@@ -135,269 +146,249 @@ impl ConsumerGroup {
generation: 0,
name,
members: IdSlab::new(),
+ assignments: BTreeMap::new(),
}
}
- /// Redistribute the topic's current partitions round-robin across members.
- /// Reads the live partition set each call, so it reflects repartitions and
- /// bumps the generation on any membership change.
- ///
- /// Deliberately asymmetric with the join path (`rebalance_cooperative`):
- /// this clears partitions and pending revocations with no cooperative
- /// drain, so a member's in-flight (polled-but-uncommitted) partitions move
- /// immediately and the new owner re-reads that range. That redelivery is
- /// exactly what the join path avoids via pending revocations -- but the
- /// callers here (leave, remove-member on disconnect, repartition) are cases
- /// where the departing member cannot drain anyway, so an eager move is
- /// correct and the redelivery is benign under at-least-once.
- pub fn rebalance_members(&mut self, partition_ids: &[usize]) {
+ /// Metadata chooses successors; only a durable partition installation activates them.
+ pub fn rebalance_members(&mut self, partitions: &[Partition], metadata_op: u64, now: u64) {
self.generation += 1;
-
- let member_count = self.members.len();
- let member_keys: Vec<usize> = self.members.iter().map(|(id, _)| id).collect();
- for &member_id in &member_keys {
- if let Some(member) = self.members.get_mut(member_id) {
- member.partitions.clear();
- member.pending_revocations.clear();
+ let members: Vec<(u128, u64)> = self
+ .members
+ .iter()
+ .filter_map(|(_, member)| {
+ member
+ .session
+ .filter(|session| *session != 0)
+ .map(|session| (member.client_id, session))
+ })
+ .collect();
+ let partition_ids: std::collections::BTreeSet<usize> =
+ partitions.iter().map(|partition| partition.id).collect();
+ self.assignments
+ .retain(|partition_id, _| partition_ids.contains(partition_id));
+ let desired_members = self.balanced_member_indices(partitions, &members);
+ for (index, partition) in partitions.iter().enumerate() {
+ let incarnation = partition.created_revision;
+ let (client_id, session) =
+ desired_members[index].map_or((0, 0), |index| members[index]);
+ let desired = ConsumerGroupOwner {
+ client_id,
+ session,
+ generation: self.generation,
+ };
+ let assignment =
+ self.assignments
+ .entry(partition.id)
+ .or_insert(ConsumerGroupAssignment {
+ incarnation,
+ owner: None,
+ pending: None,
+ });
+ if assignment.incarnation != incarnation {
+ assignment.incarnation = incarnation;
+ assignment.owner = None;
+ assignment.pending = None;
}
+ let same_member = |owner: ConsumerGroupOwner| {
+ owner.client_id == client_id && owner.session == session
+ };
+ if let Some(pending) = assignment.pending {
+ if same_member(pending.owner) {
+ continue;
+ }
+ } else if assignment.owner.is_some_and(same_member)
+ || (assignment.owner.is_none() && desired.is_unassigned())
+ {
+ continue;
+ }
+ let skip_drain = assignment
+ .owner
+ .is_none_or(|owner| !members.contains(&(owner.client_id, owner.session)));
+ assignment.pending = Some(PendingOwnership {
+ owner: desired,
+ metadata_op,
+ created_at: now,
+ skip_drain,
+ });
}
+ self.rebuild_member_assignments();
+ }
- if member_count == 0 {
- return;
+ pub fn complete_revocation(
+ &mut self,
+ partition_id: usize,
+ installation: &InstallConsumerGroupOwnerRequest,
+ partition_op: u64,
+ ) -> bool {
+ if installation.group_id != self.id || partition_op == 0 {
+ return false;
}
-
- for (i, &partition_id) in partition_ids.iter().enumerate() {
- let target = i % member_count;
- if let Some(&member_id) = member_keys.get(target)
- && let Some(member) = self.members.get_mut(member_id)
+ let Some(assignment) = self.assignments.get_mut(&partition_id) else {
+ return false;
+ };
+ let Some(pending) = assignment.pending else {
+ return false;
+ };
+ if assignment.incarnation != installation.incarnation
+ || pending.owner != installation.owner
+ || pending.metadata_op != installation.metadata_op
+ {
+ return false;
+ }
+ let previous = assignment.owner;
+ assignment.owner = (!pending.owner.is_unassigned()).then_some(pending.owner);
+ assignment.pending = None;
+ self.generation += 1;
+ for (_, member) in &mut self.members {
+ if previous.is_some_and(|owner| {
+ owner.client_id == member.client_id && Some(owner.session) == member.session
+ }) {
+ member
+ .partitions
+ .retain(|assigned| *assigned != partition_id);
+ member
+ .pending_revocations
+ .retain(|pending| *pending != partition_id);
+ }
+ if pending.owner.client_id == member.client_id
+ && Some(pending.owner.session) == member.session
{
member.partitions.push(partition_id);
+ member.partitions.sort_unstable();
}
}
+ true
}
- /// Cooperative rebalance (on member join): assign unassigned partitions to
- /// idle members immediately, and mark each over-assigned member's excess as
- /// pending revocation toward an idle member -- the partition only moves once
- /// the source's consumer drains it (see the reconciler) or it times out, so
- /// no in-flight batch is dropped. `now` is the replicated op timestamp
- /// (micros), kept identical across replicas for deterministic apply.
- #[allow(clippy::too_many_lines)]
- pub fn rebalance_cooperative(
- &mut self,
- partition_ids: &[usize],
- in_flight: &std::collections::HashSet<usize>,
- now: u64,
+ /// Replaces this group's entries in `pending_revocations` with its open transitions.
+ #[allow(clippy::cast_possible_truncation)]
+ pub(crate) fn index_pending_revocations(
+ &self,
+ stream_id: usize,
+ topic_id: usize,
+ pending_revocations: &mut BTreeMap<
+ (usize, usize, u64, usize),
+ ConsumerGroupOwnershipTransition,
+ >,
) {
- self.generation += 1;
-
- let member_count = self.members.len();
- if member_count == 0 || partition_ids.is_empty() {
- return;
- }
-
- let mut assigned: std::collections::HashSet<usize> = std::collections::HashSet::new();
- for (_, member) in &self.members {
- for &partition_id in &member.partitions {
- assigned.insert(partition_id);
- }
+ pending_revocations
+ .extract_if(
+ (stream_id, topic_id, self.id, 0)..=(stream_id, topic_id, self.id, usize::MAX),
+ |_, _| true,
+ )
+ .for_each(drop);
+ for (&partition_id, assignment) in &self.assignments {
+ let Some(pending) = assignment.pending else {
+ continue;
+ };
+ pending_revocations.insert(
+ (stream_id, topic_id, self.id, partition_id),
+ ConsumerGroupOwnershipTransition {
+ stream_id: stream_id as u32,
+ topic_id: topic_id as u32,
+ partition_id: partition_id as u32,
+ source: assignment.owner,
+ installation: InstallConsumerGroupOwnerRequest {
+ incarnation: assignment.incarnation,
+ group_id: self.id,
+ owner: pending.owner,
+ metadata_op: pending.metadata_op,
+ },
+ created_at: pending.created_at,
+ skip_drain: pending.skip_drain,
+ },
+ );
}
+ }
- // Step 1: hand unassigned partitions straight to idle members.
- let unassigned: Vec<usize> = partition_ids
+ fn balanced_member_indices(
+ &self,
+ partitions: &[Partition],
+ members: &[(u128, u64)],
+ ) -> Vec<Option<usize>> {
+ let mut loads = vec![0usize; members.len()];
+ let mut desired_members: Vec<Option<usize>> = partitions
.iter()
- .copied()
- .filter(|partition_id| !assigned.contains(partition_id))
- .collect();
- if !unassigned.is_empty() {
- let idle: Vec<usize> = self
- .members
- .iter()
- .filter(|(_, member)| member.partitions.is_empty())
- .map(|(slab_id, _)| slab_id)
- .collect();
- if !idle.is_empty() {
- for (i, partition_id) in unassigned.into_iter().enumerate() {
- let slab_id = idle[i % idle.len()];
- if let Some(member) = self.members.get_mut(slab_id) {
- member.partitions.push(partition_id);
- }
+ .map(|partition| {
+ let assignment = self.assignments.get(&partition.id)?;
+ if assignment.incarnation != partition.created_revision {
+ return None;
}
- }
- }
-
- // Step 2: mark excess on over-assigned members as pending revocation.
- let partition_count = partition_ids.len();
- let fair_share = partition_count / member_count;
- let remainder = partition_count % member_count;
-
- let revocation_targets: std::collections::HashSet<usize> = self
- .members
- .iter()
- .flat_map(|(_, member)| {
- member
- .pending_revocations
+ let owner = assignment
+ .pending
+ .map(|pending| pending.owner)
+ .or(assignment.owner)?;
+ let index = members
.iter()
- .map(|revocation| revocation.target_member)
+ .position(|member| *member == (owner.client_id, owner.session))?;
+ loads[index] += 1;
+ Some(index)
})
.collect();
- let idle: Vec<usize> = self
- .members
- .iter()
- .filter(|(slab_id, member)| {
- member.partitions.is_empty() && !revocation_targets.contains(slab_id)
- })
- .map(|(slab_id, _)| slab_id)
- .collect();
- if idle.is_empty() {
- return;
- }
-
- // Pass 1: collect excess (round-robin into idle members in pass 2 so a
- // single over-assigned member's excess spreads evenly, not all to one).
- let member_keys: Vec<usize> = self.members.iter().map(|(slab_id, _)| slab_id).collect();
- let mut members_with_remainder = remainder;
- let mut all_excess: Vec<(usize, usize)> = Vec::new();
- for &slab_id in &member_keys {
- let Some(member) = self.members.get(slab_id) else {
- continue;
- };
- let pending: std::collections::HashSet<usize> = member
- .pending_revocations
- .iter()
- .map(|revocation| revocation.partition_id)
- .collect();
- let effective: Vec<usize> = member
- .partitions
- .iter()
- .copied()
- .filter(|partition_id| !pending.contains(partition_id))
- .collect();
- let max_allowed = if members_with_remainder > 0 {
- fair_share + 1
- } else {
- fair_share
- };
- if effective.len() <= max_allowed {
- if effective.len() > fair_share {
- members_with_remainder = members_with_remainder.saturating_sub(1);
- }
- continue;
- }
- let excess_count = effective.len() - max_allowed;
- if members_with_remainder > 0 && effective.len() > fair_share {
- members_with_remainder = members_with_remainder.saturating_sub(1);
- }
- for partition_id in effective.into_iter().rev().take(excess_count) {
- all_excess.push((partition_id, slab_id));
+ if !members.is_empty() {
+ let fair_share = partitions.len() / members.len();
+ let mut capacities = vec![fair_share; members.len()];
+ let mut ranked_members: Vec<usize> = (0..members.len()).collect();
+ ranked_members.sort_by_key(|index| std::cmp::Reverse(loads[*index]));
+ for index in ranked_members
+ .into_iter()
+ .take(partitions.len() % members.len())
+ {
+ capacities[index] += 1;
}
- }
-
- // Pass 2: distribute the collected excess round-robin over idle members.
- // An in-flight partition (the source polled but hasn't committed it) is
- // pending-revoked so the source can drain it first; a never-polled or
- // already-drained partition has nothing in flight, so it moves now.
- for (i, (partition_id, source_slab)) in all_excess.into_iter().enumerate() {
- let target_slab = idle[i % idle.len()];
- if in_flight.contains(&partition_id) {
- if let Some(source) = self.members.get_mut(source_slab) {
- source.pending_revocations.push(PendingRevocation {
- partition_id,
- target_member: target_slab,
- created_at: now,
- });
- }
- } else {
- if let Some(source) = self.members.get_mut(source_slab) {
- source.partitions.retain(|&p| p != partition_id);
+ for desired in desired_members.iter_mut().rev() {
+ if let Some(index) = *desired
+ && loads[index] > capacities[index]
+ {
+ loads[index] -= 1;
+ *desired = None;
}
- if let Some(target) = self.members.get_mut(target_slab) {
- target.partitions.push(partition_id);
+ }
+ for desired in &mut desired_members {
+ if desired.is_none()
+ && let Some(index) = loads
+ .iter()
+ .enumerate()
+ .filter(|(index, load)| **load < capacities[*index])
+ .min_by_key(|(_, load)| **load)
+ .map(|(index, _)| index)
+ {
+ loads[index] += 1;
+ *desired = Some(index);
}
}
}
+ desired_members
}
- /// Complete a pending revocation: move `partition_id` from the member with
- /// `source_client_id` to its recorded target. Bumps the generation so both
- /// clients re-sync. Returns `false` if no such pending revocation exists
- /// (already completed / stale) or the target member is gone.
- pub fn complete_revocation(&mut self, source_client_id: u128, partition_id: usize) -> bool {
- let Some(source_slab) = self
- .members
- .iter()
- .find(|(_, member)| member.client_id == source_client_id)
- .map(|(slab_id, _)| slab_id)
- else {
- return false;
- };
- let Some(source) = self.members.get_mut(source_slab) else {
- return false;
- };
- let Some(pos) = source
- .pending_revocations
- .iter()
- .position(|revocation| revocation.partition_id == partition_id)
- else {
- return false;
- };
- let revocation = source.pending_revocations.remove(pos);
- source.partitions.retain(|&p| p != partition_id);
- if let Some(target) = self.members.get_mut(revocation.target_member) {
- target.partitions.push(partition_id);
- self.generation += 1;
- } else {
- // Target gone: full rebalance recovers a consistent assignment.
- let mut all: std::collections::HashSet<usize> = std::collections::HashSet::new();
- for (_, member) in &self.members {
- for &p in &member.partitions {
- all.insert(p);
+ fn rebuild_member_assignments(&mut self) {
+ for (_, member) in &mut self.members {
+ member.partitions.clear();
+ member.pending_revocations.clear();
+ for (&partition_id, assignment) in &self.assignments {
+ if assignment.owner.is_some_and(|owner| {
+ owner.client_id == member.client_id && Some(owner.session) == member.session
+ }) {
+ member.partitions.push(partition_id);
+ if assignment.pending.is_some() {
+ member.pending_revocations.push(partition_id);
+ }
}
}
- all.insert(partition_id);
- let mut partition_ids: Vec<usize> = all.into_iter().collect();
- partition_ids.sort_unstable();
- self.rebalance_members(&partition_ids);
}
- true
- }
-
- /// `(source_client_id, partition_id, created_at)` for every pending
- /// revocation in this group, for the reconciler's completion check.
- #[must_use]
- pub fn pending_revocations(&self) -> Vec<(u128, usize, u64)> {
- self.members
- .iter()
- .flat_map(|(_, member)| {
- member.pending_revocations.iter().map(move |revocation| {
- (
- member.client_id,
- revocation.partition_id,
- revocation.created_at,
- )
- })
- })
- .collect()
}
}
-/// Replicated `JoinConsumerGroup`, enriched by the primary's home shard.
-///
-/// The wire request carries only identifiers; the apply needs the client id
-/// (which member is joining) and can't read it from the consensus header. The
-/// home shard also gathers `in_flight` -- the group's partitions with
-/// uncommitted polled data -- so the cooperative rebalance pending-revokes only
-/// those and hands off never-polled/drained partitions immediately (the apply
-/// has no partition-plane access to classify them itself).
+/// Replicated join enriched with the authenticated member identity.
+/// Drain decisions belong to the partition's installed owner.
#[derive(Debug, Clone)]
pub struct JoinConsumerGroupRequest {
pub stream_id: WireIdentifier,
pub topic_id: WireIdentifier,
pub group_id: WireIdentifier,
pub client_id: u128,
- pub in_flight: Vec<u32>,
- /// Authenticated bind epoch, absent only in legacy records. `PrepareHeader` does not retain it.
- pub session: Option<u64>,
+ pub session: u64,
}
impl WireEncode for JoinConsumerGroupRequest {
@@ -405,10 +396,8 @@ impl WireEncode for JoinConsumerGroupRequest {
self.stream_id.encoded_size()
+ self.topic_id.encoded_size()
+ self.group_id.encoded_size()
- + 16
- + 4
- + self.in_flight.len() * 4
- + self.session.map_or(0, |_| size_of::<u64>())
+ + size_of::<u128>()
+ + size_of::<u64>()
}
fn encode(&self, buf: &mut BytesMut) {
@@ -416,16 +405,7 @@ impl WireEncode for JoinConsumerGroupRequest {
self.topic_id.encode(buf);
self.group_id.encode(buf);
buf.put_u128_le(self.client_id);
- // `encoded_size` + the loop below use the true length, so a silent
- // clamp here would desync the count header from the body. `in_flight`
- // is bounded by the topic's partition count, so this never fires.
- buf.put_u32_le(u32::try_from(self.in_flight.len()).expect("in_flight count fits u32"));
- for partition_id in &self.in_flight {
- buf.put_u32_le(*partition_id);
- }
- if let Some(session) = self.session {
- buf.put_u64_le(session);
- }
+ buf.put_u64_le(self.session);
}
}
@@ -437,34 +417,20 @@ impl WireDecode for JoinConsumerGroupRequest {
let (group_id, n) = WireIdentifier::decode(&buf[pos..])?;
pos += n;
let client_id = read_u128_le(buf, pos)?;
- pos += 16;
- let count = read_u32_le(buf, pos)? as usize;
- pos += 4;
- // Cap against the bytes actually left: `count` is read straight off a
- // replicated prepare, so an unbounded `with_capacity` would let a
- // corrupt/bit-rotted count (near u32::MAX) allocate gigabytes and abort
- // every backup that applies it.
- let mut in_flight =
- Vec::with_capacity(bounded_capacity(count, buf.len().saturating_sub(pos), 4));
- for _ in 0..count {
- in_flight.push(read_u32_le(buf, pos)?);
- pos += 4;
+ pos += size_of::<u128>();
+ let session = read_u64_le(buf, pos)?;
+ pos += size_of::<u64>();
+ if client_id == 0 || session == 0 || pos != buf.len() {
+ return Err(iggy_binary_protocol::WireError::Validation(
+ "invalid consumer group member identity or trailing bytes".into(),
+ ));
}
- // Old WAL records end after in_flight; new joins retain the request epoch.
- let session = if pos == buf.len() {
- None
- } else {
- let session = read_u64_le(buf, pos)?;
- pos += size_of::<u64>();
- Some(session).filter(|&epoch| epoch != 0)
- };
Ok((
Self {
stream_id,
topic_id,
group_id,
client_id,
- in_flight,
session,
},
pos,
@@ -536,6 +502,10 @@ impl StateHandler for CreateConsumerGroupRequest {
let Some(topic_id) = state.resolve_topic_id(stream_id, &self.topic_id) else {
return ApplyReply::err(CreateConsumerGroupResult::TopicNotFound);
};
+ if state.lifecycle_blocks_consumer(stream_id, topic_id, None) {
+ return ApplyReply::err(iggy_common::IggyError::LifecycleBusy.as_code());
+ }
+
let Some(topic) = state
.items
.get_mut(stream_id)
@@ -601,6 +571,25 @@ impl StateHandler for DeleteConsumerGroupRequest {
let Some(group_id) = topic.resolve_group_id(&self.group_id) else {
return ApplyReply::err(DeleteConsumerGroupResult::ConsumerGroupNotFound);
};
+ let Ok(wire_group_id) = u32::try_from(group_id) else {
+ return ApplyReply::err(iggy_common::IggyError::InvalidCommand.as_code());
+ };
+ if let Some(reply) = state.begin_lifecycle(
+ stream_id,
+ Some(topic_id),
+ LifecycleAction::DeleteConsumerGroup {
+ group_id: wire_group_id,
+ },
+ ) {
+ return reply;
+ }
+ let Some(topic) = state
+ .items
+ .get_mut(stream_id)
+ .and_then(|stream| stream.topics.get_mut(topic_id))
+ else {
+ return ApplyReply::err(DeleteConsumerGroupResult::TopicNotFound);
+ };
let Some(group) = topic.consumer_groups.remove(&group_id) else {
return ApplyReply::err(DeleteConsumerGroupResult::ConsumerGroupNotFound);
};
@@ -627,6 +616,16 @@ impl StateHandler for JoinConsumerGroupRequest {
let Some(topic_id) = state.resolve_topic_id(stream_id, &self.topic_id) else {
return ApplyReply::err(JoinConsumerGroupResult::TopicNotFound);
};
+
+ let group_id = state
+ .items
+ .get(stream_id)
+ .and_then(|stream| stream.topics.get(topic_id))
+ .and_then(|topic| topic.resolve_group_id(&self.group_id));
+ if state.lifecycle_blocks_consumer(stream_id, topic_id, group_id) {
+ return ApplyReply::err(iggy_common::IggyError::LifecycleBusy.as_code());
+ }
+
let Some(topic) = state
.items
.get_mut(stream_id)
@@ -637,9 +636,6 @@ impl StateHandler for JoinConsumerGroupRequest {
let Some(group_id) = topic.resolve_group_id(&self.group_id) else {
return ApplyReply::err(JoinConsumerGroupResult::ConsumerGroupNotFound);
};
- // Snapshot the live partition ids before taking a mutable borrow of the
- // group (both borrow the topic).
- let partition_ids: Vec<usize> = topic.partitions.iter().map(|p| p.id).collect();
let Some(group) = topic.consumer_groups.get_mut(&group_id) else {
return ApplyReply::err(JoinConsumerGroupResult::ConsumerGroupNotFound);
};
@@ -649,22 +645,28 @@ impl StateHandler for JoinConsumerGroupRequest {
.iter_mut()
.find(|(_, member)| member.client_id == self.client_id);
if let Some((_, member)) = already {
- member.session = self.session.or(member.session);
+ let session = Some(self.session);
+ if member.session != session {
+ member.session = session;
+ group.rebalance_members(
+ &topic.partitions,
+ state.apply_context.metadata_op,
+ timestamp.as_micros(),
+ );
+ state.recompute_consumer_group_metadata();
+ }
return ApplyReply::ok(Bytes::new());
}
let member_key = group
.members
.insert(ConsumerGroupMember::new(0, self.client_id));
group.members[member_key].id = member_key;
- group.members[member_key].session = self.session;
- // Cooperative handoff using the home-shard-gathered `in_flight` set:
- // never-polled/drained excess moves to the new member immediately (so a
- // fresh group distributes synchronously at join), while partitions with
- // uncommitted in-flight data are pending-revoked for the reconciler to
- // complete once the source drains them (or on timeout).
- let in_flight: std::collections::HashSet<usize> =
- self.in_flight.iter().map(|&p| p as usize).collect();
- group.rebalance_cooperative(&partition_ids, &in_flight, timestamp.as_micros());
+ group.members[member_key].session = Some(self.session);
+ group.rebalance_members(
+ &topic.partitions,
+ state.apply_context.metadata_op,
+ timestamp.as_micros(),
+ );
state.recompute_consumer_group_metadata();
ApplyReply::ok(Bytes::new())
}
@@ -672,11 +674,7 @@ impl StateHandler for JoinConsumerGroupRequest {
impl StateHandler for LeaveConsumerGroupRequest {
type State = StreamsInner;
- fn apply(
- &self,
- state: &mut StreamsInner,
- _timestamp: iggy_common::IggyTimestamp,
- ) -> ApplyReply {
+ fn apply(&self, state: &mut StreamsInner, timestamp: iggy_common::IggyTimestamp) -> ApplyReply {
// Same level-by-level resolution as Join, surfacing a loud rejection
// instead of the old silent-OK no-op when a level is gone.
let Some(stream_id) = state.resolve_stream_id(&self.stream_id) else {
@@ -685,6 +683,16 @@ impl StateHandler for LeaveConsumerGroupRequest {
let Some(topic_id) = state.resolve_topic_id(stream_id, &self.topic_id) else {
return ApplyReply::err(LeaveConsumerGroupResult::TopicNotFound);
};
+
+ let group_id = state
+ .items
+ .get(stream_id)
+ .and_then(|stream| stream.topics.get(topic_id))
+ .and_then(|topic| topic.resolve_group_id(&self.group_id));
+ if state.lifecycle_blocks_consumer(stream_id, topic_id, group_id) {
+ return ApplyReply::err(iggy_common::IggyError::LifecycleBusy.as_code());
+ }
+
let Some(topic) = state
.items
.get_mut(stream_id)
@@ -695,7 +703,6 @@ impl StateHandler for LeaveConsumerGroupRequest {
let Some(group_id) = topic.resolve_group_id(&self.group_id) else {
return ApplyReply::err(LeaveConsumerGroupResult::ConsumerGroupNotFound);
};
- let partition_ids: Vec<usize> = topic.partitions.iter().map(|p| p.id).collect();
let Some(group) = topic.consumer_groups.get_mut(&group_id) else {
return ApplyReply::err(LeaveConsumerGroupResult::ConsumerGroupNotFound);
};
@@ -712,7 +719,11 @@ impl StateHandler for LeaveConsumerGroupRequest {
return ApplyReply::err(LeaveConsumerGroupResult::ConsumerGroupMemberNotFound);
};
group.members.remove(key);
- group.rebalance_members(&partition_ids);
+ group.rebalance_members(
+ &topic.partitions,
+ state.apply_context.metadata_op,
+ timestamp.as_micros(),
+ );
state.recompute_consumer_group_metadata();
ApplyReply::ok(Bytes::new())
}
@@ -747,11 +758,7 @@ impl WireDecode for RemoveConsumerGroupMemberRequest {
impl StateHandler for RemoveConsumerGroupMemberRequest {
type State = StreamsInner;
- fn apply(
- &self,
- state: &mut StreamsInner,
- _timestamp: iggy_common::IggyTimestamp,
- ) -> ApplyReply {
+ fn apply(&self, state: &mut StreamsInner, timestamp: iggy_common::IggyTimestamp) -> ApplyReply {
let mut any_removed = false;
for (_, stream) in &mut state.items {
for (_, topic) in &mut stream.topics {
@@ -761,15 +768,13 @@ impl StateHandler for RemoveConsumerGroupMemberRequest {
.iter()
.find(|(_, m)| m.client_id == self.client_id)
.map(|(key, _)| key);
- // Collect the partition set only when this group actually
- // holds the member. This runs on every logout inside the
- // no-await commit critical section on every replica, so the
- // alloc stays out of the common (no-match) path.
if let Some(key) = member_key {
- let partition_ids: Vec<usize> =
- topic.partitions.iter().map(|p| p.id).collect();
group.members.remove(key);
- group.rebalance_members(&partition_ids);
+ group.rebalance_members(
+ &topic.partitions,
+ state.apply_context.metadata_op,
+ timestamp.as_micros(),
+ );
any_removed = true;
}
}
@@ -784,52 +789,61 @@ impl StateHandler for RemoveConsumerGroupMemberRequest {
}
}
-/// Reconciler-originated completion of a pending cooperative revocation.
-///
-/// Moves `partition_id` from the member `source_client_id` to its recorded
-/// target, once the source consumer has drained the partition (committed up to
-/// what it polled) or the revocation timed out.
+/// Durable partition evidence for one exact pending ownership transition.
#[derive(Debug, Clone)]
pub struct CompleteConsumerGroupRevocationRequest {
pub stream_id: WireIdentifier,
pub topic_id: WireIdentifier,
- pub group_id: u64,
- pub source_client_id: u128,
pub partition_id: u32,
+ pub installation: InstallConsumerGroupOwnerRequest,
+ pub partition_op: u64,
}
impl WireEncode for CompleteConsumerGroupRevocationRequest {
fn encoded_size(&self) -> usize {
- self.stream_id.encoded_size() + self.topic_id.encoded_size() + 8 + 16 + 4
+ self.stream_id.encoded_size()
+ + self.topic_id.encoded_size()
+ + size_of::<u32>()
+ + self.installation.encoded_size()
+ + size_of::<u64>()
}
-
fn encode(&self, buf: &mut BytesMut) {
self.stream_id.encode(buf);
self.topic_id.encode(buf);
- buf.put_u64_le(self.group_id);
- buf.put_u128_le(self.source_client_id);
buf.put_u32_le(self.partition_id);
+ self.installation.encode(buf);
+ buf.put_u64_le(self.partition_op);
}
}
impl WireDecode for CompleteConsumerGroupRevocationRequest {
- fn decode(buf: &[u8]) -> Result<(Self, usize), iggy_binary_protocol::WireError> {
+ fn decode_from(buf: &[u8]) -> Result<Self, iggy_binary_protocol::WireError> {
+ let (request, consumed) = Self::decode(buf)?;
+ if consumed != buf.len() || request.partition_op == 0 {
+ return Err(iggy_binary_protocol::WireError::Validation(
+ "invalid ownership completion length or partition operation".into(),
+ ));
+ }
+ Ok(request)
+ }
+
+ fn decode(buf: &[u8]) -> Result<(Self, usize), iggy_binary_protocol::error::WireError> {
let (stream_id, mut pos) = WireIdentifier::decode(buf)?;
- let (topic_id, n) = WireIdentifier::decode(&buf[pos..])?;
- pos += n;
- let group_id = read_u64_le(buf, pos)?;
- pos += 8;
- let source_client_id = read_u128_le(buf, pos)?;
- pos += 16;
+ let (topic_id, consumed) = WireIdentifier::decode(&buf[pos..])?;
+ pos += consumed;
let partition_id = read_u32_le(buf, pos)?;
- pos += 4;
+ pos += size_of::<u32>();
+ let (installation, consumed) = InstallConsumerGroupOwnerRequest::decode(&buf[pos..])?;
+ pos += consumed;
+ let partition_op = read_u64_le(buf, pos)?;
+ pos += size_of::<u64>();
Ok((
Self {
stream_id,
topic_id,
- group_id,
- source_client_id,
partition_id,
+ installation,
+ partition_op,
},
pos,
))
@@ -843,18 +857,37 @@ impl StateHandler for CompleteConsumerGroupRevocationRequest {
state: &mut StreamsInner,
_timestamp: iggy_common::IggyTimestamp,
) -> ApplyReply {
- let Some(topic) = state.topic_mut(&self.stream_id, &self.topic_id) else {
+ let Some(stream_id) = state.resolve_stream_id(&self.stream_id) else {
+ return ApplyReply::ok(Bytes::new());
+ };
+ let Some(topic_id) = state.resolve_topic_id(stream_id, &self.topic_id) else {
+ return ApplyReply::ok(Bytes::new());
+ };
+ let Some(topic) = state
+ .items
+ .get_mut(stream_id)
+ .and_then(|stream| stream.topics.get_mut(topic_id))
+ else {
return ApplyReply::ok(Bytes::new());
};
let completed = topic
.consumer_groups
- .get_mut(&self.group_id)
+ .get_mut(&self.installation.group_id)
.is_some_and(|group| {
- group.complete_revocation(self.source_client_id, self.partition_id as usize)
+ group.complete_revocation(
+ self.partition_id as usize,
+ &self.installation,
+ self.partition_op,
+ )
});
if completed {
state.revision = state.revision.wrapping_add(1);
- state.recompute_consumer_group_metadata();
+ state.pending_revocations.remove(&(
+ stream_id,
+ topic_id,
+ self.installation.group_id,
+ self.partition_id as usize,
+ ));
}
ApplyReply::ok(Bytes::new())
}
@@ -870,17 +903,34 @@ pub struct RefreshConsumerGroupSessionRequest {
impl StateHandler for RefreshConsumerGroupSessionRequest {
type State = StreamsInner;
- fn apply(&self, state: &mut StreamsInner, _timestamp: IggyTimestamp) -> ApplyReply {
+ fn apply(&self, state: &mut StreamsInner, timestamp: IggyTimestamp) -> ApplyReply {
if let Some(memberships) = state.consumer_group_members.get(&self.client_id) {
for &(stream_id, topic_id, group_id, member_id) in memberships {
- if let Some(member) = state
+ let Some(topic) = state
.items
.get_mut(stream_id)
.and_then(|stream| stream.topics.get_mut(topic_id))
- .and_then(|topic| topic.consumer_groups.get_mut(&group_id))
- .and_then(|group| group.members.get_mut(member_id))
- {
+ else {
+ continue;
+ };
+ let Some(group) = topic.consumer_groups.get_mut(&group_id) else {
+ continue;
+ };
+ let Some(member) = group.members.get_mut(member_id) else {
+ continue;
+ };
+ if member.session != Some(self.session) {
member.session = Some(self.session);
+ group.rebalance_members(
+ &topic.partitions,
+ state.apply_context.metadata_op,
+ timestamp.as_micros(),
+ );
+ group.index_pending_revocations(
+ stream_id,
+ topic_id,
+ &mut state.pending_revocations,
+ );
}
}
}
@@ -888,112 +938,73 @@ impl StateHandler for RefreshConsumerGroupSessionRequest {
}
}
-/// Consumer group member snapshot representation for serialization.
-///
-/// Snapshots are encoded by `rmp_serde::to_vec` as **positional arrays** (no
-/// field-name map), so `#[serde(default)]` only fills a missing **trailing**
-/// element -- a new field is back/forward-compatible only if appended last. A
-/// field inserted mid-struct shifts every later position and corrupts decode.
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ConsumerGroupMemberSnapshot {
pub id: usize,
pub client_id: u128,
- pub partitions: Vec<usize>,
- /// `(partition_id, target_member_slab, created_at_micros)` per pending
- /// cooperative revocation. Trailing + `serde(default)` so a snapshot that
- /// predates this field still decodes (the default fills the absent
- /// trailing element).
- #[serde(default)]
- pub pending_revocations: Vec<(usize, usize, u64)>,
- #[serde(default)]
pub session: Option<u64>,
}
-/// Consumer group snapshot representation for serialization (nested under the
-/// topic snapshot). Positional array, same trailing-only rule as
-/// [`ConsumerGroupMemberSnapshot`].
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct ConsumerGroupSnapshot {
pub id: u64,
- // No `serde(default)`: `generation` is not the trailing field, so a default
- // could not fill it positionally anyway, and every consumer-group snapshot
- // (new with co-located groups) always writes it.
pub generation: u64,
pub name: String,
pub members: Vec<(usize, ConsumerGroupMemberSnapshot)>,
+ pub assignments: BTreeMap<usize, ConsumerGroupAssignment>,
}
impl ConsumerGroupSnapshot {
#[must_use]
pub fn from_group(group: &ConsumerGroup) -> Self {
- let members = group
- .members
- .iter()
- .map(|(member_id, member)| {
- (
- member_id,
- ConsumerGroupMemberSnapshot {
- id: member.id,
- client_id: member.client_id,
- session: member.session,
- partitions: member.partitions.clone(),
- pending_revocations: member
- .pending_revocations
- .iter()
- .map(|revocation| {
- (
- revocation.partition_id,
- revocation.target_member,
- revocation.created_at,
- )
- })
- .collect(),
- },
- )
- })
- .collect();
Self {
id: group.id,
generation: group.generation,
name: group.name.to_string(),
- members,
+ members: group
+ .members
+ .iter()
+ .map(|(member_id, member)| {
+ (
+ member_id,
+ ConsumerGroupMemberSnapshot {
+ id: member.id,
+ client_id: member.client_id,
+ session: member.session,
+ },
+ )
+ })
+ .collect(),
+ assignments: group.assignments.clone(),
}
}
#[must_use]
pub fn into_group(self) -> ConsumerGroup {
- let members: IdSlab<ConsumerGroupMember> = self
- .members
- .into_iter()
- .map(|(member_key, member_snap)| {
- (
- member_key,
- ConsumerGroupMember {
- id: member_snap.id,
- client_id: member_snap.client_id,
- session: member_snap.session,
- partitions: member_snap.partitions,
- pending_revocations: member_snap
- .pending_revocations
- .into_iter()
- .map(
- |(partition_id, target_member, created_at)| PendingRevocation {
- partition_id,
- target_member,
- created_at,
- },
- )
- .collect(),
- },
- )
- })
- .collect();
- ConsumerGroup {
+ let mut group = ConsumerGroup {
id: self.id,
generation: self.generation,
- name: Arc::from(self.name.as_str()),
- members,
- }
+ name: Arc::from(self.name),
+ members: self
+ .members
+ .into_iter()
+ .map(|(key, member)| {
+ (
+ key,
+ ConsumerGroupMember {
+ id: member.id,
+ client_id: member.client_id,
+ session: member.session,
+ partitions: Vec::new(),
+ pending_revocations: Vec::new(),
+ },
+ )
+ })
+ .collect(),
+ assignments: self.assignments,
+ };
+ group.rebuild_member_assignments();
+ group
}
}
@@ -1010,6 +1021,208 @@ mod tests {
use iggy_binary_protocol::{WireName, WireOptions};
use iggy_common::IggyTimestamp;
+ #[test]
+ fn given_balanced_owners_when_fourth_member_joins_should_move_only_three_partitions() {
+ let mut group = ConsumerGroup::new(0, Arc::from("group"));
+ let partitions: Vec<_> = (0..12)
+ .map(|id| Partition::new(id, id as u64 + 1, IggyTimestamp::from(1), 1, 0))
+ .collect();
+ let complete = |group: &mut ConsumerGroup| {
+ let pending: Vec<_> = group
+ .assignments
+ .iter()
+ .filter_map(|(&id, assignment)| {
+ let pending = assignment.pending?;
+ Some((
+ id,
+ InstallConsumerGroupOwnerRequest {
+ incarnation: assignment.incarnation,
+ group_id: group.id,
+ owner: pending.owner,
+ metadata_op: pending.metadata_op,
+ },
+ ))
+ })
+ .collect();
+ for (id, installation) in pending {
+ assert!(group.complete_revocation(id, &installation, 1));
+ }
+ };
+ for id in 0..3 {
+ let mut member = ConsumerGroupMember::new(id, id as u128 + 1);
+ member.session = Some(id as u64 + 1);
+ group.members.insert(member);
+ group.rebalance_members(&partitions, id as u64 + 10, 100);
+ complete(&mut group);
+ }
+ assert!(
+ group
+ .members
+ .iter()
+ .all(|(_, member)| member.partitions.len() == 4)
+ );
+ let before = group.assignments.clone();
+ let mut member = ConsumerGroupMember::new(3, 4);
+ member.session = Some(4);
+ group.members.insert(member);
+ group.rebalance_members(&partitions, 20, 200);
+ assert_eq!(
+ group
+ .assignments
+ .values()
+ .filter(|assignment| assignment.pending.is_some())
+ .count(),
+ 3
+ );
+ for (&id, assignment) in &group.assignments {
+ assert_eq!(assignment.owner, before[&id].owner);
+ }
+ complete(&mut group);
+ assert!(
+ group
+ .members
+ .iter()
+ .all(|(_, member)| member.partitions.len() == 3)
+ );
+ let before = group.assignments.clone();
+ group.members.remove(3);
+ group.rebalance_members(&partitions, 30, 300);
+ assert_eq!(
+ group
+ .assignments
+ .values()
+ .filter(|assignment| assignment.pending.is_some())
+ .count(),
+ 3
+ );
+ for (&id, assignment) in &group.assignments {
+ if assignment.pending.is_none() {
+ assert_eq!(assignment.owner, before[&id].owner);
+ }
+ }
+ complete(&mut group);
+ assert!(
+ group
+ .members
+ .iter()
+ .all(|(_, member)| member.partitions.len() == 4)
+ );
+ }
+
+ #[test]
+ fn given_durable_owner_when_reassigned_should_fence_superseded_installations() {
+ let mut group = ConsumerGroup::new(0, Arc::from("group"));
+ let mut first = ConsumerGroupMember::new(0, 7);
+ first.session = Some(11);
+ group.members.insert(first);
+ let mut partitions: Vec<_> = (0..2)
+ .map(|id| Partition::new(id, id as u64 + 1, IggyTimestamp::from(1), 1, 0))
+ .collect();
+ group.rebalance_members(&partitions, 12, 100);
+ assert_eq!(group.members[0].partitions, [] as [usize; 0]);
+ for partition_id in 0..2 {
+ let assignment = &group.assignments[&partition_id];
+ let pending = assignment.pending.unwrap();
+ assert!(pending.skip_drain);
+ let installation = InstallConsumerGroupOwnerRequest {
+ incarnation: assignment.incarnation,
+ group_id: group.id,
+ owner: pending.owner,
+ metadata_op: pending.metadata_op,
+ };
+ assert!(group.complete_revocation(partition_id, &installation, 1));
+ }
+ let unchanged = group.assignments[&0].owner;
+ let old_owner = group.assignments[&1].owner;
+ let mut second = ConsumerGroupMember::new(1, 8);
+ second.session = Some(13);
+ group.members.insert(second);
+ group.rebalance_members(&partitions, 14, 200);
+ assert_eq!(group.assignments[&0].owner, unchanged);
+ assert!(group.assignments[&0].pending.is_none());
+ assert_eq!(group.assignments[&1].owner, old_owner);
+ assert_eq!(group.members[0].pollable_partitions(), [0]);
+ assert_eq!(group.members[1].partitions, [] as [usize; 0]);
+ let assignment = &group.assignments[&1];
+ let pending = assignment.pending.unwrap();
+ assert!(!pending.skip_drain);
+ let superseded = InstallConsumerGroupOwnerRequest {
+ incarnation: assignment.incarnation,
+ group_id: group.id,
+ owner: pending.owner,
+ metadata_op: pending.metadata_op,
+ };
+ group.rebalance_members(&partitions, 15, 300);
+ assert_eq!(group.assignments[&1].pending, Some(pending));
+ group.members.remove(1);
+ group.rebalance_members(&partitions, 16, 400);
+ assert!(!group.complete_revocation(1, &superseded, 2));
+ let replacement = group.assignments[&1].pending.unwrap();
+ assert_eq!(replacement.owner.client_id, old_owner.unwrap().client_id);
+ assert!(replacement.owner.generation > superseded.owner.generation);
+ let mut installation = InstallConsumerGroupOwnerRequest {
+ incarnation: superseded.incarnation,
+ group_id: group.id,
+ owner: replacement.owner,
+ metadata_op: replacement.metadata_op,
+ };
+ installation.metadata_op -= 1;
+ assert!(!group.complete_revocation(1, &installation, 3));
+ installation.metadata_op += 1;
+ assert!(group.complete_revocation(1, &installation, 3));
+ assert!(!group.complete_revocation(1, &superseded, 2));
+ assert_eq!(group.members[0].pollable_partitions(), [0, 1]);
+
+ partitions[1].created_revision += 1;
+ group.rebalance_members(&partitions, 17, 500);
+ assert_eq!(group.assignments[&0].owner, unchanged);
+ assert!(group.assignments[&1].owner.is_none());
+ assert!(group.assignments[&1].pending.unwrap().skip_drain);
+ assert!(!group.complete_revocation(1, &installation, 3));
+ group.members.remove(0);
+ group.rebalance_members(&partitions, 18, 600);
+ for (&partition_id, assignment) in &group.assignments.clone() {
+ let pending = assignment.pending.unwrap();
+ assert!(pending.owner.is_unassigned());
+ assert!(pending.skip_drain);
+ let installation = InstallConsumerGroupOwnerRequest {
+ incarnation: assignment.incarnation,
+ group_id: group.id,
+ owner: pending.owner,
+ metadata_op: pending.metadata_op,
+ };
+ assert!(group.complete_revocation(partition_id, &installation, 4));
+ assert!(group.assignments[&partition_id].owner.is_none());
+ }
+ }
+
+ #[test]
+ fn given_session_refresh_when_rebalancing_should_index_revocations_like_full_rebuild() {
+ const CLIENT: u128 = 7;
+ const OTHER: u128 = 8;
+ const SESSION: u64 = 11;
+ let mut state = streams_with_topic();
+ assert_eq!(create_group(&mut state, "first").code, 0);
+ assert_eq!(create_group(&mut state, "second").code, 0);
+ assert_eq!(join(&mut state, 0, 0, 0, CLIENT).code, 0);
+ assert_eq!(join(&mut state, 0, 0, 1, OTHER).code, 0);
+ state.apply_context.metadata_op += 1;
+ let refresh = RefreshConsumerGroupSessionRequest {
+ client_id: CLIENT,
+ session: SESSION,
+ };
+ assert_eq!(
+ StateHandler::apply(&refresh, &mut state, IggyTimestamp::now()).code,
+ 0
+ );
+ let refreshed = state.pending_revocations.clone();
+ assert_eq!(refreshed.len(), 2);
+ assert_eq!(refreshed[&(0, 0, 0, 0)].installation.owner.session, SESSION);
+ assert_eq!(refreshed[&(0, 0, 1, 0)].installation.owner.client_id, OTHER);
+ state.recompute_consumer_group_metadata();
+ assert_eq!(state.pending_revocations, refreshed);
+ }
+
#[test]
fn session_refresh_index_tracks_membership_removal_and_rejoin() {
const CLIENT: u128 = 7;
@@ -1025,7 +1238,7 @@ mod tests {
assert_eq!(leave(&mut state, 0, 0, 0, CLIENT).code, 0);
assert_eq!(state.consumer_group_members[&CLIENT].len(), 1);
assert_eq!(
- StateHandler::apply(
+ crate::stm::lifecycle::apply_with_lifecycle_completion(
&DeleteConsumerGroupRequest {
stream_id: WireIdentifier::numeric(0),
topic_id: WireIdentifier::numeric(0),
@@ -1039,7 +1252,7 @@ mod tests {
);
assert!(!state.consumer_group_members.contains_key(&CLIENT));
assert_eq!(
- StateHandler::apply(
+ crate::stm::lifecycle::apply_with_lifecycle_completion(
&RemoveConsumerGroupMemberRequest { client_id: OTHER },
&mut state,
IggyTimestamp::now(),
@@ -1050,7 +1263,12 @@ mod tests {
assert!(state.consumer_group_members.is_empty());
assert_eq!(join(&mut state, 0, 0, 0, CLIENT).code, 0);
let streams = Streams::from(state);
- streams.refresh_consumer_group_session(CLIENT, 11);
+ streams.refresh_consumer_group_session(
+ CLIENT,
+ 11,
+ 11,
+ iggy_common::IggyTimestamp::default(),
+ );
assert_eq!(streams.consumer_group_session(CLIENT), Some(11));
assert_eq!(streams.consumer_group_session(OTHER), None);
}
@@ -1063,7 +1281,7 @@ mod tests {
assert_eq!(create_group(&mut state, "group").code, 0);
assert_eq!(join(&mut state, 0, 0, 0, CLIENT).code, 0);
let reply = if delete_stream {
- StateHandler::apply(
+ crate::stm::lifecycle::apply_with_lifecycle_completion(
&DeleteStreamRequest {
stream_id: WireIdentifier::numeric(0),
},
@@ -1071,7 +1289,7 @@ mod tests {
IggyTimestamp::now(),
)
} else {
- StateHandler::apply(
+ crate::stm::lifecycle::apply_with_lifecycle_completion(
&DeleteTopicRequest {
stream_id: WireIdentifier::numeric(0),
topic_id: WireIdentifier::numeric(0),
@@ -1101,15 +1319,25 @@ mod tests {
restored.restore_in_place(original.to_snapshot());
for streams in [boot, Streams::from(restored)] {
for epoch in [11, 12, 13] {
- streams.refresh_consumer_group_session(CLIENT, epoch);
- streams.refresh_consumer_group_session(u128::MAX, epoch);
+ streams.refresh_consumer_group_session(
+ CLIENT,
+ epoch,
+ epoch,
+ iggy_common::IggyTimestamp::default(),
+ );
+ streams.refresh_consumer_group_session(
+ u128::MAX,
+ epoch,
+ epoch,
+ iggy_common::IggyTimestamp::default(),
+ );
assert!(streams.read(|inner| {
let topic = &inner.items[0].topics[0];
for group in topic.consumer_groups.values() {
for (_, member) in &group.members {
assert_eq!(
member.session,
- (member.client_id == CLIENT).then_some(epoch)
+ Some(if member.client_id == CLIENT { epoch } else { 1 })
);
}
}
@@ -1120,65 +1348,70 @@ mod tests {
}
#[test]
- fn member_session_snapshot_round_trip_preserves_fence_and_reads_legacy_members() {
+ fn given_pending_ownership_when_restoring_snapshot_should_preserve_inactive_successor() {
const CLIENT: u128 = 7;
const SESSION: u64 = 11;
let mut group = ConsumerGroup::new(0, Arc::from("group"));
let mut member = ConsumerGroupMember::new(0, CLIENT);
member.session = Some(SESSION);
- member.partitions.push(2);
group.members.insert(member);
- let snapshot = ConsumerGroupSnapshot::from_group(&group);
- let encoded = rmp_serde::to_vec(&snapshot).unwrap();
- let restored = rmp_serde::from_slice::<ConsumerGroupSnapshot>(&encoded)
+ let partition = Partition::new(2, 1, IggyTimestamp::from(1), 1, 0);
+ group.rebalance_members(&[partition], 12, 1);
+ let encoded = rmp_serde::to_vec(&ConsumerGroupSnapshot::from_group(&group)).unwrap();
+ let mut restored = rmp_serde::from_slice::<ConsumerGroupSnapshot>(&encoded)
.unwrap()
.into_group();
assert_eq!(restored.members[0].session, Some(SESSION));
+ assert_eq!(restored.members[0].partitions, [] as [usize; 0]);
+ assert_eq!(restored.assignments, group.assignments);
+ let assignment = &restored.assignments[&2];
+ let pending = assignment.pending.unwrap();
+ let installation = InstallConsumerGroupOwnerRequest {
+ incarnation: assignment.incarnation,
+ group_id: restored.id,
+ owner: pending.owner,
+ metadata_op: pending.metadata_op,
+ };
+ assert!(!restored.complete_revocation(2, &installation, 0));
+ assert_eq!(restored.members[0].partitions, [] as [usize; 0]);
+ assert!(restored.complete_revocation(2, &installation, 21));
assert_eq!(restored.members[0].partitions, [2]);
+ let encoded = rmp_serde::to_vec(&ConsumerGroupSnapshot::from_group(&restored)).unwrap();
+ let restored = rmp_serde::from_slice::<ConsumerGroupSnapshot>(&encoded)
+ .unwrap()
+ .into_group();
+ assert_eq!(restored.members[0].partitions, [2]);
+ assert!(restored.assignments[&2].pending.is_none());
- let legacy = rmp_serde::to_vec(&(
- 0usize,
- CLIENT,
- vec![2usize],
- Vec::<(usize, usize, u64)>::new(),
- ))
- .unwrap();
- let restored = rmp_serde::from_slice::<ConsumerGroupMemberSnapshot>(&legacy).unwrap();
- assert_eq!(restored.session, None);
- assert_eq!(restored.client_id, CLIENT);
- assert_eq!(restored.partitions, [2]);
+ let legacy = rmp_serde::to_vec(&(0usize, CLIENT, vec![2usize])).unwrap();
+ assert!(rmp_serde::from_slice::<ConsumerGroupMemberSnapshot>(&legacy).is_err());
}
#[test]
- fn replicated_join_preserves_session_and_decodes_legacy_payloads() {
- let mut join = JoinConsumerGroupRequest {
+ fn replicated_join_requires_exact_live_session_identity() {
+ let join = JoinConsumerGroupRequest {
stream_id: WireIdentifier::numeric(0),
topic_id: WireIdentifier::numeric(0),
group_id: WireIdentifier::numeric(0),
client_id: 7,
- in_flight: vec![1, 2],
- session: None,
+ session: 11,
};
- let legacy = join.to_bytes();
- assert_eq!(
- JoinConsumerGroupRequest::decode_from(&legacy)
- .unwrap()
- .session,
- None
- );
- join.session = Some(11);
let encoded = join.to_bytes();
let decoded = JoinConsumerGroupRequest::decode_from(&encoded).unwrap();
+ assert_eq!(decoded.client_id, join.client_id);
assert_eq!(decoded.session, join.session);
- assert_eq!(decoded.in_flight, join.in_flight);
- for length in legacy.len() + 1..encoded.len() {
+ for length in 0..encoded.len() {
assert!(JoinConsumerGroupRequest::decode_from(&encoded[..length]).is_err());
}
+ let mut invalid = encoded.to_vec();
+ invalid.push(0);
+ assert!(JoinConsumerGroupRequest::decode_from(&invalid).is_err());
+ invalid = encoded.to_vec();
+ let end = invalid.len();
+ invalid[end - size_of::<u64>()..].fill(0);
+ assert!(JoinConsumerGroupRequest::decode_from(&invalid).is_err());
}
- // Groups co-locate in the topic node, so an apply resolves its parent through
- // `topic_mut`. Build a `StreamsInner` holding stream 0 / topic 0 via the same
- // handlers the commit path drives.
fn streams_with_topic() -> StreamsInner {
let mut inner = StreamsInner::new();
let _ = StateHandler::apply(
@@ -1226,14 +1459,15 @@ mod tests {
group: u32,
client_id: u128,
) -> ApplyReply {
+ state.apply_context.metadata_op += 1;
StateHandler::apply(
&JoinConsumerGroupRequest {
stream_id: WireIdentifier::numeric(stream),
topic_id: WireIdentifier::numeric(topic),
group_id: WireIdentifier::numeric(group),
client_id,
- in_flight: Vec::new(),
- session: None,
+
+ session: 1,
},
state,
IggyTimestamp::now(),
diff --git a/core/metadata/src/stm/lifecycle.rs b/core/metadata/src/stm/lifecycle.rs
new file mode 100644
index 000000000..b626bffaf
--- /dev/null
+++ b/core/metadata/src/stm/lifecycle.rs
@@ -0,0 +1,1175 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use std::collections::btree_map::Entry;
+use std::collections::{BTreeMap, BTreeSet};
+
+use bytes::{BufMut, Bytes, BytesMut};
+use iggy_binary_protocol::codec::{read_u32_le, read_u64_le};
+use iggy_binary_protocol::primitives::partition_history::ConsumerGroupOwner;
+use iggy_binary_protocol::requests::consumer_groups::DeleteConsumerGroupRequest;
+use iggy_binary_protocol::requests::partitions::{
+ DeletePartitionsRequest, InstallConsumerGroupOwnerRequest, TransitionPartitionHistoryRequest,
+};
+use iggy_binary_protocol::requests::streams::DeleteStreamRequest;
+use iggy_binary_protocol::requests::topics::DeleteTopicRequest;
+use iggy_binary_protocol::{Operation, WireDecode, WireEncode, WireError, WireIdentifier};
+use iggy_common::{IggyError, IggyTimestamp};
+use serde::{Deserialize, Serialize};
+
+use crate::stm::result::ApplyReply;
+use crate::stm::snapshot::SnapshotError;
+use crate::stm::stream::{Streams, StreamsInner, StreamsSnapshot};
+use crate::stm::{ApplyContext, StateHandler};
+
+#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
+pub enum LifecycleAction {
+ DeleteStream,
+ DeleteTopic,
+ DeletePartitions { count: u32 },
+ DeleteConsumerGroup { group_id: u32 },
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
+pub enum LifecycleFence {
+ History(TransitionPartitionHistoryRequest),
+ Owner(InstallConsumerGroupOwnerRequest),
+}
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct LifecyclePartition {
+ pub topic_id: u32,
+ pub partition_id: u32,
+ pub fence: LifecycleFence,
+ pub partition_op: Option<u64>,
+}
+
+/// Retained until every affected partition has durably installed its exact fence.
+/// The original request identity stays stable through retry, failover and restore.
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct LifecycleIntent {
+ pub context: ApplyContext,
+ pub stream_id: u32,
+ pub topic_id: Option<u32>,
+ pub action: LifecycleAction,
+ pub partitions: Vec<LifecyclePartition>,
+ #[serde(skip)]
+ partition_index: BTreeMap<(u32, u32), usize>,
+ #[serde(skip)]
+ remaining_partitions: usize,
+}
+
+impl LifecycleIntent {
+ pub(crate) fn rebuild_partition_index(&mut self) {
+ self.partition_index = self
+ .partitions
+ .iter()
+ .enumerate()
+ .map(|(index, partition)| ((partition.topic_id, partition.partition_id), index))
+ .collect();
+ self.remaining_partitions = self
+ .partitions
+ .iter()
+ .filter(|partition| partition.partition_op.is_none())
+ .count();
+ }
+}
+
+#[derive(Debug, Clone, Copy)]
+pub struct CompleteLifecycleRequest {
+ pub metadata_op: u64,
+ pub stream_id: u32,
+ pub topic_id: u32,
+ pub partition_id: u32,
+ pub partition_op: u64,
+}
+
+impl WireEncode for CompleteLifecycleRequest {
+ fn encoded_size(&self) -> usize {
+ 2 * size_of::<u64>() + 3 * size_of::<u32>()
+ }
+
+ fn encode(&self, buf: &mut BytesMut) {
+ buf.put_u64_le(self.metadata_op);
+ buf.put_u32_le(self.stream_id);
+ buf.put_u32_le(self.topic_id);
+ buf.put_u32_le(self.partition_id);
+ buf.put_u64_le(self.partition_op);
+ }
+}
+
+impl WireDecode for CompleteLifecycleRequest {
+ fn decode_from(buf: &[u8]) -> Result<Self, WireError> {
+ let (request, consumed) = Self::decode(buf)?;
+ if consumed != buf.len() {
+ return Err(WireError::Validation(
+ "trailing lifecycle completion bytes".into(),
+ ));
+ }
+ Ok(request)
+ }
+
+ fn decode(buf: &[u8]) -> Result<(Self, usize), WireError> {
+ let metadata_op = read_u64_le(buf, 0)?;
+ let stream_id = read_u32_le(buf, 8)?;
+ let topic_id = read_u32_le(buf, 12)?;
+ let partition_id = read_u32_le(buf, 16)?;
+ let partition_op = read_u64_le(buf, 20)?;
+ if metadata_op == 0 || partition_op == 0 {
+ return Err(WireError::Validation(
+ "missing lifecycle completion identity".into(),
+ ));
+ }
+ let request = Self {
+ metadata_op,
+ stream_id,
+ topic_id,
+ partition_id,
+ partition_op,
+ };
+ let size = request.encoded_size();
+ Ok((request, size))
+ }
+}
+
+impl Streams {
+ pub fn lifecycle_pending(&self, metadata_op: u64) -> bool {
+ self.read(|state| state.lifecycle_intents.contains_key(&metadata_op))
+ }
+
+ pub fn has_pending_lifecycles(&self) -> bool {
+ self.read(|state| !state.lifecycle_intents.is_empty())
+ }
+
+ pub fn pending_lifecycles(&self) -> Vec<LifecycleIntent> {
+ self.read(|state| state.lifecycle_intents.values().cloned().collect())
+ }
+
+ pub fn lifecycle_blocks(&self, stream_id: usize, topic_id: Option<usize>) -> bool {
+ self.read(|state| state.lifecycle_blocks(stream_id, topic_id))
+ }
+
+ pub fn lifecycle_blocks_consumer(
+ &self,
+ stream_id: usize,
+ topic_id: usize,
+ group_id: Option<u64>,
+ ) -> bool {
+ self.read(|state| state.lifecycle_blocks_consumer(stream_id, topic_id, group_id))
+ }
+
+ pub fn request_lifecycle_blocked(&self, operation: Operation, body: &[u8]) -> bool {
+ let topic_scoped = match operation {
+ Operation::UpdateStream | Operation::DeleteStream | Operation::CreateTopic => false,
+ Operation::UpdateTopic
+ | Operation::DeleteTopic
+ | Operation::CreatePartitions
+ | Operation::DeletePartitions
+ | Operation::CreateConsumerGroup
+ | Operation::DeleteConsumerGroup
+ | Operation::JoinConsumerGroup
+ | Operation::LeaveConsumerGroup => true,
+ _ => return false,
+ };
+ let Ok((stream, consumed)) = WireIdentifier::decode(body) else {
+ return false;
+ };
+ self.read(|state| {
+ let Some(stream_id) = state.resolve_stream_id(&stream) else {
+ return false;
+ };
+ if operation == Operation::CreateTopic {
+ return state.lifecycle_blocks_topic_creation(stream_id);
+ }
+ if !topic_scoped {
+ return state.lifecycle_blocks(stream_id, None);
+ }
+ let Ok((topic, topic_bytes)) = WireIdentifier::decode(&body[consumed..]) else {
+ return false;
+ };
+ let Some(topic_id) = state.resolve_topic_id(stream_id, &topic) else {
+ return false;
+ };
+ if operation == Operation::CreateConsumerGroup {
+ return state.lifecycle_blocks_consumer(stream_id, topic_id, None);
+ }
+ if matches!(
+ operation,
+ Operation::JoinConsumerGroup | Operation::LeaveConsumerGroup
+ ) {
+ let Ok((group, _)) = WireIdentifier::decode(&body[consumed + topic_bytes..]) else {
+ return false;
+ };
+ let group_id = state
+ .items
+ .get(stream_id)
+ .and_then(|stream| stream.topics.get(topic_id))
+ .and_then(|topic| topic.resolve_group_id(&group));
+ return state.lifecycle_blocks_consumer(stream_id, topic_id, group_id);
+ }
+ state.lifecycle_blocks(stream_id, Some(topic_id))
+ })
+ }
+}
+
+impl StreamsInner {
+ pub(crate) fn lifecycle_blocks(&self, stream_id: usize, topic_id: Option<usize>) -> bool {
+ self.lifecycle_blockers(stream_id, topic_id)
+ .next()
+ .is_some()
+ }
+
+ /// A topic-scoped intent leaves the rest of its stream writable, so only a
+ /// stream-wide intent blocks a new topic. Admission and apply share this rule.
+ pub(crate) fn lifecycle_blocks_topic_creation(&self, stream_id: usize) -> bool {
+ self.lifecycle_blockers(stream_id, None)
+ .any(|intent| intent.topic_id.is_none())
+ }
+
+ pub(crate) fn lifecycle_blocks_consumer(
+ &self,
+ stream_id: usize,
+ topic_id: usize,
+ group_id: Option<u64>,
+ ) -> bool {
+ self.lifecycle_blockers(stream_id, Some(topic_id))
+ .any(|intent| {
+ !matches!(intent.action, LifecycleAction::DeleteConsumerGroup { group_id: deleted }
+ if Some(u64::from(deleted)) != group_id)
+ })
+ }
+
+ fn lifecycle_blockers(
+ &self,
+ stream_id: usize,
+ topic_id: Option<usize>,
+ ) -> impl Iterator<Item = &LifecycleIntent> {
+ self.lifecycle_intents.values().filter(move |intent| {
+ !self.finalizing_lifecycle
+ && intent.stream_id as usize == stream_id
+ && (intent.topic_id.is_none()
+ || topic_id.is_none()
+ || intent.topic_id.map(|id| id as usize) == topic_id)
+ })
+ }
+
+ /// Called after command validation, before its destructive metadata effect.
+ /// Empty scopes finish in this same apply; all other scopes retain their rows.
+ #[allow(clippy::too_many_lines)]
+ pub(crate) fn begin_lifecycle(
+ &mut self,
+ stream_id: usize,
+ topic_id: Option<usize>,
+ action: LifecycleAction,
+ ) -> Option<ApplyReply> {
+ if self.finalizing_lifecycle {
+ return None;
+ }
+ if self.lifecycle_blocks(stream_id, topic_id) {
+ return Some(ApplyReply::err(IggyError::LifecycleBusy.as_code()));
+ }
+ let stream = self.items.get(stream_id)?;
+ let Ok(wire_stream_id) = u32::try_from(stream_id) else {
+ return Some(ApplyReply::err(IggyError::InvalidCommand.as_code()));
+ };
+ let Ok(wire_topic_id) = topic_id.map(u32::try_from).transpose() else {
+ return Some(ApplyReply::err(IggyError::InvalidCommand.as_code()));
+ };
+ let mut partitions = Vec::new();
+ for (id, topic) in &stream.topics {
+ if topic_id.is_some_and(|wanted| wanted != id) {
+ continue;
+ }
+ let first = match action {
+ LifecycleAction::DeletePartitions { count } => {
+ topic.partitions.len().checked_sub(count as usize)?
+ }
+ _ => 0,
+ };
+ for partition in &topic.partitions[first..] {
+ let incarnation = partition.created_revision;
+ let fence = if let LifecycleAction::DeleteConsumerGroup { group_id } = action {
+ let group = topic.consumer_groups.get(&u64::from(group_id))?;
+ let Some(generation) = group.generation.checked_add(1) else {
+ return Some(ApplyReply::err(IggyError::InvalidCommand.as_code()));
+ };
+ LifecycleFence::Owner(InstallConsumerGroupOwnerRequest {
+ incarnation,
+ group_id: u64::from(group_id),
+ owner: ConsumerGroupOwner {
+ client_id: 0,
+ session: 0,
+ generation,
+ },
+ metadata_op: self.apply_context.metadata_op,
+ })
+ } else {
+ LifecycleFence::History(TransitionPartitionHistoryRequest {
+ incarnation,
+ metadata_op: self.apply_context.metadata_op,
+ })
+ };
+ let (Ok(topic_id), Ok(partition_id)) =
+ (u32::try_from(id), u32::try_from(partition.id))
+ else {
+ return Some(ApplyReply::err(IggyError::InvalidCommand.as_code()));
+ };
+ partitions.push(LifecyclePartition {
+ topic_id,
+ partition_id,
+ fence,
+ partition_op: None,
+ });
+ }
+ }
+ if partitions.is_empty() {
+ return None;
+ }
+ if self.apply_context.metadata_op == 0 {
+ return Some(ApplyReply::err(IggyError::InvalidCommand.as_code()));
+ }
+ let mut intent = LifecycleIntent {
+ context: self.apply_context,
+ stream_id: wire_stream_id,
+ topic_id: wire_topic_id,
+ action,
+ partitions,
+ partition_index: BTreeMap::new(),
+ remaining_partitions: 0,
+ };
+ intent.rebuild_partition_index();
+ self.lifecycle_intents
+ .insert(self.apply_context.metadata_op, intent);
+ self.revision = self.revision.wrapping_add(1);
+ Some(ApplyReply::ok(Bytes::new()))
+ }
+}
+
+impl StateHandler for CompleteLifecycleRequest {
+ type State = StreamsInner;
+
+ fn apply(&self, state: &mut StreamsInner, timestamp: IggyTimestamp) -> ApplyReply {
+ let Entry::Occupied(mut entry) = state.lifecycle_intents.entry(self.metadata_op) else {
+ return ApplyReply::ok(Bytes::new());
+ };
+ let intent = entry.get_mut();
+ if intent.stream_id != self.stream_id {
+ return ApplyReply::err(IggyError::InvalidCommand.as_code());
+ }
+ let Some(&index) = intent
+ .partition_index
+ .get(&(self.topic_id, self.partition_id))
+ else {
+ return ApplyReply::err(IggyError::InvalidCommand.as_code());
+ };
+ let Some(partition) = intent.partitions.get_mut(index) else {
+ return ApplyReply::err(IggyError::InvalidCommand.as_code());
+ };
+ if partition
+ .partition_op
+ .is_some_and(|op| op != self.partition_op)
+ {
+ return ApplyReply::err(IggyError::InvalidCommand.as_code());
+ }
+ let previous_completion = partition.partition_op;
+ partition.partition_op = Some(self.partition_op);
+ if previous_completion.is_none() {
+ intent.remaining_partitions -= 1;
+ }
+ if intent.remaining_partitions != 0 {
+ return ApplyReply::ok(Bytes::new());
+ }
+ let mut intent = entry.remove();
+ let stream_id = WireIdentifier::numeric(intent.stream_id);
+ let topic_id = WireIdentifier::numeric(intent.topic_id.unwrap_or_default());
+ let context = state.apply_context;
+ state.apply_context = intent.context;
+ state.finalizing_lifecycle = true;
+ let reply = match intent.action {
+ LifecycleAction::DeleteStream => {
+ DeleteStreamRequest { stream_id }.apply(state, timestamp)
+ }
+ LifecycleAction::DeleteTopic => DeleteTopicRequest {
+ stream_id,
+ topic_id,
+ }
+ .apply(state, timestamp),
+ LifecycleAction::DeletePartitions { count } => DeletePartitionsRequest {
+ stream_id,
+ topic_id,
+ partitions_count: count,
+ }
+ .apply(state, timestamp),
+ LifecycleAction::DeleteConsumerGroup { group_id } => DeleteConsumerGroupRequest {
+ stream_id,
+ topic_id,
+ group_id: WireIdentifier::numeric(group_id),
+ }
+ .apply(state, timestamp),
+ };
+ state.finalizing_lifecycle = false;
+ state.apply_context = context;
+ if reply.code != 0 {
+ // Keep the final report pending so reconciliation can retry finalization.
+ if let Some(partition) = intent.partitions.get_mut(index) {
+ partition.partition_op = previous_completion;
+ if previous_completion.is_none() {
+ intent.remaining_partitions += 1;
+ }
+ }
+ state.lifecycle_intents.insert(self.metadata_op, intent);
+ }
+ reply
+ }
+}
+
+impl StreamsSnapshot {
+ #[allow(clippy::too_many_lines)]
+ pub(crate) fn validate_ownership_history(&self, metadata_op: u64) -> Result<(), SnapshotError> {
+ for (_, stream) in &self.items {
+ for (_, topic) in &stream.topics {
+ for (group_id, group) in &topic.consumer_groups {
+ if group.id != *group_id || *group_id >= topic.next_consumer_group_id {
+ return Err(SnapshotError::InvalidState(
+ "consumer group allocation is inconsistent",
+ ));
+ }
+ let covers_all_partitions = group.assignments.len() == topic.partitions.len();
+ if (!group.members.is_empty() || !group.assignments.is_empty())
+ && !covers_all_partitions
+ {
+ return Err(SnapshotError::InvalidState(
+ "incomplete consumer ownership coverage",
+ ));
+ }
+ let mut members = BTreeSet::new();
+ for (_, member) in &group.members {
+ if member.client_id == 0
+ || member.session.is_none_or(|session| session == 0)
+ || !members.insert((member.client_id, member.session))
+ {
+ return Err(SnapshotError::InvalidState(
+ "invalid consumer member identity",
+ ));
+ }
+ }
+ for (&partition_id, assignment) in &group.assignments {
+ let partition = topic
+ .partitions
+ .iter()
+ .find(|partition| partition.id == partition_id)
+ .ok_or(SnapshotError::InvalidState(
+ "owner refers to an absent partition",
+ ))?;
+ if assignment.incarnation != partition.created_revision {
+ return Err(SnapshotError::InvalidState(
+ "owner belongs to another history",
+ ));
+ }
+ if assignment.owner.is_some_and(|owner| {
+ owner.is_unassigned()
+ || owner.client_id == 0
+ || owner.session == 0
+ || owner.generation == 0
+ || owner.generation > group.generation
+ || (assignment.pending.is_none()
+ && !members.contains(&(owner.client_id, Some(owner.session))))
+ }) {
+ return Err(SnapshotError::InvalidState(
+ "invalid installed owner identity",
+ ));
+ }
+ if let Some(pending) = assignment.pending
+ && ((pending.owner.client_id == 0) != (pending.owner.session == 0)
+ || pending.owner.generation == 0
+ || pending.owner.generation > group.generation
+ || pending.metadata_op == 0
+ || pending.metadata_op > metadata_op
+ || (!pending.owner.is_unassigned()
+ && !members.contains(&(
+ pending.owner.client_id,
+ Some(pending.owner.session),
+ )))
+ || assignment.owner.is_some_and(|owner| {
+ pending.owner.generation <= owner.generation
+ }))
+ {
+ return Err(SnapshotError::InvalidState(
+ "invalid pending owner installation",
+ ));
+ }
+ }
+ }
+ }
+ }
+ let mut scopes = BTreeSet::new();
+ for (&op, intent) in &self.lifecycle_intents {
+ if op == 0
+ || op != intent.context.metadata_op
+ || op > metadata_op
+ || intent.partitions.is_empty()
+ || intent
+ .partitions
+ .iter()
+ .all(|target| target.partition_op.is_some())
+ {
+ return Err(SnapshotError::InvalidState(
+ "invalid lifecycle identity or completion",
+ ));
+ }
+ let stream_wide = matches!(intent.action, LifecycleAction::DeleteStream);
+ if stream_wide != intent.topic_id.is_none() {
+ return Err(SnapshotError::InvalidState(
+ "lifecycle scope does not match its action",
+ ));
+ }
+ let stream = self
+ .items
+ .iter()
+ .find(|(id, _)| *id == intent.stream_id as usize)
+ .map(|(_, stream)| stream)
+ .ok_or(SnapshotError::InvalidState("lifecycle stream is absent"))?;
+ let mut expected = BTreeSet::new();
+ for (topic_id, topic) in &stream.topics {
+ if intent.topic_id.is_some_and(|id| id as usize != *topic_id) {
+ continue;
+ }
+ if !scopes.insert((intent.stream_id, *topic_id)) {
+ return Err(SnapshotError::InvalidState("overlapping lifecycle intents"));
+ }
+ let first = if let LifecycleAction::DeletePartitions { count } = intent.action {
+ if count == 0 {
+ return Err(SnapshotError::InvalidState("empty partition deletion"));
+ }
+ topic.partitions.len().checked_sub(count as usize).ok_or(
+ SnapshotError::InvalidState("partition deletion exceeds its topic"),
+ )?
+ } else {
+ 0
+ };
+ for partition in &topic.partitions[first..] {
+ expected.insert((*topic_id, partition.id));
+ }
+ }
+ for target in &intent.partitions {
+ let topic = stream
+ .topics
+ .iter()
+ .find(|(id, _)| *id == target.topic_id as usize)
+ .map(|(_, topic)| topic)
+ .ok_or(SnapshotError::InvalidState("lifecycle topic is absent"))?;
+ let partition = topic
+ .partitions
+ .iter()
+ .find(|partition| partition.id == target.partition_id as usize)
+ .ok_or(SnapshotError::InvalidState("lifecycle partition is absent"))?;
+ if !expected.remove(&(target.topic_id as usize, target.partition_id as usize))
+ || target.partition_op == Some(0)
+ {
+ return Err(SnapshotError::InvalidState(
+ "duplicate or unexpected lifecycle partition",
+ ));
+ }
+ match (intent.action, target.fence) {
+ (
+ LifecycleAction::DeleteConsumerGroup { group_id },
+ LifecycleFence::Owner(installation),
+ ) => {
+ let group = topic
+ .consumer_groups
+ .iter()
+ .find(|(id, _)| *id == u64::from(group_id))
+ .map(|(_, group)| group)
+ .ok_or(SnapshotError::InvalidState(
+ "deleted consumer group is absent",
+ ))?;
+ if installation.incarnation != partition.created_revision
+ || installation.metadata_op != op
+ || installation.group_id != u64::from(group_id)
+ || !installation.owner.is_unassigned()
+ || installation.owner.generation == 0
+ || installation.owner.generation > group.generation.saturating_add(1)
+ {
+ return Err(SnapshotError::InvalidState(
+ "incorrect consumer group deletion fence",
+ ));
+ }
+ }
+ (LifecycleAction::DeleteConsumerGroup { .. }, _)
+ | (_, LifecycleFence::Owner(_)) => {
+ return Err(SnapshotError::InvalidState(
+ "incorrect lifecycle fence kind",
+ ));
+ }
+ (_, LifecycleFence::History(transition)) => {
+ if transition.incarnation != partition.created_revision
+ || transition.metadata_op != op
+ {
+ return Err(SnapshotError::InvalidState(
+ "incorrect partition history fence",
+ ));
+ }
+ }
+ }
+ }
+ if !expected.is_empty() {
+ return Err(SnapshotError::InvalidState(
+ "lifecycle omits affected partitions",
+ ));
+ }
+ }
+ Ok(())
+ }
+}
+
+#[cfg(test)]
+pub(crate) fn apply_with_lifecycle_completion<R: StateHandler<State = StreamsInner>>(
+ request: &R,
+ state: &mut StreamsInner,
+ timestamp: IggyTimestamp,
+) -> ApplyReply {
+ state.apply_context.metadata_op += 1;
+ let op = state.apply_context.metadata_op;
+ let reply = request.apply(state, timestamp);
+ if reply.code != 0 {
+ return reply;
+ }
+ let Some(intent) = state.lifecycle_intents.get(&op).cloned() else {
+ return reply;
+ };
+ for (index, partition) in intent.partitions.iter().enumerate() {
+ state.apply_context.metadata_op += 1;
+ let result = CompleteLifecycleRequest {
+ metadata_op: op,
+ stream_id: intent.stream_id,
+ topic_id: partition.topic_id,
+ partition_id: partition.partition_id,
+ partition_op: u64::try_from(index).unwrap() + 1,
+ }
+ .apply(state, timestamp);
+ assert_eq!(
+ result.code, 0,
+ "fixture must complete each matching partition fence"
+ );
+ }
+ assert!(!state.lifecycle_intents.contains_key(&op));
+ reply
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::stm::State;
+ use crate::stm::consumer_group::{
+ CompleteConsumerGroupRevocationRequest, ConsumerGroup, ConsumerGroupMember,
+ ConsumerGroupSnapshot,
+ };
+ use crate::stm::snapshot::Snapshotable;
+ use iggy_binary_protocol::primitives::partition_assignment::CreatedPartitionAssignment;
+ use iggy_binary_protocol::requests::consumer_groups::JoinConsumerGroupRequest;
+ use iggy_binary_protocol::requests::topics::{
+ CreateTopicRequest, CreateTopicWithAssignmentsRequest,
+ };
+ use iggy_binary_protocol::{Operation, PrepareHeader, WireName, WireOptions};
+ use iggy_common::Either;
+ use server_common::Message;
+ use server_common::sharding::IggyNamespace;
+ use std::sync::Arc;
+
+ const INTENT_OP: u64 = 50;
+
+ fn streams() -> Streams {
+ let streams: Streams = StreamsInner::new().into();
+ streams.seed_namespace(IggyNamespace::new(0, 0, 1), 2, 0);
+ streams
+ }
+
+ fn context() -> ApplyContext {
+ ApplyContext {
+ metadata_op: INTENT_OP,
+ client_id: 17,
+ session: 5,
+ request: 8,
+ }
+ }
+
+ fn apply(
+ streams: &Streams,
+ operation: Operation,
+ request: &impl WireEncode,
+ context: ApplyContext,
+ ) -> ApplyReply {
+ let body = request.to_bytes();
+ let size = iggy_binary_protocol::HEADER_SIZE + body.len();
+ let mut prepare = Message::<PrepareHeader>::new(size).transmute_header(
+ |_, header: &mut PrepareHeader| {
+ header.command = iggy_binary_protocol::Command::Prepare;
+ header.operation = operation;
+ header.op = context.metadata_op;
+ header.client = context.client_id;
+ header.session = context.session;
+ header.request = context.request;
+ header.timestamp = 100;
+ header.size = u32::try_from(size).unwrap();
+ },
+ );
+ prepare.as_mut_slice()[iggy_binary_protocol::HEADER_SIZE..].copy_from_slice(&body);
+ match State::apply(streams, prepare).unwrap() {
+ Either::Left(reply) => reply,
+ Either::Right(_) => panic!("lifecycle operation was not dispatched"),
+ }
+ }
+
+ fn complete(streams: &Streams, partition_id: u32, partition_op: u64) {
+ let reply = apply(
+ streams,
+ Operation::CompleteLifecycle,
+ &CompleteLifecycleRequest {
+ metadata_op: INTENT_OP,
+ stream_id: 0,
+ topic_id: 0,
+ partition_id,
+ partition_op,
+ },
+ ApplyContext {
+ metadata_op: INTENT_OP + 1 + u64::from(partition_id),
+ ..ApplyContext::default()
+ },
+ );
+ assert_eq!(reply.code, 0);
+ }
+
+ #[test]
+ fn given_invalid_lifecycle_snapshot_when_restoring_should_reject_without_mutating_state() {
+ let streams = streams();
+ assert_eq!(
+ apply(
+ &streams,
+ Operation::DeleteTopic,
+ &DeleteTopicRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ },
+ context()
+ )
+ .code,
+ 0
+ );
+ let valid = streams.to_snapshot();
+ #[allow(clippy::type_complexity)]
+ let mutations: &[(&str, fn(&mut LifecycleIntent))] = &[
+ ("missing partition", |intent| {
+ intent.partitions.pop();
+ }),
+ ("duplicate partition", |intent| {
+ intent.partitions.push(intent.partitions[0].clone());
+ }),
+ ("wrong scope", |intent| {
+ intent.topic_id = Some(7);
+ }),
+ ("zero completion", |intent| {
+ intent.partitions[0].partition_op = Some(0);
+ }),
+ ("false completion", |intent| {
+ for partition in &mut intent.partitions {
+ partition.partition_op = Some(1);
+ }
+ }),
+ ("wrong history", |intent| {
+ if let LifecycleFence::History(transition) = &mut intent.partitions[0].fence {
+ transition.incarnation += 1;
+ }
+ }),
+ ];
+ for (label, mutate) in mutations {
+ let mut invalid = valid.clone();
+ mutate(invalid.lifecycle_intents.get_mut(&INTENT_OP).unwrap());
+ let mut snapshot = crate::stm::snapshot::MetadataSnapshot::new(INTENT_OP);
+ snapshot.streams = Some(invalid);
+ let encoded = snapshot.encode().unwrap();
+ assert!(
+ crate::stm::snapshot::MetadataSnapshot::decode(&encoded).is_err(),
+ "{label}"
+ );
+ assert!(
+ crate::stm::snapshot::RestoreSnapshotInPlace::restore_snapshot_in_place(
+ &streams, &snapshot
+ )
+ .is_err(),
+ "{label}"
+ );
+ assert_eq!(
+ rmp_serde::to_vec(&streams.to_snapshot()).unwrap(),
+ rmp_serde::to_vec(&valid).unwrap(),
+ "{label}"
+ );
+ }
+ }
+
+ #[test]
+ fn given_invalid_owner_snapshot_when_restoring_should_reject_missing_or_unrelated_authority() {
+ let streams = streams();
+ let partitions = streams.read(|inner| inner.items[0].topics[0].partitions.clone());
+ let mut group = ConsumerGroup::new(0, Arc::from("group"));
+ let mut member = ConsumerGroupMember::new(0, 17);
+ member.session = Some(5);
+ group.members.insert(member);
+ group.rebalance_members(&partitions, INTENT_OP, 100);
+ let assignment = &group.assignments[&0];
+ let pending = assignment.pending.unwrap();
+ assert!(group.complete_revocation(
+ 0,
+ &InstallConsumerGroupOwnerRequest {
+ incarnation: assignment.incarnation,
+ group_id: 0,
+ owner: pending.owner,
+ metadata_op: pending.metadata_op,
+ },
+ 12
+ ));
+ let mut valid = streams.to_snapshot();
+ let topic = &mut valid.items[0].1.topics[0].1;
+ topic.next_consumer_group_id = 1;
+ topic
+ .consumer_groups
+ .push((0, ConsumerGroupSnapshot::from_group(&group)));
+ assert!(valid.validate_ownership_history(INTENT_OP).is_ok());
+ for mutation in 0..4 {
+ let mut invalid = valid.clone();
+ let group = &mut invalid.items[0].1.topics[0].1.consumer_groups[0].1;
+ match mutation {
+ 0 => {
+ group.assignments.remove(&1);
+ }
+ 1 => {
+ group
+ .assignments
+ .get_mut(&0)
+ .unwrap()
+ .owner
+ .as_mut()
+ .unwrap()
+ .session += 1;
+ }
+ 2 => {
+ group
+ .assignments
+ .get_mut(&1)
+ .unwrap()
+ .pending
+ .as_mut()
+ .unwrap()
+ .owner
+ .client_id += 1;
+ }
+ _ => {
+ group.members[0].1.session = None;
+ }
+ }
+ let mut snapshot = crate::stm::snapshot::MetadataSnapshot::new(INTENT_OP);
+ snapshot.streams = Some(invalid);
+ assert!(
+ crate::stm::snapshot::MetadataSnapshot::decode(&snapshot.encode().unwrap())
+ .is_err()
+ );
+ }
+ }
+
+ #[test]
+ fn given_partial_delete_when_restored_should_retain_identity_and_wait_for_every_partition() {
+ let streams = streams();
+ let namespaces = [IggyNamespace::new(0, 0, 0), IggyNamespace::new(0, 0, 1)];
+ let reply = apply(
+ &streams,
+ Operation::DeleteTopic,
+ &DeleteTopicRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ },
+ context(),
+ );
+ assert_eq!(reply.code, 0);
+ assert!(streams.lifecycle_pending(INTENT_OP));
+ complete(&streams, 0, 12);
+ for namespace in namespaces {
+ assert!(streams.created_revision_for_namespace(namespace).is_some());
+ }
+
+ let snapshot = streams.to_snapshot();
+ let encoded = rmp_serde::to_vec_named(&snapshot).unwrap();
+ let restored = Streams::from_snapshot(rmp_serde::from_slice(&encoded).unwrap()).unwrap();
+ let pending = restored.pending_lifecycles();
+ assert_eq!(pending.len(), 1);
+ assert_eq!(pending[0].context.client_id, 17);
+ assert_eq!(pending[0].context.session, 5);
+ assert_eq!(pending[0].context.request, 8);
+ assert_eq!(pending[0].partitions[0].partition_op, Some(12));
+ assert_eq!(pending[0].remaining_partitions, 1);
+ complete(&restored, 0, 12);
+ assert!(restored.lifecycle_pending(INTENT_OP));
+ assert_eq!(restored.pending_lifecycles()[0].remaining_partitions, 1);
+ for namespace in namespaces {
+ assert!(restored.created_revision_for_namespace(namespace).is_some());
+ }
+ complete(&restored, 1, 23);
+ assert!(!restored.lifecycle_pending(INTENT_OP));
+ for namespace in namespaces {
+ assert_eq!(restored.created_revision_for_namespace(namespace), None);
+ }
+ }
+
+ #[test]
+ fn given_restored_ownership_transitions_when_completed_should_retain_other_targets() {
+ let streams = streams();
+ let partitions = streams.read(|inner| inner.items[0].topics[0].partitions.clone());
+ let mut snapshot = streams.to_snapshot();
+ let topic = &mut snapshot.items[0].1.topics[0].1;
+ for group_id in 0..2 {
+ let mut group = ConsumerGroup::new(group_id, Arc::from(format!("group-{group_id}")));
+ let mut member = ConsumerGroupMember::new(0, 17);
+ member.session = Some(5);
+ group.members.insert(member);
+ group.rebalance_members(&partitions, INTENT_OP - 1, 100);
+ topic
+ .consumer_groups
+ .push((group_id, ConsumerGroupSnapshot::from_group(&group)));
+ }
+ topic.next_consumer_group_id = 2;
+ let encoded = rmp_serde::to_vec_named(&snapshot).unwrap();
+ let restored = Streams::from_snapshot(rmp_serde::from_slice(&encoded).unwrap()).unwrap();
+ let pending = restored.consumer_group_pending_revocations();
+ assert_eq!(pending.len(), 4);
+ let target = pending[0];
+ let completion = CompleteConsumerGroupRevocationRequest {
+ stream_id: WireIdentifier::numeric(target.stream_id),
+ topic_id: WireIdentifier::numeric(target.topic_id),
+ partition_id: target.partition_id,
+ installation: target.installation,
+ partition_op: 12,
+ };
+ assert_eq!(
+ apply(
+ &restored,
+ Operation::CompleteConsumerGroupRevocation,
+ &completion,
+ context()
+ )
+ .code,
+ 0
+ );
+ let remaining = restored.consumer_group_pending_revocations();
+ assert_eq!(remaining.len(), 3);
+ assert!(
+ !remaining
+ .iter()
+ .any(|transition| transition.partition_id == target.partition_id
+ && transition.installation.group_id == target.installation.group_id)
+ );
+ assert_eq!(
+ apply(
+ &restored,
+ Operation::CompleteConsumerGroupRevocation,
+ &completion,
+ context()
+ )
+ .code,
+ 0
+ );
+ assert_eq!(restored.consumer_group_pending_revocations().len(), 3);
+ let encoded = rmp_serde::to_vec_named(&restored.to_snapshot()).unwrap();
+ let restored = Streams::from_snapshot(rmp_serde::from_slice(&encoded).unwrap()).unwrap();
+ assert_eq!(restored.consumer_group_pending_revocations().len(), 3);
+ }
+
+ #[test]
+ fn given_pending_group_deletion_when_other_consumers_arrive_should_isolate_the_group() {
+ let streams = streams();
+ let partitions = streams.read(|inner| inner.items[0].topics[0].partitions.clone());
+ let mut snapshot = streams.to_snapshot();
+ let topic = &mut snapshot.items[0].1.topics[0].1;
+ for group_id in 0..2 {
+ let mut group = ConsumerGroup::new(group_id, Arc::from(format!("group-{group_id}")));
+ group.rebalance_members(&partitions, INTENT_OP - 1, 100);
+ topic
+ .consumer_groups
+ .push((group_id, ConsumerGroupSnapshot::from_group(&group)));
+ }
+ topic.next_consumer_group_id = 2;
+ let streams = Streams::from_snapshot(snapshot).unwrap();
+ assert_eq!(
+ apply(
+ &streams,
+ Operation::DeleteConsumerGroup,
+ &DeleteConsumerGroupRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ group_id: WireIdentifier::numeric(0),
+ },
+ context()
+ )
+ .code,
+ 0
+ );
+ assert!(streams.lifecycle_pending(INTENT_OP));
+ assert!(streams.lifecycle_blocks_consumer(0, 0, Some(0)));
+ assert!(!streams.lifecycle_blocks_consumer(0, 0, Some(1)));
+ assert!(!streams.lifecycle_blocks_consumer(0, 0, None));
+ for group_id in 0..2 {
+ let request = JoinConsumerGroupRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ group_id: WireIdentifier::numeric(group_id),
+ };
+ assert_eq!(
+ streams
+ .request_lifecycle_blocked(Operation::JoinConsumerGroup, &request.to_bytes()),
+ group_id == 0
+ );
+ }
+ assert!(!streams.request_lifecycle_blocked(Operation::SendMessages, &[]));
+ assert!(
+ streams.request_lifecycle_blocked(
+ Operation::DeleteTopic,
+ &DeleteTopicRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ }
+ .to_bytes()
+ )
+ );
+ complete(&streams, 0, 12);
+ assert!(streams.lifecycle_blocks_consumer(0, 0, Some(0)));
+ complete(&streams, 1, 23);
+ assert!(!streams.lifecycle_blocks_consumer(0, 0, Some(0)));
+ assert!(!streams.lifecycle_pending(INTENT_OP));
+ assert_eq!(
+ streams.read(|inner| inner.items[0].topics[0]
+ .consumer_groups
+ .keys()
+ .copied()
+ .collect::<Vec<_>>()),
+ vec![1]
+ );
+ }
+
+ #[test]
+ fn given_pending_delete_when_conflicting_delete_arrives_should_retain_retirement_coverage() {
+ let streams = streams();
+ let namespace = IggyNamespace::new(0, 0, 0);
+ let incarnation = streams.created_revision_for_namespace(namespace).unwrap();
+ let reply = apply(
+ &streams,
+ Operation::DeleteTopic,
+ &DeleteTopicRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ },
+ context(),
+ );
+ assert_eq!(reply.code, 0);
+ complete(&streams, 0, 12);
+ assert_eq!(
+ streams.created_revision_for_namespace(namespace),
+ Some(incarnation)
+ );
+ let conflict = apply(
+ &streams,
+ Operation::DeletePartitions,
+ &DeletePartitionsRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ partitions_count: 1,
+ },
+ ApplyContext {
+ metadata_op: 60,
+ ..context()
+ },
+ );
+ assert_eq!(conflict.code, IggyError::LifecycleBusy.as_code());
+ assert_eq!(streams.pending_lifecycles().len(), 1);
+ complete(&streams, 1, 23);
+ assert_eq!(streams.created_revision_for_namespace(namespace), None);
+ streams.seed_namespace(namespace, 3, 0);
+ assert!(streams.created_revision_for_namespace(namespace).unwrap() > incarnation);
+ }
+
+ #[test]
+ fn given_pending_intent_when_topic_is_created_should_admit_and_apply_alike() {
+ let create_topic = CreateTopicWithAssignmentsRequest {
+ created_view: 0,
+ request: CreateTopicRequest {
+ stream_id: WireIdentifier::numeric(0),
+ partitions_count: 1,
+ name: WireName::new("other").unwrap(),
+ options: WireOptions::empty(),
+ },
+ derived_options: WireOptions::empty(),
+ partitions: vec![CreatedPartitionAssignment {
+ partition_id: 0,
+ consensus_group_id: 1,
+ }],
+ };
+ for stream_wide in [false, true] {
+ let streams = streams();
+ let intent = if stream_wide {
+ apply(
+ &streams,
+ Operation::DeleteStream,
+ &DeleteStreamRequest {
+ stream_id: WireIdentifier::numeric(0),
+ },
+ context(),
+ )
+ } else {
+ apply(
+ &streams,
+ Operation::DeleteTopic,
+ &DeleteTopicRequest {
+ stream_id: WireIdentifier::numeric(0),
+ topic_id: WireIdentifier::numeric(0),
+ },
+ context(),
+ )
+ };
+ assert_eq!(intent.code, 0);
+ assert!(streams.lifecycle_pending(INTENT_OP));
+ assert_eq!(
+ streams.request_lifecycle_blocked(
+ Operation::CreateTopic,
+ &create_topic.request.to_bytes()
+ ),
+ stream_wide,
+ "admission, stream_wide={stream_wide}"
+ );
+ let created = apply(
+ &streams,
+ Operation::CreateTopicWithAssignments,
+ &create_topic,
+ ApplyContext {
+ metadata_op: INTENT_OP + 10,
+ ..context()
+ },
+ );
+ let expected = if stream_wide {
+ IggyError::LifecycleBusy.as_code()
+ } else {
+ 0
+ };
+ assert_eq!(created.code, expected, "apply, stream_wide={stream_wide}");
+ }
+ }
+}
diff --git a/core/metadata/src/stm/mod.rs b/core/metadata/src/stm/mod.rs
index 198414663..d525ab686 100644
--- a/core/metadata/src/stm/mod.rs
+++ b/core/metadata/src/stm/mod.rs
@@ -18,11 +18,13 @@
pub mod authz;
pub mod consumer_group;
pub mod id_slab;
+pub mod lifecycle;
pub mod mux;
pub mod result;
pub mod snapshot;
pub mod stream;
pub mod user;
+use iggy_binary_protocol::PrepareHeader;
use iggy_common::Either;
use left_right::{Absorb, ReadHandle, ReadHandleFactory, WriteHandle};
use std::cell::Cell;
@@ -31,6 +33,27 @@ use std::sync::Arc;
pub use left_right::ReadHandleFactory as LeftRightFactory;
+/// Identity of the committed operation driving a metadata transition.
+/// The Streams revision is an incarnation counter and cannot replace `metadata_op`.
+#[derive(Debug, Clone, Copy, Default, serde::Serialize, serde::Deserialize)]
+pub struct ApplyContext {
+ pub metadata_op: u64,
+ pub client_id: u128,
+ pub session: u64,
+ pub request: u64,
+}
+
+impl From<&PrepareHeader> for ApplyContext {
+ fn from(header: &PrepareHeader) -> Self {
+ Self {
+ metadata_op: header.op,
+ client_id: header.client,
+ session: header.session,
+ request: header.request,
+ }
+ }
+}
+
pub struct WriteCell<T, O>
where
T: Absorb<O>,
@@ -308,6 +331,7 @@ macro_rules! define_state {
$field_vis $field_name: $field_type,
)*
pub(crate) last_result: Option<$crate::stm::result::ApplyReply>,
+ pub(crate) apply_context: $crate::stm::ApplyContext,
}
impl [<$state Inner>] {
@@ -403,10 +427,10 @@ macro_rules! collect_handlers {
#[derive(Debug, Clone)]
pub enum [<$state Command>] {
$(
- $operation([<$operation Request>], ::iggy_common::IggyTimestamp),
+ $operation([<$operation Request>], ::iggy_common::IggyTimestamp, $crate::stm::ApplyContext),
)*
$($(
- $internal([<$internal Request>], ::iggy_common::IggyTimestamp),
+ $internal([<$internal Request>], ::iggy_common::IggyTimestamp, $crate::stm::ApplyContext),
)*)?
/// Replace the whole state from a snapshot section, in place.
/// Never parsed off the wire (state transfer installs it via
@@ -429,9 +453,9 @@ macro_rules! collect_handlers {
// Both scalars copied out of one header read. `header()`
// re-validates the bit pattern on each call, and the borrow has
// to end before the pass-through arm can move `input` on.
- let (operation, timestamp) = {
+ let (operation, timestamp, context) = {
let header = input.header();
- (header.operation, header.timestamp)
+ (header.operation, header.timestamp, $crate::stm::ApplyContext::from(header))
};
match operation {
@@ -443,7 +467,7 @@ macro_rules! collect_handlers {
let cmd = [<$operation Request>]::decode_from(input.body())
.map_err(|_| ::iggy_common::IggyError::InvalidCommand)?;
let ts = ::iggy_common::IggyTimestamp::from(timestamp);
- Ok(Either::Left([<$state Command>]::$operation(cmd, ts)))
+ Ok(Either::Left([<$state Command>]::$operation(cmd, ts, context)))
},
)*
_ => Ok(Either::Right(input)),
@@ -455,12 +479,14 @@ macro_rules! collect_handlers {
fn dispatch(&mut self, cmd: &[<$state Command>]) {
self.last_result = Some(match cmd {
$(
- [<$state Command>]::$operation(payload, ts) => {
+ [<$state Command>]::$operation(payload, ts, context) => {
+ self.apply_context = *context;
$crate::stm::StateHandler::apply(payload, self, *ts)
},
)*
$($(
- [<$state Command>]::$internal(payload, ts) => {
+ [<$state Command>]::$internal(payload, ts, context) => {
+ self.apply_context = *context;
$crate::stm::StateHandler::apply(payload, self, *ts)
},
)*)?
diff --git a/core/metadata/src/stm/result.rs b/core/metadata/src/stm/result.rs
index 57c72ac8a..4bd7aaaf2 100644
--- a/core/metadata/src/stm/result.rs
+++ b/core/metadata/src/stm/result.rs
@@ -176,6 +176,7 @@ result_enum!(UpdateTopicResult {
StreamNotFound = 1009,
TopicNotFound = 2010,
NameAlreadyExists = 2013,
+ UnsupportedOptionKey = 4041,
InvalidOptionValue = 4042,
});
result_enum!(DeleteTopicResult {
@@ -191,12 +192,14 @@ result_enum!(CreatePartitionsResult {
StreamNotFound = 1009,
TopicNotFound = 2010,
InvalidPartitionsCount = 2019,
+ PartitionResizeDisabled = 2023,
PartitionIdSpaceExhausted = 3013,
});
result_enum!(DeletePartitionsResult {
StreamNotFound = 1009,
TopicNotFound = 2010,
InvalidPartitionsCount = 2019,
+ PartitionResizeDisabled = 2023,
});
// `TruncatePartition` is the committed form of a client `DeleteSegments`; an
// unresolvable target commits as a rejection so the request sequence stays
@@ -274,23 +277,28 @@ result_enum!(LeaveConsumerGroupResult {
/// `IggyError::Unauthorized`. Any control-plane op can commit as an in-apply
/// authorization no-op, so this code is valid for every op regardless of its
-/// own result enum. Pinned to the wire discriminant in
+/// own result enum. Pinned, like the two codes below, to the wire discriminant in
/// [`tests::given_result_discriminants_then_match_iggy_error_codes`].
const UNAUTHORIZED_CODE: u32 = 41;
+/// `IggyError::LifecycleBusy`, committed by the lifecycle apply gates.
+const LIFECYCLE_BUSY_CODE: u32 = 88;
+/// `IggyError::InvalidCommand`, committed by the lifecycle apply paths.
+const INVALID_COMMAND_CODE: u32 = 3;
/// True if `code` is one this op's result enum declares (`Ok = 0` included), or
-/// the global `UNAUTHORIZED_CODE`.
+/// one of the global codes `UNAUTHORIZED_CODE`, `LIFECYCLE_BUSY_CODE` and
+/// `INVALID_COMMAND_CODE`.
///
/// The state machine only commits codes from the op's own enum, plus the
-/// in-apply authorization gate's global `Unauthorized` denial, so an
+/// global in-apply denials of the authorization and lifecycle gates, so an
/// unrecognized one is a server bug, not a race (a race still yields a declared
/// code). Enriched internal ops map to their client-op enum; ops with no result
/// section (partition plane) return `true`.
#[must_use]
pub const fn result_code_recognized(operation: Operation, code: u32) -> bool {
- // The in-apply RBAC gate can deny any control-plane op, committing an
- // `Unauthorized` no-op, so accept it globally rather than per-op enum.
- if code == UNAUTHORIZED_CODE {
+ // The in-apply RBAC and lifecycle gates can deny any control-plane op,
+ // committing a no-op, so accept their codes globally rather than per-op enum.
+ if code == UNAUTHORIZED_CODE || code == LIFECYCLE_BUSY_CODE || code == INVALID_COMMAND_CODE {
return true;
}
match operation {
@@ -418,6 +426,16 @@ mod tests {
Operation::CreateConsumerGroup,
UNAUTHORIZED_CODE
));
+ // The lifecycle apply gates commit `LifecycleBusy` for ops whose enums do
+ // not declare it.
+ assert!(result_code_recognized(
+ Operation::DeleteTopic,
+ IggyError::LifecycleBusy.as_code()
+ ));
+ assert!(result_code_recognized(
+ Operation::JoinConsumerGroup,
+ IggyError::LifecycleBusy.as_code()
+ ));
}
// Pins every result-enum discriminant to the `IggyError` variant its doc
@@ -673,5 +691,8 @@ mod tests {
// Unauthorized (41) - the global in-apply RBAC denial code.
assert_eq!(UNAUTHORIZED_CODE, IggyError::Unauthorized.as_code());
+ // LifecycleBusy (88) and InvalidCommand (3) - the global lifecycle gate codes.
+ assert_eq!(LIFECYCLE_BUSY_CODE, IggyError::LifecycleBusy.as_code());
+ assert_eq!(INVALID_COMMAND_CODE, IggyError::InvalidCommand.as_code());
}
}
diff --git a/core/metadata/src/stm/snapshot.rs b/core/metadata/src/stm/snapshot.rs
index 1988997fb..b27fbc42d 100644
--- a/core/metadata/src/stm/snapshot.rs
+++ b/core/metadata/src/stm/snapshot.rs
@@ -53,7 +53,9 @@ use crate::stm::user::UsersSnapshot;
/// Version 8: streams persist the revision of their partition incarnations.
///
/// Version 10: `PartitionSnapshot` dropped `purge_generation`.
-pub const SNAPSHOT_FORMAT_VERSION: u32 = 10;
+///
+/// Version 11: streams also persist pending lifecycle intents.
+pub const SNAPSHOT_FORMAT_VERSION: u32 = 11;
/// Oldest format version [`MetadataSnapshot::decode`] still reads.
///
@@ -78,6 +80,7 @@ const _: () = assert!(SNAPSHOT_WRITER_RELEASE > 0);
#[derive(Debug)]
pub enum SnapshotError {
+ InvalidState(&'static str),
/// Serialization failed.
Serialize(rmp_serde::encode::Error),
/// Deserialization failed.
@@ -98,17 +101,25 @@ pub enum SnapshotError {
/// The snapshot's integrity trailer does not match its payload: a torn or
/// bit-rotted checkpoint. Refuse to restore from it rather than feed corrupt state
/// into the state machine.
- ChecksumMismatch { expected: u128, actual: u128 },
+ ChecksumMismatch {
+ expected: u128,
+ actual: u128,
+ },
/// Snapshot integrity framing is absent or damaged.
InvalidTrailer,
/// Snapshot file is too short to contain a valid checksum.
- Truncated { size: u64 },
+ Truncated {
+ size: u64,
+ },
/// The snapshot was written in a format version this build does not read: a
/// different build wrote it, on this disk or on a state-transfer peer. Refuse it
/// rather than let msgpack read one field's bytes as another's. The version is
/// peeked ahead of the rest of the payload, so this fires even when nothing past
/// it is recognizable.
- UnsupportedFormatVersion { found: u32, expected: u32 },
+ UnsupportedFormatVersion {
+ found: u32,
+ expected: u32,
+ },
/// A state-transfer descriptor's frontiers contradict its artifacts: a
/// `commit_op` below the snapshot the same offer ships, or a client-table
/// frontier outside the snapshot-to-commit range. Both are impossible from a
@@ -142,6 +153,7 @@ pub enum PersistStage {
impl fmt::Display for SnapshotError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
+ Self::InvalidState(reason) => write!(f, "invalid snapshot state: {reason}"),
Self::Serialize(e) => write!(f, "snapshot serialization failed: {e}"),
Self::Deserialize(e) => write!(f, "snapshot deserialization failed: {e}"),
Self::ClientTable(e) => write!(f, "snapshot client table is invalid: {e}"),
@@ -192,7 +204,8 @@ impl std::error::Error for SnapshotError {
Self::Deserialize(e) => Some(e),
Self::ClientTable(e) => Some(e),
Self::Io(e) | Self::Persist { source: e, .. } => Some(e),
- Self::ChecksumMismatch { .. }
+ Self::InvalidState(..)
+ | Self::ChecksumMismatch { .. }
| Self::MissingClientTable
| Self::InvalidTrailer
| Self::Truncated { .. }
@@ -303,7 +316,16 @@ impl MetadataSnapshot {
expected: SNAPSHOT_FORMAT_VERSION,
});
}
- rmp_serde::from_slice(bytes).map_err(SnapshotError::Deserialize)
+ let snapshot: Self = rmp_serde::from_slice(bytes).map_err(SnapshotError::Deserialize)?;
+ snapshot.validate()?;
+ Ok(snapshot)
+ }
+
+ pub(crate) fn validate(&self) -> Result<(), SnapshotError> {
+ if let Some(streams) = &self.streams {
+ streams.validate_ownership_history(self.sequence_number)?;
+ }
+ Ok(())
}
}
@@ -490,6 +512,7 @@ macro_rules! impl_fill_restore {
) -> Result<Self, $crate::stm::snapshot::SnapshotError> {
use serde::de::Error as _;
use $crate::stm::snapshot::{SnapshotError, Snapshotable};
+ snapshot.validate()?;
let snap = snapshot.$field.clone().ok_or_else(|| {
SnapshotError::Deserialize(rmp_serde::decode::Error::custom(format_args!(
"Snapshot Restore Error: {}",
@@ -510,6 +533,7 @@ macro_rules! impl_fill_restore {
use serde::de::Error as _;
use $crate::stm::snapshot::SnapshotError;
paste::paste! {
+ snapshot.validate()?;
let snap = snapshot.$field.clone().ok_or_else(|| {
SnapshotError::Deserialize(rmp_serde::decode::Error::custom(
format_args!("Snapshot Restore Error: {}", stringify!($field)),
@@ -531,6 +555,7 @@ macro_rules! impl_fill_restore {
) -> Result<(), $crate::stm::snapshot::SnapshotError> {
use serde::de::Error as _;
use $crate::stm::snapshot::SnapshotError;
+ snapshot.validate()?;
if snapshot.$field.is_none() {
return Err(SnapshotError::Deserialize(
rmp_serde::decode::Error::custom(format_args!(
@@ -578,7 +603,7 @@ mod tests {
// operator's boot reading one field's bytes as another's. Changing either
// number is the reminder to change the other.
const FIELD_COUNT: u32 = 7;
- const PINNED_VERSION: u32 = 10;
+ const PINNED_VERSION: u32 = 11;
let encoded = MetadataSnapshot::new(0).encode().unwrap();
let mut cursor = encoded.as_slice();
@@ -602,7 +627,7 @@ mod tests {
// actually grows need their own pins.
const TOPIC_FIELD_COUNT: u32 = 11;
const STREAM_FIELD_COUNT: u32 = 6;
- const STREAMS_FIELD_COUNT: u32 = 3;
+ const STREAMS_FIELD_COUNT: u32 = 4;
const USER_FIELD_COUNT: u32 = 7;
// Version 4 appended `fences`; it is defaulted on read, which is what
// keeps version 3 readable, so a further append here needs the same
@@ -686,6 +711,7 @@ mod tests {
items: vec![(0, stream)],
revision: 1,
namespace_revision: 1,
+ lifecycle_intents: std::collections::BTreeMap::default(),
};
let encoded = rmp_serde::to_vec(&streams).unwrap();
assert_eq!(
@@ -827,6 +853,7 @@ mod tests {
let mut snapshot = MetadataSnapshot::new(100);
snapshot.streams = Some(StreamsSnapshot {
+ lifecycle_intents: std::collections::BTreeMap::default(),
revision: 0,
namespace_revision: 0,
items: vec![(
@@ -946,6 +973,7 @@ mod tests {
};
let streams_snap = StreamsSnapshot {
+ lifecycle_intents: std::collections::BTreeMap::default(),
revision: 0,
namespace_revision: 0,
items: vec![
diff --git a/core/metadata/src/stm/stream.rs b/core/metadata/src/stm/stream.rs
index e1a9871cd..567f8db9d 100644
--- a/core/metadata/src/stm/stream.rs
+++ b/core/metadata/src/stm/stream.rs
@@ -15,23 +15,25 @@
// specific language governing permissions and limitations
// under the License.
-use crate::stm::StateHandler;
use crate::stm::consumer_group::{
- CompleteConsumerGroupRevocationRequest, ConsumerGroup, ConsumerGroupSnapshot,
- JoinConsumerGroupRequest, LeaveConsumerGroupRequest, RefreshConsumerGroupSessionRequest,
- RemoveConsumerGroupMemberRequest,
+ CompleteConsumerGroupRevocationRequest, ConsumerGroup, ConsumerGroupOwnershipTransition,
+ ConsumerGroupSnapshot, JoinConsumerGroupRequest, LeaveConsumerGroupRequest,
+ RefreshConsumerGroupSessionRequest, RemoveConsumerGroupMemberRequest,
};
use crate::stm::id_slab::IdSlab;
+use crate::stm::lifecycle::{CompleteLifecycleRequest, LifecycleAction, LifecycleIntent};
use crate::stm::result::{
ApplyReply, CreatePartitionsResult, CreateStreamResult, CreateTopicResult,
DeletePartitionsResult, DeleteStreamResult, DeleteTopicResult, TruncatePartitionResult,
UpdateStreamResult, UpdateTopicResult,
};
use crate::stm::snapshot::Snapshotable;
+use crate::stm::{ApplyContext, StateHandler};
use crate::{collect_handlers, define_state, impl_fill_restore};
use ahash::{AHashMap, AHashSet};
use bytes::{BufMut, Bytes, BytesMut};
use iggy_binary_protocol::codec::{WireDecode, WireEncode};
+use std::collections::BTreeMap;
// Only `seed_namespace` (sim/test-gated) uses this at module scope, so keep the
// import under the same gate. The test module re-imports it independently.
#[cfg(any(test, feature = "simulator"))]
@@ -51,6 +53,7 @@ use iggy_binary_protocol::requests::streams::{
// Only the slab-seeding helpers build a bare `CreateTopicRequest`; without
// their cfg the import is dead and `-p <crate>` clippy (which skips the
// simulator feature) rejects it.
+use iggy_binary_protocol::primitives::partition_history::PartitionContext;
#[cfg(any(test, feature = "simulator"))]
use iggy_binary_protocol::requests::topics::CreateTopicRequest;
use iggy_binary_protocol::requests::topics::{
@@ -66,9 +69,9 @@ use iggy_binary_protocol::responses::topics::get_topic::PartitionResponse;
use iggy_binary_protocol::{WireIdentifier, WireName};
use iggy_common::wire_conversions::{resource_options_from_wire, resource_options_to_wire_split};
use iggy_common::{
- CompressionAlgorithm, IggyByteSize, IggyExpiry, IggyTimestamp, MaxTopicSize, PartitionStats,
- ResourceOptions, StreamStats, TopicCreateOptions, TopicRuntimeOptions, TopicStats,
- topic_option_keys,
+ CompressionAlgorithm, IggyByteSize, IggyExpiry, IggyTimestamp, MaxTopicSize,
+ PartitionResizePolicy, PartitionStats, ResourceOptions, StreamStats, TopicCreateOptions,
+ TopicRuntimeOptions, TopicStats, topic_option_keys,
};
use serde::{Deserialize, Serialize};
use server_common::sharding::{IggyNamespace, MAX_PARTITIONS, MAX_STREAMS, MAX_TOPICS};
@@ -256,18 +259,9 @@ impl Topic {
}
}
- /// Re-run round-robin assignment for every consumer group under this topic
- /// against the current partition set. Called after a partition-count change
- /// (`CreatePartitions`/`DeletePartitions`) so groups pick up added
- /// partitions and drop removed ones; each `rebalance_members` bumps the
- /// group generation so stale clients re-sync.
- pub fn rebalance_consumer_groups(&mut self) {
- if self.consumer_groups.is_empty() {
- return;
- }
- let partition_ids: Vec<usize> = self.partitions.iter().map(|p| p.id).collect();
+ pub fn rebalance_consumer_groups(&mut self, metadata_op: u64, timestamp: IggyTimestamp) {
for group in self.consumer_groups.values_mut() {
- group.rebalance_members(&partition_ids);
+ group.rebalance_members(&self.partitions, metadata_op, timestamp.as_micros());
}
}
@@ -774,13 +768,10 @@ define_state! {
// Retirement proofs survive truncation, but not a changed set of
// partition incarnations.
pub namespace_revision: u64,
- // Total pending cooperative revocations across all groups, recomputed
- // once per commit by `post_apply`. The consensus tick reads it O(1)
- // every 10ms instead of walking every stream/topic/group/member to
- // decide whether to wake the reconciler. Deterministic (same ops, same
- // recompute on every replica).
- pub(crate) pending_revocations_count: u64,
- // Derived with the revocation count after membership changes and restore.
+ pub lifecycle_intents: BTreeMap<u64, LifecycleIntent>,
+ pub(crate) finalizing_lifecycle: bool,
+ // Rebuilt after membership changes and restore; ticks visit only open transitions.
+ pub(crate) pending_revocations: BTreeMap<(usize, usize, u64, usize), ConsumerGroupOwnershipTransition>,
// Locations are (stream, topic, group, member); never persisted.
pub(crate) consumer_group_members: AHashMap<u128, Vec<(usize, usize, u64, usize)>>,
// Shared aggregate stats, one `Arc` per stream/topic across both
@@ -890,6 +881,10 @@ impl StateHandler for TruncatePartitionRequest {
let Some(topic_id) = state.resolve_topic_id(stream_id, &self.topic_id) else {
return ApplyReply::err(TruncatePartitionResult::TopicNotFound);
};
+ if state.lifecycle_blocks(stream_id, Some(topic_id)) {
+ return ApplyReply::err(iggy_common::IggyError::LifecycleBusy.as_code());
+ }
+
let Some(stream) = state.items.get_mut(stream_id) else {
return ApplyReply::err(TruncatePartitionResult::StreamNotFound);
};
@@ -947,23 +942,29 @@ collect_handlers! {
RemoveConsumerGroupMember,
CompleteConsumerGroupRevocation,
TruncatePartition,
+ CompleteLifecycle,
}
internal { RefreshConsumerGroupSession }
}
impl StreamsInner {
/// Rebuild derived group metadata after membership/revocation changes or restore.
- /// Session refreshes use the membership index without rescanning unrelated clients.
+ /// A session refresh changes no membership, so it re-indexes only the groups it
+ /// rebalances instead of calling this.
pub(crate) fn recompute_consumer_group_metadata(&mut self) {
- let mut count: u64 = 0;
+ self.pending_revocations.clear();
for memberships in self.consumer_group_members.values_mut() {
memberships.clear();
}
for (stream_id, stream) in &self.items {
for (topic_id, topic) in &stream.topics {
for group in topic.consumer_groups.values() {
+ group.index_pending_revocations(
+ stream_id,
+ topic_id,
+ &mut self.pending_revocations,
+ );
for (member_id, member) in &group.members {
- count += member.pending_revocations.len() as u64;
self.consumer_group_members
.entry(member.client_id)
.or_default()
@@ -972,7 +973,6 @@ impl StreamsInner {
}
}
}
- self.pending_revocations_count = count;
self.consumer_group_members
.retain(|_, memberships| !memberships.is_empty());
}
@@ -1012,19 +1012,6 @@ impl StreamsInner {
WireIdentifier::String(name) => stream.topic_index.get(name.as_str()).copied(),
}
}
-
- /// Mutable topic resolved from (stream, topic) identifiers -- the
- /// consumer-group `StateHandler`s in [`crate::stm::consumer_group`] operate
- /// through this.
- pub(crate) fn topic_mut(
- &mut self,
- stream_id: &WireIdentifier,
- topic_id: &WireIdentifier,
- ) -> Option<&mut Topic> {
- let stream_id = self.resolve_stream_id(stream_id)?;
- let topic_id = self.resolve_topic_id(stream_id, topic_id)?;
- self.items.get_mut(stream_id)?.topics.get_mut(topic_id)
- }
}
/// Metadata identity for a consumer operation, independent of unrelated groups.
@@ -1036,6 +1023,18 @@ pub struct PollMetadata {
}
impl PollMetadata {
+ #[must_use]
+ pub const fn context(&self, metadata_op: u64) -> PartitionContext {
+ PartitionContext {
+ incarnation: self.created_revision,
+ owner_generation: match self.group {
+ Some((_, generation)) => generation,
+ None => 0,
+ },
+ metadata_op,
+ }
+ }
+
#[must_use]
pub fn matches_partition(&self, created_revision: Option<u64>) -> bool {
created_revision == Some(self.created_revision)
@@ -1088,6 +1087,13 @@ impl Streams {
require_pollable: bool,
) -> Option<PollMetadata> {
self.read(|inner| {
+ if inner.lifecycle_blocks_consumer(
+ namespace.stream_id(),
+ namespace.topic_id(),
+ group_id,
+ ) {
+ return None;
+ }
let topic = inner
.items
.get(namespace.stream_id())?
@@ -1106,7 +1112,14 @@ impl Streams {
}) {
return None;
}
- Some((group.id, group.generation))
+ let assignment = group.assignments.get(&namespace.partition_id())?;
+ let owner = assignment.owner?;
+ if owner.client_id != client_id
+ || assignment.incarnation != partition.created_revision
+ {
+ return None;
+ }
+ Some((group.id, owner.generation))
} else {
None
};
@@ -1346,12 +1359,13 @@ impl Streams {
}
/// Whether any consumer group has a pending cooperative revocation. O(1):
- /// reads the `pending_revocations_count` that `post_apply` maintains per
+ /// reads the open-transition index maintained after each membership
/// commit. The consensus tick polls this every 10ms to wake the reconciler
/// promptly when a source drains a revoked partition, so it must not walk.
#[must_use]
pub fn has_pending_revocations(&self) -> bool {
- self.inner.read(|inner| inner.pending_revocations_count > 0)
+ self.inner
+ .read(|inner| !inner.pending_revocations.is_empty())
}
/// The topic's current partition ids, for the join-time in-flight gather
@@ -1372,67 +1386,11 @@ impl Streams {
})
}
- /// Partitions currently owned by some live member of the group (union over
- /// members, pending-revoked included since the source still owns them until
- /// completion). The join-time in-flight gather uses this to tell a genuine
- /// in-flight hold (a live member polled past its commit) from a stale
- /// `last_polled` left by a since-removed member: only an owned partition can
- /// be in flight, so an unowned one with uncommitted data is the dead-member
- /// residue of a reconnect and must be reassigned, not protected.
#[must_use]
#[allow(clippy::cast_possible_truncation)]
- pub fn consumer_group_assigned_partitions(
- &self,
- stream_id: &WireIdentifier,
- topic_id: &WireIdentifier,
- group_id: &WireIdentifier,
- ) -> Option<std::collections::HashSet<u32>> {
- self.inner.read(|inner| {
- let stream_id = inner.resolve_stream_id(stream_id)?;
- let topic_id = inner.resolve_topic_id(stream_id, topic_id)?;
- let topic = inner.items.get(stream_id)?.topics.get(topic_id)?;
- let group = topic
- .consumer_groups
... 22150 lines suppressed ...