You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "krishvishal (via GitHub)" <gi...@apache.org> on 2026/10/03 05:52:25 UTC
[PR] feat(server): store offsets of groups managed outside Iggy [iggy]
krishvishal opened a new pull request, #4392:
URL: https://github.com/apache/iggy/pull/4392
## Which issue does this PR address?
Relates to #3542
## Rationale
The Kafka gateway needs to keep a committed offset for every Kafka group, topic and partition. The accepted design in `gateways/kafka/docs/OFFSET_STORAGE.md` stores it as an Iggy consumer group offset under `kafka.cg.<group>`. That kind does not fit
the gateway:
- The server lets only a member that owns the partition write a group offset, and the gateway never joins the group.
- The server refuses an offset past the last message. Kafka commits the next offset to read, which is one past the last message, and it can commit 0 on an empty partition.
- A group offset holds back retention. A Kafka committed offset never does.
- An Iggy consumer that polls that key with `Next` skips one record per partition, and only a naming rule stops it.
The gateway handlers and the update to that design doc come in a separate PR.
## What changed?
Group offsets had to come from a member of an Iggy consumer group, inside the partition's range, and they held back retention. A new consumer kind, `ExternalGroup` (wire value 3, `Consumer::external_group` in the Rust SDK), now stores offsets for a group that lives outside Iggy. The key is the numeric id of an existing Iggy consumer group, so the gateway still creates that group but never joins it. These offsets have their own key space, so they never share a key with the group's own offsets.
For this kind the server:
- accepts a store or a delete for any existing group, with no membership check
- stores the value as sent, with no range check and no clamp at boot
- refuses a poll with `FeatureUnavailable`
- leaves it out of the retention barrier
- deletes the offsets of a deleted group, through the same replicated delete as group offsets
The server replicates these offsets, recovers them at boot from `offsets/external_groups/`, carries them in partition state transfer, and clears them on a purge. Boot creates that directory for a partition that an older build made. The HTTP client
returns `FeatureUnavailable` for this kind, because the REST API cannot name a consumer kind. The CLI accepts `--kind external-group`.
Some changes elsewhere follow from the new kind:
- A caught-up primary now serves a state transfer of an empty partition with committed ops. A client can store and then delete an offset on a partition that never took a message. Before, a backup behind the repair floor had no way past those ops.
- A purge whose offset directory sync fails leaves its generation unrecorded, so the reconciler runs it again. Boot never clamps an external offset, so without this a crash can bring back a deleted one.
- A state transfer install syncs `offsets/` after the kind directories. The commit walk syncs it once per walk, not once per kind.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal commented on code in PR #4392:
URL: https://github.com/apache/iggy/pull/4392#discussion_r4186241452
##########
core/common/src/types/consumer/consumer_kind.rs:
##########
@@ -88,15 +91,55 @@ impl Consumer {
id,
}
}
+
+ /// Creates a new `ExternalGroup` from the `Identifier` of an Iggy consumer group.
Review Comment:
Fixed: the docs of `Consumer::external_group`, `ConsumerOffsetClient` and `MessageClient::poll_messages` name `FeatureUnavailable`.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -752,8 +747,9 @@ where
/// Set the per-kind durable consumer-offset limit for this partition.
pub fn set_consumer_offsets_max(&mut self, offsets_max: usize) {
- self.consumer_offset_capacity.set_limit(offsets_max);
- self.consumer_group_offset_capacity.set_limit(offsets_max);
+ for capacity in &self.consumer_offset_capacities {
+ capacity.set_limit(offsets_max);
Review Comment:
Fixed: the `consumer_offsets_max` comment in `config.toml` counts external groups as a third kind.
##########
core/integration/tests/server/cg.rs:
##########
@@ -81,6 +81,14 @@ async fn offset_cleanup(harness: &TestHarness) {
consumer_group_offset_cleanup_scenario::run(harness).await;
}
+#[iggy_harness(
+ test_client_transport = [Tcp, WebSocket, Quic],
Review Comment:
Fixed: a unit test covers all four HTTP client methods, and a CLI test sets an offset with `--kind external-group` and reads it back.
##########
core/partitions/src/partition_storage.rs:
##########
@@ -798,7 +838,11 @@ async fn load_partition_consumer_group_offsets<S: DurableStorage>(
Ok(RecoveredOffsets::default())
}
Err(source) => Err(PartitionRecoveryError::ConsumerOffsetsLoad {
- consumer_kind: "consumer group",
+ consumer_kind: if kind == ConsumerKind::ExternalGroup {
Review Comment:
Fixed: the error labels use `kind.as_str()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #4392:
URL: https://github.com/apache/iggy/pull/4392#issuecomment-5966290926
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/4392?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `92.92543%` with `74 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 87.90%. Comparing base ([`2d7fddb`](https://app.codecov.io/gh/apache/iggy/commit/2d7fddba5552640518aa7f1372f69af9c77e436a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a53cdc6`](https://app.codecov.io/gh/apache/iggy/commit/a53cdc69af406d0f77fb4f375001a3eb7c8614bb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/4392?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [core/partitions/src/iggy\_partition.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Figgy_partition.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9pZ2d5X3BhcnRpdGlvbi5ycw==) | 95.14% | [23 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/partitions/src/partition\_storage.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Fpartition_storage.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9wYXJ0aXRpb25fc3RvcmFnZS5ycw==) | 78.44% | [23 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/partitions/src/state\_transfer.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Fstate_transfer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9zdGF0ZV90cmFuc2Zlci5ycw==) | 85.71% | [10 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...nds/binary\_consumer\_offsets/get\_consumer\_offset.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fcli%2Fsrc%2Fcommands%2Fbinary_consumer_offsets%2Fget_consumer_offset.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jbGkvc3JjL2NvbW1hbmRzL2JpbmFyeV9jb25zdW1lcl9vZmZzZXRzL2dldF9jb25zdW1lcl9vZmZzZXQucnM=) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/sdk/src/http/consumer\_offsets.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fsdk%2Fsrc%2Fhttp%2Fconsumer_offsets.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZGsvc3JjL2h0dHAvY29uc3VtZXJfb2Zmc2V0cy5ycw==) | 83.33% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/server/src/dispatch/partition.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fdispatch%2Fpartition.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL2Rpc3BhdGNoL3BhcnRpdGlvbi5ycw==) | 96.66% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [core/server/src/namespace.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fnamespace.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL25hbWVzcGFjZS5ycw==) | 85.71% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #4392 +/- ##
============================================
- Coverage 87.96% 87.90% -0.06%
Complexity 1579 1579
============================================
Files 1290 1290
Lines 230145 230276 +131
Branches 193480 193612 +132
============================================
- Hits 202440 202429 -11
- Misses 22980 23092 +112
- Partials 4725 4755 +30
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/4392/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/4392/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `89.00% <92.92%> (-0.07%)` | :arrow_down: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/4392/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `68.74% <ø> (ø)` | |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/4392/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.64% <ø> (+0.02%)` | :arrow_up: |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/4392/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `91.24% <ø> (ø)` | |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/4392/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.67% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/4392/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `96.57% <ø> (-0.03%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/4392/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.26% <ø> (+0.07%)` | :arrow_up: |
| [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/4392?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core/binary\_protocol/src/primitives/consumer.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fprimitives%2Fconsumer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL3ByaW1pdGl2ZXMvY29uc3VtZXIucnM=) | `100.00% <100.00%> (ø)` | |
| [core/common/src/types/consumer/consumer\_kind.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fcommon%2Fsrc%2Ftypes%2Fconsumer%2Fconsumer_kind.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb21tb24vc3JjL3R5cGVzL2NvbnN1bWVyL2NvbnN1bWVyX2tpbmQucnM=) | `59.67% <100.00%> (+12.80%)` | :arrow_up: |
| [core/partitions/src/consumer\_offset\_capacity.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Fconsumer_offset_capacity.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9jb25zdW1lcl9vZmZzZXRfY2FwYWNpdHkucnM=) | `99.00% <100.00%> (+1.59%)` | :arrow_up: |
| [core/partitions/src/iggy\_partitions.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Figgy_partitions.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9pZ2d5X3BhcnRpdGlvbnMucnM=) | `87.82% <100.00%> (+0.25%)` | :arrow_up: |
| [core/partitions/src/lib.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9saWIucnM=) | `0.00% <ø> (ø)` | |
| [core/partitions/src/offset\_recovery.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Foffset_recovery.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9vZmZzZXRfcmVjb3ZlcnkucnM=) | `73.85% <100.00%> (+0.74%)` | :arrow_up: |
| [core/partitions/src/persistence.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Fpersistence.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy9wZXJzaXN0ZW5jZS5ycw==) | `92.05% <100.00%> (ø)` | |
| [core/partitions/src/types.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fpartitions%2Fsrc%2Ftypes.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9wYXJ0aXRpb25zL3NyYy90eXBlcy5ycw==) | `83.09% <100.00%> (+1.14%)` | :arrow_up: |
| [core/sdk/src/http/messages.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fsdk%2Fsrc%2Fhttp%2Fmessages.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZGsvc3JjL2h0dHAvbWVzc2FnZXMucnM=) | `92.55% <ø> (ø)` | |
| [core/server/src/consumer\_group.rs](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fconsumer_group.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL2NvbnN1bWVyX2dyb3VwLnJz) | `98.34% <100.00%> (+<0.01%)` | :arrow_up: |
| ... and [16 more](https://app.codecov.io/gh/apache/iggy/pull/4392?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [42 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/4392/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal commented on code in PR #4392:
URL: https://github.com/apache/iggy/pull/4392#discussion_r4188047947
##########
core/partitions/src/state_transfer.rs:
##########
@@ -64,7 +64,11 @@ use std::sync::atomic::Ordering;
/// Current state-transfer offsets format, including the prepare-chain anchor.
pub(crate) const CONSUMER_OFFSETS_MAGIC: [u8; 4] = *b"ICO1";
-pub(crate) const CONSUMER_OFFSETS_VERSION: u8 = 1;
+/// Version 2: the external group count and section. `server-0.9.0` sends
+/// version 1, which has neither, so an artifact exchanged with a peer on that
+/// release fails with `UnsupportedVersion` instead of decoding 4 bytes out of
+/// place.
+pub(crate) const CONSUMER_OFFSETS_VERSION: u8 = 2;
Review Comment:
Fixed: decode accepts versions 1 and 2, and encode writes version 1 when there is no external offset. A test checks the exact `server-0.9.0` bytes in both directions.
##########
core/server/src/namespace.rs:
##########
@@ -251,11 +253,17 @@ where
S: 'static,
SB: SuperblockStore + 'static,
{
- if consumer.kind != KIND_CONSUMER_GROUP {
- return Ok(());
+ let streams = shard.plane.metadata().mux_stm.streams();
+ match consumer.kind {
+ KIND_CONSUMER_GROUP => {}
+ KIND_EXTERNAL_GROUP => {
Review Comment:
Partly: added the upgrade note to `core/server/README.md`. A cluster-wide gate needs each replica's release, and no header carries it yet.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2092,17 +2146,14 @@ where
.retain(|start_offset, _| live.contains_key(start_offset));
}
- // An empty chain at frontier 0 tells the receiver to unlink its own, so
- // serve it only when a recorded purge says the emptiness is the truth.
- // `install_state_transfer`'s `purge_advances` check re-decides that
- // against the metadata plane and refuses the rest.
+ // An empty chain at frontier 0 is still served. The `NothingCommitted`
Review Comment:
Fixed with both guards: hubcio's gate stays, and the install also refuses an offer that holds nothing while the receiver holds committed offsets, with the test you asked for.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8022,38 +7981,42 @@ where
}
// Directory fsync so those unlinks stick, mirroring the install path: a
// crash right after the purge otherwise resurrects the offset files at
- // boot, and while recovery clamps a resurrected offset down to the
- // rebuilt head, "consumed through 0" is not the intended "no entry at
- // all" -- that consumer skips the first post-purge message. Logged on
- // failure, sharper than the partition-dir fsync above: the generation
- // write below still runs, so a crash would resurrect these files with
- // no boot re-purge left to clear them.
- for dir in self
- .consumer_offsets_path
- .clone()
- .into_iter()
- .chain(self.consumer_group_offsets_path.clone())
- {
- if let Err(error) = storage.sync_directory(Path::new(&dir)).await {
- warn!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- namespace_raw = namespace.inner(),
- generation,
- dir = %dir,
- %error,
- "purge could not fsync an offsets dir; a crash may resurrect \
- deleted offset files with the purge already recorded"
- );
+ // boot. Recovery clamps a resurrected Iggy offset down to the rebuilt
+ // head, but "consumed through 0" is not the intended "no entry at all"
+ // -- that consumer skips the first post-purge message -- and boot never
+ // clamps an external group offset, so one of 1000 skips 1000 messages.
+ // A failed sync therefore leaves the generation unrecorded below, and
+ // the reconciler re-issues the purge.
+ let mut offset_dirs_synced = true;
+ for dir in self.consumer_offset_dirs.iter().flatten() {
+ match storage.sync_directory(Path::new(dir)).await {
+ Ok(()) => {}
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
+ // Recovery accepts a missing directory, and a directory
+ // that is gone holds no unlink to make durable.
+ }
+ Err(error) => {
Review Comment:
Fixed: a dir sync failure other than `NotFound` now returns `OffsetsNotDurable`, and the router fences the partition.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8099,6 +8062,11 @@ where
.len();
self.evict_committed_prefix(fenced_prefix).await;
+ if !offset_dirs_synced {
Review Comment:
Fixed in round 2. Now only a failed `purge.gen` write is retried, and the retry redoes only that write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #4392:
URL: https://github.com/apache/iggy/pull/4392#discussion_r4180852838
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8022,38 +7981,35 @@ where
}
// Directory fsync so those unlinks stick, mirroring the install path: a
// crash right after the purge otherwise resurrects the offset files at
- // boot, and while recovery clamps a resurrected offset down to the
- // rebuilt head, "consumed through 0" is not the intended "no entry at
- // all" -- that consumer skips the first post-purge message. Logged on
- // failure, sharper than the partition-dir fsync above: the generation
- // write below still runs, so a crash would resurrect these files with
- // no boot re-purge left to clear them.
- for dir in self
- .consumer_offsets_path
- .clone()
- .into_iter()
- .chain(self.consumer_group_offsets_path.clone())
- {
- if let Err(error) = storage.sync_directory(Path::new(&dir)).await {
+ // boot. Recovery clamps a resurrected Iggy offset down to the rebuilt
+ // head, but "consumed through 0" is not the intended "no entry at all"
+ // -- that consumer skips the first post-purge message -- and boot never
+ // clamps an external group offset, so one of 1000 skips 1000 messages.
+ // A failed sync therefore leaves the generation unrecorded below, and
+ // the reconciler re-issues the purge.
+ let mut offset_dirs_synced = true;
+ for dir in self.consumer_offset_dirs.iter().flatten() {
+ if let Err(error) = storage.sync_directory(Path::new(dir)).await {
Review Comment:
warning: When the offsets directory is already gone, `sync_directory` returns `NotFound`, which the purge counts as a failed sync, so every re-issued purge fails and the replica withholds `PrepareOk` for that partition. Tolerate `ErrorKind::NotFound` here, as `flush_consumer_offset_directories_for` does.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -403,9 +409,12 @@ impl ConsumerOffsetsWire {
let next_offset = cursor.u64()?;
let consumer_count = cursor.u32()?;
let group_count = cursor.u32()?;
+ let external_group_count = cursor.u32()?;
Review Comment:
warning: `decode` reads a fourth count field that an older build's artifact does not carry, but `CONSUMER_OFFSETS_VERSION` stays 1, so the version guard accepts both layouts and a mixed-version peer misparses the artifact. Bump the version to 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal commented on code in PR #4392:
URL: https://github.com/apache/iggy/pull/4392#discussion_r4186240545
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -1106,14 +1095,8 @@ where
}
}
}
- let mut directories = Vec::with_capacity(4);
- for directory in [
- &self.consumer_offsets_path,
- &self.consumer_group_offsets_path,
- ]
- .into_iter()
- .flatten()
- {
+ let mut directories = Vec::with_capacity(ConsumerKind::COUNT + 2);
+ for directory in self.consumer_offset_dirs.iter().flatten() {
Review Comment:
Not fixed, on purpose: a delete leaves the dirty set, so skipping the directory can lose an unsynced unlink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "ryerraguntla (via GitHub)" <gi...@apache.org>.
ryerraguntla commented on PR #4392:
URL: https://github.com/apache/iggy/pull/4392#issuecomment-5988312645
/skill team-review-slim
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal merged PR #4392:
URL: https://github.com/apache/iggy/pull/4392
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "hubcio (via GitHub)" <gi...@apache.org>.
hubcio commented on code in PR #4392:
URL: https://github.com/apache/iggy/pull/4392#discussion_r4184303283
##########
core/sdk/src/http/consumer_offsets.rs:
##########
@@ -101,3 +104,12 @@ impl ConsumerOffsetClient for HttpClient {
fn get_path(stream_id: &str, topic_id: &str) -> String {
format!("streams/{stream_id}/topics/{topic_id}/consumer-offsets")
}
+
+/// The REST API cannot name a consumer kind, so an external group would land on a plain
+/// consumer's offset.
+pub(crate) fn refuse_external_group(consumer: &Consumer) -> Result<(), IggyError> {
Review Comment:
warning: HTTP drops the consumer kind, so `ConsumerGroup` offset calls and polls also silently act as a plain consumer - the same reason this guard exists. refuse everything but `ConsumerKind::Consumer` here, or open a follow-up.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -885,25 +890,37 @@ where
let partition_id = wire.partition_id.unwrap_or(0);
let namespace =
resolve_partition_namespace(shard, &wire.stream_id, &wire.topic_id, Some(partition_id))?;
+ if wire.consumer.kind != KIND_CONSUMER_GROUP && wire.consumer.kind != KIND_EXTERNAL_GROUP {
+ let consumer = polling_consumer_from_wire(&wire.consumer, partition_id)?;
+ return Ok(Some((
+ namespace,
+ partition_id,
+ PartitionRead::ConsumerOffset { consumer },
+ )));
+ }
// A group offset is keyed by the group's monotonic id (any client may read
// it, member or not), the same key the write path is rewritten to. An
// unresolved group (e.g. deleted) has no offset, so the read reports None.
- let consumer = if wire.consumer.kind == KIND_CONSUMER_GROUP {
- let Some(group_id) = shard
- .plane
- .metadata()
- .mux_stm
- .streams()
- .resolve_consumer_group_id(&wire.stream_id, &wire.topic_id, &wire.consumer.id)
- else {
- return Ok(None);
- };
- #[allow(clippy::cast_possible_truncation)]
- PollingConsumer::ConsumerGroup(group_id as usize, partition_id as usize)
+ let Some(group_id) = shard
+ .plane
+ .metadata()
+ .mux_stm
+ .streams()
+ .resolve_consumer_group_id(&wire.stream_id, &wire.topic_id, &wire.consumer.id)
+ else {
+ return Ok(None);
+ };
+ #[allow(clippy::cast_possible_truncation)]
+ let read = if wire.consumer.kind == KIND_EXTERNAL_GROUP {
+ PartitionRead::ExternalGroupOffset {
Review Comment:
warning: the SDK sends this read to the metadata leader, which can be a lagging backup for this partition, so a kafka `OffsetFetch` right after `OffsetCommit` can return the old offset. route it to the primary or document the follower read.
##########
core/sdk/src/stream_builder/build/build_iggy_consumer.rs:
##########
@@ -58,6 +58,8 @@ pub(crate) async fn build_iggy_consumer(
let mut builder = match consumer_kind {
ConsumerKind::Consumer => client.consumer(consumer_name, stream, topic, partition)?,
ConsumerKind::ConsumerGroup => client.consumer_group(consumer_name, stream, topic)?,
+ // Holds offsets only, it cannot poll.
+ ConsumerKind::ExternalGroup => return Err(IggyError::InvalidConfiguration),
Review Comment:
nit: this rejects `ExternalGroup` only after the stream, topic or producer was already built, and returns a bare `InvalidConfiguration` with no log. reject it before any server call and log which field is wrong.
##########
core/integration/tests/server/cg.rs:
##########
@@ -81,6 +81,14 @@ async fn offset_cleanup(harness: &TestHarness) {
consumer_group_offset_cleanup_scenario::run(harness).await;
}
+#[iggy_harness(
+ test_client_transport = [Tcp, WebSocket, Quic],
Review Comment:
nit: `Http` is left out here, so nothing tests the client-side `refuse_external_group` guard or the CLI's `--kind external-group`. add a unit test for the guard and a CLI set-then-get case.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8099,6 +8062,11 @@ where
.len();
self.evict_committed_prefix(fenced_prefix).await;
+ if !offset_dirs_synced {
Review Comment:
warning: any sync error other than `NotFound` leaves the purge unrecorded, so the reconciler's retry wipes sends this replica applied since - peers keep them. when the chain is already reset, retry only the dir sync and the `purge.gen` write.
also at line 8099.
##########
core/common/src/types/consumer/consumer_kind.rs:
##########
@@ -88,15 +91,55 @@ impl Consumer {
id,
}
}
+
+ /// Creates a new `ExternalGroup` from the `Identifier` of an Iggy consumer group.
Review Comment:
nit: these docs say external groups are never polled but don't name the errors - HTTP calls and polls both return `FeatureUnavailable`. note the HTTP case here and on `ConsumerOffsetClient`, and the poll case on `MessageClient::poll_messages`.
##########
core/partitions/src/partition_storage.rs:
##########
@@ -798,7 +838,11 @@ async fn load_partition_consumer_group_offsets<S: DurableStorage>(
Ok(RecoveredOffsets::default())
}
Err(source) => Err(PartitionRecoveryError::ConsumerOffsetsLoad {
- consumer_kind: "consumer group",
+ consumer_kind: if kind == ConsumerKind::ExternalGroup {
Review Comment:
simplification: pick the label with `kind.as_str()` instead of this if/else, and use `ConsumerKind::ExternalGroup.as_str()` at line 304 - drops the duplicated literal. the message text becomes `consumer_group`, which no test or doc depends on.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2092,17 +2146,14 @@ where
.retain(|start_offset, _| live.contains_key(start_offset));
}
- // An empty chain at frontier 0 tells the receiver to unlink its own, so
- // serve it only when a recorded purge says the emptiness is the truth.
- // `install_state_transfer`'s `purge_advances` check re-decides that
- // against the metadata plane and refuses the rest.
+ // An empty chain at frontier 0 is still served. The `NothingCommitted`
Review Comment:
warning: without the old `NothingCommitted` check, a zero-byte replica whose floor moved through an offsets-only window serves an empty chain, and a receiver at frontier 0 installs it, claiming history it doesn't hold. keep refusing when nothing backs the floor.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -1106,14 +1095,8 @@ where
}
}
}
- let mut directories = Vec::with_capacity(4);
- for directory in [
- &self.consumer_offsets_path,
- &self.consumer_group_offsets_path,
- ]
- .into_iter()
- .flatten()
- {
+ let mut directories = Vec::with_capacity(ConsumerKind::COUNT + 2);
+ for directory in self.consumer_offset_dirs.iter().flatten() {
Review Comment:
nit: every WAL checkpoint now syncs a fifth directory, even with no external offsets. only multi-replica `Persisted` partitions pay it, so it's minor - skip `external_groups/` while it's empty, or leave it.
##########
core/simulator/src/storage/purge.rs:
##########
@@ -25,7 +25,12 @@
//! The two controls make that distinction observable. Without a purge, recovery
//! must preserve bookmark 2 and return messages 3 and 4. After a completed purge,
//! recovery must find no bookmarks and return all five messages, 0 through 4.
-//! Both controls cover individual consumers and groups under both offset policies.
+//! Both controls cover individual consumers, groups, and external groups under
+//! both offset policies. An external group is never polled, so its bookmark is
+//! read directly. Boot never clamps it, so a resurrected one would survive.
+//!
+//! A third test purges after one consumer or group directory is gone, a state
+//! that recovery accepts. The purge must still record its generation.
Review Comment:
nit: the new test only covers the `NotFound` case. nothing drives the other sync-failure arm, where the generation stays unrecorded and the purge gets retried - add a test that fails one offsets dir sync and checks the retry.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -752,8 +747,9 @@ where
/// Set the per-kind durable consumer-offset limit for this partition.
pub fn set_consumer_offsets_max(&mut self, offsets_max: usize) {
- self.consumer_offset_capacity.set_limit(offsets_max);
- self.consumer_group_offset_capacity.set_limit(offsets_max);
+ for capacity in &self.consumer_offset_capacities {
+ capacity.set_limit(offsets_max);
Review Comment:
nit: every kind gets its own `consumer_offsets_max` budget here, so external groups add a third, but the comment at `core/server/config.toml:983` still talks about two kinds. mention external groups there so operators size for three.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal commented on code in PR #4392:
URL: https://github.com/apache/iggy/pull/4392#discussion_r4188049504
##########
core/partitions/src/partition_storage.rs:
##########
@@ -323,53 +283,92 @@ pub async fn configure_consumer_offsets_with_storage<S: DurableStorage, B: Messa
{
let guard = consumer_group_offsets.pin();
for (group_id, offset) in recovered_groups.entries {
- let recovered_offset = offset.offset.load(Ordering::Relaxed);
- if recovered_offset > offset_space_ceiling {
- warn!(
- consumer_group_id = group_id.0,
- recovered_offset,
- current_offset,
- offset_space_ceiling,
- stream_id,
- topic_id,
- partition_id,
- "recovered consumer group offset ahead of partition data; clamping"
- );
- offset.offset.store(current_offset, Ordering::Relaxed);
- }
- let committed_offset = offset.offset.load(Ordering::Relaxed);
- partition.seed_recovered_consumer_offset(
+ seed_recovered_offset(
+ partition,
+ namespace,
ConsumerKind::ConsumerGroup,
- offset.consumer_id,
- committed_offset,
- recovered_offset,
+ &offset,
+ Some(offset_space_ceiling),
+ current_offset,
);
guard.insert(group_id, offset);
}
}
+ // A partition made before this kind existed has no directory for it. The checkpoint syncs
+ // every offset directory, so it has to exist before the first one.
+ storage
+ .create_directories(Path::new(&external_group_offsets_path))
+ .await
+ .map_err(|_| PartitionRecoveryError::ConsumerOffsetsLoad {
Review Comment:
Fixed: the new `CreateOffsetsDirectory` error keeps the io error and prints it.
##########
core/sdk/src/http/consumer_offsets.rs:
##########
@@ -101,3 +104,12 @@ impl ConsumerOffsetClient for HttpClient {
fn get_path(stream_id: &str, topic_id: &str) -> String {
format!("streams/{stream_id}/topics/{topic_id}/consumer-offsets")
}
+
+/// The REST API cannot name a consumer kind, so an external group would land on a plain
+/// consumer's offset.
+pub(crate) fn refuse_external_group(consumer: &Consumer) -> Result<(), IggyError> {
Review Comment:
Already fixed: the unit test from `838231e89` covers it.
##########
core/sdk/src/stream_builder/build/build_iggy_consumer.rs:
##########
@@ -58,6 +58,8 @@ pub(crate) async fn build_iggy_consumer(
let mut builder = match consumer_kind {
ConsumerKind::Consumer => client.consumer(consumer_name, stream, topic, partition)?,
ConsumerKind::ConsumerGroup => client.consumer_group(consumer_name, stream, topic)?,
+ // Holds offsets only, it cannot poll.
+ ConsumerKind::ExternalGroup => return Err(IggyError::InvalidConfiguration),
Review Comment:
Fixed: the stream builder returns `FeatureUnavailable`, like the server poll and the HTTP SDK.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -3301,38 +3264,39 @@ where
.collect()
}
- /// Snapshot dead group keys for deletion through the partition's VSR log.
- /// A local unlink could free the primary's quota while backups retained
- /// every older generation, so reclamation uses the same ordered delete as
- /// an explicit consumer-offset request.
+ /// Snapshot the committed offsets of every kind keyed by consumer group
+ /// whose group `is_live` rejects, ascending by id within each kind, for
+ /// deletion through the partition's VSR log. A local unlink could free the
+ /// primary's quota while backups retained every older generation, so
+ /// reclamation uses the same ordered delete as an explicit consumer-offset
+ /// request.
#[must_use]
- pub fn dead_consumer_group_offset_ids(&self, is_live: impl Fn(u64) -> bool) -> Vec<u32> {
+ pub fn dead_group_offset_keys(
+ &self,
+ is_live: impl Fn(u64) -> bool,
+ ) -> Vec<(ConsumerKind, u32)> {
if !self.consensus.is_primary() || !self.consensus.is_normal() {
return Vec::new();
}
- // A stranded file already failed normal loading or unlink. Reissuing
- // replicated deletes every reconciliation pass cannot make its
- // filesystem writable and would create a permanent commit loop.
- // A single-replica explicit deletion can retry after repair. On a
- // replicated partition the file must be repaired or removed locally,
- // because older peers do not recognize a delete for a map-missing key.
- let capacity = self.consumer_offset_capacity_for(ConsumerKind::ConsumerGroup);
let mut dead = Vec::new();
- for key in self.consumer_group_offsets.pin().keys() {
- let Ok(id) = u32::try_from(key.0) else {
- continue;
- };
- if !is_live(u64::from(id))
- && !capacity.is_stranded(id)
- && self
- .durable_consumer_offsets
- .contains(ConsumerKind::ConsumerGroup, id)
- {
- dead.push(id);
- }
+ for kind in [ConsumerKind::ConsumerGroup, ConsumerKind::ExternalGroup] {
Review Comment:
Fixed: dead keys are sorted by group id, with both kinds of one group side by side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "krishvishal (via GitHub)" <gi...@apache.org>.
krishvishal commented on code in PR #4392:
URL: https://github.com/apache/iggy/pull/4392#discussion_r4186239568
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8099,6 +8062,11 @@ where
.len();
self.evict_committed_prefix(fenced_prefix).await;
+ if !offset_dirs_synced {
Review Comment:
Fixed: a retry for a generation whose reset already ran redoes only the offsets dir sync and the `purge.gen` write, and an install or a converge clears the marker.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2092,17 +2146,14 @@ where
.retain(|start_offset, _| live.contains_key(start_offset));
}
- // An empty chain at frontier 0 tells the receiver to unlink its own, so
- // serve it only when a recorded purge says the emptiness is the truth.
- // `install_state_transfer`'s `purge_advances` check re-decides that
- // against the metadata plane and refuses the rest.
+ // An empty chain at frontier 0 is still served. The `NothingCommitted`
Review Comment:
Fixed: the refusal is back for an offer with no segments, `next_offset` 0 and purge generation 0.
##########
core/server/src/dispatch/partition.rs:
##########
@@ -885,25 +890,37 @@ where
let partition_id = wire.partition_id.unwrap_or(0);
let namespace =
resolve_partition_namespace(shard, &wire.stream_id, &wire.topic_id, Some(partition_id))?;
+ if wire.consumer.kind != KIND_CONSUMER_GROUP && wire.consumer.kind != KIND_EXTERNAL_GROUP {
+ let consumer = polling_consumer_from_wire(&wire.consumer, partition_id)?;
+ return Ok(Some((
+ namespace,
+ partition_id,
+ PartitionRead::ConsumerOffset { consumer },
+ )));
+ }
// A group offset is keyed by the group's monotonic id (any client may read
// it, member or not), the same key the write path is rewritten to. An
// unresolved group (e.g. deleted) has no offset, so the read reports None.
- let consumer = if wire.consumer.kind == KIND_CONSUMER_GROUP {
- let Some(group_id) = shard
- .plane
- .metadata()
- .mux_stm
- .streams()
- .resolve_consumer_group_id(&wire.stream_id, &wire.topic_id, &wire.consumer.id)
- else {
- return Ok(None);
- };
- #[allow(clippy::cast_possible_truncation)]
- PollingConsumer::ConsumerGroup(group_id as usize, partition_id as usize)
+ let Some(group_id) = shard
+ .plane
+ .metadata()
+ .mux_stm
+ .streams()
+ .resolve_consumer_group_id(&wire.stream_id, &wire.topic_id, &wire.consumer.id)
+ else {
+ return Ok(None);
+ };
+ #[allow(clippy::cast_possible_truncation)]
+ let read = if wire.consumer.kind == KIND_EXTERNAL_GROUP {
+ PartitionRead::ExternalGroupOffset {
Review Comment:
Docs only, your second option: `get_consumer_offset` documents the follower read, and routing to the primary is in a drafted follow-up issue.
##########
core/sdk/src/http/consumer_offsets.rs:
##########
@@ -101,3 +104,12 @@ impl ConsumerOffsetClient for HttpClient {
fn get_path(stream_id: &str, topic_id: &str) -> String {
format!("streams/{stream_id}/topics/{topic_id}/consumer-offsets")
}
+
+/// The REST API cannot name a consumer kind, so an external group would land on a plain
+/// consumer's offset.
+pub(crate) fn refuse_external_group(consumer: &Consumer) -> Result<(), IggyError> {
Review Comment:
Not fixed here: the follow-up issue is drafted but not filed (your second option), because refusing it changes behavior for current HTTP callers.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -1106,14 +1095,8 @@ where
}
}
}
- let mut directories = Vec::with_capacity(4);
- for directory in [
- &self.consumer_offsets_path,
- &self.consumer_group_offsets_path,
- ]
- .into_iter()
- .flatten()
- {
+ let mut directories = Vec::with_capacity(ConsumerKind::COUNT + 2);
+ for directory in self.consumer_offset_dirs.iter().flatten() {
Review Comment:
Not fixed, on purpose (your "or leave it"): a delete leaves the dirty set, so skipping the directory can lose an unsynced unlink.
##########
core/simulator/src/storage/purge.rs:
##########
@@ -25,7 +25,12 @@
//! The two controls make that distinction observable. Without a purge, recovery
//! must preserve bookmark 2 and return messages 3 and 4. After a completed purge,
//! recovery must find no bookmarks and return all five messages, 0 through 4.
-//! Both controls cover individual consumers and groups under both offset policies.
+//! Both controls cover individual consumers, groups, and external groups under
+//! both offset policies. An external group is never polled, so its bookmark is
+//! read directly. Boot never clamps it, so a resurrected one would survive.
+//!
+//! A third test purges after one consumer or group directory is gone, a state
+//! that recovery accepts. The purge must still record its generation.
Review Comment:
Fixed: a simulator test fails the external group dir sync across a power loss, and a partition test drives the retry through `purge()`.
##########
core/sdk/src/stream_builder/build/build_iggy_consumer.rs:
##########
@@ -58,6 +58,8 @@ pub(crate) async fn build_iggy_consumer(
let mut builder = match consumer_kind {
ConsumerKind::Consumer => client.consumer(consumer_name, stream, topic, partition)?,
ConsumerKind::ConsumerGroup => client.consumer_group(consumer_name, stream, topic)?,
+ // Holds offsets only, it cannot poll.
+ ConsumerKind::ExternalGroup => return Err(IggyError::InvalidConfiguration),
Review Comment:
Fixed: `IggyConsumerConfig::validate` logs the bad `consumer_kind` and runs first in all four builder entry points.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server): store offsets of groups managed outside Iggy [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #4392:
URL: https://github.com/apache/iggy/pull/4392#discussion_r4184460583
##########
core/partitions/src/state_transfer.rs:
##########
@@ -2092,17 +2146,14 @@ where
.retain(|start_offset, _| live.contains_key(start_offset));
}
- // An empty chain at frontier 0 tells the receiver to unlink its own, so
- // serve it only when a recorded purge says the emptiness is the truth.
- // `install_state_transfer`'s `purge_advances` check re-decides that
- // against the metadata plane and refuses the rest.
+ // An empty chain at frontier 0 is still served. The `NothingCommitted`
Review Comment:
this drops the `segments.is_empty() && next_offset == 0 && purge_generation == 0` gate from d35000d07, and the PR now makes offsets-only partitions possible. A primary that lost its dir with `commit_max > 0` serves an empty offer to a restarted replica (commit_min = commit_max = 0, so `StaleTransfer` is inert), `OfferRewindsDurableData` only keys on `local_next_offset > 0`, and the install sweep unlinks the receiver's committed external offsets.
Don't restore the blanket gate, extend the anti-rewind check to the durable offset table, with a test where the receiver holds only external offsets. The comment at :3398 still calls this shape the phantom the gate stops.
##########
core/partitions/src/state_transfer.rs:
##########
@@ -64,7 +64,11 @@ use std::sync::atomic::Ordering;
/// Current state-transfer offsets format, including the prepare-chain anchor.
pub(crate) const CONSUMER_OFFSETS_MAGIC: [u8; 4] = *b"ICO1";
-pub(crate) const CONSUMER_OFFSETS_VERSION: u8 = 1;
+/// Version 2: the external group count and section. `server-0.9.0` sends
+/// version 1, which has neither, so an artifact exchanged with a peer on that
+/// release fails with `UnsupportedVersion` instead of decoding 4 bytes out of
+/// place.
+pub(crate) const CONSUMER_OFFSETS_VERSION: u8 = 2;
Review Comment:
decode accepts only `== 2` and encode always writes v2, so every partition state transfer between a 0.9.0 node and a new one fails with `UnsupportedVersion` in both directions, even with zero external offsets. That breaks the rolling upgrade `core/server/README.md` documents.
Accept `1 | 2` on decode (v1 = empty external section), write v1 when the external section is empty, and add a v1 round-trip test.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8022,38 +7981,42 @@ where
}
// Directory fsync so those unlinks stick, mirroring the install path: a
// crash right after the purge otherwise resurrects the offset files at
- // boot, and while recovery clamps a resurrected offset down to the
- // rebuilt head, "consumed through 0" is not the intended "no entry at
- // all" -- that consumer skips the first post-purge message. Logged on
- // failure, sharper than the partition-dir fsync above: the generation
- // write below still runs, so a crash would resurrect these files with
- // no boot re-purge left to clear them.
- for dir in self
- .consumer_offsets_path
- .clone()
- .into_iter()
- .chain(self.consumer_group_offsets_path.clone())
- {
- if let Err(error) = storage.sync_directory(Path::new(&dir)).await {
- warn!(
- target: "iggy.partitions.diag",
- plane = "partitions",
- namespace_raw = namespace.inner(),
- generation,
- dir = %dir,
- %error,
- "purge could not fsync an offsets dir; a crash may resurrect \
- deleted offset files with the purge already recorded"
- );
+ // boot. Recovery clamps a resurrected Iggy offset down to the rebuilt
+ // head, but "consumed through 0" is not the intended "no entry at all"
+ // -- that consumer skips the first post-purge message -- and boot never
+ // clamps an external group offset, so one of 1000 skips 1000 messages.
+ // A failed sync therefore leaves the generation unrecorded below, and
+ // the reconciler re-issues the purge.
+ let mut offset_dirs_synced = true;
+ for dir in self.consumer_offset_dirs.iter().flatten() {
+ match storage.sync_directory(Path::new(dir)).await {
+ Ok(()) => {}
+ Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
+ // Recovery accepts a missing directory, and a directory
+ // that is gone holds no unlink to make durable.
+ }
+ Err(error) => {
Review Comment:
`sync_directory` opens a fresh handle each time, so after a writeback EIO a later fsync on the retried purge can succeed without the earlier unlinks being durable (the comment at state_transfer.rs:3344 warns about exactly this). `purge.gen` then gets recorded, and since boot never clamps external offsets a crash can resurrect a deleted one.
On a non-NotFound error, fence the partition like `Unserviceable` instead of relying on the retry.
##########
core/server/src/namespace.rs:
##########
@@ -251,11 +253,17 @@ where
S: 'static,
SB: SuperblockStore + 'static,
{
- if consumer.kind != KIND_CONSUMER_GROUP {
- return Ok(());
+ let streams = shard.plane.metadata().mux_stm.streams();
+ match consumer.kind {
+ KIND_CONSUMER_GROUP => {}
+ KIND_EXTERNAL_GROUP => {
Review Comment:
a new primary admits kind 3 here, but a 0.9.0 backup fails `WireConsumer::decode` on it before `journal_append`. No PrepareOk, repair fails the same way, so with 2 of 3 replicas still old the partition stalls.
At minimum add an upgrade note to the README (upgrade the whole cluster before any `ExternalGroup` store). A cluster-wide admission gate here would be the real fix.
##########
core/sdk/src/http/consumer_offsets.rs:
##########
@@ -101,3 +104,12 @@ impl ConsumerOffsetClient for HttpClient {
fn get_path(stream_id: &str, topic_id: &str) -> String {
format!("streams/{stream_id}/topics/{topic_id}/consumer-offsets")
}
+
+/// The REST API cannot name a consumer kind, so an external group would land on a plain
+/// consumer's offset.
+pub(crate) fn refuse_external_group(consumer: &Consumer) -> Result<(), IggyError> {
Review Comment:
nit: nothing tests this. `Consumer::kind` is `#[serde(skip)]` on the REST path, so if the guard goes, an external offset silently lands on a plain consumer key. add a unit test.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -3301,38 +3264,39 @@ where
.collect()
}
- /// Snapshot dead group keys for deletion through the partition's VSR log.
- /// A local unlink could free the primary's quota while backups retained
- /// every older generation, so reclamation uses the same ordered delete as
- /// an explicit consumer-offset request.
+ /// Snapshot the committed offsets of every kind keyed by consumer group
+ /// whose group `is_live` rejects, ascending by id within each kind, for
+ /// deletion through the partition's VSR log. A local unlink could free the
+ /// primary's quota while backups retained every older generation, so
+ /// reclamation uses the same ordered delete as an explicit consumer-offset
+ /// request.
#[must_use]
- pub fn dead_consumer_group_offset_ids(&self, is_live: impl Fn(u64) -> bool) -> Vec<u32> {
+ pub fn dead_group_offset_keys(
+ &self,
+ is_live: impl Fn(u64) -> bool,
+ ) -> Vec<(ConsumerKind, u32)> {
if !self.consensus.is_primary() || !self.consensus.is_normal() {
return Vec::new();
}
- // A stranded file already failed normal loading or unlink. Reissuing
- // replicated deletes every reconciliation pass cannot make its
- // filesystem writable and would create a permanent commit loop.
- // A single-replica explicit deletion can retry after repair. On a
- // replicated partition the file must be repaired or removed locally,
- // because older peers do not recognize a delete for a map-missing key.
- let capacity = self.consumer_offset_capacity_for(ConsumerKind::ConsumerGroup);
let mut dead = Vec::new();
- for key in self.consumer_group_offsets.pin().keys() {
- let Ok(id) = u32::try_from(key.0) else {
- continue;
- };
- if !is_live(u64::from(id))
- && !capacity.is_stranded(id)
- && self
- .durable_consumer_offsets
- .contains(ConsumerKind::ConsumerGroup, id)
- {
- dead.push(id);
- }
+ for kind in [ConsumerKind::ConsumerGroup, ConsumerKind::ExternalGroup] {
Review Comment:
nit: consumer-group keys go first under the 32-per-pass cap, so a big backlog of dead CG keys delays external-group deletes by whole passes. interleave, or note it is intended.
##########
core/partitions/src/partition_storage.rs:
##########
@@ -323,53 +283,92 @@ pub async fn configure_consumer_offsets_with_storage<S: DurableStorage, B: Messa
{
let guard = consumer_group_offsets.pin();
for (group_id, offset) in recovered_groups.entries {
- let recovered_offset = offset.offset.load(Ordering::Relaxed);
- if recovered_offset > offset_space_ceiling {
- warn!(
- consumer_group_id = group_id.0,
- recovered_offset,
- current_offset,
- offset_space_ceiling,
- stream_id,
- topic_id,
- partition_id,
- "recovered consumer group offset ahead of partition data; clamping"
- );
- offset.offset.store(current_offset, Ordering::Relaxed);
- }
- let committed_offset = offset.offset.load(Ordering::Relaxed);
- partition.seed_recovered_consumer_offset(
+ seed_recovered_offset(
+ partition,
+ namespace,
ConsumerKind::ConsumerGroup,
- offset.consumer_id,
- committed_offset,
- recovered_offset,
+ &offset,
+ Some(offset_space_ceiling),
+ current_offset,
);
guard.insert(group_id, offset);
}
}
+ // A partition made before this kind existed has no directory for it. The checkpoint syncs
+ // every offset directory, so it has to exist before the first one.
+ storage
+ .create_directories(Path::new(&external_group_offsets_path))
+ .await
+ .map_err(|_| PartitionRecoveryError::ConsumerOffsetsLoad {
Review Comment:
nit: `map_err(|_| ...)` drops the io error, so a failed create on the first boot after upgrade says nothing about why. keep the source in the error.
##########
core/sdk/src/stream_builder/build/build_iggy_consumer.rs:
##########
@@ -58,6 +58,8 @@ pub(crate) async fn build_iggy_consumer(
let mut builder = match consumer_kind {
ConsumerKind::Consumer => client.consumer(consumer_name, stream, topic, partition)?,
ConsumerKind::ConsumerGroup => client.consumer_group(consumer_name, stream, topic)?,
+ // Holds offsets only, it cannot poll.
+ ConsumerKind::ExternalGroup => return Err(IggyError::InvalidConfiguration),
Review Comment:
nit: the same misuse returns `InvalidConfiguration` here but `FeatureUnavailable` from the server poll (dispatch/partition.rs:815) and the HTTP SDK, and `FeatureUnavailable` now also means "wrong kind". pick one code for it.
##########
core/partitions/src/iggy_partition.rs:
##########
@@ -8099,6 +8062,11 @@ where
.len();
self.evict_committed_prefix(fenced_prefix).await;
+ if !offset_dirs_synced {
Review Comment:
the re-issued purge wipes everything this replica committed since the first attempt (messages, external stores). `purge_deferred` only gates the PrepareOk send, local commits still apply, and peers keep those ops, so counters and the next `base_offset` diverge here. Under persistent EIO every reconciler pass wipes again with no backoff.
Retry only the dir syncs plus the `purge.gen` write (remember the wiped generation in memory), or fence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org