You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "krishvishal (via GitHub)" <gi...@apache.org> on 2026/04/21 23:28:44 UTC

[PR] feat(shard): add v2 consumer-offset commands with AckLevel [iggy]

krishvishal opened a new pull request, #3151:
URL: https://github.com/apache/iggy/pull/3151

   ## Summary
   - Introduce `AckLevel { NoAck, Quorum }` and v2 wire commands `StoreConsumerOffset2` / `DeleteConsumerOffset2` that carry an explicit ack byte; v1 commands and wire format are untouched.
   - Unify the server's two offset-write paths so `PollMessages` auto-commit and explicit stores share a single shard entry point, with `AckLevel` selecting the policy.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(shard): add v2 consumer-offset commands with AckLevel [iggy]

Posted by "atharvalade (via GitHub)" <gi...@apache.org>.
atharvalade commented on code in PR #3151:
URL: https://github.com/apache/iggy/pull/3151#discussion_r3139490421


##########
core/simulator/src/client.rs:
##########
@@ -181,6 +181,44 @@ impl SimClient {
         self.build_request_with_namespace(Operation::DeleteConsumerOffset, &payload, namespace)
     }
 
+    /// v2 of `store_consumer_offset` carrying an explicit `AckLevel` byte.
+    ///
+    /// Only the simulator emits this opcode today; the partitions plane
+    /// accepts it alongside v1. The ack byte is reserved for future
+    /// cluster-side commit-timing semantics.
+    pub fn store_consumer_offset_v2(
+        &self,
+        namespace: IggyNamespace,
+        consumer_kind: u8,
+        consumer_id: u32,
+        offset: u64,
+        ack: AckLevel,
+    ) -> Message<RequestHeader> {
+        let mut payload = Vec::with_capacity(14);
+        payload.push(consumer_kind);
+        payload.extend_from_slice(&consumer_id.to_le_bytes());
+        payload.extend_from_slice(&offset.to_le_bytes());
+        payload.push(ack.as_u8());
+
+        self.build_request_with_namespace(Operation::StoreConsumerOffset2, &payload, namespace)
+    }
+
+    /// v2 of `delete_consumer_offset` carrying an explicit `AckLevel` byte.
+    pub fn delete_consumer_offset_v2(
+        &self,
+        namespace: IggyNamespace,
+        consumer_kind: u8,
+        consumer_id: u32,
+        ack: AckLevel,
+    ) -> Message<RequestHeader> {
+        let mut payload = Vec::with_capacity(6);
+        payload.push(consumer_kind);
+        payload.extend_from_slice(&consumer_id.to_le_bytes());
+        payload.push(ack.as_u8());
+
+        self.build_request_with_namespace(Operation::DeleteConsumerOffset2, &payload, namespace)
+    }

Review Comment:
   `store_consumer_offset_v2` and `delete_consumer_offset_v2` are defined here but never called anywhere in the codebase. Similarly, `StoreConsumerOffset2Request` / `DeleteConsumerOffset2Request` in binary_protocol are only exercised in their own unit tests.
   
   This means the v2 path has zero end-to-end coverage. The wire types have roundtrip tests which is great, but there's nothing proving a v2 message built by the simulator actually flows through the consensus pipeline and gets parsed correctly by `parse_consumer_offset_payload`. I think at minimum there should be a simulator-level test (or even just a unit test in `iggy_partition.rs`) that builds a v2 store/delete message and runs it through the partition parser to confirm the bytes land in the right places. Otherwise this is a lot of scaffolding with no integration confidence.



##########
core/partitions/src/iggy_partition.rs:
##########
@@ -1421,7 +1430,9 @@ where
                         })?;
                 Ok((kind, consumer_id, Some(offset)))
             }
-            Operation::DeleteConsumerOffset => Ok((kind, consumer_id, None)),
+            Operation::DeleteConsumerOffset | Operation::DeleteConsumerOffset2 => {
+                Ok((kind, consumer_id, None))
+            }
             _ => Err(IggyError::InvalidCommand),
         }
     }

Review Comment:
   This is the only place that actually processes v2 commands (both `parse_consumer_offset_request` and `parse_staged_consumer_offset_commit` delegate here), and it handles `StoreConsumerOffset2` in the same branch as v1 without ever reading the trailing ack byte. For store, parsing stops at `body[5..13]` (byte 12). For delete, it stops at `body[1..5]` (byte 4). The ack byte just falls off the end.
   
   Two concerns here. First, `AckLevel` has literally no effect on behavior right now, but the PR description says it "selects the policy." I think the description should be explicit that the ack byte is wire-reserved only and behavioral branching comes later. Otherwise future contributors might assume this already works.
   
   Second, and more importantly from a protocol correctness angle: an invalid ack value (e.g. `0xFF`) would be silently accepted with no validation error. Wouldn't it be better to at least validate the byte here, even if you don't use it yet? Something like reading the ack byte and calling `AckLevel::from_code()` to reject garbage early. That way you get fail-fast behavior for malformed messages instead of silently swallowing bad data through the pipeline.



##########
core/integration/tests/server/scenarios/authentication_scenario.rs:
##########
@@ -139,6 +139,16 @@ async fn test_all_commands_require_auth(client: &IggyClient) {
         ) {
             continue;
         }
+        // v2 consumer-offset ops are registered in the dispatch table for the
+        // consensus/simulator pathway but are not wired into the legacy binary
+        // server's dispatch. They'll move into server-ng alongside the rest of
+        // the v2 surface; re-enable these codes here once that lands.
+        if matches!(
+            code,
+            STORE_CONSUMER_OFFSET_2_CODE | DELETE_CONSUMER_OFFSET_2_CODE
+        ) {
+            continue;
+        }

Review Comment:
   The comment says "re-enable these codes here once that lands" but there's no issue/ticket reference to track it. In my experience these `continue` skips tend to get forgotten once the PR is merged. Would it be worth adding a `// TODO(#XXXX)` with an issue number so this shows up in tracking? Or even a `compile_error!` behind a feature flag that fires when server-ng v2 dispatch lands, forcing someone to come back here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(shard): add v2 consumer-offset commands with AckLevel [iggy]

Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #3151:
URL: https://github.com/apache/iggy/pull/3151#discussion_r3122358256


##########
core/common/src/traits/consumer_offset_client.rs:
##########
@@ -53,4 +54,34 @@ pub trait ConsumerOffsetClient {
         topic_id: &Identifier,
         partition_id: Option<u32>,
     ) -> Result<(), IggyError>;
+

Review Comment:
   We don't want those methods to be part of the SDK traits, as they are not supported by the current `server`, instead it has to be stashed together similarly to new `SendMessages` command, where it exists as a new command in the `binary_protocol` crate, but the only user of it is the `simulator` client.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(shard): add v2 consumer-offset commands with AckLevel [iggy]

Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on PR #3151:
URL: https://github.com/apache/iggy/pull/3151#issuecomment-4294491677

   This command suppose to be implemented only in the components that are used by the cluster which is in development, not by our core server. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(shard): add v2 consumer-offset commands with AckLevel [iggy]

Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex merged PR #3151:
URL: https://github.com/apache/iggy/pull/3151


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(shard): add v2 consumer-offset commands with AckLevel [iggy]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3151:
URL: https://github.com/apache/iggy/pull/3151#issuecomment-4292530734

   ## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/3151?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 72.16%. Comparing base ([`54fe2bb`](https://app.codecov.io/gh/apache/iggy/commit/54fe2bb9f22b455bc29d25cc746c3d2a52469a97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6291d5f`](https://app.codecov.io/gh/apache/iggy/commit/6291d5f061264e51b1abc1717353d36ad9c781a1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3151      +/-   ##
   ============================================
   - Coverage     73.29%   72.16%   -1.13%     
     Complexity      943      943              
   ============================================
     Files          1126     1063      -63     
     Lines         98435    93521    -4914     
     Branches      75608    75415     -193     
   ============================================
   - Hits          72148    67492    -4656     
   + Misses        23683    23425     -258     
     Partials       2604     2604              
   ```
   
   | [Components](https://app.codecov.io/gh/apache/iggy/pull/3151/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/3151/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `74.16% <ø> (-0.03%)` | :arrow_down: |
   | [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/3151/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.30% <ø> (ø)` | |
   | [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/3151/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.44% <ø> (ø)` | |
   | [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/3151/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.43% <ø> (ø)` | |
   | [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/3151/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.29% <ø> (-4.15%)` | :arrow_down: |
   | [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/3151/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `39.41% <ø> (ø)` | |
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/3151?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [core/binary\_protocol/src/codes.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fcodes.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL2NvZGVzLnJz) | `100.00% <ø> (ø)` | |
   | [core/binary\_protocol/src/consensus/operation.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fconsensus%2Foperation.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL2NvbnNlbnN1cy9vcGVyYXRpb24ucnM=) | `95.91% <ø> (-0.05%)` | :arrow_down: |
   | [core/binary\_protocol/src/dispatch.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fbinary_protocol%2Fsrc%2Fdispatch.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9iaW5hcnlfcHJvdG9jb2wvc3JjL2Rpc3BhdGNoLnJz) | `89.52% <ø> (-1.63%)` | :arrow_down: |
   | [...common/src/traits/binary\_impls/consumer\_offsets.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fcommon%2Fsrc%2Ftraits%2Fbinary_impls%2Fconsumer_offsets.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb21tb24vc3JjL3RyYWl0cy9iaW5hcnlfaW1wbHMvY29uc3VtZXJfb2Zmc2V0cy5ycw==) | `100.00% <ø> (ø)` | |
   | [core/consensus/src/observability.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fconsensus%2Fsrc%2Fobservability.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25zZW5zdXMvc3JjL29ic2VydmFiaWxpdHkucnM=) | `40.00% <ø> (ø)` | |
   | [...c/client\_wrappers/binary\_consumer\_offset\_client.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fsdk%2Fsrc%2Fclient_wrappers%2Fbinary_consumer_offset_client.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZGsvc3JjL2NsaWVudF93cmFwcGVycy9iaW5hcnlfY29uc3VtZXJfb2Zmc2V0X2NsaWVudC5ycw==) | `100.00% <ø> (ø)` | |
   | [core/sdk/src/clients/binary\_consumer\_offset.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fsdk%2Fsrc%2Fclients%2Fbinary_consumer_offset.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZGsvc3JjL2NsaWVudHMvYmluYXJ5X2NvbnN1bWVyX29mZnNldC5ycw==) | `100.00% <ø> (ø)` | |
   | [core/sdk/src/http/consumer\_offsets.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fsdk%2Fsrc%2Fhttp%2Fconsumer_offsets.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZGsvc3JjL2h0dHAvY29uc3VtZXJfb2Zmc2V0cy5ycw==) | `100.00% <ø> (ø)` | |
   | [core/server/src/binary/dispatch.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fbinary%2Fdispatch.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL2JpbmFyeS9kaXNwYXRjaC5ycw==) | `78.90% <ø> (ø)` | |
   | [...consumer\_offsets/delete\_consumer\_offset\_handler.rs](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree&filepath=core%2Fserver%2Fsrc%2Fbinary%2Fhandlers%2Fconsumer_offsets%2Fdelete_consumer_offset_handler.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zZXJ2ZXIvc3JjL2JpbmFyeS9oYW5kbGVycy9jb25zdW1lcl9vZmZzZXRzL2RlbGV0ZV9jb25zdW1lcl9vZmZzZXRfaGFuZGxlci5ycw==) | `65.21% <ø> (-1.45%)` | :arrow_down: |
   | ... and [5 more](https://app.codecov.io/gh/apache/iggy/pull/3151?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
   
   ... and [81 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/3151/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(shard): add v2 consumer-offset commands with AckLevel [iggy]

Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #3151:
URL: https://github.com/apache/iggy/pull/3151#discussion_r3146125857


##########
core/partitions/src/iggy_partition.rs:
##########
@@ -1421,7 +1430,9 @@ where
                         })?;
                 Ok((kind, consumer_id, Some(offset)))
             }
-            Operation::DeleteConsumerOffset => Ok((kind, consumer_id, None)),
+            Operation::DeleteConsumerOffset | Operation::DeleteConsumerOffset2 => {
+                Ok((kind, consumer_id, None))
+            }
             _ => Err(IggyError::InvalidCommand),
         }
     }

Review Comment:
   Yeah, we need to make sure we use the `AckLevel` as that is the main difference between the old request and new one. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org