You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "krishvishal (via GitHub)" <gi...@apache.org> on 2026/06/23 09:20:10 UTC
[PR] feat(server-ng): commit-time metadata validation with result codes [iggy]
krishvishal opened a new pull request, #3553:
URL: https://github.com/apache/iggy/pull/3553
## Why
`server-ng` previously validated metadata business rules during **preflight**: primary-only, before replication. Invalid requests were then silently dropped.
This caused two issues:
**1. VSR correctness.** A rejection must be a deterministic function of the committed log. It should be computed during apply on every replica and recorded so that all replicas agree. Preflight validation uses a primary-local snapshot, and its verdict is never replicated. After a view change, the new primary is not guaranteed to reach the same decision.
**2. Infinite retry.** The in-process path mapped the preflight `Err` to `Canceled`, leaving the home shard silent. As a result, the SDK kept retrying permanently invalid requests forever, such as `CreatePartitions` on a missing topic.
## What Changed
**Result taxonomy.** Added `core/metadata/src/stm/result.rs`. There is now one closed `#[repr(u32)]` enum per operation: `Ok = 0`, with other discriminants reused from the existing `IggyError` code space. This keeps SDK error mapping compatible with existing codes.
**Apply now returns committed results.** `StateHandler::apply` now returns `ApplyReply { code, body }` instead of `Bytes`. Every previous silent `Bytes::new()` no-op is now represented as a committed result code. `MuxStateMachine::update` now reserves `Err` for decode/corruption failures only.
**Result is encoded in the reply body.** The result rides in the reply body. The reply body now starts with a sparse result section, `[count][index, result]*`, followed by the payload. This is written in place via `build_reply_message_with`, using one allocation and one payload copy. There is no `ReplyHeader` layout change.
**Eviction instead of silent drop.** Structurally invalid requests are now evicted rather than silently dropped: `not-client-allowed` maps to `InvalidRequestOperation`, while undecodable/overflow maps to `InvalidRequestBody`.
**Removed preflight partition-count read.** Deleted the `current_partition_count` preflight read. Parent existence is now checked at commit time and returned as `CreatePartitionsResult::{Stream, Topic}NotFound`, removing the TOCTOU window.
**Consensus frame size floor.** Every consensus frame’s `size` must now span its header. This is asserted during `consensus_message` construction, `PrepareOk` projection, and in the simulator. This prevents reply-body slicing from underflowing.
**Simulator changes.** The simulator now uses outcome-first generation: operations target a chosen outcome, including error outcomes; `on_reply` decodes the committed code from the body; and the decoded code is asserted to be a declared result code. The strict targeted-equals-committed oracle is gated to serial runs only: `client_count == 1 && CLIENT_REQUEST_QUEUE_MAX == 1`.
## Wire / SDK Impact
`ReplyHeader` is unchanged. This is not a `#[repr(C)]` layout change. However, the **reply body format changes**. A result section now precedes the payload, so body decoding changes for success replies as well. There is no released `server-ng` wire format to break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server-ng): commit-time metadata validation with result codes [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex merged PR #3553:
URL: https://github.com/apache/iggy/pull/3553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server-ng): commit-time metadata validation with result codes [iggy]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3553:
URL: https://github.com/apache/iggy/pull/3553#issuecomment-4778438081
## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/3553?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 72.73%. Comparing base ([`74d62eb`](https://app.codecov.io/gh/apache/iggy/commit/74d62eb8969e84ed46728110f1d75fb73ccb6001?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`3fa6043`](https://app.codecov.io/gh/apache/iggy/commit/3fa6043d3ac815488676b5a4d0fca9c9f84aa2b4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3553 +/- ##
============================================
- Coverage 74.41% 72.73% -1.69%
Complexity 937 937
============================================
Files 1243 1180 -63
Lines 125987 121001 -4986
Branches 101854 101469 -385
============================================
- Hits 93756 88006 -5750
- Misses 29218 30074 +856
+ Partials 3013 2921 -92
```
| [Components](https://app.codecov.io/gh/apache/iggy/pull/3553/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/3553/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `75.29% <ø> (+0.11%)` | :arrow_up: |
| [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/3553/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.44% <ø> (ø)` | |
| [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/3553/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.10% <ø> (ø)` | |
| [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/3553/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `88.88% <ø> (ø)` | |
| [PHP SDK](https://app.codecov.io/gh/apache/iggy/pull/3553/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.29% <ø> (ø)` | |
| [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/3553/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.27% <ø> (-3.86%)` | :arrow_down: |
| [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/3553/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `15.66% <ø> (-24.48%)` | :arrow_down: |
| [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/3553?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core/common/src/error/iggy\_error.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fcommon%2Fsrc%2Ferror%2Figgy_error.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb21tb24vc3JjL2Vycm9yL2lnZ3lfZXJyb3IucnM=) | `100.00% <ø> (ø)` | |
| [core/consensus/src/impls.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fconsensus%2Fsrc%2Fimpls.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25zZW5zdXMvc3JjL2ltcGxzLnJz) | `79.03% <ø> (+0.06%)` | :arrow_up: |
| [core/consensus/src/metadata\_helpers.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fconsensus%2Fsrc%2Fmetadata_helpers.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25zZW5zdXMvc3JjL21ldGFkYXRhX2hlbHBlcnMucnM=) | `91.49% <ø> (-0.02%)` | :arrow_down: |
| [core/consensus/src/plane\_helpers.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fconsensus%2Fsrc%2Fplane_helpers.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9jb25zZW5zdXMvc3JjL3BsYW5lX2hlbHBlcnMucnM=) | `92.33% <ø> (-0.06%)` | :arrow_down: |
| [core/metadata/src/impls/metadata.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fmetadata%2Fsrc%2Fimpls%2Fmetadata.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9tZXRhZGF0YS9zcmMvaW1wbHMvbWV0YWRhdGEucnM=) | `39.33% <ø> (+0.26%)` | :arrow_up: |
| [core/metadata/src/stm/consumer\_group.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fmetadata%2Fsrc%2Fstm%2Fconsumer_group.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9tZXRhZGF0YS9zcmMvc3RtL2NvbnN1bWVyX2dyb3VwLnJz) | `19.09% <ø> (+1.21%)` | :arrow_up: |
| [core/metadata/src/stm/mod.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fmetadata%2Fsrc%2Fstm%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9tZXRhZGF0YS9zcmMvc3RtL21vZC5ycw==) | `75.44% <ø> (ø)` | |
| [core/metadata/src/stm/mux.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fmetadata%2Fsrc%2Fstm%2Fmux.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9tZXRhZGF0YS9zcmMvc3RtL211eC5ycw==) | `94.59% <ø> (-0.05%)` | :arrow_down: |
| [core/metadata/src/stm/stream.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fmetadata%2Fsrc%2Fstm%2Fstream.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9tZXRhZGF0YS9zcmMvc3RtL3N0cmVhbS5ycw==) | `67.82% <ø> (+1.87%)` | :arrow_up: |
| [core/metadata/src/stm/user.rs](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree&filepath=core%2Fmetadata%2Fsrc%2Fstm%2Fuser.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9tZXRhZGF0YS9zcmMvc3RtL3VzZXIucnM=) | `62.77% <ø> (+2.05%)` | :arrow_up: |
| ... and [34 more](https://app.codecov.io/gh/apache/iggy/pull/3553?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
... and [123 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/3553/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(server-ng): commit-time metadata validation with result codes [iggy]
Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on PR #3553:
URL: https://github.com/apache/iggy/pull/3553#issuecomment-4789619775
I think is worth addressing as this PR is using the custom decoder in the SDK (hidden behind the `vsr` feature flag) to take into account those status error codes and map them to `IggyError`. Alternatively you could put that inside of the `binary_protocol` crate, and reuse that inside of `simulator` for the mapping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org