You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by ma...@apache.org on 2026/07/16 11:22:17 UTC

(iggy) branch master updated: docs(server-ng): refine partition-materialization barrier TODO (#3677)

This is an automated email from the ASF dual-hosted git repository.

mmodzelewski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iggy.git


The following commit(s) were added to refs/heads/master by this push:
     new 313c50bba docs(server-ng): refine partition-materialization barrier TODO (#3677)
313c50bba is described below

commit 313c50bba41efcbf7b6129971fa8e750bfc5ee2a
Author: Grzegorz Koszyk <11...@users.noreply.github.com>
AuthorDate: Thu Jul 16 13:22:13 2026 +0200

    docs(server-ng): refine partition-materialization barrier TODO (#3677)
    
    Co-authored-by: Maciej Modzelewski <ma...@modzelewski.dev>
---
 core/server-ng/src/partition_reconciler.rs | 36 ++++++++++++++++++++++++++----
 1 file changed, 32 insertions(+), 4 deletions(-)

diff --git a/core/server-ng/src/partition_reconciler.rs b/core/server-ng/src/partition_reconciler.rs
index 998537c5b..37a94b32f 100644
--- a/core/server-ng/src/partition_reconciler.rs
+++ b/core/server-ng/src/partition_reconciler.rs
@@ -45,10 +45,38 @@
 //! every `create_topic`.
 //!
 //! TODO: block VSR-ification of `topics.rs` / `partitions.rs`
-//! `binary_impls` on a materialization barrier (cross-shard
-//! `MaterializedAck` from each assigned shard back to shard 0; shard 0
-//! holds the reply in `client_table` until all acks arrive). Sync-block
-//! alternative was deferred.
+//! `binary_impls` on a materialization barrier. Two changes, both
+//! required together (one without the other does not close the race):
+//!
+//! 1. **Owner becomes the sole writer of its own `shards_table` row.**
+//!    Today any non-owning shard's reconciler independently seeds an
+//!    `InsertRouted` row the moment it observes committed metadata (see
+//!    the bullet above) -- a pure hash computation, no coordination with
+//!    the owner. That is fine for routing (`calculate_shard_assignment`
+//!    is a static function of the namespace, identical on every shard,
+//!    no placement decision to propagate) but it means a row can exist
+//!    before the owner's own `build_partition_fresh` has finished.
+//!    Non-owning shards must stop writing this row ahead of the owner.
+//! 2. **Owner pushes the row to every other shard once materialised**,
+//!    instead of each shard independently guessing it. Cheap: this is a
+//!    same-node, cross-shard-core message (the existing `ReconcileOp`
+//!    inter-shard channel already carries `InsertOwned`/`ConfirmRemove`;
+//!    extend it with a push variant), not a network round trip to
+//!    another replica.
+//!
+//! With both in place, `shards_table.shard_for(ns).is_some()` on ANY
+//! shard implies the owner has already materialised the partition, so
+//! `dispatch::wait_for_partition_routable`'s second-phase `partition_read`
+//! probe (the owner-readiness check a router-side reader currently has to
+//! do by hand, since the table alone can't be trusted) becomes
+//! unnecessary; a single `shards_table` poll is a sufficient barrier for
+//! both server-ng-shard routing AND the pump/client reply. The heavier
+//! "shard 0 holds the client reply until every assigned shard acks"
+//! design was the original idea here; this is a smaller, cheaper
+//! alternative scoped to the local (same-node) table-visibility problem
+//! only, not the reply-timing one -- the create-topic reply can still
+//! ship on metadata commit as it does today, since the retry loop that
+//! consumes `shards_table` is what actually needs the invariant.
 
 use crate::bootstrap::ServerNgShard;
 use crate::partition_helpers::{build_partition_fresh, delete_partitions_from_disk};