You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iggy.apache.org by "krishvishal (via GitHub)" <gi...@apache.org> on 2026/03/29 19:16:38 UTC

[PR] feat(simulator): wire Network into tick loop with per-replica outboxes and replica_crash [iggy]

krishvishal opened a new pull request, #3049:
URL: https://github.com/apache/iggy/pull/3049

     ## Summary
   
   - Replace shared `MemBus` FIFO queue with per-replica `SimOutbox` instances that stage outbound messages locally
   - Wire `PacketSimulator`-backed `Network` into the simulator tick loop for delivery with configurable delay, loss, and partitioning
   - Add `replica_crash()` which disables network links, discards outbox, tracks crashed nodes via `HashSet<u8>`
   
   Implements the outbox-only architecture from #3017. `replica_restart` is defered since it requires consensus durability support that does not yet exist.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(simulator): wire Network into tick loop with per-replica outboxes and replica_crash [iggy]

Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex merged PR #3049:
URL: https://github.com/apache/iggy/pull/3049


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(simulator): wire Network into tick loop with per-replica outboxes and replica_crash [iggy]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #3049:
URL: https://github.com/apache/iggy/pull/3049#issuecomment-4150902039

   ## [Codecov](https://app.codecov.io/gh/apache/iggy/pull/3049?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :white_check_mark: All modified and coverable lines are covered by tests.
   :white_check_mark: Project coverage is 70.65%. Comparing base ([`411a697`](https://app.codecov.io/gh/apache/iggy/commit/411a697eb184ebc33d7ce4b862f51db49ce46ea5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`001356b`](https://app.codecov.io/gh/apache/iggy/commit/001356b7658567c8ead1c43f4680b7653098d975?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3049      +/-   ##
   ============================================
   - Coverage     71.74%   70.65%   -1.10%     
     Complexity      943      943              
   ============================================
     Files          1121     1058      -63     
     Lines         93800    88825    -4975     
     Branches      71124    70869     -255     
   ============================================
   - Hits          67301    62755    -4546     
   + Misses        23863    23434     -429     
     Partials       2636     2636              
   ```
   
   | [Components](https://app.codecov.io/gh/apache/iggy/pull/3049/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [Rust Core](https://app.codecov.io/gh/apache/iggy/pull/3049/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `72.53% <ø> (+0.17%)` | :arrow_up: |
   | [Java SDK](https://app.codecov.io/gh/apache/iggy/pull/3049/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.30% <ø> (ø)` | |
   | [C# SDK](https://app.codecov.io/gh/apache/iggy/pull/3049/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.64% <ø> (ø)` | |
   | [Python SDK](https://app.codecov.io/gh/apache/iggy/pull/3049/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.43% <ø> (ø)` | |
   | [Node SDK](https://app.codecov.io/gh/apache/iggy/pull/3049/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `87.29% <ø> (-4.03%)` | :arrow_down: |
   | [Go SDK](https://app.codecov.io/gh/apache/iggy/pull/3049/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `38.68% <ø> (ø)` | |
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/iggy/pull/3049?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [core/shard/src/lib.rs](https://app.codecov.io/gh/apache/iggy/pull/3049?src=pr&el=tree&filepath=core%2Fshard%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zaGFyZC9zcmMvbGliLnJz) | `0.00% <ø> (ø)` | |
   | [core/simulator/src/bus.rs](https://app.codecov.io/gh/apache/iggy/pull/3049?src=pr&el=tree&filepath=core%2Fsimulator%2Fsrc%2Fbus.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zaW11bGF0b3Ivc3JjL2J1cy5ycw==) | `0.00% <ø> (ø)` | |
   | [core/simulator/src/lib.rs](https://app.codecov.io/gh/apache/iggy/pull/3049?src=pr&el=tree&filepath=core%2Fsimulator%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zaW11bGF0b3Ivc3JjL2xpYi5ycw==) | `0.00% <ø> (ø)` | |
   | [core/simulator/src/main.rs](https://app.codecov.io/gh/apache/iggy/pull/3049?src=pr&el=tree&filepath=core%2Fsimulator%2Fsrc%2Fmain.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zaW11bGF0b3Ivc3JjL21haW4ucnM=) | `0.00% <ø> (ø)` | |
   | [core/simulator/src/network.rs](https://app.codecov.io/gh/apache/iggy/pull/3049?src=pr&el=tree&filepath=core%2Fsimulator%2Fsrc%2Fnetwork.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zaW11bGF0b3Ivc3JjL25ldHdvcmsucnM=) | `0.00% <ø> (ø)` | |
   | [core/simulator/src/packet.rs](https://app.codecov.io/gh/apache/iggy/pull/3049?src=pr&el=tree&filepath=core%2Fsimulator%2Fsrc%2Fpacket.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zaW11bGF0b3Ivc3JjL3BhY2tldC5ycw==) | `79.17% <ø> (ø)` | |
   | [core/simulator/src/replica.rs](https://app.codecov.io/gh/apache/iggy/pull/3049?src=pr&el=tree&filepath=core%2Fsimulator%2Fsrc%2Freplica.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y29yZS9zaW11bGF0b3Ivc3JjL3JlcGxpY2EucnM=) | `0.00% <ø> (ø)` | |
   
   ... and [81 files with indirect coverage changes](https://app.codecov.io/gh/apache/iggy/pull/3049/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   </details>
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(simulator): wire Network into tick loop with per-replica outboxes and replica_crash [iggy]

Posted by "numinnex (via GitHub)" <gi...@apache.org>.
numinnex commented on code in PR #3049:
URL: https://github.com/apache/iggy/pull/3049#discussion_r3015915558


##########
core/simulator/src/lib.rs:
##########
@@ -23,122 +23,206 @@ pub mod packet;
 pub mod ready_queue;
 pub mod replica;
 
-use bus::MemBus;
+use bus::SimOutbox;
 use consensus::PartitionsHandle;
 use iggy_binary_protocol::{GenericHeader, Message, ReplyHeader};
 use iggy_common::sharding::IggyNamespace;
 use iggy_common::{IggyError, IggyMessagesBatchSet};
 use message_bus::MessageBus;
+use network::Network;
+use packet::{PacketSimulatorOptions, ProcessId};
 use partitions::{Partition, PartitionOffsets, PollingArgs, PollingConsumer};
 use replica::{Replica, new_replica};
+use std::collections::HashSet;
 use std::sync::Arc;
 
 pub struct Simulator {
+    /// All replicas, indexed by replica id. Always fully populated — crashed
+    /// replicas are kept alive but skipped during dispatch.
     pub replicas: Vec<Replica>,
-    pub message_bus: Arc<MemBus>,
+    /// Per-replica outbox, indexed by replica id. Shared with consensus inside
+    /// each replica via [`SharedSimOutbox`](bus::SharedSimOutbox).
+    pub outboxes: Vec<Arc<SimOutbox>>,
+    /// Set of replica ids that are currently crashed. Dispatch and outbox drain
+    /// are skipped for these ids.
+    pub crashed: HashSet<u8>,
+    pub network: Network,
+    pub replica_count: u8,
+    pub client_ids: Vec<u128>,
 }
 
 impl Simulator {
-    /// Initialize a partition with its own consensus group on all replicas.
-    pub fn init_partition(&mut self, namespace: iggy_common::sharding::IggyNamespace) {
-        for replica in &mut self.replicas {
-            replica.init_partition(namespace);
-        }
-    }
-
+    /// Create a new simulator with per-replica outboxes routed through a [`Network`].
     #[allow(clippy::cast_possible_truncation)]
-    pub fn new(replica_count: usize, clients: impl Iterator<Item = u128>) -> Self {
-        let mut message_bus = MemBus::new();
-        for client in clients {
-            message_bus.add_client(client, ());
-        }
+    pub fn new(
+        replica_count: usize,
+        clients: impl Iterator<Item = u128>,
+        network_options: PacketSimulatorOptions,
+    ) -> Self {
+        let client_ids: Vec<u128> = clients.collect();
+        let mut network = Network::new(network_options);
 
-        for i in 0..replica_count as u8 {
-            message_bus.add_replica(i);
+        for &cid in &client_ids {
+            network.register_client(cid);
         }
 
-        let message_bus = Arc::new(message_bus);
-        let replicas = (0..replica_count)
-            .map(|i| {
-                new_replica(
-                    i as u8,
-                    format!("replica-{i}"),
-                    &message_bus,
-                    replica_count as u8,
-                )
-            })
-            .collect();
+        let rc = replica_count as u8;
+        let mut replicas = Vec::with_capacity(replica_count);
+        let mut outboxes = Vec::with_capacity(replica_count);
+
+        for i in 0..replica_count {
+            let id = i as u8;
+            let mut bus = SimOutbox::new(id);
+            for &cid in &client_ids {
+                bus.add_client(cid, ());
+            }
+            for j in 0..rc {
+                bus.add_replica(j);
+            }
+            let outbox = Arc::new(bus);
+            let replica = new_replica(id, format!("replica-{i}"), &outbox, rc);
+            replicas.push(replica);
+            outboxes.push(outbox);
+        }
 
         Self {
             replicas,
-            message_bus,
+            outboxes,
+            crashed: HashSet::new(),
+            network,
+            replica_count: rc,
+            client_ids,
         }
     }
 
+    /// Initialize a partition with its own consensus group on all live replicas.
     #[allow(clippy::cast_possible_truncation)]
-    pub fn with_message_bus(replica_count: usize, mut message_bus: MemBus) -> Self {
-        for i in 0..replica_count as u8 {
-            message_bus.add_replica(i);
-        }
-
-        let message_bus = Arc::new(message_bus);
-        let replicas = (0..replica_count)
-            .map(|i| {
-                new_replica(
-                    i as u8,
-                    format!("replica-{i}"),
-                    &message_bus,
-                    replica_count as u8,
-                )
-            })
-            .collect();
-
-        Self {
-            replicas,
-            message_bus,
+    pub fn init_partition(&mut self, namespace: IggyNamespace) {
+        for (i, replica) in self.replicas.iter_mut().enumerate() {
+            if !self.crashed.contains(&(i as u8)) {
+                replica.init_partition(namespace);
+            }
         }
     }
-}
 
-impl Simulator {
+    /// Advance the simulation by one tick.
+    ///
+    /// Returns all client replies delivered during this tick.
+    ///
+    /// The tick has three phases that never borrow replicas and network
+    /// simultaneously:
+    ///
+    /// 1. **Deliver**: `network.step()` returns ready packets; each is
+    ///    dispatched to its target replica (or collected as a client reply).
+    /// 2. **Drain**: each live replica's outbox is drained and fed into
+    ///    `network.submit()`.
+    /// 3. **Tick**: `network.tick()` advances network time (partitions,
+    ///    clogging, etc.).
+    ///
     /// # Panics
-    /// Panics if a client response message has an invalid command type.
-    #[allow(clippy::future_not_send)]
-    pub async fn step(&self) -> Option<Message<ReplyHeader>> {
-        if let Some(envelope) = self.message_bus.receive() {
-            if let Some(_client_id) = envelope.to_client {
-                let reply: Message<ReplyHeader> = envelope
-                    .message
-                    .try_into_typed()
-                    .expect("invalid message, wrong command type for an client response");
-                return Some(reply);
+    /// Panics if a packet addressed to a client cannot be converted to a
+    /// `ReplyHeader` message.
+    #[allow(clippy::future_not_send, clippy::cast_possible_truncation)]
+    pub async fn step(&mut self) -> Vec<Message<ReplyHeader>> {
+        let mut client_replies = Vec::new();
+
+        // Phase 1: Deliver ready packets from the network.
+        let packets = self.network.step();
+        for packet in &packets {
+            match packet.to {
+                ProcessId::Replica(id) => {
+                    if !self.crashed.contains(&id)
+                        && let Some(replica) = self.replicas.get(id as usize)
+                    {
+                        Self::dispatch_to_replica(replica, packet.message.clone()).await;

Review Comment:
   I think something that we could potentially do there is to create an dedicated `compio` task per dispatch, to simulate multiple peers concurrently sending messages to replica. 



##########
core/simulator/src/main.rs:
##########
@@ -16,32 +16,27 @@
 // under the License.
 
 use iggy_binary_protocol::{Message, ReplyHeader};
-use iggy_common::PollingStrategy;
 use iggy_common::sharding::IggyNamespace;
-use iggy_common::{IggyByteSize, MemoryPool, MemoryPoolConfigOther};
-use message_bus::MessageBus;
+use iggy_common::{IggyByteSize, MemoryPool, MemoryPoolConfigOther, PollingStrategy};
 use partitions::{PollingArgs, PollingConsumer};
-use simulator::{Simulator, client::SimClient};
-use std::collections::VecDeque;
-use std::sync::{Arc, Mutex};
-
-/// Shared response queue for client replies
-#[derive(Default)]
-pub struct Responses {
-    queue: VecDeque<Message<ReplyHeader>>,
-}
-
-impl Responses {
-    pub fn push(&mut self, msg: Message<ReplyHeader>) {
-        self.queue.push_back(msg);
-    }
-
-    pub fn pop(&mut self) -> Option<Message<ReplyHeader>> {
-        self.queue.pop_front()
+use simulator::Simulator;
+use simulator::client::SimClient;
+use simulator::packet::PacketSimulatorOptions;
+
+/// Step the simulator until at least one client reply is received,
+/// or `max_ticks` is reached. Returns all collected replies.
+fn step_until_reply(sim: &mut Simulator, max_ticks: u64) -> Vec<Message<ReplyHeader>> {

Review Comment:
   We could probably move to start using `compio`, rather than default executor from `futures` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@iggy.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org