[PR] [improve][test] Add network conditions, network faults and geo-replication to the performance tests [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari opened a new pull request, #26859:
URL: https://github.com/apache/pulsar/pull/26859

   ### Motivation
   
   The Pulsar Performance Testing Framework runs the cluster and its workloads on one host, over Docker's bridge network, where a round trip takes a tenth of a millisecond and nothing is lost. Real deployments don't look like that: IoT gateways reach the cluster over the internet or a cellular network, brokers and bookies sit in different availability zones, links get congested, and brokers, bookies and ZooKeeper get cut off. Geo-replicated clusters in different regions are connected over such networks too. How Pulsar's latency, throughput and recovery behave under those conditions, and whether every message still arrives in order, couldn't be tested with the framework.
   
   This PR lets a scenario run over an impaired network: delay, jitter, packet loss and limited bandwidth between chosen containers, for the whole run or for a time, and timed faults: partitions, connection resets and silent disconnections. A scenario can also add a second cluster that the topics replicate to, with applications that consume the replicated messages there. The applications already check every device's messages, so a run with faults is a correctness test as well as a performance test.
   
   The impairments use Linux traffic control (`tc` with `netem`) on the containers' network interfaces instead of a TCP proxy such as Toxiproxy:
   
   - netem acts on packets, so TCP reacts as on a real network: a lost segment is retransmitted. A proxy terminates TCP, its delays and bandwidth act on two separate connections, and its packet loss drops chunks of the byte stream that no retransmission recovers.
   - Nothing in the cluster changes: no configuration, image or address. A proxy would need advertised listeners for the brokers, its own ports for the bookies, and the metadata URLs pointed at it. So the same impairments work with released 4.x and 5.x images, with `-Pperformance.clusterPulsarImage` and `compareImages`.
   
   ### Modifications
   
   - **Scenario format**: a new `network` section with impairments as a mapping by name, so that overlays compose and `--set` changes one setting.
     - Each impairment names its endpoints with `between: [A, B]`, or `from` and `to`. The endpoints are `gateways`, `applications`, `clients`, `brokers`, `broker-N`, `bookies`, `bookie-N`, `zookeeper`, `configuration-store` and `metadata`, and with a remote cluster `remote-brokers`, `remote-broker-N`, `remote-bookies`, `remote-bookie-N`, `remote-zookeeper`, `remote-applications`, `local-region` and `remote-region`.
     - It is a condition (`delay`, `jitter`, `loss`, `lossBurst`, `rate` with a bounded `queue`), a `partition`, a `reset`, or a `silentDisconnect`.
     - It lasts the whole run, or starts at `start` after the measurement's start or the gateways' start (`anchor`), for an optional `duration`.
     - `NetworkSettings` validates it: unknown keys, endpoints, units and ranges, settings that don't fit the kind, a start after the measurement's estimated end, overlapping conditions on the same pair of containers, and a device's band limit.
   - **`NetworkImpairments`** applies them from a privileged sidecar in the Docker engine host's network and PID namespaces, as the `perf stat` sidecar runs.
     - Delay, jitter, loss and partitions act on the receiver's host-side veth, filtered by the senders' IPs, so the sender's TCP stack doesn't see the queue, as on a real path. A rate acts on the sender's own interface, with a bounded queue, so it gives the sender backpressure, as a saturated uplink does.
     - Each device gets a `prio` qdisc with a netem band per impairment. Applying or removing an impairment adds or deletes its `u32` filters. A partition is netem's 100 % loss, with filters that take precedence over a condition's.
     - A reset destroys the TCP sockets with `ss -K`, which resets the connections on both ends.
     - A silent disconnection lists the connections from `from` to `to`, drops each one's packets both ways by its addresses and ports, and destroys the sockets on the `from` side. `from` reconnects at once, while `to` keeps half-open connections until its keep-alive notices, as after a NAT gateway or a firewall forgot its connections. Long filter scripts run from a file in the sidecar.
     - Loss bursts use netem's Gilbert-Elliott model. netem's correlated random loss (`loss random 1% 25%`) dropped almost no packet on Linux 7.1.
     - Jitter keeps the packets' order by default, via netem's rate path at a rate that never limits. netem's independent delays reordered 147 of 400 pings at 50 ± 20 ms, and TCP takes reordering for loss. `reorder: true` opts out.
     - TCP and generic segmentation offloads are turned off on the senders of a loss, a jitter or a rate, for the whole run (`offloads: auto|off|on`). Otherwise netem drops and delays GSO super-packets: at 1 % loss, about nine segments per drop.
     - The sidecar probes the kernel and its tools in a throwaway network namespace before the cluster starts.
     - Cleanup: every filter and qdisc is removed and the offloads restored, also on failure and with `PULSAR_CONTAINERS_LEAVE_RUNNING`.
     - A failed impairment fails the run at once. An impairment that never ran, or a reset that matched no connection, makes the launcher exit with an error after the report.
   - **Launcher**:
     - The gateways get a new `--await-run-start` option, so the whole-run conditions are installed before they create their clients.
     - Measurement-anchored impairments use the existing measurement gate, which now releases the measurement only once its impairments are applied.
     - `awaitWorkloads` also waits on the impairments' failure.
     - The impairments' events become Grafana annotations.
     - New setting `cluster.brokers.bootstrap: all` lists every broker in the workloads' service URL, so that a partitioned broker isn't the clients' only way in.
   - **Geo-replication**: a new `cluster.remote` section starts a second cluster that shares the configuration store with the first, so that the namespace `public/default` replicates the workload's topics to it.
     - The remote cluster copies the first cluster's brokers and bookies; `cluster.remote.brokers` and `cluster.remote.bookies` change them.
     - Remote applications, in a container of their own, consume the replicated messages in the remote cluster with the workload's subscriptions and checks; `cluster.remote.applications` sets their count, pods and environment. The run fails when one misses a message or receives one out of order.
     - The run report has a Remote applications section with their correctness and latency. The progress shows the remote applications on a line of their own, with the remote subscriptions' backlog and the replication backlog. `remote-topic-stats.csv` has the remote topics' stats and `replication-stats.csv` each topic's replicators, once a second.
     - The remote cluster's containers are network endpoints, so the geo overlays compose with the others, which act on the first cluster.
   - **Workloads**:
     - The Pulsar client's warnings, such as reconnections and timeouts, go to `pulsar-client.log.txt` in each workload's output directory. Before, they were dropped, since the tools had no SLF4J binding. `PULSAR_CLIENT_LOG_LEVEL` sets the level.
     - The applications tell redelivered duplicates (the same message ID) from messages persisted twice (another message ID), for a device's latest message. The ordering violation samples record the redelivery count and the receipt time.
   - **Report**: a Network conditions section with each impairment, when it was applied and removed relative to the measurement, its queues' packets and drops, and each impaired container's TCP retransmissions. The data comes from `network-summary.json`; `network-events.jsonl` records every event as it happens.
   - **Scenarios**: overlays under `scenarios/network/`, each describing what it models and what to look for:
     - conditions: `wan-clients`, `mobile-gateways`, `constrained-uplink`, `cross-zone-storage`, `slow-bookie`, `lossy-storage`, `constrained-consumers`;
     - faults: `bookie-partition`, `broker-partition`, `broker-isolation`, `metadata-partition`, `metadata-takeover`, `return-path-blackhole`, `connection-resets`, `consumer-reconnect-storm`, `silent-client-disconnect`, `silent-broker-disconnect`, `storage-outage`, `bookie-metadata-outage`, `zookeeper-outage`;
     - geo-replication: `geo-replication`, which adds the remote cluster with four remote applications and an hour's retention, and `geo-wan`, `geo-partition`, `geo-silent-disconnect` and `geo-constrained-link`, which extend it;
     - `failover-cluster`, a two-broker, four-bookie cluster with E/W/A 3/3/2 that the fault overlays extend.
   - **Docs**: `docs/network-conditions.md`, with what each overlay can be used for and a geo-replication section, entries in the scenario format and the README, and a section in the framework's agent guide on network and fault experiments: preparing the host, checking that the impairments acted, the correctness failures and the duplicates' classification, where the complete logs are, lining up the clocks, the JIT's effect on a fault's first occurrence, heap dumps and profiles of a fault, and geo-replication runs.
   
   ### What the scenarios are for
   
   Each overlay models a network condition or a fault that production clusters meet, and checks Pulsar's correctness, stalls and recovery under it: the publish and end-to-end latency over WANs and lossy links, failover after a broker, a bookie or the metadata store becomes unreachable, the brokers' handling of connections that vanish without notice, consumers that fall behind, mass reconnections, and geo-replication over a WAN, a congested link and a link failure. `docs/network-conditions.md` lists what each one can be used for.
   
   Running them against master on `iot-telemetry-small` found several problems in Pulsar, to be reported as separate issues: among them, stalls of unrelated producers when many Key_Shared consumers reconnect, a broker running out of heap while it holds the old consumers of connections that vanished, Key_Shared ordering violations in the same situation, slow producer failover after a broker loses its metadata session, bookies shutting down when they lose ZooKeeper because of the embedded AutoRecovery, and geo-replication losing messages after the connections between the clusters break.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   This change added tests and can be verified as follows:
   
   - Unit tests:
     - `NetworkSettingsTest`: parsing, endpoints, pairs, validation, overlaps, the band limit.
     - `NetworkImpairmentsTest`: the impairments' state machine on a fake host: whole-run and timed impairments, missed and skipped ones, failures during the set-up and the run, resets, silent disconnections, and the cleanup, including after a partial set-up.
     - `TcCommandsTest`: the generated commands, hexadecimal class IDs, queue sizes, and parsing `ip`, `ethtool` and `tc` output.
     - `NetworkReportTest`: the report section.
     - `ClusterSettingsTest`: the bootstrap setting and the remote cluster.
     - `RemoteApplicationsTest`: the remote applications' workload and the containers' names.
     - `ProgressMonitorTest`: the remote applications' line and the age of stale samples.
     - `MeasurementSecondsTest`: the measurement's estimated length, which bounds the impairments' start times.
     - `RunReportTest`: the Remote applications section.
     - `RemoteApplicationsTest`: also that the launcher's container names match what the endpoints expand to, and that a remote cluster needs the topics in `public/default`.
     - `MeasurementControlTest`: `/run/start`.
     - `ScenarioFilesTest`: every network overlay on the small and the high-rate scenarios.
     - `DeviceSequenceTrackerTest`: telling redelivered duplicates from messages persisted twice.
   - `./gradlew :tests:performance:launcher:networkCheck`, a new task, runs `NetworkImpairmentsCheck` against real containers with `ping` and `iperf3`. It covers a delay on one destination and none on others, a timed partition on the timeline, a silent disconnection in each direction, a timed silent disconnection of a target without connections, the jitter's order with and without `reorder`, 1 % loss (0.7–1.3 % measured) and a 20 Mbit/s rate, a reset, and the cleanup after each: no impairment left on the containers' interfaces or their host-side veths, and the offloads restored. It passes on Linux 7.1 with Docker 29.8.
   - A smoke run of `iot-telemetry-small` with 20 ms between the gateways and the brokers raised the publish latency from about 1 ms to 41 ms at the full 1,000 msg/s, and every message was delivered in order.
   - A run of `iot-telemetry-small` with `network/geo-replication` delivered every message to the four remote applications, in order, about 1 ms behind the first cluster's applications.
   
   ### Does this pull request potentially affect one of the following parts:
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][test] Add network conditions, network faults and geo-replication to the performance tests [pulsar]

["dao-jun (via GitHub)" <gi...@apache.org>]

dao-jun commented on PR #26859:
URL: https://github.com/apache/pulsar/pull/26859#issuecomment-6121779014

   The introduction of the chaos test will greatly boost Pulsar's resilience.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [improve][test] Add network conditions, network faults and geo-replication to the performance tests [pulsar]

["lhotari (via GitHub)" <gi...@apache.org>]

lhotari merged PR #26859:
URL: https://github.com/apache/pulsar/pull/26859


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org