[jira] [Created] (HDDS-16627) Add a deterministic simulation test for SCM

["Chu Cheng Li (Jira)" <ji...@apache.org>]

Chu Cheng Li created HDDS-16627:
-----------------------------------

             Summary: Add a deterministic simulation test for SCM
                 Key: HDDS-16627
                 URL: https://issues.apache.org/jira/browse/HDDS-16627
             Project: Apache Ozone
          Issue Type: Test
          Components: SCM
            Reporter: Chu Cheng Li


SCM bugs that involve datanode failures, decommission or maintenance, and SCM restarts are hard to find and hard to reproduce. Whether they happen depends on the order in which the node health check, the event handlers, ReplicationManager and the pipeline services see each change. MiniOzoneCluster tests are too slow to try many such orders, and a failed run usually cannot be repeated in the same order.

This adds a deterministic simulation test for SCM, TestScmSimulation in hdds-server-scm. It runs the real StorageContainerManager in-process against simulated datanodes:
* Simulated datanodes register and heartbeat through SCM's datanode protocol server, send container and pipeline reports, and carry out the commands SCM sends back.
* SCM is built with a simulated clock, seeded random sources, and an event queue whose handlers run on the simulation's scheduler. The simulation runs SCM's background services itself: the node health check, ReplicationManager, the background pipeline creator, the pipeline scrubber and the decommission monitor.
* Everything runs on one thread, so a run is fully determined by its seed. Rerunning a seed gives exactly the same trace. Three hours of simulated time take about 1-2 seconds.
* During the chaos phase, a client writes to containers while datanodes crash, get partitioned, are decommissioned (one at a time or a whole rack), enter maintenance or join, replicas are lost or corrupted, and SCM restarts. Then the faults stop, every datanode comes back, and SCM gets time to repair the cluster.
* Checked during the run: SCM never deletes the last replica, the last healthy replica, or a replica newer than every other healthy one; open containers are on open pipelines; healthy datanodes are in the network topology; no event handler or service throws.
* Checked at the end: every container is correctly replicated, every decommission and maintenance finishes, SCM and the datanodes agree on the replicas, and SCM sends no replicate or delete commands for the next 15 simulated minutes.

Not simulated: SCM HA, block deletion, EC containers, the container balancer and security.

A few production classes change so that the test can inject what it needs:
* SCMConfigurator can set the event queue, the clock and the pipeline ID generator.
* The clock is passed down to the node manager and to the container and pipeline state maps.
* NetworkTopologyImpl can take a seeded Random, and PipelinePlacementPolicy is no longer final.
* RatisUnderReplicationHandler rotates the vulnerable UNHEALTHY replicas instead of shuffling them.

The default run uses seeds 1, 2 and 3. To rerun a failed seed, or to try new ones:
{noformat}
mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos -Dscm.simulation.seed=SEED
mvn -pl :hdds-server-scm test -Dtest=TestScmSimulation#chaos -Dscm.simulation.random.seeds=100 -Dscm.simulation.racks=random
{noformat}

Bugs found with this test are linked to this issue as "Discovered while testing".



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org