You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Ivan Andika (Jira)" <ji...@apache.org> on 2026/09/22 09:05:00 UTC
[jira] [Updated] (HDDS-15501) Distributed System Testing in Ozone
[ https://issues.apache.org/jira/browse/HDDS-15501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Andika updated HDDS-15501:
-------------------------------
Description:
Currently, we only test Ozone using the traditional UT, IT, Acceptance Tests. We had a MiniOzoneChaosCluster (fault injection testing), but it seems unused. Additionally, we also have byteman injection testing but it seems that it is a part of acceptance tests and therefore the testing feedback loop is not that tight. I propose to introduce a distributed system testing and proofs system so that we can have the Ozone spec as the shared mental model. Some of the regressions for issues like breaking majority commit contract (HDDS-15052) or handling disk failures (HDDS-12564, HDDS-12151) is not detected since we don't have the spec as the source of truth. Additionally sometimes simply we use informal reasonings to guide our implementation and fixes which can cause regressions (for example, a lot of ReplicationManager fixes are only done only when there is an issue in productions).
This is a parent task for the effort to introduce distributed system testing and proofs to test the correctness of Ozone implementation. For example,
* Snapshot and snapshot diff
* Partial write commit and all commit semantics
* Container state transitions (under failures)
* Replication manager & container balancer correctness
** Replication manager should not cause infinite replications, it should end in a steady state
* Container state and replica management
** How to appease eventually consistent container replica management (through heartbeat) and strongly consistent container state (through Ratis)
* End to end block deletion guarantee
** Ensure that blocks are deleted and not orphaned
* Ensure that container replica numOfKeys and numOfBytes is accurate
* OM bucket quota accuracy issue: Ensure that OM bucket quota should not become negative
* OM and SCM linearizability guarantee with the table cache and (double) transacstion buffer mechanisms
* Container reconciliation guarantee
* SCM distributed sequence ID generator correctness
** ID generated by ID generator is globally unique in presence of failures
Distributed system testing tools:
- Jepsen, Ellen, Maelstorm
- Fray
- Hypothesis (Hegel)
- Antithesis (paid)
Distributed system proofs:
- TLA+
-- Specula ([https://github.com/specula-org/Specula]) : AI generated TLA+ already used to find some Raft bugs
- Lean4
- P framework
Real systems
- 3FS ([https://github.com/deepseek-ai/3FS/tree/main/specs]) - uses P framework
- AWS S3
-- [https://cacm.acm.org/practice/systems-correctness-practices-at-amazon-web-services/]
-- [https://p-org.github.io/P/casestudies/#case-studies]
-- [https://dl.acm.org/doi/10.1145/3477132.3483540]
- etcd robustness test ([https://github.com/etcd-io/etcd/tree/main/tests/robustness]) - Uses antithesis (among other things)
I prefer if we can start with P framework since some storage systems already used it and it seems it already had some JUnit integration.
In the future, we can support Deterministic Simulation Testing ([https://antithesis.com/docs/resources/deterministic_simulation_testing]). However, this requires a lot of efforts and very invasive since we need to make all Ozone implementations to be testable by the simulation framework so it's not going to happen anytime soon. Some of the deterministic simulation testing framework is Madsim (Rust), Turmoil (Rust), BUGGIFY (C++, FoundationDB), Tickloom (Java, [https://github.com/unmeshjoshi/tickloom]), Vortex (Zig, used by TigerBeetle). We can start taking look at Tickloom for testing in Java.
Having a real industry-recognized spec helps to instil confidence in Ozone robustness. More importantly, distributed system testing allows us to have confidence that our changes will not introduce critical issues (as long as the system is covered by the test). It also forces us to be more rigorous in our design and implementation. The hope for the Ozone community is that we can have distributed system testing as a crucial baseline skill for all Ozone community members. If in the future, we decide to introduce a new component or rework Ozone (e.g. porting it to Rust, etc), I hope that it's built on the distributed system testing foundation (e.g. we build systems on top of DST first) to ensure that Ozone correctness remain preserved. It's going to be difficult, but I believe it's worth it long term.
Other resources
* https://danluu.com/agentic-testing/
was:
Currently, we only test Ozone using the traditional UT, IT, Acceptance Tests. We had a MiniOzoneChaosCluster (fault injection testing), but it seems unused. Additionally, we also have byteman injection testing but it seems that it is a part of acceptance tests and therefore the testing feedback loop is not that tight. I propose to introduce a distributed system testing and proofs system so that we can have the Ozone spec as the shared mental model. Some of the regressions for issues like breaking majority commit contract (HDDS-15052) or handling disk failures (HDDS-12564, HDDS-12151) is not detected since we don't have the spec as the source of truth. Additionally sometimes simply we use our intuitions to guide our implementation and fixes which can cause regressions (for example, a lot of ReplicationManager fixes are only done only when there is an issue in productions).
This is a parent task for the effort to introduce distributed system testing and proofs to test the correctness of Ozone implementation. For example,
* Snapshot and snapshot diff
* Partial write commit and all commit semantics
* Container state transitions (under failures)
* Replication manager & container balancer correctness
** Replication manager should not cause infinite replications, it should end in a steady state
* Container state and replica management
** How to appease eventually consistent container replica management (through heartbeat) and strongly consistent container state (through Ratis)
* End to end block deletion guarantee
** Ensure that blocks are deleted and not orphaned
* Ensure that container replica numOfKeys and numOfBytes is accurate
* OM bucket quota accuracy issue: Ensure that OM bucket quota should not become negative
* OM and SCM linearizability guarantee with the table cache and (double) transacstion buffer mechanisms
* Container reconciliation guarantee
* SCM distributed sequence ID generator correctness
** ID generated by ID generator is globally unique in presence of failures
Distributed system testing tools:
- Jepsen, Ellen, Maelstorm
- Fray
- Hypothesis (Hegel)
- Antithesis (paid)
Distributed system proofs:
- TLA+
-- Specula ([https://github.com/specula-org/Specula]) : AI generated TLA+ already used to find some Raft bugs
- Lean4
- P framework
Real systems
- 3FS ([https://github.com/deepseek-ai/3FS/tree/main/specs]) - uses P framework
- AWS S3
-- [https://cacm.acm.org/practice/systems-correctness-practices-at-amazon-web-services/]
-- [https://p-org.github.io/P/casestudies/#case-studies]
-- [https://dl.acm.org/doi/10.1145/3477132.3483540]
- etcd robustness test ([https://github.com/etcd-io/etcd/tree/main/tests/robustness]) - Uses antithesis (among other things)
I prefer if we can start with P framework since some storage systems already used it and it seems it already had some JUnit integration.
In the future, we can support Deterministic Simulation Testing ([https://antithesis.com/docs/resources/deterministic_simulation_testing]). However, this requires a lot of efforts and very invasive since we need to make all Ozone implementations to be testable by the simulation framework so it's not going to happen anytime soon. Some of the deterministic simulation testing framework is Madsim (Rust), Turmoil (Rust), BUGGIFY (C++, FoundationDB), Tickloom (Java, [https://github.com/unmeshjoshi/tickloom]), Vortex (Zig, used by TigerBeetle). We can start taking look at Tickloom for testing in Java.
Having a real industry-recognized spec helps to instil confidence in Ozone robustness. More importantly, distributed system testing allows us to have confidence that our changes will not introduce critical issues (as long as the system is covered by the test). It also forces us to be more rigorous in our design and implementation. The hope for the Ozone community is that we can have distributed system testing as a crucial baseline skill for all Ozone community members. If in the future, we decide to introduce a new component or rework Ozone (e.g. porting it to Rust, etc), I hope that it's built on the distributed system testing foundation (e.g. we build systems on top of DST first) to ensure that Ozone correctness remain preserved. It's going to be difficult, but I believe it's worth it long term.
> Distributed System Testing in Ozone
> -----------------------------------
>
> Key: HDDS-15501
> URL: https://issues.apache.org/jira/browse/HDDS-15501
> Project: Apache Ozone
> Issue Type: Epic
> Components: test
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> Currently, we only test Ozone using the traditional UT, IT, Acceptance Tests. We had a MiniOzoneChaosCluster (fault injection testing), but it seems unused. Additionally, we also have byteman injection testing but it seems that it is a part of acceptance tests and therefore the testing feedback loop is not that tight. I propose to introduce a distributed system testing and proofs system so that we can have the Ozone spec as the shared mental model. Some of the regressions for issues like breaking majority commit contract (HDDS-15052) or handling disk failures (HDDS-12564, HDDS-12151) is not detected since we don't have the spec as the source of truth. Additionally sometimes simply we use informal reasonings to guide our implementation and fixes which can cause regressions (for example, a lot of ReplicationManager fixes are only done only when there is an issue in productions).
> This is a parent task for the effort to introduce distributed system testing and proofs to test the correctness of Ozone implementation. For example,
> * Snapshot and snapshot diff
> * Partial write commit and all commit semantics
> * Container state transitions (under failures)
> * Replication manager & container balancer correctness
> ** Replication manager should not cause infinite replications, it should end in a steady state
> * Container state and replica management
> ** How to appease eventually consistent container replica management (through heartbeat) and strongly consistent container state (through Ratis)
> * End to end block deletion guarantee
> ** Ensure that blocks are deleted and not orphaned
> * Ensure that container replica numOfKeys and numOfBytes is accurate
> * OM bucket quota accuracy issue: Ensure that OM bucket quota should not become negative
> * OM and SCM linearizability guarantee with the table cache and (double) transacstion buffer mechanisms
> * Container reconciliation guarantee
> * SCM distributed sequence ID generator correctness
> ** ID generated by ID generator is globally unique in presence of failures
> Distributed system testing tools:
> - Jepsen, Ellen, Maelstorm
> - Fray
> - Hypothesis (Hegel)
> - Antithesis (paid)
> Distributed system proofs:
> - TLA+
> -- Specula ([https://github.com/specula-org/Specula]) : AI generated TLA+ already used to find some Raft bugs
> - Lean4
> - P framework
> Real systems
> - 3FS ([https://github.com/deepseek-ai/3FS/tree/main/specs]) - uses P framework
> - AWS S3
> -- [https://cacm.acm.org/practice/systems-correctness-practices-at-amazon-web-services/]
> -- [https://p-org.github.io/P/casestudies/#case-studies]
> -- [https://dl.acm.org/doi/10.1145/3477132.3483540]
> - etcd robustness test ([https://github.com/etcd-io/etcd/tree/main/tests/robustness]) - Uses antithesis (among other things)
> I prefer if we can start with P framework since some storage systems already used it and it seems it already had some JUnit integration.
> In the future, we can support Deterministic Simulation Testing ([https://antithesis.com/docs/resources/deterministic_simulation_testing]). However, this requires a lot of efforts and very invasive since we need to make all Ozone implementations to be testable by the simulation framework so it's not going to happen anytime soon. Some of the deterministic simulation testing framework is Madsim (Rust), Turmoil (Rust), BUGGIFY (C++, FoundationDB), Tickloom (Java, [https://github.com/unmeshjoshi/tickloom]), Vortex (Zig, used by TigerBeetle). We can start taking look at Tickloom for testing in Java.
> Having a real industry-recognized spec helps to instil confidence in Ozone robustness. More importantly, distributed system testing allows us to have confidence that our changes will not introduce critical issues (as long as the system is covered by the test). It also forces us to be more rigorous in our design and implementation. The hope for the Ozone community is that we can have distributed system testing as a crucial baseline skill for all Ozone community members. If in the future, we decide to introduce a new component or rework Ozone (e.g. porting it to Rust, etc), I hope that it's built on the distributed system testing foundation (e.g. we build systems on top of DST first) to ensure that Ozone correctness remain preserved. It's going to be difficult, but I believe it's worth it long term.
> Other resources
> * https://danluu.com/agentic-testing/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org