[GitHub] [zookeeper] maoling commented on pull request #1135: ZOOKEEPER-3601:introduce the fault injection framework: Byteman for ZooKeeper

[GitBox <gi...@apache.org>]

maoling commented on pull request #1135:
URL: https://github.com/apache/zookeeper/pull/1135#issuecomment-793273459


   - @functioner  the functionality is still working(Glad to see you used the Byteman to reproduce the issue in the ZOOKEEPER-4203)
   - @anmolnar  It seems that this patch is very useful to help users and developers, do you have a plan to merge it?
   
   ### Sharing some other chaos engineering approaches currently I used for ZooKeeper:
   
   #### [blockade](https://github.com/worstcase/blockade):
   I use it for injecting network latency and network partition(very useful, I love it). I write lots of specific scripts which help me create the custom network partition pattern I want.
   
   #### [chaos-mesh](https://chaos-mesh.org/docs):
   I use it for general chaos work and it has a good web UI and is integrated with k8s. But AFAIK, I cannot customize network partition
   
   #### [jepsen](https://github.com/jepsen-io/jepsen):
   A framework for distributed systems verification, with fault injection(mainly used for consistency check). I used it for linearizable read consistency check in [ZOOKEEPER-3600](https://issues.apache.org/jira/browse/ZOOKEEPER-3600)
   
   #### [Byteman](https://byteman.jboss.org/):
   It's what this patch wants to introduce. Byteman is a tool which makes it easy to trace, monitor and test the behaviour of Java application and JDK runtime code. It injects Java code into your application methods or into Java runtime methods without the need for you to recompile, repackage or even redeploy your application. Injection can be performed at JVM startup or after startup while the application is still running. Injected code can access any of your data and call any application methods, including where they are private. You can inject code almost anywhere you want and there is no need to prepare the original source code in advance. You can even remove injected code and reinstall different changes while the application continues to execute.
   
   #### [chaosblade](https://github.com/chaosblade-io/chaosblade):
   ChaosBlade is a injection tool that follows the principles of chaos engineering and chaos experimental models to help enterprises improve the fault tolerance of distributed systems and ensure business continuity during the process of enterprises going to cloud or moving to cloud native systems. The scenarios include:
   Basic resources: such as CPU, memory, network, disk, process and other experimental scenarios;
   Java applications: such as databases, caches, messages, JVM itself, microservices, etc. You can also specify any class method to inject various complex experimental scenarios;
   
   #### [Stress for Linux](https://pkgs.org/download/stress):
   A tool which imposes a configurable amount of load on your system where ZooKeeper locates
   
   #### [namazu-swarm](https://github.com/osrg/namazu-swarm):
   Namazu Swarm is a part of Namazu, a programmable fuzzy scheduler for testing distributed systems. AFAIK, this project has not been maintained for a long time and it's not easy to install it.
   
   ### Other resources from other projects:
   
   #### [etcd]:
   https://coreos.com/blog/new-functional-testing-in-etcd.html
   https://coreos.com/blog/testing-distributed-systems-in-go.html
   #### [Kafka]:
   https://cwiki.apache.org/confluence/display/KAFKA/Fault+Injection
   #### [Apache Ozone]:
   https://blog.cloudera.com/apache-ozone-fault-injection-framework/


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org