You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ratis.apache.org by el...@apache.org on 2019/03/11 18:07:51 UTC

[incubator-ratis] branch master updated: RATIS-463. Basic Distributed Test Harness for Ratis

This is an automated email from the ASF dual-hosted git repository.

elserj pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-ratis.git


The following commit(s) were added to refs/heads/master by this push:
     new 35838f0  RATIS-463. Basic Distributed Test Harness for Ratis
35838f0 is described below

commit 35838f032a4096d78843130fa1435bcddf5ce961
Author: Clay Baenziger <cw...@clayb.net>
AuthorDate: Wed Dec 5 02:50:27 2018 -0500

    RATIS-463. Basic Distributed Test Harness for Ratis
    
    Signed-off-by: Josh Elser <el...@apache.org>
---
 README.md                                          |   2 +
 dev-support/vagrant/.gitignore                     |   5 +
 dev-support/vagrant/README.md                      |  52 +++
 dev-support/vagrant/Vagrantfile                    | 197 ++++++++++
 dev-support/vagrant/bin/start_ratis_load_gen.sh    |  31 ++
 dev-support/vagrant/bin/start_ratis_server.sh      |  27 ++
 dev-support/vagrant/docs/vagrantfile.bpmn          | 420 +++++++++++++++++++++
 dev-support/vagrant/docs/vagrantfile_bpmn.svg      |  17 +
 dev-support/vagrant/namazu_configs/hdd_config.toml |  26 ++
 dev-support/vagrant/run_all_tests.sh               |  66 ++++
 dev-support/vagrant/screenrcs/namazu_hdd_screenrc  |  32 ++
 .../screenrcs/ratis_ratis-hdd-slowdown_screenrc    |  33 ++
 .../vagrant/screenrcs/ratis_ratis-server_screenrc  |  33 ++
 13 files changed, 941 insertions(+)

diff --git a/README.md b/README.md
index bc82248..0e19693 100644
--- a/README.md
+++ b/README.md
@@ -60,6 +60,8 @@ For example `ratis-examples/src/main/bin/client.sh arithmetic get --name b --pee
 
 PS: the peer is a id, ipaddress pair seperated by ':', for eg. n0:172.26.32.224:6000
 
+### Pre-Setup Vagrant Pseudo Cluster
+One can see the interactions of a three server Ratis cluster with a load-generator running against it by using the `run_all_tests.sh` script found in [dev-support/vagrant/]. See the [dev-support/vagrant/README.md] for more on dependencies and what is setup. This will allow one to try a fully setup three server Ratis cluster on a single VM image, preventing resource contention with your development host and allowing failure injection too.
 
 # Reference
 [1] _Diego Ongaro and John Ousterhout. 2014. In search of an understandable consensus algorithm. In Proceedings of the 2014 USENIX conference on USENIX Annual Technical Conference (USENIX ATC'14), Garth Gibson and Nickolai Zeldovich (Eds.). USENIX Association, Berkeley, CA, USA, 305-320._
diff --git a/dev-support/vagrant/.gitignore b/dev-support/vagrant/.gitignore
new file mode 100644
index 0000000..6bf78c5
--- /dev/null
+++ b/dev-support/vagrant/.gitignore
@@ -0,0 +1,5 @@
+.*.sw*
+*~
+*.box
+*.log
+.vagrant
diff --git a/dev-support/vagrant/README.md b/dev-support/vagrant/README.md
new file mode 100644
index 0000000..9f51805
--- /dev/null
+++ b/dev-support/vagrant/README.md
@@ -0,0 +1,52 @@
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+# What is this?
+
+This is a series of scripts for [Vagrant](https://vagrantup.com) to stand-up and test [Apache (Incubating) Ratis](https://ratis.incubator.apache.org/) servers and clients. One needs to have Vagrant and [VirtualBox](https://virtualbox.org) installed to stand-up these tests environments.
+
+# What is provided?
+
+This provides a multi-host `Vagrantfile` which provides the following described VM's:
+
+## `ratis-build` VM
+This provides a built version of Ratis with the [Namazu](https://github.com/osrg/namazu) test framework as well
+
+## `ratis-servers` VM
+This leverages the built Ratis servers to run a three server cluster and load generator. The Ratis servers are listening on ports 6000, 6001, 6002. The four processes are started and daemonized using [GNU Screen](https://www.gnu.org/software/screen/). The daemons further log to files in the home directory of the test user.
+
+## `ratis-hdd-slowdown` VM
+This VM starts the three Ratis servers and load genearator as the `ratis-servers` VM. However, the three servers contend with their storage directories made pathological (slow and error-prone) by Namazu. The configuration of pathology in namazu can be tuned in [hdd_config.toml](./namazu_configs/hdd_config.toml).
+
+The test VM's can be stoped and all daemons restarted via: `vagrant up --provision <VM name>`
+One can login to the VM and read the message-of-the-day for instructions on how to read the daemon logs; or connect to the Screen session.
+
+# How to get started:
+There is a shell script `run_all_tests.sh` which provides a single entrypoint for building or cleaning up all tests.
+To visualize the flow of building all tests, a BPMN diagram of the intended process flow is: ![Vagrantfile BPMN Flow][Vagrantfile_BPMN]
+
+Execute `run_all_tests.sh` with option `build`:
+* Builds the `ratis-build` VM
+* Packages a [Vagrant box](https://www.vagrantup.com/docs/boxes.html) to build test VMs off of
+* Builds all test VMs and suspends them on success
+
+Execute `run_all_tests.sh` with option `clean`:
+* Destroys all test VMs
+* Destroys the `ratis-build` VM
+* Removes the `ratis-test` from Vagrant
+* Removes the `ratistest.box` from the local file-system
+
+Run the tests for a machine with `vagrant resume`:
+* e.g. `vagrant resume ratis-servers && vagrant ssh ratis-servers`
+
+[Vagrantfile_BPMN]: ./docs/vagrantfile_bpmn.svg "Vagrantfile Steps in BPMN (created with https://demo.bpmn.io)"
diff --git a/dev-support/vagrant/Vagrantfile b/dev-support/vagrant/Vagrantfile
new file mode 100644
index 0000000..78a00c8
--- /dev/null
+++ b/dev-support/vagrant/Vagrantfile
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+#
+# This is a Vagrantfile to automatically provision a test environment
+#
+# See http://www.vagrantup.com/ for info on Vagrant.
+#
+
+VAGRANT_HOME = '/home/vagrant/'
+TEST_DATA = ::File.join(VAGRANT_HOME, 'test_data')
+RATIS_PATH = ::File.join(VAGRANT_HOME, 'incubator-ratis')
+SCREEN_RC_PATH = ::File.join(RATIS_PATH, 'dev-support/vagrant/screenrcs')
+
+# Settings common to all VMs
+common_vm_settings = Proc.new do |vm|
+  vm.customize ['modifyvm', :id, '--nictype2', '82543GC']
+  vm.customize ['modifyvm', :id, '--largepages', 'on']
+  vm.customize ['modifyvm', :id, '--nestedpaging', 'on']
+  vm.customize ['modifyvm', :id, '--vtxvpid', 'on']
+  vm.customize ['modifyvm', :id, '--hwvirtex', 'on']
+  vm.customize ['modifyvm', :id, '--ioapic', 'on']
+end
+
+# Settings common to all test VMs
+common_test_vm_settings = Proc.new do |testvm|
+  # link in and build Ratis
+  testvm.vm.synced_folder '../..', RATIS_PATH, type: 'rsync', rsync__exclude: '**/.vagrant/**'
+
+  test_vm_memory = ENV.fetch 'TEST_VM_MEM', (15 * 1024)
+  # starting screen wants a tty; for more see https://github.com/hashicorp/vagrant/issues/1673
+  testvm.ssh.pty = true
+  testvm.vm.provision :shell, privileged: false, name: 'Run Ratis Servers', inline: <<-EOH
+    pkill -u vagrant -f ratis || true
+    # divide 95% of the VM memory in to thirds for the servers
+    export JAVA_OPTS="-Xmx #{(test_vm_memory.to_i * 0.95) / 3}"
+    # screen wants a tty; use screen(1) for that
+    script -c "screen -c #{SCREEN_RC_PATH}/ratis_$(hostname)_screenrc" /dev/stdout
+  EOH
+  testvm.ssh.pty = false
+
+  testvm.vm.box = 'ratis-test'
+  testvm.vm.box_url = 'ratistest.box'
+
+  testvm.vm.provider :virtualbox do |vb|
+    vb.gui = false
+
+    vb.cpus = ENV.fetch 'TEST_VM_CPUs', 5
+    vb.memory = test_vm_memory
+    vb.linked_clone = true
+    common_vm_settings.call(vb)
+  end
+end
+
+Vagrant.configure('2') do |config|
+  config.vm.define 'ratis-build'.to_sym do |ratisbuild|
+    ratisbuild.vm.hostname = 'ratis-build'
+    # setup a local Maven settings.xml if available
+    if File.exist?(File.expand_path('~/.m2/settings.xml'))
+      config.vm.provision 'shell', privileged: false, inline: 'mkdir -p ~/.m2'
+
+      config.vm.provision 'file', source: '~/.m2/settings.xml',
+                                  destination: "#{VAGRANT_HOME}/.m2/settings.xml"
+    end
+
+    # install packages
+    ratisbuild.vm.provision :shell, name: 'Install Packages', inline: <<-EOH
+      set -e
+      # setup /usr/local/bin for non-packaged software
+      if [[ $(egrep -c 'PATH.*/usr/local/bin' /etc/environment) -eq 0 ]]; then
+        echo 'export PATH=${PATH}:/usr/local/bin' >> /etc/environment
+      fi
+
+      # install Java
+      apt-get update
+      apt-get -y install openjdk-8-jdk-headless
+
+      # install Maven
+      mkdir -p /usr/local
+      wget --continue http://apache.mirrors.ionfish.org/maven/maven-3/3.6.0/binaries/apache-maven-3.6.0-bin.tar.gz
+      tar -xzf apache-maven-3.6.0-bin.tar.gz -C /usr/local
+      [ -L /usr/local/bin/mvn ] || ln -s /usr/local/apache-maven-3.6.0/bin/mvn /usr/local/bin/mvn
+
+      # Namazu dependencies
+      apt-get install -y git libnetfilter-queue-dev libzmq3-dev
+      wget --continue https://dl.google.com/go/go1.11.2.linux-amd64.tar.gz
+      tar -xzf go1.11.2.linux-amd64.tar.gz -C /usr/local
+      [ -L /usr/local/bin/go ] || ln -s /usr/local/go/bin/go /usr/local/bin/go
+    EOH
+
+    # download and build Namazu
+    ratisbuild.vm.provision :shell, privileged: false, name: 'Build Namazu', inline: <<-EOH
+      set -e
+      cd ~/
+      [ '!' -d namazu ] && git clone https://github.com/osrg/namazu
+      cd namazu
+      export GOROOT=/usr/local/go
+      export GOPATH=`pwd`
+      # for some reason seelog fails to pull automatically
+      go get -u github.com/cihub/seelog
+      ./build
+    EOH
+
+    # link in and build Ratis
+    config.vm.synced_folder '../..', RATIS_PATH
+
+    ratisbuild.vm.provision :shell, privileged: false, name: 'Build Ratis', inline: <<-EOH
+      set -e
+      # load proxies or other environment specifics loaded via a Vagrantfile.local
+      # or otherwise into /etc/environment
+      . /etc/environment
+      cd #{RATIS_PATH}
+      mvn package -DskipTests
+    EOH
+
+    ratisbuild.vm.provider :virtualbox do |vb|
+      vb.gui = false
+      vb.memory = ENV.fetch 'BUILD_VM_MEM', 2 * 1024
+      vb.cpus = ENV.fetch 'BUILD_VM_CPUs', 1
+      common_vm_settings.call(vb)
+    end
+
+    ratisbuild.vm.box = 'ubuntu/bionic64'
+  end
+
+  # Configure a generic VM with three Ratis servers
+  config.vm.define 'ratis-servers'.to_sym do |server|
+    server.vm.hostname = 'ratis-server'
+    motd = %(Welcome to the Ratis test VM
+             ========================================
+             This VM provides the following:
+             * screen -x -- this will connect you to a GNU Screen session running all three Ratis daemons
+             * clean-up and restart on your hypervisor with: vagrant up --provision ratisserver
+             ========================================
+            )
+    server.vm.provision :shell, name: 'Update MOTD', inline: <<-EOH.gsub(/^\s+/, '').strip
+      set -e
+      cat <<EOF >/etc/motd
+      #{motd.gsub(/^\s+/, ' ').strip}
+      EOF
+    EOH
+
+    # normal test VM spin-up steps
+    common_test_vm_settings.call(server)
+  end
+
+  # Configure a pathological VM with three Ratis servers running on bad disks
+  config.vm.define 'ratis-hdd-slowdown'.to_sym do |hdd|
+    hdd.vm.hostname = 'ratis-hdd-slowdown'
+    motd = %(Welcome to the Ratis flakey disk test VM
+             ========================================
+             This VM provides the following:
+             * screen -x -- this will connect you to a GNU Screen session running all three Ratis daemons
+             * sudo screen -x -- this will connect you to a GNU Screen session running the Namazu fuzzing daemon
+             * clean-up and restart on your hypervisor with: vagrant up --provision ratishddslowdown
+             ========================================
+            )
+
+    hdd.vm.provision :shell, name: 'Update MOTD', inline: <<-EOH.gsub(/^\s+/, '').strip
+      cat <<EOF >/etc/motd
+      #{motd.gsub(/^\s+/, ' ').strip}
+      EOF
+    EOH
+
+    hdd.vm.provision :shell, name: 'Prepare Namazu Daemon', inline: <<-EOH
+      set -e
+      pkill -u root -f namazu || true
+      fusermount -u #{TEST_DATA}/data0_slowed/ || true
+      fusermount -u #{TEST_DATA}/data1_slowed/ || true
+      fusermount -u #{TEST_DATA}/data2_slowed/ || true
+      mkdir -p #{TEST_DATA}/data{0,0_slowed,1,1_slowed,2,2_slowed}
+      chown vagrant #{TEST_DATA}/data{0,0_slowed,1,1_slowed,2,2_slowed}
+    EOH
+
+    hdd.ssh.pty = true
+    # screen wants a tty; use screen(1) for that
+    hdd.vm.provision :shell, name: 'Run Namazu Daemon', inline: <<-EOH
+      script -c "screen -c #{SCREEN_RC_PATH}/namazu_hdd_screenrc" /dev/stdout
+    EOH
+    hdd.ssh.pty = false
+
+    # normal test VM spin-up steps
+    common_test_vm_settings.call(hdd)
+  end
+end
diff --git a/dev-support/vagrant/bin/start_ratis_load_gen.sh b/dev-support/vagrant/bin/start_ratis_load_gen.sh
new file mode 100755
index 0000000..62f21b6
--- /dev/null
+++ b/dev-support/vagrant/bin/start_ratis_load_gen.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Trivial script to call the Ratis example load gen from GNU Screen
+
+HOME=/home/vagrant
+peers=$1
+
+cd ${HOME}/incubator-ratis
+
+# run the load generator
+./ratis-examples/src/main/bin/client.sh filestore loadgen --size 1048576 --numFiles 100 --peers $peers 2>&1 | \
+  tee ${HOME}/loadgen.log
+
+# verify all logs checksum the same
+echo "Verification of all Ratis file server logs have the same checksum across all storage directories:"
+find ${HOME}/test_data/data? -type f -a -name 'file-*' -exec md5sum \{\} \+ | sed 's/ .*//' | sort | uniq -c
diff --git a/dev-support/vagrant/bin/start_ratis_server.sh b/dev-support/vagrant/bin/start_ratis_server.sh
new file mode 100755
index 0000000..6d8bf80
--- /dev/null
+++ b/dev-support/vagrant/bin/start_ratis_server.sh
@@ -0,0 +1,27 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Trivial script to call the Ratis example server from GNU Screen
+
+HOME=/home/vagrant
+storage=$1
+id=$2
+peers=$3
+
+cd ${HOME}/incubator-ratis/
+java -jar `find ./ -name 'ratis-examples*-SNAPSHOT.jar'` filestore server --storage $storage --id $id --peers $peers 2>&1 | \
+  tee ${HOME}/server_${id}.log
diff --git a/dev-support/vagrant/docs/vagrantfile.bpmn b/dev-support/vagrant/docs/vagrantfile.bpmn
new file mode 100644
index 0000000..d8d91e1
--- /dev/null
+++ b/dev-support/vagrant/docs/vagrantfile.bpmn
@@ -0,0 +1,420 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+<bpmn:definitions xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:bpmn="http://www.omg.org/spec/BPMN/20100524/MODEL" xmlns:bpmndi="http://www.omg.org/spec/BPMN/20100524/DI" xmlns:dc="http://www.omg.org/spec/DD/20100524/DC" xmlns:di="http://www.omg.org/spec/DD/20100524/DI" id="Definitions_1sfadzh" targetNamespace="http://bpmn.io/schema/bpmn">
+  <bpmn:process id="Process_1" isExecutable="false">
+    <bpmn:subProcess id="SubProcess_0z7pqrc" name="Build ratis-build VM&#10;">
+      <bpmn:incoming>SequenceFlow_0urd70f</bpmn:incoming>
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_0y3qocg" name="Download Java&#10;">
+        <bpmn:incoming>SequenceFlow_0omwrdp</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_1gsb6y6</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:startEvent id="StartEvent_06uerma" name="vagrant up ratis-build --provision">
+        <bpmn:outgoing>SequenceFlow_0omwrdp</bpmn:outgoing>
+      </bpmn:startEvent>
+      <bpmn:sequenceFlow id="SequenceFlow_0omwrdp" sourceRef="StartEvent_06uerma" targetRef="IntermediateThrowEvent_0y3qocg" />
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_1aqnusk" name="Install Maven&#10;">
+        <bpmn:incoming>SequenceFlow_1gsb6y6</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_0ba1g32</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:sequenceFlow id="SequenceFlow_1gsb6y6" sourceRef="IntermediateThrowEvent_0y3qocg" targetRef="IntermediateThrowEvent_1aqnusk" />
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_0jqkmbb" name="Install Namazu Dependencies">
+        <bpmn:incoming>SequenceFlow_0ba1g32</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_1kbg34m</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:sequenceFlow id="SequenceFlow_0ba1g32" sourceRef="IntermediateThrowEvent_1aqnusk" targetRef="IntermediateThrowEvent_0jqkmbb" />
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_0g8jagu" name="Build Namazu&#10;">
+        <bpmn:incoming>SequenceFlow_1kbg34m</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_1d5wr7n</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:sequenceFlow id="SequenceFlow_1kbg34m" sourceRef="IntermediateThrowEvent_0jqkmbb" targetRef="IntermediateThrowEvent_0g8jagu" />
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_1v6diup" name="Build Ratis&#10;">
+        <bpmn:incoming>SequenceFlow_1d5wr7n</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_0l4bff2</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:sequenceFlow id="SequenceFlow_1d5wr7n" sourceRef="IntermediateThrowEvent_0g8jagu" targetRef="IntermediateThrowEvent_1v6diup" />
+      <bpmn:endEvent id="EndEvent_1si4vk8" name="vagrant halt&#10;ratis-test&#10;">
+        <bpmn:incoming>SequenceFlow_0l4bff2</bpmn:incoming>
+      </bpmn:endEvent>
+      <bpmn:sequenceFlow id="SequenceFlow_0l4bff2" sourceRef="IntermediateThrowEvent_1v6diup" targetRef="EndEvent_1si4vk8" />
+    </bpmn:subProcess>
+    <bpmn:subProcess id="SubProcess_1vox2fm" name="Build ratis-servers VM&#10;">
+      <bpmn:incoming>SequenceFlow_18d45r2</bpmn:incoming>
+      <bpmn:startEvent id="StartEvent_118f43t" name="vagrant up&#10;ratis-servers --provision&#10;">
+        <bpmn:outgoing>SequenceFlow_1p4qi6s</bpmn:outgoing>
+      </bpmn:startEvent>
+      <bpmn:subProcess id="SubProcess_0h4f1v2" name="Common VM Processes&#10;">
+        <bpmn:incoming>SequenceFlow_1p4qi6s</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_0rls8pr</bpmn:outgoing>
+        <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_1gjprfe" name="Run VM specific Ratis launch (via screenrc)&#10;">
+          <bpmn:incoming>SequenceFlow_09emzly</bpmn:incoming>
+        </bpmn:intermediateThrowEvent>
+        <bpmn:sequenceFlow id="SequenceFlow_09emzly" sourceRef="StartEvent_0set2bm" targetRef="IntermediateThrowEvent_1gjprfe" />
+        <bpmn:intermediateThrowEvent id="StartEvent_0set2bm" name="Kill any ratis processes&#10;">
+          <bpmn:outgoing>SequenceFlow_09emzly</bpmn:outgoing>
+        </bpmn:intermediateThrowEvent>
+      </bpmn:subProcess>
+      <bpmn:sequenceFlow id="SequenceFlow_1p4qi6s" sourceRef="StartEvent_118f43t" targetRef="SubProcess_0h4f1v2" />
+      <bpmn:endEvent id="EndEvent_0liwtzf" name="vagrant suspend ratis-servers&#10;">
+        <bpmn:incoming>SequenceFlow_0rls8pr</bpmn:incoming>
+      </bpmn:endEvent>
+      <bpmn:sequenceFlow id="SequenceFlow_0rls8pr" sourceRef="SubProcess_0h4f1v2" targetRef="EndEvent_0liwtzf" />
+      <bpmn:textAnnotation id="TextAnnotation_04s3sts">
+        <bpmn:text>sets Vagrant box to ratis-test &amp; sets memory, CPU</bpmn:text>
+      </bpmn:textAnnotation>
+      <bpmn:association id="Association_1pw72r1" sourceRef="SubProcess_0h4f1v2" targetRef="TextAnnotation_04s3sts" />
+    </bpmn:subProcess>
+    <bpmn:subProcess id="SubProcess_1nxh3nk" name="Build ratis-hdd-slowdown VM&#10;">
+      <bpmn:incoming>SequenceFlow_1lvmnzt</bpmn:incoming>
+      <bpmn:startEvent id="StartEvent_0l3hmxf" name="vagrant up ratis-hdd-slowdown --provision&#10;">
+        <bpmn:outgoing>SequenceFlow_05xfi9i</bpmn:outgoing>
+      </bpmn:startEvent>
+      <bpmn:endEvent id="EndEvent_0h7nqya" name="vagrant suspend ratis-hdd-slowdown&#10;">
+        <bpmn:incoming>SequenceFlow_1rtfyit</bpmn:incoming>
+      </bpmn:endEvent>
+      <bpmn:task id="Task_13zqryo" name="Common VM Processes&#10;">
+        <bpmn:incoming>SequenceFlow_1odg4c0</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_1rtfyit</bpmn:outgoing>
+      </bpmn:task>
+      <bpmn:subProcess id="SubProcess_0brvgq2" name="Prepare Namazu Daemon&#10;">
+        <bpmn:incoming>SequenceFlow_05xfi9i</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_0y93jhx</bpmn:outgoing>
+        <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_01t8cjj" name=" Make test dirs&#10;">
+          <bpmn:outgoing>SequenceFlow_1wxxary</bpmn:outgoing>
+        </bpmn:intermediateThrowEvent>
+        <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_0kuy86e" name="Kill any namazu processes&#10;">
+          <bpmn:incoming>SequenceFlow_1wxxary</bpmn:incoming>
+          <bpmn:outgoing>SequenceFlow_1kpvmtl</bpmn:outgoing>
+        </bpmn:intermediateThrowEvent>
+        <bpmn:sequenceFlow id="SequenceFlow_1wxxary" sourceRef="IntermediateThrowEvent_01t8cjj" targetRef="IntermediateThrowEvent_0kuy86e" />
+        <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_07tr02j" name="Unmount all Fuse mounts&#10;">
+          <bpmn:incoming>SequenceFlow_1kpvmtl</bpmn:incoming>
+        </bpmn:intermediateThrowEvent>
+        <bpmn:sequenceFlow id="SequenceFlow_1kpvmtl" sourceRef="IntermediateThrowEvent_0kuy86e" targetRef="IntermediateThrowEvent_07tr02j" />
+      </bpmn:subProcess>
+      <bpmn:sequenceFlow id="SequenceFlow_05xfi9i" sourceRef="StartEvent_0l3hmxf" targetRef="SubProcess_0brvgq2" />
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_08jwjmk" name="Run namazu (via screenrc)&#10;">
+        <bpmn:incoming>SequenceFlow_0y93jhx</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_1odg4c0</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:sequenceFlow id="SequenceFlow_1odg4c0" sourceRef="IntermediateThrowEvent_08jwjmk" targetRef="Task_13zqryo" />
+      <bpmn:sequenceFlow id="SequenceFlow_1rtfyit" sourceRef="Task_13zqryo" targetRef="EndEvent_0h7nqya" />
+      <bpmn:sequenceFlow id="SequenceFlow_0y93jhx" sourceRef="SubProcess_0brvgq2" targetRef="IntermediateThrowEvent_08jwjmk" />
+    </bpmn:subProcess>
+    <bpmn:subProcess id="SubProcess_003i1wn" name="run_all_tests.sh:build&#10;">
+      <bpmn:outgoing>SequenceFlow_0urd70f</bpmn:outgoing>
+      <bpmn:outgoing>SequenceFlow_18d45r2</bpmn:outgoing>
+      <bpmn:outgoing>SequenceFlow_1lvmnzt</bpmn:outgoing>
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_0n4ow8g" name="Build&#10;ratis-servers VM&#10;">
+        <bpmn:incoming>SequenceFlow_0pqdxj9</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_171shbg</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_1cmdrjz" name="vagrant package ratis-build&#10;">
+        <bpmn:incoming>SequenceFlow_1ormzc8</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_0pqdxj9</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_1pdyawc" name="Build ratis-build VM&#10;">
+        <bpmn:incoming>SequenceFlow_142v0rf</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_1ormzc8</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:startEvent id="StartEvent_1pvbxq8">
+        <bpmn:outgoing>SequenceFlow_142v0rf</bpmn:outgoing>
+      </bpmn:startEvent>
+      <bpmn:endEvent id="EndEvent_0w9o8ng">
+        <bpmn:incoming>SequenceFlow_1kxnv1n</bpmn:incoming>
+      </bpmn:endEvent>
+      <bpmn:intermediateThrowEvent id="IntermediateThrowEvent_0ppv1mn" name="Build ratis-hdd-slowdownVM&#10;">
+        <bpmn:incoming>SequenceFlow_171shbg</bpmn:incoming>
+        <bpmn:outgoing>SequenceFlow_1kxnv1n</bpmn:outgoing>
+      </bpmn:intermediateThrowEvent>
+      <bpmn:sequenceFlow id="SequenceFlow_171shbg" sourceRef="IntermediateThrowEvent_0n4ow8g" targetRef="IntermediateThrowEvent_0ppv1mn" />
+      <bpmn:sequenceFlow id="SequenceFlow_0pqdxj9" sourceRef="IntermediateThrowEvent_1cmdrjz" targetRef="IntermediateThrowEvent_0n4ow8g" />
+      <bpmn:sequenceFlow id="SequenceFlow_1ormzc8" sourceRef="IntermediateThrowEvent_1pdyawc" targetRef="IntermediateThrowEvent_1cmdrjz" />
+      <bpmn:sequenceFlow id="SequenceFlow_142v0rf" sourceRef="StartEvent_1pvbxq8" targetRef="IntermediateThrowEvent_1pdyawc" />
+      <bpmn:sequenceFlow id="SequenceFlow_1kxnv1n" sourceRef="IntermediateThrowEvent_0ppv1mn" targetRef="EndEvent_0w9o8ng" />
+    </bpmn:subProcess>
+    <bpmn:sequenceFlow id="SequenceFlow_0urd70f" sourceRef="SubProcess_003i1wn" targetRef="SubProcess_0z7pqrc" />
+    <bpmn:sequenceFlow id="SequenceFlow_18d45r2" sourceRef="SubProcess_003i1wn" targetRef="SubProcess_1vox2fm" />
+    <bpmn:sequenceFlow id="SequenceFlow_1lvmnzt" sourceRef="SubProcess_003i1wn" targetRef="SubProcess_1nxh3nk" />
+  </bpmn:process>
+  <bpmndi:BPMNDiagram id="BPMNDiagram_1">
+    <bpmndi:BPMNPlane id="BPMNPlane_1" bpmnElement="Process_1">
+      <bpmndi:BPMNShape id="SubProcess_0z7pqrc_di" bpmnElement="SubProcess_0z7pqrc" isExpanded="true">
+        <dc:Bounds x="158" y="-274" width="380" height="293" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_0y3qocg_di" bpmnElement="IntermediateThrowEvent_0y3qocg">
+        <dc:Bounds x="226" y="-71" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="207" y="-26" width="75" height="27" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="StartEvent_06uerma_di" bpmnElement="StartEvent_06uerma">
+        <dc:Bounds x="226" y="-198" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="204" y="-244" width="80" height="27" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_0omwrdp_di" bpmnElement="SequenceFlow_0omwrdp">
+        <di:waypoint x="244" y="-162" />
+        <di:waypoint x="244" y="-71" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_1aqnusk_di" bpmnElement="IntermediateThrowEvent_1aqnusk">
+        <dc:Bounds x="307" y="-198" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="293" y="-220" width="64" height="27" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_1gsb6y6_di" bpmnElement="SequenceFlow_1gsb6y6">
+        <di:waypoint x="262" y="-53" />
+        <di:waypoint x="285" y="-53" />
+        <di:waypoint x="285" y="-180" />
+        <di:waypoint x="307" y="-180" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_0jqkmbb_di" bpmnElement="IntermediateThrowEvent_0jqkmbb">
+        <dc:Bounds x="307" y="-71" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="289" y="-28" width="72" height="27" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_0ba1g32_di" bpmnElement="SequenceFlow_0ba1g32">
+        <di:waypoint x="325" y="-162" />
+        <di:waypoint x="325" y="-71" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_0g8jagu_di" bpmnElement="IntermediateThrowEvent_0g8jagu">
+        <dc:Bounds x="392" y="-198" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="376" y="-218" width="67" height="27" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_1kbg34m_di" bpmnElement="SequenceFlow_1kbg34m">
+        <di:waypoint x="343" y="-53" />
+        <di:waypoint x="368" y="-53" />
+        <di:waypoint x="368" y="-180" />
+        <di:waypoint x="392" y="-180" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_1v6diup_di" bpmnElement="IntermediateThrowEvent_1v6diup">
+        <dc:Bounds x="392" y="-71" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="385" y="-28" width="51" height="27" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_1d5wr7n_di" bpmnElement="SequenceFlow_1d5wr7n">
+        <di:waypoint x="410" y="-162" />
+        <di:waypoint x="410" y="-71" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="EndEvent_1si4vk8_di" bpmnElement="EndEvent_1si4vk8">
+        <dc:Bounds x="464" y="-198" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="462" y="-154" width="57" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_0l4bff2_di" bpmnElement="SequenceFlow_0l4bff2">
+        <di:waypoint x="428" y="-53" />
+        <di:waypoint x="446" y="-53" />
+        <di:waypoint x="446" y="-180" />
+        <di:waypoint x="464" y="-180" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="SubProcess_1vox2fm_di" bpmnElement="SubProcess_1vox2fm" isExpanded="true">
+        <dc:Bounds x="114" y="50" width="468" height="213" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="StartEvent_118f43t_di" bpmnElement="StartEvent_118f43t">
+        <dc:Bounds x="146" y="151" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="132" y="90" width="63" height="53" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="SubProcess_0h4f1v2_di" bpmnElement="SubProcess_0h4f1v2" isExpanded="true">
+        <dc:Bounds x="221" y="103" width="199" height="131" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_1p4qi6s_di" bpmnElement="SequenceFlow_1p4qi6s">
+        <di:waypoint x="182" y="169" />
+        <di:waypoint x="221" y="169" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_1gjprfe_di" bpmnElement="IntermediateThrowEvent_1gjprfe">
+        <dc:Bounds x="342" y="142" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="320" y="185" width="81" height="53" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_09emzly_di" bpmnElement="SequenceFlow_09emzly">
+        <di:waypoint x="277" y="160" />
+        <di:waypoint x="342" y="160" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_16roajq_di" bpmnElement="StartEvent_0set2bm">
+        <dc:Bounds x="241" y="142" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="231" y="185" width="58" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="TextAnnotation_04s3sts_di" bpmnElement="TextAnnotation_04s3sts">
+        <dc:Bounds x="462" y="71" width="100" height="70" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="Association_1pw72r1_di" bpmnElement="Association_1pw72r1">
+        <di:waypoint x="420" y="136" />
+        <di:waypoint x="462" y="122" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="EndEvent_0liwtzf_di" bpmnElement="EndEvent_0liwtzf">
+        <dc:Bounds x="459" y="151" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="436" y="194" width="82" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_0rls8pr_di" bpmnElement="SequenceFlow_0rls8pr">
+        <di:waypoint x="420" y="169" />
+        <di:waypoint x="459" y="169" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="SubProcess_1nxh3nk_di" bpmnElement="SubProcess_1nxh3nk" isExpanded="true">
+        <dc:Bounds x="138" y="284" width="419" height="316" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="StartEvent_0l3hmxf_di" bpmnElement="StartEvent_0l3hmxf">
+        <dc:Bounds x="181" y="366" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="159" y="305" width="80" height="53" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="EndEvent_0h7nqya_di" bpmnElement="EndEvent_0h7nqya">
+        <dc:Bounds x="477" y="341" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="454" y="384" width="82" height="53" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="Task_13zqryo_di" bpmnElement="Task_13zqryo">
+        <dc:Bounds x="349" y="319" width="100" height="80" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="SubProcess_0brvgq2_di" bpmnElement="SubProcess_0brvgq2" isExpanded="true">
+        <dc:Bounds x="189" y="455" width="334" height="120" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_05xfi9i_di" bpmnElement="SequenceFlow_05xfi9i">
+        <di:waypoint x="199" y="402" />
+        <di:waypoint x="199" y="455" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_01t8cjj_di" bpmnElement="IntermediateThrowEvent_01t8cjj">
+        <dc:Bounds x="214" y="475" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="197" y="518" width="72" height="27" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_0kuy86e_di" bpmnElement="IntermediateThrowEvent_0kuy86e">
+        <dc:Bounds x="331" y="475" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="311" y="518" width="76" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_1wxxary_di" bpmnElement="SequenceFlow_1wxxary">
+        <di:waypoint x="250" y="493" />
+        <di:waypoint x="331" y="493" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_07tr02j_di" bpmnElement="IntermediateThrowEvent_07tr02j">
+        <dc:Bounds x="448" y="475" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="424" y="518" width="85" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_1kpvmtl_di" bpmnElement="SequenceFlow_1kpvmtl">
+        <di:waypoint x="367" y="493" />
+        <di:waypoint x="448" y="493" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_08jwjmk_di" bpmnElement="IntermediateThrowEvent_08jwjmk">
+        <dc:Bounds x="275" y="341" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="251" y="309" width="83" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_1odg4c0_di" bpmnElement="SequenceFlow_1odg4c0">
+        <di:waypoint x="311" y="359" />
+        <di:waypoint x="349" y="359" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="SequenceFlow_1rtfyit_di" bpmnElement="SequenceFlow_1rtfyit">
+        <di:waypoint x="449" y="359" />
+        <di:waypoint x="477" y="359" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="SequenceFlow_0y93jhx_di" bpmnElement="SequenceFlow_0y93jhx">
+        <di:waypoint x="356" y="455" />
+        <di:waypoint x="356" y="416" />
+        <di:waypoint x="293" y="416" />
+        <di:waypoint x="293" y="377" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNShape id="SubProcess_003i1wn_di" bpmnElement="SubProcess_003i1wn" isExpanded="true">
+        <dc:Bounds x="-315" y="-83" width="325" height="448" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_0n4ow8g_di" bpmnElement="IntermediateThrowEvent_0n4ow8g">
+        <dc:Bounds x="-189" y="162" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="-282" y="160" width="82" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_1cmdrjz_di" bpmnElement="IntermediateThrowEvent_1cmdrjz">
+        <dc:Bounds x="-189" y="97" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="-284" y="95" width="82" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_1pdyawc_di" bpmnElement="IntermediateThrowEvent_1pdyawc">
+        <dc:Bounds x="-189" y="38" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="-277" y="42" width="73" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="StartEvent_1pvbxq8_di" bpmnElement="StartEvent_1pvbxq8">
+        <dc:Bounds x="-189" y="-23" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="10" y="-160.5" width="52" height="53" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="EndEvent_0w9o8ng_di" bpmnElement="EndEvent_0w9o8ng">
+        <dc:Bounds x="-189" y="299" width="36" height="36" />
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNShape id="IntermediateThrowEvent_0ppv1mn_di" bpmnElement="IntermediateThrowEvent_0ppv1mn">
+        <dc:Bounds x="-189" y="226" width="36" height="36" />
+        <bpmndi:BPMNLabel>
+          <dc:Bounds x="-277" y="224" width="73" height="40" />
+        </bpmndi:BPMNLabel>
+      </bpmndi:BPMNShape>
+      <bpmndi:BPMNEdge id="SequenceFlow_171shbg_di" bpmnElement="SequenceFlow_171shbg">
+        <di:waypoint x="-171" y="198" />
+        <di:waypoint x="-171" y="226" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="SequenceFlow_0pqdxj9_di" bpmnElement="SequenceFlow_0pqdxj9">
+        <di:waypoint x="-171" y="133" />
+        <di:waypoint x="-171" y="162" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="SequenceFlow_1ormzc8_di" bpmnElement="SequenceFlow_1ormzc8">
+        <di:waypoint x="-171" y="74" />
+        <di:waypoint x="-171" y="97" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="SequenceFlow_142v0rf_di" bpmnElement="SequenceFlow_142v0rf">
+        <di:waypoint x="-171" y="13" />
+        <di:waypoint x="-171" y="38" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="SequenceFlow_1kxnv1n_di" bpmnElement="SequenceFlow_1kxnv1n">
+        <di:waypoint x="-171" y="262" />
+        <di:waypoint x="-171" y="299" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="SequenceFlow_0urd70f_di" bpmnElement="SequenceFlow_0urd70f">
+        <di:waypoint x="-152" y="-83" />
+        <di:waypoint x="-152" y="-127" />
+        <di:waypoint x="158" y="-127" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="SequenceFlow_18d45r2_di" bpmnElement="SequenceFlow_18d45r2">
+        <di:waypoint x="10" y="141" />
+        <di:waypoint x="114" y="141" />
+      </bpmndi:BPMNEdge>
+      <bpmndi:BPMNEdge id="SequenceFlow_1lvmnzt_di" bpmnElement="SequenceFlow_1lvmnzt">
+        <di:waypoint x="-152" y="365" />
+        <di:waypoint x="-152" y="442" />
+        <di:waypoint x="138" y="442" />
+      </bpmndi:BPMNEdge>
+    </bpmndi:BPMNPlane>
+  </bpmndi:BPMNDiagram>
+</bpmn:definitions>
diff --git a/dev-support/vagrant/docs/vagrantfile_bpmn.svg b/dev-support/vagrant/docs/vagrantfile_bpmn.svg
new file mode 100644
index 0000000..1d242ce
--- /dev/null
+++ b/dev-support/vagrant/docs/vagrantfile_bpmn.svg
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!--
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+<!-- created with bpmn-js / http://bpmn.io -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="909" height="886" viewBox="-321 -280 909 886" version="1.1"><defs><marker id="sequenceflow-end-white-black-d4t3chy4ucmg80j3tw5bo8x0e" viewBox="0 0 20 20" refX="11" refY="10" markerWidth="10" markerHeight="10" orient="auto"><path d="M 1 5 L 11 10 L 1 15 Z" style="fill: black; stroke-width: 1px; stroke-linecap: round; stroke-dasharray: 10000, 1; stroke: black;"/></marker></defs><g class="djs-group"><g [...]
diff --git a/dev-support/vagrant/namazu_configs/hdd_config.toml b/dev-support/vagrant/namazu_configs/hdd_config.toml
new file mode 100644
index 0000000..892b012
--- /dev/null
+++ b/dev-support/vagrant/namazu_configs/hdd_config.toml
@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Policy for a "flakey" disk to stress performance and recovery
+explorePolicy = "random"
+
+[explorePolicyParam]
+  # for the Filesystem inspector, event will be non-deterministically delayed.
+  # minInterval and maxInterval are bounds for the non-deterministic delays
+  minInterval = "10ms"
+  maxInterval = "300ms"
+
+  # for Filesystem inspectors, you can specify fault-injection probability (0.0-1.0).
+  faultActionProbability = 0.0
diff --git a/dev-support/vagrant/run_all_tests.sh b/dev-support/vagrant/run_all_tests.sh
new file mode 100755
index 0000000..cab97ae
--- /dev/null
+++ b/dev-support/vagrant/run_all_tests.sh
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+testvms="ratis-servers ratis-hdd-slowdown"
+
+box_path=$(dirname ${BASH_SOURCE[0]})/ratistest.box
+
+if [[ $1 == "build" ]]; then
+  # build everything
+  echo "============================================"
+  echo "Building the ratisbuild VM:"
+  echo "============================================"
+  vagrant up ratis-build --provision
+  vagrant halt ratis-build
+  [ '!' -e $box_path ] && vagrant package ratis-build --output $box_path
+
+  echo "============================================"
+  echo "Building the test-suite VMs:"
+  echo "============================================"
+  for vm in $testvms; do
+    echo "============================================"
+    echo "Building test-suite VM: $vm"
+    echo "============================================"
+    vagrant up $vm --provision
+    vagrant suspend $vm
+  done
+  clear
+  echo "============================================"
+  echo "Build complete"
+  echo "Run vagrant resume <vm name> to start a particular environment"
+  echo "Run vagrant ssh <vm name> to enter a particular environment"
+  echo "============================================"
+  vagrant status
+elif [[ $1 == "clean" ]]; then
+  echo "============================================"
+  echo "Cleaning-up all test artifacts"
+  echo "============================================"
+  vagrant destroy -f ratis-build || true
+  for vm in $testvms; do
+    vagrant destroy -f $vm || true
+  done
+  vagrant box remove ratis-test || true
+  rm -f $box_path
+  echo "============================================"
+  echo "Clean-up complete"
+  echo "============================================"
+else
+  echo "$(basename $0): Usage: $(basename $0) build"
+  echo "                       $(basename $0) clean"
+fi
diff --git a/dev-support/vagrant/screenrcs/namazu_hdd_screenrc b/dev-support/vagrant/screenrcs/namazu_hdd_screenrc
new file mode 100644
index 0000000..90281dc
--- /dev/null
+++ b/dev-support/vagrant/screenrcs/namazu_hdd_screenrc
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# huge scrollback buffer
+defscrollback 50000
+
+# no welcome message
+startup_message off
+
+# default windows
+screen -t Disk0 0 /home/vagrant/namazu/bin/nmz inspectors fs -original-dir /home/vagrant/test_data/data0 -mount-point /home/vagrant/test_data/data0_slowed/ -autopilot /home/vagrant/incubator-ratis/dev-support/vagrant/namazu_configs/hdd_config.toml
+screen -t Disk1 0 /home/vagrant/namazu/bin/nmz inspectors fs -original-dir /home/vagrant/test_data/data1 -mount-point /home/vagrant/test_data/data1_slowed/ -autopilot /home/vagrant/incubator-ratis/dev-support/vagrant/namazu_configs/hdd_config.toml
+screen -t Disk2 0 /home/vagrant/namazu/bin/nmz inspectors fs -original-dir /home/vagrant/test_data/data2 -mount-point /home/vagrant/test_data/data2_slowed/ -autopilot /home/vagrant/incubator-ratis/dev-support/vagrant/namazu_configs/hdd_config.toml
+
+autodetach on
+detach
+
+# Don't kill window after the process died
+# 'k' kills window, 'r' restarts
+zombie kr
diff --git a/dev-support/vagrant/screenrcs/ratis_ratis-hdd-slowdown_screenrc b/dev-support/vagrant/screenrcs/ratis_ratis-hdd-slowdown_screenrc
new file mode 100644
index 0000000..c51091d
--- /dev/null
+++ b/dev-support/vagrant/screenrcs/ratis_ratis-hdd-slowdown_screenrc
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# huge scrollback buffer
+defscrollback 50000
+
+# no welcome message
+startup_message off
+
+# default windows
+screen -t Server0 0 /home/vagrant/incubator-ratis/dev-support/vagrant/bin/start_ratis_server.sh /home/vagrant/test_data/data0_slowed n0 n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
+screen -t Server1 1 /home/vagrant/incubator-ratis/dev-support/vagrant/bin/start_ratis_server.sh /home/vagrant/test_data/data1_slowed n1 n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
+screen -t Server2 2 /home/vagrant/incubator-ratis/dev-support/vagrant/bin/start_ratis_server.sh /home/vagrant/test_data/data2_slowed n2 n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
+screen -t LoadGen 3 /home/vagrant/incubator-ratis/dev-support/vagrant/bin/start_ratis_load_gen.sh n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
+
+autodetach on
+detach
+
+# Don't kill window after the process died
+# 'k' kills window, 'r' restarts
+zombie kr
diff --git a/dev-support/vagrant/screenrcs/ratis_ratis-server_screenrc b/dev-support/vagrant/screenrcs/ratis_ratis-server_screenrc
new file mode 100644
index 0000000..1829f1e
--- /dev/null
+++ b/dev-support/vagrant/screenrcs/ratis_ratis-server_screenrc
@@ -0,0 +1,33 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# huge scrollback buffer
+defscrollback 50000
+
+# no welcome message
+startup_message off
+
+# default windows
+screen -t Server0 0 /home/vagrant/incubator-ratis/dev-support/vagrant/bin/start_ratis_server.sh /home/vagrant/test_data/data0 n0 n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
+screen -t Server1 1 /home/vagrant/incubator-ratis/dev-support/vagrant/bin/start_ratis_server.sh /home/vagrant/test_data/data1 n1 n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
+screen -t Server2 2 /home/vagrant/incubator-ratis/dev-support/vagrant/bin/start_ratis_server.sh /home/vagrant/test_data/data2 n2 n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
+screen -t LoadGen 3 /home/vagrant/incubator-ratis/dev-support/vagrant/bin/start_ratis_load_gen.sh n0:127.0.0.1:6000,n1:127.0.0.1:6001,n2:127.0.0.1:6002
+
+autodetach on
+detach
+
+# Don't kill window after the process died
+# 'k' kills window, 'r' restarts
+zombie kr