You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@commons.apache.org by ah...@apache.org on 2019/10/23 16:57:09 UTC
[commons-rng] 05/09: Updated stress test documentation for
PractRand.
This is an automated email from the ASF dual-hosted git repository.
aherbert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/commons-rng.git
commit 9307eacc97d9f54299afda41639349b343eaac9e
Author: aherbert <ah...@apache.org>
AuthorDate: Wed Oct 23 14:18:46 2019 +0100
Updated stress test documentation for PractRand.
---
.../examples-stress/stress_test.md | 140 +++++++++++++++++----
1 file changed, 117 insertions(+), 23 deletions(-)
diff --git a/commons-rng-examples/examples-stress/stress_test.md b/commons-rng-examples/examples-stress/stress_test.md
index dbb0d90..898d7a8 100644
--- a/commons-rng-examples/examples-stress/stress_test.md
+++ b/commons-rng-examples/examples-stress/stress_test.md
@@ -19,7 +19,8 @@ Apache Commons RNG Stress Test Example
======================================
The stress test module contains an application for calling external tools that perform stringent
-uniformity tests. The following shows an example of how to run **Dieharder** and **TestU01**.
+uniformity tests. The following shows an example of how to run **Dieharder**, **TestU01** and
+**PractRand**.
Installation on Linux/MacOS
---------------------------
@@ -32,20 +33,20 @@ This assumes that the source code has been checked out and the current working d
This can be installed from [TestU01](http://simul.iro.umontreal.ca/testu01/tu01.html) or using
the available packages (linux):
- > apt-get install libtestu01 libtestu01-dev testu01-bin
+ apt-get install libtestu01 libtestu01-dev testu01-bin
If installing **TestU01** from source then it is recommended to do this using an install directory
to avoid header clashes as the install puts headers directly into `<install directory>/include`:
- > ./configure --prefix=/usr/local/testu01
- > make
- > make install
+ ./configure --prefix=/usr/local/testu01
+ make
+ make install
A simple bridge is provided to read the integers from `stdin` and pass them to the **TestU01** library.
This can be compiled using one of the following (depending on the library names):
- > gcc src/main/c/stdin2testu01.c -o stdin2testu01 -ltestu01 -lprobdist -lmylib -lm
- > gcc src/main/c/stdin2testu01.c -o stdin2testu01 -ltestu01 \
+ gcc src/main/c/stdin2testu01.c -o stdin2testu01 -ltestu01 -lprobdist -lmylib -lm
+ gcc src/main/c/stdin2testu01.c -o stdin2testu01 -ltestu01 \
-ltestu01probdist -ltestu01mylib -lm
Note: If **TestU01** was installed from source then the `stdin2testu01.c` may have to be modified to
@@ -55,7 +56,7 @@ use different headers. Open the source and update to:
Then compile with the `<install directory>/include` and `lib` directories:
- > gcc src/main/c/stdin2testu01.c -o stdin2testu01 \
+ gcc src/main/c/stdin2testu01.c -o stdin2testu01 \
-I/usr/local/testu01/include \
-L/usr/local/testu01/lib \
-ltestu01 -lprobdist -lmylib -lm
@@ -65,12 +66,40 @@ Then compile with the `<install directory>/include` and `lib` directories:
This can be installed from [Dieharder](http://webhome.phy.duke.edu/~rgb/General/dieharder.php) or
using the available packages:
- > apt-get install dieharder
- > brew install dieharder
+ apt-get install dieharder
+ brew install dieharder
The `dieharder` executable can read integers from `stdin` for analysis so no additional steps are
required.
+### PractRand
+
+This must be installed using the source from [PractRand](http://pracrand.sourceforge.net/). These
+instructions apply to `v0.93` on linux. The latest stable build `v0.94` does not compile the
+`RNG_test` source.
+
+`PractRand` prints results to standard output. Version 0.93 did not flush the output after
+reporting test results. This prevents following the latest results and can cause
+missing results if the test aborts. This can happen due to memory allocation failures on long
+running tests which can use gigabytes of memory. A patch file is provided
+[RNG_test_0.93.patch](./src/main/resources/patch/RNG_test_0.93.patch) to flush the output.
+This can be applied from the root directory of the PractRand v0.93 source archive:
+
+ patch -p0 < RNG_test_0.93.patch
+
+Flushing of test results to output has been corrected in version 0.94+.
+
+Building **PractRand** for multi-threaded support:
+
+ g++ -std=c++14 -c src/*.cpp src/RNGs/*.cpp src/RNGs/other/*.cpp -O3 -Iinclude -pthread
+ ar rcs libPractRand.a *.o
+ rm -f *.o
+ g++ -std=c++14 -o RNG_test tools/RNG_test.cpp libPractRand.a -O3 -Iinclude -pthread
+
+Note: Earlier compilers may substitute `-std=c++14` for `-std=c++11`.
+
+The `RNG_test` tool is used to accept raw bits via standard input for testing.
+
Test platform Endianness
------------------------
@@ -84,6 +113,12 @@ The **Dieharder** application and `stdin2testu01` bridge application for the **T
test suites read binary data using a c-library function that requires the data to be in the native
byte order. These test suites will be supported automatically.
+The **PractRand** application will read binary data using the `stdin32` or `stdin64` arguments
+for 32-bit or 64-bit data. **PractRand** defaults use smart testing based on the RNG metadata to
+put special emphasis on testing the least significant bits of the input data and so 64-bit
+generators should be tested using `stdin64`. This also requires that the Commons RNG stress test
+application is run in 64-bit output mode.
+
If using a alternative test suite the endian format expected by the test suite must be verified.
More details of how to do this are provided in the [endianness](./endianness.md) page.
@@ -92,19 +127,21 @@ Running on Linux/MacOS
Build the `examples-stress` jar file:
- > mvn package -P examples-stress
+ mvn package -P examples-stress
This will create a single jar file with all the dependencies in the `target` directory.
The jar file is executable and provides usage information with the `--help` or `-h` option:
- > java -jar target/examples-stress.jar -h
+ java -jar target/examples-stress.jar -h
The list of generators that can be tested can be customised. The default is to use all known
generators that do not require arguments in addition to a random seed. To create a template list
use the `list` command:
- > java -jar target/examples-stress.jar list
+ java -jar target/examples-stress.jar list
+
+This will produce a list of generators:
# ID RandomSource trials [constructor arguments ...]
1 JDK 1
@@ -119,32 +156,89 @@ To run the stress test use the `stress` command and provide the following argume
| Argument | Description | Example |
| --------- | ----------- | ------- |
-| executable | The test tool | dieharder, stdin2testu01 |
-| ... | Arguments for the test tool | dieharder: -a -g 200 -Y 1 -k 2 <br/> stdin2testu01: SmallCrush, Crush, BigCrush |
+| executable | The test tool | dieharder, stdin2testu01, PractRand |
+| ... | Arguments for the test tool | dieharder: -a -g 200 -Y 1 -k 2 <br/> stdin2testu01: SmallCrush, Crush, BigCrush <br/> RNG_test stdin32 -tf 1 -te 0 -tlmin 1KB -tlmax 4TB |
-The command sets defaults for other options that can be changed, for example the output results
-file prefix, the number of concurrent tasks, byte-order or the number of trials per generator.
+The `stress` command defaults for other options that can be changed, for example the output
+results file prefix, the number of concurrent tasks, byte-order or the number of trials per
+generator.
Use the `--help` option to show the available options.
### TestU01
- > java -jar target/examples-stress.jar stress \
+ java -jar target/examples-stress.jar stress \
--prefix target/tu_ \
./stdin2testu01 \
BigCrush
### Dieharder
- > java -jar target/examples-stress.jar stress \
+ java -jar target/examples-stress.jar stress \
--prefix target/dh_ \
/usr/bin/dieharder \
-a -g 200 -Y 1 -k 2
+### PractRand
+
+**PractRand** must be run separately for 32-bit or 64-bit generators. This requires a subset of
+each to be specified in a list. For example to perform 3 trials of each generator:
+
+ java -jar target/examples-stress.jar list -t 3 \
+ --provider int > rng32.list
+ java -jar target/examples-stress.jar stress \
+ -l rng32.list \
+ --prefix target/pr_ \
+ --threads 3 \
+ RNG_test stdin32 \
+ -tf 1 -te 0 -tlmin 1 KB -tl max 4TB -multithreaded
+
+ java -jar target/examples-stress.jar list -t 3 \
+ --provider long > rng64.list
+ java -jar target/examples-stress.jar stress \
+ -l rng64.list \
+ --raw64
+ --prefix target/pr_ \
+ --threads 3 \
+ RNG_test stdin64 \
+ -tf 1 -te 0 -tlmin 1 KB -tl max 4TB -multithreaded
+
+Note the use of `--raw64` and ``threads 3` for the `stress` command and the `stdin32` or `stdin64` and `-multithreaded` arguments to **PractRand**.
+
+**PractRand** runs for a long time as the RNG output is tested in doubling sizes starting from 1 KB.
+The default maximum is 32 TB of RNG output. At each doubling of data an interim test analysis is
+performed. The tests compare the current aggregated output to previous tests. Thus memory
+consumption grows with the test duration. Trials have shown the following:
+
+| Argument | Max memory (KB) |
+| --------- | ---------- |
+| stdin32 -tf 1 -te 0 -tlmax 1TB | 732792 |
+| stdin32 -tf 2 -te 0 -tlmax 1TB | 1271424 |
+| stdin64 -tf 1 -te 0 -tlmax 1TB | 800148 |
+| stdin64 -tf 2 -te 0 -tlmax 1TB | 1304116 |
+| stdin32 -tf 1 -te 0 -tlmax 4TB | ? |
+| stdin32 -tf 2 -te 0 -tlmax 4TB | 2314232 |
+| stdin64 -tf 1 -te 0 -tlmax 4TB | ? |
+| stdin64 -tf 2 -te 0 -tlmax 4TB | ? |
+
+Given the large memory consumption on long running tests the machine resources may be better used
+when RAM limited by running **PractRand** multi-threaded instead of running more single-threaded
+concurrent tasks.
+**PractRand** -multithreaded mode has an approximate 3-fold speed up although thread use fluctuates
+from 1 to 5 threads. Allowing for an extra thread to control the java RNG this requires
+approximately 3 threads per **PractRand** and 4 in total per test task. By default the `stress`
+command will use 1 thread per test application. The number of threads can be increased using the
+``--threads` option. Note that the `stress` command cannot allocate threads to the process.
+Instead it adjusts the number of concurrent tasks it will execute as
+ceil(processors / threadsPerTask).
+
+### Viewing results
+
The output results can be viewed using the `results` command:
- > java -jar target/examples-stress.jar results \
+ java -jar target/examples-stress.jar results \
target/tu_* \
- target/dh_*
+ target/dh_* \
+ target/pr_* \
Various formats are available. Use the `--help` option to show the available options.
@@ -160,13 +254,13 @@ byte-reversed if desired.
The `list` command can output available generators by provider type. For example to output the
64-bit providers to a `rng64.list` file:
- > java -jar target/examples-stress.jar list \
+ java -jar target/examples-stress.jar list \
--provider long > rng64.list
The list can then be used to test 64-bit providers, for example to test the lower 32-bits in reverse
order using **BigCrush**:
- > java -jar target/examples-stress.jar stress \
+ java -jar target/examples-stress.jar stress \
--list rng64.list \
--low-bits \
--reverse-bits \