You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@commons.apache.org by ah...@apache.org on 2019/10/23 16:57:11 UTC
[commons-rng] 07/09: Update endianness documentation.
This is an automated email from the ASF dual-hosted git repository.
aherbert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/commons-rng.git
commit 3c6751af94d1221a3f04222bd85b5a0d2ddb04a6
Author: aherbert <ah...@apache.org>
AuthorDate: Wed Oct 23 15:29:07 2019 +0100
Update endianness documentation.
---
commons-rng-examples/examples-stress/endianness.md | 110 +++++++++++++++++----
1 file changed, 90 insertions(+), 20 deletions(-)
diff --git a/commons-rng-examples/examples-stress/endianness.md b/commons-rng-examples/examples-stress/endianness.md
index be4e570..56567f8 100644
--- a/commons-rng-examples/examples-stress/endianness.md
+++ b/commons-rng-examples/examples-stress/endianness.md
@@ -18,16 +18,20 @@
Apache Commons RNG Stress Test Endianness
=========================================
-The stress test application will output raw binary data for generated integers. An integer is 4
-bytes and thus the byte order or [Endianness](https://en.wikipedia.org/wiki/Endianness) of the data
+The stress test application will output raw binary data for generated numbers. An integer is 4
+bytes and a long is 8 bytes and thus the byte order or
+[Endianness](https://en.wikipedia.org/wiki/Endianness) of the data
must be correct for the test application. The stress test application can support either big-endian
or little-endian format. The application will auto-detect the platform and will default to output
binary data using the native byte order.
You can determine the native platform format using the stress test application `endian` command:
- > mvn package -P examples-stress
- > java -jar target/examples-stress.jar endian
+ mvn package -P examples-stress
+ java -jar target/examples-stress.jar endian
+
+This will output:
+
[INFO] LITTLE_ENDIAN
The **Dieharder** application and `stdin2testu01` bridge application for the **TestU01**
@@ -36,7 +40,9 @@ byte order. These test suites will be supported automatically.
The following sections show examples of verifying the endianness for a test application. The first
uses **Dieharder** which can read test data in both text and binary format. The second uses the
-`stdin2testu01` bridge application to verify data passed to an application using a C API is correct.
+`stdin2testu01` bridge application to verify data passed to an application using a C API is
+correct. The third verifies that any data tests using the `output` command are applicable to
+the `stress` command.
Testing using file output
-------------------------
@@ -47,17 +53,17 @@ application to demonstrate the result.
Build the `examples-stress` jar file:
- > mvn package -P examples-stress
+ mvn package -P examples-stress
This will create a single jar file with all the dependencies in the `target` directory. The program
contains a simple command to output data from a random generator. To output data in both text
and binary format use the following commands:
- > java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
+ java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
-f DIEHARDER -o test.dh
- > java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
+ java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
-f BINARY -o test.big -b big_endian
- > java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
+ java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
-f BINARY -o test.little -b little_endian
This should produce the following output files:
@@ -70,9 +76,9 @@ This should produce the following output files:
The data can then be used to run a test within **Dieharder**:
- > dieharder -g 202 -d 0 -f test.dh
- > dieharder -g 201 -d 0 -f test.little
- > dieharder -g 201 -d 0 -f test.big
+ dieharder -g 202 -d 0 -f test.dh
+ dieharder -g 201 -d 0 -f test.little
+ dieharder -g 201 -d 0 -f test.big
Only one of the raw binary files should match the result from the text file. This is the correct
endianness.
@@ -83,17 +89,17 @@ written direct to dieharder. For the birthdays test (`-d 0` option) 20,000,000 s
In the following example the stress test application directly writes to `stdout` which is then
piped to the `dieharder` application which reads using `stdin` (`-g 200` option):
- > java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
+ java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
-f DIEHARDER -o test.dh
- > dieharder -g 202 -d 0 -f test.dh
- > java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
+ dieharder -g 202 -d 0 -f test.dh
+ java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
-f BINARY -b little_endian | dieharder -g 200 -d 0
If the results are not the same then the second command can be repeated with `-b big_endian`.
Remember to clean up the output files:
- > rm -f test.dh test.big test.little
+ rm -f test.dh test.big test.little
Testing using the Java to c bridge
----------------------------------
@@ -105,7 +111,7 @@ with a `c` language API.
Build the `examples-stress` jar file:
- > mvn package -P examples-stress
+ mvn package -P examples-stress
This will create a single jar file with all the dependencies in the `target` directory. The program
contains a simple test for the bridge between the Java application and a sub-process. This will
@@ -119,7 +125,7 @@ application are provided in the [stress test](./stress_test.md) page.
The platform endianess is auto-detected by the Java application. To run the bridge test
use the following command:
- > java -jar target/examples-stress.jar bridge ./stdin2testu01 raw32
+ java -jar target/examples-stress.jar bridge ./stdin2testu01 raw32
This should produce the following output files:
@@ -149,7 +155,7 @@ If the data has been correctly read the `bridge.data` and `bridge.out` should ma
If the endianess is incorrect then the data sent by the Java application will not match the
data read by the sub-process. For example to swap the endianness use the `-b` option:
- > java -jar target/examples-stress.jar bridge -b BIG_ENDIAN ./stdin2testu01 raw32
+ java -jar target/examples-stress.jar bridge -b BIG_ENDIAN ./stdin2testu01 raw32
In this case the little-endian plaform has been sent big-endian data and the contents of the
`bridge.out` file begin with the reverse byte order:
@@ -164,4 +170,68 @@ values do not match; the sub-process has incorrectly received the data.
Remember to clean up the output files:
- > rm -f bridge.data bridge.out bridge.err
+ rm -f bridge.data bridge.out bridge.err
+
+Testing the stress command
+--------------------------
+
+The `stress` command will open the provided application and send binary data. The data transfer
+modes should match the equivalent modes in the `output` command. However the `output` command
+can be used to output more formats and used to test data transfer to a test application that
+accepts different input (such as the **Dieharder** text and binary format).
+
+To verify that the `stress` command is sending the same data as the `output` command requires
+piping data from each to a program. This example uses `stdin2testu01` with the raw mode.
+
+Create a list for the `stress` command:
+
+ java -jar target/examples-stress.jar list | grep SPLIT_MIX_64 > rng.list
+
+Run the `stress` command:
+
+ java -jar target/examples-stress.jar stress -l rng.list \
+ -x 012345 \
+ --prefix stress_ \
+ -o overwrite \
+ ./stdin2testu01 raw32 10
+
+The `overwrite` mode will replace existing files and is used here to allow repeat testing.
+
+Run the equivalent `output` command:
+
+ java -jar target/examples-stress.jar output SPLIT_MIX_64 \
+ -x 012345 \
+ -f binary \
+ -b little_endian \
+ | ./stdin2testu01 raw32 10 > output.data
+
+Note the use of the same hex seed with the `-x` option.
+
+Compare the data in `stress_10_1` and `output.data`.
+The number data in the file should be identical if the endianness was correctly configured.
+
+ diff stress_10_1 output.data
+
+The only difference should be the the header and footer added by the `stress` command
+to the results file. If the endianness is incorrect for the `output` command then the number
+data will not match. Note that the `output` command by default uses big endian for consistency
+across all platforms.
+
+An equivalent 64-bit output would use the `--raw64` option for each command:
+
+ java -jar target/examples-stress.jar stress -l rng.list \
+ -x 012345 \
+ --prefix stress_ \
+ -o overwrite \
+ --raw64 \
+ ./stdin2testu01 raw64 10
+ java -jar target/examples-stress.jar output SPLIT_MIX_64 \
+ -x 012345 \
+ -f binary \
+ -b little_endian \
+ --raw64 \
+ | ./stdin2testu01 raw64 10 > output.data
+
+Remember to clean up the output files:
+
+ rm -f rng.list stress_10_1 output.data