You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@commons.apache.org by ah...@apache.org on 2022/03/30 10:14:14 UTC
[commons-rng] branch master updated (92a6e97 -> ea379ad)
This is an automated email from the ASF dual-hosted git repository.
aherbert pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/commons-rng.git.
from 92a6e97 Add missing <p> tags
new 3c4184b RNG-169: Avoid long to int[] that can generate a zero output seed.
new cc6b0c0 Correct longAsInt output in the stress test application
new 34b22b3 RNG-173: BaseProvider static method to extend an input array seed
new ea379ad Consolidate Int/LongProvider to use next() as the source of random bits
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/commons/rng/core/BaseProvider.java | 147 ++++++++++++++
.../commons/rng/core/source32/IntProvider.java | 8 +-
.../commons/rng/core/source64/LongProvider.java | 6 +-
.../apache/commons/rng/core/BaseProviderTest.java | 104 ++++++++++
commons-rng-examples/examples-stress/endianness.md | 20 +-
.../rng/examples/stress/BridgeTestCommand.java | 4 +-
.../commons/rng/examples/stress/OutputCommand.java | 47 +++--
.../commons/rng/examples/stress/RNGUtils.java | 218 +++++++++++++++++----
.../commons/rng/examples/stress/RngDataOutput.java | 112 ++++++++++-
.../{package-info.java => Source64Mode.java} | 25 ++-
.../rng/examples/stress/StressTestCommand.java | 70 ++++---
.../commons/rng/examples/stress/RNGUtilsTest.java | 118 +++++++++++
.../rng/examples/stress/RngDataOutputTest.java | 93 ++++++++-
.../examples-stress/stress_test.md | 43 ++--
.../commons/rng/simple/internal/Conversions.java | 8 +-
.../rng/simple/internal/ConversionsTest.java | 47 ++++-
src/changes/changes.xml | 4 +
src/main/resources/pmd/pmd-ruleset.xml | 7 +
18 files changed, 938 insertions(+), 143 deletions(-)
copy commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/{package-info.java => Source64Mode.java} (65%)
create mode 100644 commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RNGUtilsTest.java
[commons-rng] 03/04: RNG-173: BaseProvider static method to extend an input array seed
Posted by ah...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aherbert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/commons-rng.git
commit 34b22b32c9f854b94409f8981a14ccf13333c37f
Author: Alex Herbert <ah...@apache.org>
AuthorDate: Mon Mar 28 21:09:01 2022 +0100
RNG-173: BaseProvider static method to extend an input array seed
---
.../org/apache/commons/rng/core/BaseProvider.java | 147 +++++++++++++++++++++
.../apache/commons/rng/core/BaseProviderTest.java | 104 +++++++++++++++
src/changes/changes.xml | 4 +
src/main/resources/pmd/pmd-ruleset.xml | 7 +
4 files changed, 262 insertions(+)
diff --git a/commons-rng-core/src/main/java/org/apache/commons/rng/core/BaseProvider.java b/commons-rng-core/src/main/java/org/apache/commons/rng/core/BaseProvider.java
index 9313a3f..f023fa2 100644
--- a/commons-rng-core/src/main/java/org/apache/commons/rng/core/BaseProvider.java
+++ b/commons-rng-core/src/main/java/org/apache/commons/rng/core/BaseProvider.java
@@ -17,6 +17,7 @@
package org.apache.commons.rng.core;
+import java.util.Arrays;
import org.apache.commons.rng.RestorableUniformRandomProvider;
import org.apache.commons.rng.RandomProviderState;
@@ -29,6 +30,16 @@ public abstract class BaseProvider
private static final String NOT_POSITIVE = "Must be strictly positive: ";
/** 2^32. */
private static final long POW_32 = 1L << 32;
+ /**
+ * The fractional part of the the golden ratio, phi, scaled to 64-bits and rounded to odd.
+ * <pre>
+ * phi = (sqrt(5) - 1) / 2) * 2^64
+ * </pre>
+ * @see <a href="https://en.wikipedia.org/wiki/Golden_ratio">Golden ratio</a>
+ */
+ private static final long GOLDEN_RATIO_64 = 0x9e3779b97f4a7c15L;
+ /** The fractional part of the the golden ratio, phi, scaled to 32-bits and rounded to odd. */
+ private static final int GOLDEN_RATIO_32 = 0x9e3779b9;
/** {@inheritDoc} */
@Override
@@ -320,4 +331,140 @@ public abstract class BaseProvider
// Code inspired from "AbstractWell" class.
return scramble(n, 1812433253L, 30, add);
}
+
+ /**
+ * Extend the seed to the specified minimum length. If the seed is equal or greater than the
+ * minimum length, return the same seed unchanged. Otherwise:
+ * <ol>
+ * <li>Create a new array of the specified length
+ * <li>Copy all elements of the seed into the array
+ * <li>Fill the remaining values. The additional values will have at most one occurrence
+ * of zero. If the original seed is all zero, the first extended value will be non-zero.
+ * </li>
+ * </ol>
+ *
+ * <p>This method can be used in constructors that must pass their seed to the super class
+ * to avoid a duplication of seed expansion to the minimum length required by the super class
+ * and the class:
+ * <pre>
+ * public RNG extends AnotherRNG {
+ * public RNG(long[] seed) {
+ * super(seed = extendSeed(seed, SEED_SIZE));
+ * // Use seed for additional state ...
+ * }
+ * }
+ * </pre>
+ *
+ * <p>Note using the state filling procedure provided in {@link #fillState(long[], long[])}
+ * is not possible as it is an instance method. Calling a seed extension routine must use a
+ * static method.
+ *
+ * <p>This method functions as if the seed has been extended using a
+ * {@link org.apache.commons.rng.core.source64.SplitMix64 SplitMix64}
+ * generator seeded with {@code seed[0]}, or zero if the input seed length is zero.
+ * <pre>
+ * if (seed.length < length) {
+ * final long[] s = Arrays.copyOf(seed, length);
+ * final SplitMix64 rng = new SplitMix64(s[0]);
+ * for (int i = seed.length; i < length; i++) {
+ * s[i] = rng.nextLong();
+ * }
+ * return s;
+ * }</pre>
+ *
+ * @param seed Input seed
+ * @param length The minimum length
+ * @return the seed
+ */
+ protected static long[] extendSeed(long[] seed, int length) {
+ if (seed.length < length) {
+ final long[] s = Arrays.copyOf(seed, length);
+ // Fill the rest as if using a SplitMix64 RNG
+ long x = s[0];
+ for (int i = seed.length; i < length; i++) {
+ s[i] = stafford13(x += GOLDEN_RATIO_64);
+ }
+ return s;
+ }
+ return seed;
+ }
+
+ /**
+ * Extend the seed to the specified minimum length. If the seed is equal or greater than the
+ * minimum length, return the same seed unchanged. Otherwise:
+ * <ol>
+ * <li>Create a new array of the specified length
+ * <li>Copy all elements of the seed into the array
+ * <li>Fill the remaining values. The additional values will have at most one occurrence
+ * of zero. If the original seed is all zero, the first extended value will be non-zero.
+ * </li>
+ * </ol>
+ *
+ * <p>This method can be used in constructors that must pass their seed to the super class
+ * to avoid a duplication of seed expansion to the minimum length required by the super class
+ * and the class:
+ * <pre>
+ * public RNG extends AnotherRNG {
+ * public RNG(int[] seed) {
+ * super(seed = extendSeed(seed, SEED_SIZE));
+ * // Use seed for additional state ...
+ * }
+ * }
+ * </pre>
+ *
+ * <p>Note using the state filling procedure provided in {@link #fillState(int[], int[])}
+ * is not possible as it is an instance method. Calling a seed extension routine must use a
+ * static method.
+ *
+ * <p>This method functions as if the seed has been extended using a
+ * {@link org.apache.commons.rng.core.source64.SplitMix64 SplitMix64}-style 32-bit
+ * generator seeded with {@code seed[0]}, or zero if the input seed length is zero. The
+ * generator uses the 32-bit mixing function from MurmurHash3.
+ *
+ * @param seed Input seed
+ * @param length The minimum length
+ * @return the seed
+ */
+ protected static int[] extendSeed(int[] seed, int length) {
+ if (seed.length < length) {
+ final int[] s = Arrays.copyOf(seed, length);
+ // Fill the rest as if using a SplitMix64-style RNG for 32-bit output
+ int x = s[0];
+ for (int i = seed.length; i < length; i++) {
+ s[i] = murmur3(x += GOLDEN_RATIO_32);
+ }
+ return s;
+ }
+ return seed;
+ }
+
+ /**
+ * Perform variant 13 of David Stafford's 64-bit mix function.
+ * This is the mix function used in the {@link SplitMix64} RNG.
+ *
+ * <p>This is ranked first of the top 14 Stafford mixers.
+ *
+ * @param x the input value
+ * @return the output value
+ * @see <a href="http://zimbry.blogspot.com/2011/09/better-bit-mixing-improving-on.html">Better
+ * Bit Mixing - Improving on MurmurHash3's 64-bit Finalizer.</a>
+ */
+ private static long stafford13(long x) {
+ x = (x ^ (x >>> 30)) * 0xbf58476d1ce4e5b9L;
+ x = (x ^ (x >>> 27)) * 0x94d049bb133111ebL;
+ return x ^ (x >>> 31);
+ }
+
+ /**
+ * Perform the finalising 32-bit mix function of Austin Appleby's MurmurHash3.
+ *
+ * @param x the input value
+ * @return the output value
+ * @see <a href="https://github.com/aappleby/smhasher">SMHasher</a>
+ */
+ private static int murmur3(int x) {
+ x = (x ^ (x >>> 16)) * 0x85ebca6b;
+ x = (x ^ (x >>> 13)) * 0xc2b2ae35;
+ return x ^ (x >>> 16);
+ }
}
diff --git a/commons-rng-core/src/test/java/org/apache/commons/rng/core/BaseProviderTest.java b/commons-rng-core/src/test/java/org/apache/commons/rng/core/BaseProviderTest.java
index 1eeaac6..491caba 100644
--- a/commons-rng-core/src/test/java/org/apache/commons/rng/core/BaseProviderTest.java
+++ b/commons-rng-core/src/test/java/org/apache/commons/rng/core/BaseProviderTest.java
@@ -17,6 +17,13 @@
package org.apache.commons.rng.core;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import java.util.Arrays;
+import java.util.SplittableRandom;
+
+import org.apache.commons.rng.core.source64.SplitMix64;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Assumptions;
@@ -128,6 +135,103 @@ class BaseProviderTest {
}
/**
+ * Test a seed can be extended to a required size by filling with a SplitMix64 generator.
+ */
+ @ParameterizedTest
+ @ValueSource(ints = {0, 1, 2, 4, 5, 6, 7, 8, 9})
+ void testExpnadSeedLong(int length) {
+ // The seed does not matter.
+ // Create random seeds that are smaller or larger than length.
+ final SplittableRandom rng = new SplittableRandom();
+ for (long[] seed : new long[][] {
+ {},
+ rng.longs(1).toArray(),
+ rng.longs(2).toArray(),
+ rng.longs(3).toArray(),
+ rng.longs(4).toArray(),
+ rng.longs(5).toArray(),
+ rng.longs(6).toArray(),
+ rng.longs(7).toArray(),
+ rng.longs(8).toArray(),
+ rng.longs(9).toArray(),
+ }) {
+ Assertions.assertArrayEquals(expandSeed(length, seed),
+ BaseProvider.extendSeed(seed, length));
+ }
+ }
+
+ /**
+ * Expand the seed to the minimum specified length using a {@link SplitMix64} generator
+ * seeded with {@code seed[0]}, or zero if the seed length is zero.
+ *
+ * @param length the length
+ * @param seed the seed
+ * @return the seed
+ */
+ private static long[] expandSeed(int length, long... seed) {
+ if (seed.length < length) {
+ final long[] s = Arrays.copyOf(seed, length);
+ final SplitMix64 rng = new SplitMix64(s[0]);
+ for (int i = seed.length; i < length; i++) {
+ s[i] = rng.nextLong();
+ }
+ return s;
+ }
+ return seed;
+ }
+
+ /**
+ * Test a seed can be extended to a required size by filling with a SplitMix64-style
+ * generator using MurmurHash3's 32-bit mix function.
+ *
+ * <p>There is no reference RNG for this output. The test uses fixed output computed
+ * from the reference c++ function in smhasher.
+ */
+ @ParameterizedTest
+ @ValueSource(ints = {0, 1, 2, 4, 5, 6, 7, 8, 9})
+ void testExpandSeedInt(int length) {
+ // Reference output from the c++ function fmix32(uint32_t) in smhasher.
+ // https://github.com/aappleby/smhasher/blob/master/src/MurmurHash3.cpp
+ final int seedA = 0x012de1ba;
+ final int[] valuesA = {
+ 0x2f66c8b6, 0x256c0269, 0x054ef409, 0x402425ba, 0x78ebf590, 0x76bea1db,
+ 0x8bf5dcbe, 0x104ecdd4, 0x43cfc87e, 0xa33c7643, 0x4d210f56, 0xfa12093d,
+ };
+ // Values from a seed of zero
+ final int[] values0 = {
+ 0x92ca2f0e, 0x3cd6e3f3, 0x1b147dcc, 0x4c081dbf, 0x487981ab, 0xdb408c9d,
+ 0x78bc1b8f, 0xd83072e5, 0x65cbdd54, 0x1f4b8cef, 0x91783bb0, 0x0231739b,
+ };
+
+ // Create a random seed that is larger than the maximum length;
+ // start with the initial value
+ final int[] data = new SplittableRandom().ints(10).toArray();
+ data[0] = seedA;
+
+ for (int i = 0; i <= 9; i++) {
+ final int seedLength = i;
+ // Truncate the random seed
+ final int[] seed = Arrays.copyOf(data, seedLength);
+ // Create the expected output length.
+ // If too short it should be extended with values from the reference output
+ final int[] expected = Arrays.copyOf(seed, Math.max(seedLength, length));
+ if (expected.length == 0) {
+ // Edge case for zero length
+ Assertions.assertArrayEquals(new int[0],
+ BaseProvider.extendSeed(seed, length));
+ continue;
+ }
+ // Extend the truncated seed using the reference output.
+ // This may be seeded with zero or the non-zero initial value.
+ final int[] source = expected[0] == 0 ? values0 : valuesA;
+ System.arraycopy(source, 0, expected, seedLength, expected.length - seedLength);
+ Assertions.assertArrayEquals(expected,
+ BaseProvider.extendSeed(seed, length),
+ () -> String.format("%d -> %d", seedLength, length));
+ }
+ }
+
+ /**
* Dummy class for checking the behaviorof the IntProvider. Tests:
* <ul>
* <li>an incomplete implementation</li>
diff --git a/src/changes/changes.xml b/src/changes/changes.xml
index e2db3d1..0b0a08c 100644
--- a/src/changes/changes.xml
+++ b/src/changes/changes.xml
@@ -86,6 +86,10 @@ behavioural compatibility between releases; derived types
may break behavioural compatibility. Any functional changes
will be recorded in the release notes.
">
+ <action dev="aherbert" type="add" issue="173">
+ "BaseProvider": Add a static method to extend input int[] and long[] seeds to a
+ minimum length.
+ </action>
<action dev="aherbert" type="update" issue="171">
Recuce the memory footprint of the cached boolean and int source for the IntProvider
and LongProvider. This change has a performance improvement on some JDKs.
diff --git a/src/main/resources/pmd/pmd-ruleset.xml b/src/main/resources/pmd/pmd-ruleset.xml
index 88bef30..95a23d1 100644
--- a/src/main/resources/pmd/pmd-ruleset.xml
+++ b/src/main/resources/pmd/pmd-ruleset.xml
@@ -101,6 +101,13 @@
<property name="violationSuppressXPath" value="//ClassOrInterfaceDeclaration[@SimpleName='TSampler']"/>
</properties>
</rule>
+ <rule ref="category/java/bestpractices.xml/AvoidReassigningParameters">
+ <properties>
+ <!-- Hash functions are optimised for minimum byte size to allow inlining -->
+ <property name="violationSuppressXPath"
+ value="./ancestor::MethodDeclaration[@Name='stafford13' or @Name='murmur3']"/>
+ </properties>
+ </rule>
<rule ref="category/java/codestyle.xml/ClassNamingConventions">
<properties>
[commons-rng] 01/04: RNG-169: Avoid long to int[] that can generate a zero output seed.
Posted by ah...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aherbert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/commons-rng.git
commit 3c4184b981d5eaf72b066aa0784795c41533308f
Author: aherbert <ah...@apache.org>
AuthorDate: Mon Mar 28 15:01:15 2022 +0100
RNG-169: Avoid long to int[] that can generate a zero output seed.
This is relevant to small state RNGs with an int[] native seed such as
XoRoShiRo64StarStar.
---
.../commons/rng/simple/internal/Conversions.java | 8 +++-
.../rng/simple/internal/ConversionsTest.java | 47 +++++++++++++++++++++-
2 files changed, 53 insertions(+), 2 deletions(-)
diff --git a/commons-rng-simple/src/main/java/org/apache/commons/rng/simple/internal/Conversions.java b/commons-rng-simple/src/main/java/org/apache/commons/rng/simple/internal/Conversions.java
index 9b51daf..2eff8c0 100644
--- a/commons-rng-simple/src/main/java/org/apache/commons/rng/simple/internal/Conversions.java
+++ b/commons-rng-simple/src/main/java/org/apache/commons/rng/simple/internal/Conversions.java
@@ -173,12 +173,18 @@ final class Conversions {
* generate the sequence and filling the ints
* in little-endian order (least significant byte first).
*
+ * <p>A special case is made to avoid an array filled with zeros for
+ * the initial 2 positions. It is still possible to create a zero in
+ * position 0. However any RNG with an int[] native type is expected to
+ * require at least 2 int values.
+ *
* @param input Input
* @param length Array length
* @return an {@code int[]}.
*/
static int[] long2IntArray(long input, int length) {
- long v = input;
+ // Special case to avoid creating a zero-filled array of length 2.
+ long v = input == -GOLDEN_RATIO ? ~input : input;
final int[] output = new int[length];
// Process pairs
final int n = length & ~0x1;
diff --git a/commons-rng-simple/src/test/java/org/apache/commons/rng/simple/internal/ConversionsTest.java b/commons-rng-simple/src/test/java/org/apache/commons/rng/simple/internal/ConversionsTest.java
index 62fcbf6..46d1244 100644
--- a/commons-rng-simple/src/test/java/org/apache/commons/rng/simple/internal/ConversionsTest.java
+++ b/commons-rng-simple/src/test/java/org/apache/commons/rng/simple/internal/ConversionsTest.java
@@ -27,6 +27,7 @@ import org.apache.commons.rng.core.source64.SplitMix64;
import org.apache.commons.rng.core.util.NumberFactory;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.RepeatedTest;
+import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.MethodSource;
import org.junit.jupiter.params.provider.ValueSource;
@@ -36,6 +37,15 @@ import org.junit.jupiter.params.provider.ValueSource;
*/
class ConversionsTest {
/**
+ * The fractional part of the the golden ratio, phi, scaled to 64-bits and rounded to odd.
+ * <pre>
+ * phi = (sqrt(5) - 1) / 2) * 2^64
+ * </pre>
+ * @see <a href="https://en.wikipedia.org/wiki/Golden_ratio">Golden ratio</a>
+ */
+ private static final long GOLDEN_RATIO = 0x9e3779b97f4a7c15L;
+
+ /**
* Gets the lengths for the byte[] seeds to convert.
*
* @return the lengths
@@ -123,7 +133,12 @@ class ConversionsTest {
@RepeatedTest(value = 5)
void testLong2IntArray() {
- final long v = ThreadLocalRandom.current().nextLong();
+ // Avoid seed == 0 - 0x9e3779b97f4a7c15L. See testLong2IntArrayLength2NotAllZero.
+ long seed;
+ do {
+ seed = ThreadLocalRandom.current().nextLong();
+ } while (seed == -GOLDEN_RATIO);
+ final long v = seed;
getIntLengths().forEach(len -> {
final int longs = Conversions.longSizeFromIntSize(len);
// Little-endian conversion
@@ -143,6 +158,36 @@ class ConversionsTest {
});
}
+ /**
+ * Test the long2IntArray conversion avoids an input that will generate a zero from the
+ * SplitMix64-style RNG. This prevents creating an array of length 2 that is zero.
+ *
+ * <p>This special case avoids creating a small state Xor-based generator such as
+ * XoRoShiRo64StarStar with a seed of all zeros.
+ */
+ @Test
+ void testLong2IntArrayLength2NotAllZero() {
+ // The first output from the SplitMix64 is mix(seed + GOLDEN_RATIO).
+ // Create the seed to ensure a zero output from the mix function.
+ final long seed = -GOLDEN_RATIO;
+ Assertions.assertEquals(0, new SplitMix64(seed).nextLong());
+
+ // Note: This cannot occur for int2IntArray as the SplitMix64 is seeded with the int.
+ // This ignores the case of an output int[] of length 1 which could be zero.
+ // An int -> int[1] conversion is nonsensical and should not be performed by the library.
+ Assertions.assertNotEquals(0, new SplitMix64((int) seed).nextLong());
+
+ // The conversion should detect this case and a zero seed of length 2 should not happen.
+ final int[] actual = Conversions.long2IntArray(seed, 2);
+ Assertions.assertFalse(Arrays.equals(new int[2], actual));
+
+ // Longer arrays may be a partially zero as the generator state passes through
+ // the zero-point.
+ Assertions.assertArrayEquals(
+ Arrays.copyOf(Conversions.long2IntArray(seed - GOLDEN_RATIO, 2), 4),
+ Conversions.long2IntArray(seed - GOLDEN_RATIO, 4));
+ }
+
@RepeatedTest(value = 5)
void testLong2LongArray() {
final long v = ThreadLocalRandom.current().nextLong();
[commons-rng] 04/04: Consolidate Int/LongProvider to use next() as the source of random bits
Posted by ah...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aherbert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/commons-rng.git
commit ea379adf1c968405a2942ff4637183b66c53aaee
Author: aherbert <ah...@apache.org>
AuthorDate: Wed Mar 30 11:10:37 2022 +0100
Consolidate Int/LongProvider to use next() as the source of random bits
This removes the mixed use of next and nextLong or nextInt. The next
method is default source of the random bits and should always be used.
---
.../java/org/apache/commons/rng/core/source32/IntProvider.java | 8 ++++----
.../java/org/apache/commons/rng/core/source64/LongProvider.java | 6 +++---
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/commons-rng-core/src/main/java/org/apache/commons/rng/core/source32/IntProvider.java b/commons-rng-core/src/main/java/org/apache/commons/rng/core/source32/IntProvider.java
index ed4964a..37ee878 100644
--- a/commons-rng-core/src/main/java/org/apache/commons/rng/core/source32/IntProvider.java
+++ b/commons-rng-core/src/main/java/org/apache/commons/rng/core/source32/IntProvider.java
@@ -35,7 +35,7 @@ public abstract class IntProvider
/**
* Provides a bit source for booleans.
*
- * <p>A cached value from a call to {@link #nextInt()}.
+ * <p>A cached value from a call to {@link #next()}.
*
* <p>Only stores 31-bits when full as 1 bit has already been consumed.
* The sign bit is a flag that shifts down so the source eventually equals 1
@@ -117,19 +117,19 @@ public abstract class IntProvider
/** {@inheritDoc} */
@Override
public double nextDouble() {
- return NumberFactory.makeDouble(nextInt(), nextInt());
+ return NumberFactory.makeDouble(next(), next());
}
/** {@inheritDoc} */
@Override
public float nextFloat() {
- return NumberFactory.makeFloat(nextInt());
+ return NumberFactory.makeFloat(next());
}
/** {@inheritDoc} */
@Override
public long nextLong() {
- return NumberFactory.makeLong(nextInt(), nextInt());
+ return NumberFactory.makeLong(next(), next());
}
/** {@inheritDoc} */
diff --git a/commons-rng-core/src/main/java/org/apache/commons/rng/core/source64/LongProvider.java b/commons-rng-core/src/main/java/org/apache/commons/rng/core/source64/LongProvider.java
index 63bbd7f..940a58b 100644
--- a/commons-rng-core/src/main/java/org/apache/commons/rng/core/source64/LongProvider.java
+++ b/commons-rng-core/src/main/java/org/apache/commons/rng/core/source64/LongProvider.java
@@ -38,7 +38,7 @@ public abstract class LongProvider
/**
* Provides a bit source for booleans.
*
- * <p>A cached value from a call to {@link #nextLong()}.
+ * <p>A cached value from a call to {@link #next()}.
*
* <p>Only stores 63-bits when full as 1 bit has already been consumed.
* The sign bit is a flag that shifts down so the source eventually equals 1
@@ -49,7 +49,7 @@ public abstract class LongProvider
/**
* Provides a source for ints.
*
- * <p>A cached half-value value from a call to {@link #nextLong()}.
+ * <p>A cached half-value value from a call to {@link #next()}.
* The int is stored in the lower 32 bits with zeros in the upper bits.
* When empty this is set to negative to trigger a refill.
*/
@@ -135,7 +135,7 @@ public abstract class LongProvider
/** {@inheritDoc} */
@Override
public double nextDouble() {
- return NumberFactory.makeDouble(nextLong());
+ return NumberFactory.makeDouble(next());
}
/** {@inheritDoc} */
[commons-rng] 02/04: Correct longAsInt output in the stress test application
Posted by ah...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
aherbert pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/commons-rng.git
commit cc6b0c0b40499fb698730d94dc9c2d66f4176ebe
Author: aherbert <ah...@apache.org>
AuthorDate: Tue Mar 29 13:40:26 2022 +0100
Correct longAsInt output in the stress test application
RNG-171 changed the cached int source to output a long as two ints using
the low 32-bits, then high 32-bits.
Updated the RngDataOutput to support HiLo or LoHi. Added tests for each
implementation.
Dropped high-bits and low-bits command options. Added source64 option
with enum for high, low, high-Low, low-high, long, int. The default for
stress testing matches the caching implementation.
Update the test documentation to reflect the changes.
---
commons-rng-examples/examples-stress/endianness.md | 20 +-
.../rng/examples/stress/BridgeTestCommand.java | 4 +-
.../commons/rng/examples/stress/OutputCommand.java | 47 +++--
.../commons/rng/examples/stress/RNGUtils.java | 218 +++++++++++++++++----
.../commons/rng/examples/stress/RngDataOutput.java | 112 ++++++++++-
.../commons/rng/examples/stress/Source64Mode.java | 35 ++++
.../rng/examples/stress/StressTestCommand.java | 70 ++++---
.../commons/rng/examples/stress/RNGUtilsTest.java | 118 +++++++++++
.../rng/examples/stress/RngDataOutputTest.java | 93 ++++++++-
.../examples-stress/stress_test.md | 43 ++--
10 files changed, 635 insertions(+), 125 deletions(-)
diff --git a/commons-rng-examples/examples-stress/endianness.md b/commons-rng-examples/examples-stress/endianness.md
index 56567f8..95bc9b9 100644
--- a/commons-rng-examples/examples-stress/endianness.md
+++ b/commons-rng-examples/examples-stress/endianness.md
@@ -59,12 +59,12 @@ This will create a single jar file with all the dependencies in the `target` dir
contains a simple command to output data from a random generator. To output data in both text
and binary format use the following commands:
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
+ java -jar target/examples-stress.jar output KISS -s 1 -n 2048000 \
-f DIEHARDER -o test.dh
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
- -f BINARY -o test.big -b big_endian
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 100000 \
- -f BINARY -o test.little -b little_endian
+ java -jar target/examples-stress.jar output KISS -s 1 -n 1000 \
+ -f BINARY -o test.big -b big_endian --buffer-size 8192
+ java -jar target/examples-stress.jar output KISS -s 1 -n 1000 \
+ -f BINARY -o test.little -b little_endian --buffer-size 8192
This should produce the following output files:
@@ -74,6 +74,8 @@ This should produce the following output files:
| test.big | Binary file using the big-endian format |
| test.little | Binary file using the little-endian format |
+Note that the `-n` parameter is the count of numbers in text output but the count of buffers in binary output. The example above has 2048 4-byte integers per 8192 buffer. The `-n` parameter has been adjusted so the output numbers are the same.
+
The data can then be used to run a test within **Dieharder**:
dieharder -g 202 -d 0 -f test.dh
@@ -89,10 +91,10 @@ written direct to dieharder. For the birthdays test (`-d 0` option) 20,000,000 s
In the following example the stress test application directly writes to `stdout` which is then
piped to the `dieharder` application which reads using `stdin` (`-g 200` option):
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
+ java -jar target/examples-stress.jar output KISS -s 1 -n 20000000 \
-f DIEHARDER -o test.dh
dieharder -g 202 -d 0 -f test.dh
- java -jar target/examples-stress.jar output SPLIT_MIX_64 -s 1 -n 20000000 \
+ java -jar target/examples-stress.jar output KISS -s 1 -n 10000 \
-f BINARY -b little_endian | dieharder -g 200 -d 0
If the results are not the same then the second command can be repeated with `-b big_endian`.
@@ -150,7 +152,7 @@ integer value and the signed integer value. The contents will be similar to:
01011100 10001111 11110001 11000001 1552937409 1552937409
10110000 01110101 10010011 00011100 2960495388 -1334471908
-The `stdin2testu01` has been written to output the same format when using the `raw32` mode.
+The `stdin2testu01` has been written to output in the same format when using the `raw32` mode.
If the data has been correctly read the `bridge.data` and `bridge.out` should match.
If the endianess is incorrect then the data sent by the Java application will not match the
data read by the sub-process. For example to swap the endianness use the `-b` option:
@@ -215,7 +217,7 @@ The number data in the file should be identical if the endianness was correctly
The only difference should be the the header and footer added by the `stress` command
to the results file. If the endianness is incorrect for the `output` command then the number
data will not match. Note that the `output` command by default uses big endian for consistency
-across all platforms.
+across all platforms; the `stress` command uses the native byte order of the platform.
An equivalent 64-bit output would use the `--raw64` option for each command:
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/BridgeTestCommand.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/BridgeTestCommand.java
index f32e71c..70a2a04 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/BridgeTestCommand.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/BridgeTestCommand.java
@@ -131,8 +131,8 @@ class BridgeTestCommand implements Callable<Void> {
// Open the stdin of the process and write to a custom data sink.
// Note: The 'bridge' command only supports 32-bit data in order to
// demonstrate passing suitable data for TestU01 BigCrush.
- final boolean raw64 = false;
- try (RngDataOutput sink = RNGUtils.createDataOutput(rng, raw64,
+ final Source64Mode source64 = null;
+ try (RngDataOutput sink = RNGUtils.createDataOutput(rng, source64,
testingProcess.getOutputStream(), buffer.capacity() * 4, byteOrder)) {
sink.write(rng);
}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/OutputCommand.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/OutputCommand.java
index 6b0765a..bb2a7bc 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/OutputCommand.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/OutputCommand.java
@@ -126,25 +126,22 @@ class OutputCommand implements Callable<Void> {
description = {"Reverse the bits in the data (default: ${DEFAULT-VALUE})."})
private boolean reverseBits;
- /** Flag to use the upper 32-bits from the 64-bit long output. */
- @Option(names = {"--high-bits"},
- description = {"Use the upper 32-bits from the 64-bit long output.",
- "Takes precedent over --low-bits."})
- private boolean longHighBits;
-
- /** Flag to use the lower 32-bits from the 64-bit long output. */
- @Option(names = {"--low-bits"},
- description = {"Use the lower 32-bits from the 64-bit long output."})
- private boolean longLowBits;
-
/** Flag to use 64-bit long output. */
@Option(names = {"--raw64"},
description = {"Use 64-bit output (default is 32-bit).",
"This is ignored if not a native 64-bit generator.",
- "In 32-bit mode the output uses the upper then lower bits of 64-bit " +
- "generators sequentially."})
+ "Set to true sets the source64 mode to LONG."})
private boolean raw64;
+ /** Output mode for 64-bit long output. */
+ @Option(names = {"--source64"},
+ description = {"Output mode for 64-bit generators (default: ${DEFAULT-VALUE}).",
+ "This is ignored if not a native 64-bit generator.",
+ "In 32-bit mode the output uses a combination of upper and " +
+ "lower bits of the 64-bit value.",
+ "Valid values: ${COMPLETION-CANDIDATES}."})
+ private Source64Mode source64 = RNGUtils.getSource64Default();
+
/**
* The output mode for existing files.
*/
@@ -166,12 +163,20 @@ class OutputCommand implements Callable<Void> {
final Object objectSeed = createSeed();
UniformRandomProvider rng = createRNG(objectSeed);
+ // raw64 flag overrides the source64 mode
+ if (raw64) {
+ source64 = Source64Mode.LONG;
+ }
+ if (source64 == Source64Mode.LONG && !(rng instanceof RandomLongSource)) {
+ throw new ApplicationException("Not a 64-bit RNG: " + rng);
+ }
+
// Upper or lower bits from 64-bit generators must be created first.
- // This will throw if not a 64-bit generator.
- if (longHighBits) {
- rng = RNGUtils.createLongUpperBitsIntProvider(rng);
- } else if (longLowBits) {
- rng = RNGUtils.createLongLowerBitsIntProvider(rng);
+ // Note this does not test source64 != Source64Mode.LONG as the full long
+ // output split into hi-lo or lo-hi is supported by the RngDataOutput.
+ if (rng instanceof RandomLongSource &&
+ (source64 == Source64Mode.HI || source64 == Source64Mode.LO || source64 == Source64Mode.INT)) {
+ rng = RNGUtils.createIntProvider((UniformRandomProvider & RandomLongSource) rng, source64);
}
if (reverseBits) {
rng = RNGUtils.createReverseBitsProvider(rng);
@@ -305,9 +310,9 @@ class OutputCommand implements Callable<Void> {
*/
private UniformRandomProvider toOutputFormat(UniformRandomProvider rng) {
UniformRandomProvider convertedRng = rng;
- if (rng instanceof RandomLongSource && !raw64) {
+ if (rng instanceof RandomLongSource && source64 != Source64Mode.LONG) {
// Convert to 32-bit generator
- convertedRng = RNGUtils.createIntProvider(rng);
+ convertedRng = RNGUtils.createIntProvider((UniformRandomProvider & RandomLongSource) rng, source64);
}
if (byteOrder == ByteOrder.LITTLE_ENDIAN) {
convertedRng = RNGUtils.createReverseBytesProvider(convertedRng);
@@ -421,7 +426,7 @@ class OutputCommand implements Callable<Void> {
// If count is not positive use max value.
// This is effectively unlimited: program must be killed.
final long limit = (count < 1) ? Long.MAX_VALUE : count;
- try (RngDataOutput data = RNGUtils.createDataOutput(rng, raw64, out, bufferSize, byteOrder)) {
+ try (RngDataOutput data = RNGUtils.createDataOutput(rng, source64, out, bufferSize, byteOrder)) {
for (long c = 0; c < limit; c++) {
data.write(rng);
}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RNGUtils.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RNGUtils.java
index 29c09a5..4d799c9 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RNGUtils.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RNGUtils.java
@@ -33,7 +33,7 @@ import java.util.concurrent.ThreadLocalRandom;
final class RNGUtils {
/** Name prefix for bit-reversed RNGs. */
private static final String BYTE_REVERSED = "Byte-reversed ";
- /** Name prefix for byte-reversed RNGs. */
+ /** Name prefix for bit-reversed RNGs. */
private static final String BIT_REVERSED = "Bit-reversed ";
/** Name prefix for hashcode mixed RNGs. */
private static final String HASH_CODE = "HashCode ^ ";
@@ -41,15 +41,26 @@ final class RNGUtils {
private static final String TLR_MIXED = "ThreadLocalRandom ^ ";
/** Name of xor operator for xor mixed RNGs. */
private static final String XOR = " ^ ";
+ /** Message for an unrecognised source64 mode. */
+ private static final String UNRECOGNISED_SOURCE_64_MODE = "Unrecognised source64 mode: ";
/** Message for an unrecognised native output type. */
private static final String UNRECOGNISED_NATIVE_TYPE = "Unrecognised native output type: ";
- /** Message when not a RandomLongSource. */
- private static final String NOT_LONG_SOURCE = "Not a 64-bit long generator: ";
+ /** The source64 mode for the default LongProvider caching implementation. */
+ private static final Source64Mode SOURCE_64_DEFAULT = Source64Mode.LO_HI;
/** No public construction. */
private RNGUtils() {}
/**
+ * Gets the source64 mode for the default caching implementation in {@link LongProvider}.
+ *
+ * @return the source64 default mode
+ */
+ static Source64Mode getSource64Default() {
+ return SOURCE_64_DEFAULT;
+ }
+
+ /**
* Wrap the random generator with a new instance that will reverse the byte order of
* the native type. The input must be either a {@link RandomIntSource} or
* {@link RandomLongSource}.
@@ -132,6 +143,35 @@ final class RNGUtils {
}
/**
+ * Wrap the {@link RandomLongSource} with an {@link IntProvider} that will use the
+ * specified part of {@link RandomLongSource#next()} to create the int value.
+ *
+ * @param <R> The type of the generator.
+ * @param rng The random generator.
+ * @param mode the mode
+ * @return the int random generator.
+ * @throws ApplicationException If the input source native type is not 64-bit.
+ */
+ static <R extends RandomLongSource & UniformRandomProvider>
+ UniformRandomProvider createIntProvider(final R rng, Source64Mode mode) {
+ switch (mode) {
+ case INT:
+ return createIntProvider(rng);
+ case LO_HI:
+ return createLongLowerUpperBitsIntProvider(rng);
+ case HI_LO:
+ return createLongUpperLowerBitsIntProvider(rng);
+ case HI:
+ return createLongUpperBitsIntProvider(rng);
+ case LO:
+ return createLongLowerBitsIntProvider(rng);
+ case LONG:
+ default:
+ throw new IllegalArgumentException("Unsupported mode " + mode);
+ }
+ }
+
+ /**
* Wrap the random generator with an {@link IntProvider} that will use
* {@link UniformRandomProvider#nextInt()}.
* An input {@link RandomIntSource} is returned unmodified.
@@ -139,7 +179,7 @@ final class RNGUtils {
* @param rng The random generator.
* @return the int random generator.
*/
- static UniformRandomProvider createIntProvider(final UniformRandomProvider rng) {
+ private static UniformRandomProvider createIntProvider(final UniformRandomProvider rng) {
if (!(rng instanceof RandomIntSource)) {
return new IntProvider() {
@Override
@@ -157,6 +197,78 @@ final class RNGUtils {
}
/**
+ * Wrap the random generator with an {@link IntProvider} that will use the lower then upper
+ * 32-bits from {@link UniformRandomProvider#nextLong()}.
+ * An input {@link RandomIntSource} is returned unmodified.
+ *
+ * @param rng The random generator.
+ * @return the int random generator.
+ */
+ private static UniformRandomProvider createLongLowerUpperBitsIntProvider(final RandomLongSource rng) {
+ return new IntProvider() {
+ private long source = -1;
+
+ @Override
+ public int next() {
+ long next = source;
+ if (next < 0) {
+ // refill
+ next = rng.next();
+ // store hi
+ source = next >>> 32;
+ // extract low
+ return (int) next;
+ }
+ final int v = (int) next;
+ // reset
+ source = -1;
+ return v;
+ }
+
+ @Override
+ public String toString() {
+ return "Long lower-upper bits " + rng.toString();
+ }
+ };
+ }
+
+ /**
+ * Wrap the random generator with an {@link IntProvider} that will use the lower then upper
+ * 32-bits from {@link UniformRandomProvider#nextLong()}.
+ * An input {@link RandomIntSource} is returned unmodified.
+ *
+ * @param rng The random generator.
+ * @return the int random generator.
+ */
+ private static UniformRandomProvider createLongUpperLowerBitsIntProvider(final RandomLongSource rng) {
+ return new IntProvider() {
+ private long source = -1;
+
+ @Override
+ public int next() {
+ long next = source;
+ if (next < 0) {
+ // refill
+ next = rng.next();
+ // store low
+ source = next & 0xffff_ffffL;
+ // extract hi
+ return (int) (next >>> 32);
+ }
+ final int v = (int) next;
+ // reset
+ source = -1;
+ return v;
+ }
+
+ @Override
+ public String toString() {
+ return "Long upper-lower bits " + rng.toString();
+ }
+ };
+ }
+
+ /**
* Wrap the random generator with an {@link IntProvider} that will use the upper
* 32-bits of the {@code long} from {@link UniformRandomProvider#nextLong()}.
* The input must be a {@link RandomLongSource}.
@@ -165,21 +277,18 @@ final class RNGUtils {
* @return the upper bits random generator.
* @throws ApplicationException If the input source native type is not 64-bit.
*/
- static UniformRandomProvider createLongUpperBitsIntProvider(final UniformRandomProvider rng) {
- if (rng instanceof RandomLongSource) {
- return new IntProvider() {
- @Override
- public int next() {
- return (int) (rng.nextLong() >>> 32);
- }
+ private static UniformRandomProvider createLongUpperBitsIntProvider(final RandomLongSource rng) {
+ return new IntProvider() {
+ @Override
+ public int next() {
+ return (int) (rng.next() >>> 32);
+ }
- @Override
- public String toString() {
- return "Long upper-bits " + rng.toString();
- }
- };
- }
- throw new ApplicationException(NOT_LONG_SOURCE + rng);
+ @Override
+ public String toString() {
+ return "Long upper-bits " + rng.toString();
+ }
+ };
}
/**
@@ -191,21 +300,18 @@ final class RNGUtils {
* @return the lower bits random generator.
* @throws ApplicationException If the input source native type is not 64-bit.
*/
- static UniformRandomProvider createLongLowerBitsIntProvider(final UniformRandomProvider rng) {
- if (rng instanceof RandomLongSource) {
- return new IntProvider() {
- @Override
- public int next() {
- return (int) rng.nextLong();
- }
+ private static UniformRandomProvider createLongLowerBitsIntProvider(final RandomLongSource rng) {
+ return new IntProvider() {
+ @Override
+ public int next() {
+ return (int) rng.next();
+ }
- @Override
- public String toString() {
- return "Long lower-bits " + rng.toString();
- }
- };
- }
- throw new ApplicationException(NOT_LONG_SOURCE + rng);
+ @Override
+ public String toString() {
+ return "Long lower-bits " + rng.toString();
+ }
+ };
}
/**
@@ -360,32 +466,60 @@ final class RNGUtils {
*
* <p>If the RNG is a {@link RandomLongSource} then the byte output can be 32-bit or 64-bit.
* If 32-bit then the 64-bit output will be written as if 2 {@code int} values were generated
- * sequentially from the upper then lower 32-bits of the {@code long}. This setting is
- * significant depending on the byte order. If using the Java standard big-endian
- * representation the flag has no effect and the output will be the same. If using little
- * endian the output bytes will be written as:</p>
+ * sequentially from the {@code long} (order depends on the source64 mode). This setting is
+ * significant depending on the byte order. For example for a high-low source64 mode and
+ * using the Java standard big-endian representation the output is the same as the raw 64-bit
+ * output. If using little endian the output bytes will be written as:</p>
*
* <pre>
* 76543210 -> 4567 0123
* </pre>
*
+ * <h2>Note</h2>
+ *
+ * <p>The output from an implementation of RandomLongSource from the RNG core package
+ * may output the long bits as integers: in high-low order; in low-high order; using
+ * only the low bits; using only the high bits; or other combinations. This method
+ * allows testing the long output as if it were two int outputs, i.e. using the full
+ * bit output of a long provider with a stress test application that targets 32-bit
+ * random values (e.g. Test U01).
+ *
+ * <p>The results of stress testing can be used to determine if the provider
+ * implementation can use the upper, lower or both parts of the long output for int
+ * generation. In the case of the combined upper-lower output it is not expected that
+ * the order low-high or high-low is important given the stress test will consume
+ * thousands of numbers per test. The default 32-bit mode for a 64-bit source is high-low
+ * for backwards compatibility.
+ *
* @param rng The random generator.
- * @param raw64 Set to true for 64-bit byte output.
+ * @param source64 The output mode for a 64-bit source
* @param out Output stream.
* @param byteSize Number of bytes values to write.
* @param byteOrder Byte order.
* @return the data output
- * @throws ApplicationException If the input source native type is not recognised.
+ * @throws ApplicationException If the input source native type is not recognised; or if
+ * the mode for a RandomLongSource is not one of: raw; hi-lo; or lo-hi.
*/
- static RngDataOutput createDataOutput(final UniformRandomProvider rng, boolean raw64,
+ static RngDataOutput createDataOutput(final UniformRandomProvider rng, Source64Mode source64,
OutputStream out, int byteSize, ByteOrder byteOrder) {
if (rng instanceof RandomIntSource) {
return RngDataOutput.ofInt(out, byteSize / 4, byteOrder);
}
if (rng instanceof RandomLongSource) {
- return raw64 ?
- RngDataOutput.ofLong(out, byteSize / 8, byteOrder) :
- RngDataOutput.ofLongAsInt(out, byteSize / 8, byteOrder);
+ switch (source64) {
+ case HI_LO:
+ return RngDataOutput.ofLongAsHLInt(out, byteSize / 8, byteOrder);
+ case LO_HI:
+ return RngDataOutput.ofLongAsLHInt(out, byteSize / 8, byteOrder);
+ case LONG:
+ return RngDataOutput.ofLong(out, byteSize / 8, byteOrder);
+ // Note other options should have already been converted to an IntProvider
+ case INT:
+ case LO:
+ case HI:
+ default:
+ throw new ApplicationException(UNRECOGNISED_SOURCE_64_MODE + source64);
+ }
}
throw new ApplicationException(UNRECOGNISED_NATIVE_TYPE + rng);
}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RngDataOutput.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RngDataOutput.java
index 90972d6..5abeccf 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RngDataOutput.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/RngDataOutput.java
@@ -54,6 +54,11 @@ import java.nio.ByteOrder;
* {@link java.io.BufferedOutputStream#write(int) BufferedOutputStream#write(int)} that
* occur for each {@code int} value that is written to
* {@link java.io.DataOutputStream#writeInt(int) DataOutputStream#writeInt(int)}.</p>
+ *
+ * <p>This class has adaptors to write the long output from a RNG to two int values.
+ * To match the caching implementation in the the core LongProvider class this is tested
+ * to output int values in the same order as an instance of the LongProvider. Currently
+ * this outputs in order: low 32-bits, high 32-bits.
*/
abstract class RngDataOutput implements Closeable {
/** The data buffer. */
@@ -155,7 +160,8 @@ abstract class RngDataOutput implements Closeable {
}
/**
- * Write {@code long} data as two little-endian {@code int} values.
+ * Write {@code long} data as two little-endian {@code int} values, high 32-bits then
+ * low 32-bits.
* <pre>
* 76543210 -> 4567 0123
* </pre>
@@ -180,7 +186,39 @@ abstract class RngDataOutput implements Closeable {
@Override
public void fillBuffer(UniformRandomProvider rng) {
for (int i = 0; i < buffer.length; i += 8) {
- writeLongAsIntLE(i, rng.nextLong());
+ writeLongAsHighLowIntLE(i, rng.nextLong());
+ }
+ }
+ }
+
+ /**
+ * Write {@code long} data as two big-endian {@code int} values, low 32-bits then
+ * high 32-bits.
+ * <pre>
+ * 76543210 -> 3210 7654
+ * </pre>
+ *
+ * <p>This is a specialisation that allows the Java big-endian representation to be split
+ * into two big-endian values in the original order of lower then upper bits. In
+ * comparison the {@link BLongRngDataOutput} will output the same data as:
+ *
+ * <pre>
+ * 76543210 -> 7654 3210
+ * </pre>
+ */
+ private static class BLongAsLoHiIntRngDataOutput extends RngDataOutput {
+ /**
+ * @param out Output stream.
+ * @param size Buffer size.
+ */
+ BLongAsLoHiIntRngDataOutput(OutputStream out, int size) {
+ super(out, size);
+ }
+
+ @Override
+ public void fillBuffer(UniformRandomProvider rng) {
+ for (int i = 0; i < buffer.length; i += 8) {
+ writeLongAsLowHighIntBE(i, rng.nextLong());
}
}
}
@@ -258,7 +296,7 @@ abstract class RngDataOutput implements Closeable {
}
/**
- * Writes an {@code long} to the buffer as eight bytes, low byte first (big-endian).
+ * Writes an {@code long} to the buffer as eight bytes, low byte first (little-endian).
*
* @param index the index to start writing.
* @param value an {@code long} to be written.
@@ -276,22 +314,50 @@ abstract class RngDataOutput implements Closeable {
/**
* Writes an {@code long} to the buffer as two integers of four bytes, each
- * low byte first (big-endian).
+ * low byte first (little-endian). The long is written as the high 32-bits,
+ * then the low 32-bits.
+ *
+ * <p>Note: A LowHigh little-endian output is the same as {@link #writeLongLE(int, long)}.
*
* @param index the index to start writing.
* @param value an {@code long} to be written.
*/
- final void writeLongAsIntLE(int index, long value) {
+ final void writeLongAsHighLowIntLE(int index, long value) {
+ // high
buffer[index ] = (byte) (value >>> 32);
buffer[index + 1] = (byte) (value >>> 40);
buffer[index + 2] = (byte) (value >>> 48);
buffer[index + 3] = (byte) (value >>> 56);
+ // low
buffer[index + 4] = (byte) value;
buffer[index + 5] = (byte) (value >>> 8);
buffer[index + 6] = (byte) (value >>> 16);
buffer[index + 7] = (byte) (value >>> 24);
}
+ /**
+ * Writes an {@code long} to the buffer as two integers of four bytes, each
+ * high byte first (big-endian). The long is written as the low 32-bits,
+ * then the high 32-bits.
+ *
+ * <p>Note: A HighLow big-endian output is the same as {@link #writeLongBE(int, long)}.
+ *
+ * @param index the index to start writing.
+ * @param value an {@code long} to be written.
+ */
+ final void writeLongAsLowHighIntBE(int index, long value) {
+ // low
+ buffer[index ] = (byte) (value >>> 24);
+ buffer[index + 1] = (byte) (value >>> 16);
+ buffer[index + 2] = (byte) (value >>> 8);
+ buffer[index + 3] = (byte) value;
+ // high
+ buffer[index + 4] = (byte) (value >>> 56);
+ buffer[index + 5] = (byte) (value >>> 48);
+ buffer[index + 6] = (byte) (value >>> 40);
+ buffer[index + 7] = (byte) (value >>> 32);
+ }
+
@Override
public void close() throws IOException {
try (OutputStream ostream = out) {
@@ -338,7 +404,7 @@ abstract class RngDataOutput implements Closeable {
/**
* Create a new instance to write batches of data from
* {@link UniformRandomProvider#nextLong()} to the specified output as two sequential
- * {@code int} values.
+ * {@code int} values, high 32-bits then low 32-bits.
*
* <p>This will output the following bytes:</p>
*
@@ -359,11 +425,43 @@ abstract class RngDataOutput implements Closeable {
* @return the data output
*/
@SuppressWarnings("resource")
- static RngDataOutput ofLongAsInt(OutputStream out, int size, ByteOrder byteOrder) {
+ static RngDataOutput ofLongAsHLInt(OutputStream out, int size, ByteOrder byteOrder) {
// Ensure the buffer is positive and a factor of 8
final int bytes = Math.max(size * 8, 8);
return byteOrder == ByteOrder.LITTLE_ENDIAN ?
new LLongAsIntRngDataOutput(out, bytes) :
new BLongRngDataOutput(out, bytes);
}
+
+ /**
+ * Create a new instance to write batches of data from
+ * {@link UniformRandomProvider#nextLong()} to the specified output as two sequential
+ * {@code int} values, low 32-bits then high 32-bits.
+ *
+ * <p>This will output the following bytes:</p>
+ *
+ * <pre>
+ * // Little-endian
+ * 76543210 -> 0123 4567
+ *
+ * // Big-endian
+ * 76543210 -> 3210 7654
+ * </pre>
+ *
+ * <p>This ensures the output from the generator is the original lower then upper order bits
+ * for each endianess.
+ *
+ * @param out Output stream.
+ * @param size Number of values to write.
+ * @param byteOrder Byte order.
+ * @return the data output
+ */
+ @SuppressWarnings("resource")
+ static RngDataOutput ofLongAsLHInt(OutputStream out, int size, ByteOrder byteOrder) {
+ // Ensure the buffer is positive and a factor of 8
+ final int bytes = Math.max(size * 8, 8);
+ return byteOrder == ByteOrder.LITTLE_ENDIAN ?
+ new LLongRngDataOutput(out, bytes) :
+ new BLongAsLoHiIntRngDataOutput(out, bytes);
+ }
}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/Source64Mode.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/Source64Mode.java
new file mode 100644
index 0000000..357516a
--- /dev/null
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/Source64Mode.java
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.rng.examples.stress;
+
+/**
+ * The mode to output a 64-bit source of randomness.
+ */
+enum Source64Mode {
+ /** The 64-bit output from nextLong. */
+ LONG,
+ /** The 32-bit output from nextInt. */
+ INT,
+ /** The high 32-bits, then low 32-bits of the 64-bit output. */
+ HI_LO,
+ /** The low 32-bits, then high 32-bits of the 64-bit output. */
+ LO_HI,
+ /** The low 32-bits of the 64-bit output. */
+ LO,
+ /** The high 32-bits of the 64-bit output. */
+ HI;
+}
diff --git a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/StressTestCommand.java b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/StressTestCommand.java
index 62c4782..b24068e 100644
--- a/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/StressTestCommand.java
+++ b/commons-rng-examples/examples-stress/src/main/java/org/apache/commons/rng/examples/stress/StressTestCommand.java
@@ -153,25 +153,39 @@ class StressTestCommand implements Callable<Void> {
"when passing using the standard sequence."})
private boolean reverseBits;
- /** Flag to use the upper 32-bits from the 64-bit long output. */
- @Option(names = {"--high-bits"},
- description = {"Use the upper 32-bits from the 64-bit long output.",
- "Takes precedent over --low-bits."})
- private boolean longHighBits;
-
- /** Flag to use the lower 32-bits from the 64-bit long output. */
- @Option(names = {"--low-bits"},
- description = {"Use the lower 32-bits from the 64-bit long output."})
- private boolean longLowBits;
-
/** Flag to use 64-bit long output. */
@Option(names = {"--raw64"},
description = {"Use 64-bit output (default is 32-bit).",
- "This requires a 64-bit testing application and native 64-bit generators.",
- "In 32-bit mode the output uses the upper then lower bits of 64-bit " +
- "generators sequentially, each appropriately byte reversed for the platform."})
+ "This is ignored if not a native 64-bit generator.",
+ "Set to true sets the source64 mode to LONG."})
private boolean raw64;
+ /** Output mode for 64-bit long output.
+ *
+ * <p>Note: The default is set as the default caching implementation.
+ * It passes the full output of the RNG to the stress test application.
+ * Any combination random sources are performed on the full 64-bit output.
+ *
+ * <p>If using INT this will use the RNG's nextInt method.
+ * Any combination random sources are performed on the 32-bit output. Without
+ * a combination random source the output should be the same as the default if
+ * the generator uses the default caching implementation.
+ *
+ * <p>LONG and LO_HI should match binary output when LITTLE_ENDIAN. LONG and HI_LO
+ * should match binary output when BIG_ENDIAN.
+ *
+ * <p>Changing from HI_LO to LO_HI should not effect the stress test as many values are consumed
+ * per test. Using HI or LO may have a different outcome as parts of the generator output
+ * may be weak, e.g. the lower bits of linear congruential generators.
+ */
+ @Option(names = {"--source64"},
+ description = {"Output mode for 64-bit generators (default: ${DEFAULT-VALUE}).",
+ "This is ignored if not a native 64-bit generator.",
+ "In 32-bit mode the output uses a combination of upper and " +
+ "lower bits of the 64-bit value.",
+ "Valid values: ${COMPLETION-CANDIDATES}."})
+ private Source64Mode source64 = RNGUtils.getSource64Default();
+
/** The random seed as a byte[]. */
@Option(names = {"-x", "--hex-seed"},
description = {"The hex-encoded random seed.",
@@ -481,6 +495,11 @@ class StressTestCommand implements Callable<Void> {
String basePath,
Iterable<StressTestData> stressTestData,
ProgressTracker progressTracker) {
+ // raw64 flag overrides the source64 mode
+ if (raw64) {
+ source64 = Source64Mode.LONG;
+ }
+
final List<Runnable> tasks = new ArrayList<>();
for (final StressTestData testData : stressTestData) {
for (int trial = 1; trial <= testData.getTrials(); trial++) {
@@ -501,12 +520,17 @@ class StressTestCommand implements Callable<Void> {
final byte[] seed = createSeed(testData.getRandomSource());
UniformRandomProvider rng = testData.createRNG(seed);
- // Upper or lower bits from 64-bit generators must be created first.
- // This will throw if not a 64-bit generator.
- if (longHighBits) {
- rng = RNGUtils.createLongUpperBitsIntProvider(rng);
- } else if (longLowBits) {
- rng = RNGUtils.createLongLowerBitsIntProvider(rng);
+ if (source64 == Source64Mode.LONG && !(rng instanceof RandomLongSource)) {
+ throw new ApplicationException("Not a 64-bit RNG: " + rng);
+ }
+
+ // Upper or lower bits from 64-bit generators must be created first before
+ // any further combination operators.
+ // Note this does not test source64 != Source64Mode.LONG as the full long
+ // output split into hi-lo or lo-hi is supported by the RngDataOutput.
+ if (rng instanceof RandomLongSource &&
+ (source64 == Source64Mode.HI || source64 == Source64Mode.LO || source64 == Source64Mode.INT)) {
+ rng = RNGUtils.createIntProvider((UniformRandomProvider & RandomLongSource) rng, source64);
}
// Combination generators. Mainly used for testing.
@@ -669,7 +693,7 @@ class StressTestCommand implements Callable<Void> {
// of a new one)
// -- There are no pending tasks (i.e. the final submission or the end of a final task)
if (completed >= total ||
- (current >= nextReportTimestamp && (running == parallelTasks || pending == 0))) {
+ (current >= nextReportTimestamp && running == parallelTasks || pending == 0)) {
// Report
nextReportTimestamp = current + PROGRESS_INTERVAL;
final StringBuilder sb = createStringBuilderWithTimestamp(current, pending, running, completed);
@@ -991,7 +1015,7 @@ class StressTestCommand implements Callable<Void> {
final Process testingProcess = builder.start();
// Use a custom data output to write the RNG.
- try (RngDataOutput sink = RNGUtils.createDataOutput(rng, cmd.raw64,
+ try (RngDataOutput sink = RNGUtils.createDataOutput(rng, cmd.source64,
testingProcess.getOutputStream(), cmd.bufferSize, cmd.byteOrder)) {
for (;;) {
sink.write(rng);
@@ -1040,7 +1064,7 @@ class StressTestCommand implements Callable<Void> {
.append(C).append("Native byte-order: ").append(ByteOrder.nativeOrder()).append(N)
.append(C).append("Output byte-order: ").append(cmd.byteOrder).append(N);
if (rng instanceof RandomLongSource) {
- sb.append(C).append("64-bit output: ").append(cmd.raw64).append(N);
+ sb.append(C).append("64-bit output: ").append(cmd.source64).append(N);
}
sb.append(C).append(N)
.append(C).append("Analyzer: ");
diff --git a/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RNGUtilsTest.java b/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RNGUtilsTest.java
new file mode 100644
index 0000000..55084e1
--- /dev/null
+++ b/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RNGUtilsTest.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.commons.rng.examples.stress;
+
+import org.apache.commons.rng.UniformRandomProvider;
+import org.apache.commons.rng.core.source64.SplitMix64;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Tests for {@link RNGUtils}.
+ */
+class RNGUtilsTest {
+ @Test
+ void testCreateIntProviderLongThrows() {
+ final SplitMix64 rng = new SplitMix64(42);
+ Assertions.assertThrows(IllegalArgumentException.class,
+ () -> RNGUtils.createIntProvider(rng, Source64Mode.LONG));
+ }
+
+ @Test
+ void testCreateIntProviderInt() {
+ final long seed = 236784264237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.INT);
+ for (int i = 0; i < 50; i++) {
+ Assertions.assertEquals(rng1.nextInt(), rng2.nextInt());
+ }
+ }
+
+ @Test
+ void testCreateIntProviderLoHi() {
+ final long seed = 236784264237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.LO_HI);
+ for (int i = 0; i < 50; i++) {
+ final long l = rng1.nextLong();
+ final int hi = (int) (l >>> 32);
+ final int lo = (int) l;
+ Assertions.assertEquals(lo, rng2.nextInt());
+ Assertions.assertEquals(hi, rng2.nextInt());
+ }
+ }
+
+ @Test
+ void testCreateIntProviderHiLo() {
+ final long seed = 2367234237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.HI_LO);
+ for (int i = 0; i < 50; i++) {
+ final long l = rng1.nextLong();
+ final int hi = (int) (l >>> 32);
+ final int lo = (int) l;
+ Assertions.assertEquals(hi, rng2.nextInt());
+ Assertions.assertEquals(lo, rng2.nextInt());
+ }
+ }
+
+ @Test
+ void testCreateIntProviderHi() {
+ final long seed = 2367234237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.HI);
+ for (int i = 0; i < 50; i++) {
+ final int hi = (int) (rng1.nextLong() >>> 32);
+ Assertions.assertEquals(hi, rng2.nextInt());
+ }
+ }
+
+ @Test
+ void testCreateIntProviderLo() {
+ final long seed = 2367234237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), Source64Mode.LO);
+ for (int i = 0; i < 50; i++) {
+ final int lo = (int) rng1.nextLong();
+ Assertions.assertEquals(lo, rng2.nextInt());
+ }
+ }
+
+ /**
+ * Test that the default source64 mode matches the nextInt implementation for a LongProvider.
+ * If this changes then the default mode should be updated. The value is used as
+ * the default for the stress test application.
+ */
+ @Test
+ void testCreateIntProviderDefault() {
+ final long seed = 236784264237894L;
+ final UniformRandomProvider rng1 = new SplitMix64(seed);
+ final UniformRandomProvider rng2 =
+ RNGUtils.createIntProvider(new SplitMix64(seed), RNGUtils.getSource64Default());
+ for (int i = 0; i < 50; i++) {
+ final int a = rng1.nextInt();
+ final int b = rng1.nextInt();
+ Assertions.assertEquals(a, rng2.nextInt());
+ Assertions.assertEquals(b, rng2.nextInt());
+ }
+ }
+}
diff --git a/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RngDataOutputTest.java b/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RngDataOutputTest.java
index b41a747..9c5d6a0 100644
--- a/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RngDataOutputTest.java
+++ b/commons-rng-examples/examples-stress/src/test/java/org/apache/commons/rng/examples/stress/RngDataOutputTest.java
@@ -84,33 +84,54 @@ class RngDataOutputTest {
}
@Test
- void testLongAsIntBigEndian() throws IOException {
+ void testLongAsHLIntBigEndian() throws IOException {
assertRngOutput(RandomSource.SPLIT_MIX_64,
- // Convert SplitMix64 to an int provider so it is detected as requiring double the
+ // Convert to an int provider so it is detected as requiring double the
// length output.
- rng -> new IntProvider() {
+ rng -> new HiLoCachingIntProvider(rng),
+ RngDataOutputTest::writeInt,
+ RngDataOutput::ofLongAsHLInt, ByteOrder.BIG_ENDIAN);
+ }
+
+ @Test
+ void testLongAsHLIntLittleEndian() throws IOException {
+ assertRngOutput(RandomSource.SPLIT_MIX_64,
+ // Convert SplitMix64 to an int provider so it is detected as requiring double the
+ // length output. Then reverse the bytes.
+ rng -> new HiLoCachingIntProvider(rng) {
@Override
public int next() {
- return rng.nextInt();
+ return Integer.reverseBytes(super.next());
}
},
RngDataOutputTest::writeInt,
- RngDataOutput::ofLongAsInt, ByteOrder.BIG_ENDIAN);
+ RngDataOutput::ofLongAsHLInt, ByteOrder.LITTLE_ENDIAN);
}
+
@Test
- void testLongAsIntLittleEndian() throws IOException {
+ void testLongAsLHIntBigEndian() throws IOException {
assertRngOutput(RandomSource.SPLIT_MIX_64,
- // Convert SplitMix64 to an int provider so it is detected as requiring double the
+ // Convert to an int provider so it is detected as requiring double the
+ // length output.
+ rng -> new LoHiCachingIntProvider(rng),
+ RngDataOutputTest::writeInt,
+ RngDataOutput::ofLongAsLHInt, ByteOrder.BIG_ENDIAN);
+ }
+
+ @Test
+ void testLongAsLHIntLittleEndian() throws IOException {
+ assertRngOutput(RandomSource.SPLIT_MIX_64,
+ // Convert to an int provider so it is detected as requiring double the
// length output. Then reverse the bytes.
- rng -> new IntProvider() {
+ rng -> new LoHiCachingIntProvider(rng) {
@Override
public int next() {
- return Integer.reverseBytes(rng.nextInt());
+ return Integer.reverseBytes(super.next());
}
},
RngDataOutputTest::writeInt,
- RngDataOutput::ofLongAsInt, ByteOrder.LITTLE_ENDIAN);
+ RngDataOutput::ofLongAsLHInt, ByteOrder.LITTLE_ENDIAN);
}
private static void writeInt(DataOutputStream sink, UniformRandomProvider rng) {
@@ -212,4 +233,56 @@ class RngDataOutputTest {
}
return out.toByteArray();
}
+
+ private static class LoHiCachingIntProvider extends IntProvider {
+ private long source = -1;
+ private final UniformRandomProvider rng;
+
+ LoHiCachingIntProvider(UniformRandomProvider rng) {
+ this.rng = rng;
+ }
+
+ @Override
+ public int next() {
+ long next = source;
+ if (next < 0) {
+ // refill
+ next = rng.nextLong();
+ // store hi
+ source = next >>> 32;
+ // extract low
+ return (int) next;
+ }
+ final int v = (int) next;
+ // reset
+ source = -1;
+ return v;
+ }
+ }
+
+ private static class HiLoCachingIntProvider extends IntProvider {
+ private long source = -1;
+ private final UniformRandomProvider rng;
+
+ HiLoCachingIntProvider(UniformRandomProvider rng) {
+ this.rng = rng;
+ }
+
+ @Override
+ public int next() {
+ long next = source;
+ if (next < 0) {
+ // refill
+ next = rng.nextLong();
+ // store low
+ source = next & 0xffff_ffffL;
+ // extract hi
+ return (int) (next >>> 32);
+ }
+ final int v = (int) next;
+ // reset
+ source = -1;
+ return v;
+ }
+ }
}
diff --git a/commons-rng-examples/examples-stress/stress_test.md b/commons-rng-examples/examples-stress/stress_test.md
index dcd0f31..647a5d7 100644
--- a/commons-rng-examples/examples-stress/stress_test.md
+++ b/commons-rng-examples/examples-stress/stress_test.md
@@ -108,7 +108,7 @@ The `RNG_test` tool is used to accept raw bits via standard input for testing.
Test platform Endianness
------------------------
-The stress test application will output raw binary data for generated integers or longs.
+The stress test application will output raw binary data for generated integers or longs.
An integer is 4 bytes and a long is 8 bytes and thus the byte order or [Endianness](https://en.wikipedia.org/wiki/Endianness) of the data
must be correct for the test application. The stress test application can support either big-endian
or little-endian format. The application will auto-detect the platform and will default to output
@@ -170,7 +170,7 @@ To run the stress test use the `stress` command and provide the following argume
| executable | The test tool | dieharder, stdin2testu01, PractRand |
| ... | Arguments for the test tool | dieharder: -a -g 200 -Y 1 -k 2 <br/> stdin2testu01: SmallCrush, Crush, BigCrush <br/> RNG_test stdin32 -tf 1 -te 0 -tlmin 1KB -tlmax 4TB |
-The `stress` command defaults for other options that can be changed, for example the output
+The `stress` command defaults for other options can be changed, for example the output
results file prefix, the number of concurrent tasks, byte-order or the number of trials per
generator.
Use the `--help` option to show the available options.
@@ -250,24 +250,42 @@ The output results can be viewed using the `results` command:
java -jar target/examples-stress.jar results \
target/tu_* \
target/dh_* \
- target/pr_* \
+ target/pr_*
Various formats are available. Use the `--help` option to show the available options.
Test 64-bit generators
----------------------
-The available random generators output either 32-bits or 64-bits per cycle.
+The available random generators output either 32-bits or 64-bits per cycle.
Any application that supports 64-bit generators and should be tested using the `--raw64`
output mode of the `stress` command. See the example for running tests using **PractRand**
for details.
Any application that supports 32-bit generators can be tested using different subsets of the
-64-bit output. For example the test applications **Dieharder** and **TestU01** require 32-bit input.
-The standard method for a 64-bit generator is to use the upper and then lower 32-bits of each
-64-bit output. The stress test application has options to use only the upper or the lower 32-bits
-for testing. These can then be bit-reversed or byte-reversed if desired.
+64-bit output using the `--source64` option. For example the test applications **Dieharder**
+and **TestU01** require 32-bit input.
+The standard method for a 64-bit generator is to use the full 64-bits of output in the order
+provided by the Commons RNG core caching implementation of the `nextInt` method. The stress
+test application has options to use the following parts of the 64-bit output:
+
+| source64 | Description |
+| --------- | ---------- |
+| LONG | Use the full 64-bit output. This is equivalent to using the `--raw64` mode. |
+| INT | Use the 32-bit output from the RNG implementation of nextInt. |
+| HI_LO | Use the upper 32-bits, then lower 32-bits of the 64-bit output. |
+| LO_HI | Use the lower 32-bits, then upper 32-bits of the 64-bit output. |
+| HI | Use the upper 32-bits of the 64-bit output. |
+| LO | Use the lower 32-bits of the 64-bit output. |
+
+Note: The default uses an explicit mode rather than INT so that it is recorded in the stress
+test output file. If the RNG uses the default implementation for `nextInt` then output
+from INT should match LO_HI.
+
+On some platforms the LONG setting is matched by a split of the upper and lower parts.
+The binary output from LONG should match LO_HI if using little-endian format;
+it should match HI_LO if using big-endian format.
The `list` command can output available generators by provider type. For example to output the
64-bit providers to a `rng64.list` file:
@@ -280,11 +298,14 @@ order using **BigCrush**:
java -jar target/examples-stress.jar stress \
--list rng64.list \
- --low-bits \
+ --source64 LO \
--reverse-bits \
--prefix target/tu_lo_r_ \
./stdin2testu01 \
BigCrush
-If a 32-bit provider is used with the `--low-bits` or `--upper-bits` options then an error
-message is shown.
+If a 32-bit provider is used with the `--source64=LONG` or `--raw64` options then
+an error message is shown as this is explicitly for 64-bit output. Other `source64`
+options split the 64-bit output for testing with a 32-bit stress test application. These
+options are ignored for any 32-bit provider; this allows testing 64-bit and 32-bit
+providers in the same batch execution.