You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Alex Herbert <al...@gmail.com> on 2019/10/07 13:43:49 UTC
[rng] BigCrush results
I have found a bug in the stress test application that affects all the
BigCrush results.
This was found when updating the stress test application to support
64-bit testing within PractRand, and testing of upper and lower 32-bits
from 64-bit out. An initial attempt to show that the lower 32-bits, bit
reversed, of the XOR_SHIFT_1024_S systematically fail Linear Complexity
tests did not work.
The bug was a dual reversal of the platform byte-order. Code was added
to reverse the byte order in the link c application that read from stdin
and passes numbers to BigCrush. Later code was added to reverse the byte
order in the Java application so allowing Dieharder to work. The
original c application should have been updated but was not (despite the
fact I thought I had done this). So the byte order was not correct after
a dual reversal and the BigCrush results refer to a byte reversed output
sequence.
I fixed the code and have verified that the stress test application can
identify Linear Complexity failures in XOR_SHIFT_1024_S.
I am rerunning BigCrush for the other generators.
Testing of generators with PractRand is still under trial and will
probably not make it for the 1.3 release. The only outstanding item for
1.3 is continued work on updating the user guide.
Alex
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [rng] BigCrush results
Posted by Gilles Sadowski <gi...@gmail.com>.
Hello.
Le ven. 11 oct. 2019 à 15:54, Alex Herbert <al...@gmail.com> a écrit :
>
>
> On 07/10/2019 15:00, Gilles Sadowski wrote:
> > Hi.
> >
> > Le lun. 7 oct. 2019 à 15:54, Alex Herbert <al...@gmail.com> a écrit :
> >> I have found a bug in the stress test application that affects all the
> >> BigCrush results.
> >>
> >> This was found when updating the stress test application to support
> >> 64-bit testing within PractRand, and testing of upper and lower 32-bits
> >> from 64-bit out. An initial attempt to show that the lower 32-bits, bit
> >> reversed, of the XOR_SHIFT_1024_S systematically fail Linear Complexity
> >> tests did not work.
> >>
> >> The bug was a dual reversal of the platform byte-order. Code was added
> >> to reverse the byte order in the link c application that read from stdin
> >> and passes numbers to BigCrush. Later code was added to reverse the byte
> >> order in the Java application so allowing Dieharder to work. The
> >> original c application should have been updated but was not (despite the
> >> fact I thought I had done this). So the byte order was not correct after
> >> a dual reversal and the BigCrush results refer to a byte reversed output
> >> sequence.
> >>
> >> I fixed the code and have verified that the stress test application can
> >> identify Linear Complexity failures in XOR_SHIFT_1024_S.
> >>
> >> I am rerunning BigCrush for the other generators.
> >>
> >> Testing of generators with PractRand is still under trial and will
> >> probably not make it for the 1.3 release. The only outstanding item for
> >> 1.3 is continued work on updating the user guide.
> > The userguide (and web site) can be updated at any time (between
> > releases); it is not be a blocker.
>
> BigCrush has finished. Not much changed.
>
> I've added an option to look for systematic failures to the stress test
> application. It is shown in the below table using the intersection
> symbol (view these tables using a fixed width font):
>
> RNG New ∩ Old ∩
> JDK 50,51,52,49,51 48 74,72,74,73,74 70
> WELL_512_A 6,7,8,6,6 6 7,6,6,6,6 6
> WELL_1024_A 5,4,5,5,4 4 4,4,5,4,4 4
> WELL_19937_A 2,2,3,3,3 2 3,3,2,2,2 2
> WELL_19937_C 4,2,2,2,2 2 2,2,3,2,2 2
> WELL_44497_A 3,2,2,2,3 2 2,2,2,2,3 2
> WELL_44497_B 2,2,2,2,2 2 2,3,2,2,2 2
> MT 2,3,2,2,3 2 2,3,2,2,2 2
> ISAAC 1,1,0,0,1 0 0,1,2,0,0 0
> SPLIT_MIX_64 0,0,0,1,0 0 1,0,0,0,0 0
> XOR_SHIFT_1024_S 1,0,0,0,2 0 0,0,0,0,0 0
> TWO_CMRES 0,1,0,0,0 0 4,3,3,5,4 3
> MT_64 2,2,2,2,2 2 2,3,2,2,2 2
> MWC_256 0,1,1,1,0 0 0,0,0,2,0 0
> KISS 1,1,0,0,0 0 0,0,0,0,0 0
> XOR_SHIFT_1024_S_PHI 0,2,0,0,1 0 0,0,0,0,0 0
> XO_RO_SHI_RO_64_S 1,2,3,1,1 1 1,1,2,1,3 1
> XO_RO_SHI_RO_64_SS 0,1,0,0,0 0 0,0,0,0,0 0
> XO_SHI_RO_128_PLUS 0,0,1,0,0 0 1,2,2,1,1 1
> XO_SHI_RO_128_SS 1,0,1,0,0 0 0,1,0,0,0 0
> XO_RO_SHI_RO_128_PLUS 1,0,0,0,0 0 0,1,0,0,0 0
> XO_RO_SHI_RO_128_SS 1,1,1,0,0 0 1,0,1,0,0 0
> XO_SHI_RO_256_PLUS 1,0,0,0,0 0 0,0,0,0,0 0
> XO_SHI_RO_256_SS 0,0,0,0,1 0 0,1,0,2,1 0
> XO_SHI_RO_512_PLUS 0,2,0,0,0 0 0,0,0,2,2 0
> XO_SHI_RO_512_SS 0,0,0,0,0 0 0,1,0,1,0 0
> PCG_XSH_RR_32 0,0,0,0,0 0 0,0,0,0,0 0
> PCG_XSH_RS_32 0,1,2,1,0 0 1,0,1,1,0 0
> PCG_RXS_M_XS_64 0,1,0,0,0 0 0,0,1,0,0 0
> PCG_MCG_XSH_RR_32 0,0,0,0,0 0 0,0,0,0,0 0
> PCG_MCG_XSH_RS_32 2,1,0,1,0 0 1,1,0,0,0 0
> MSWS 0,0,0,1,2 0 0,0,0,1,0 0
> SFC_32 0,0,0,1,1 0 0,1,0,0,2 0
> SFC_64 0,0,0,1,2 0 0,1,0,0,1 0
> JSF_32 0,0,0,1,2 0 0,0,0,1,1 0
> JSF_64 0,0,2,0,0 0 1,1,0,0,0 0
> XO_SHI_RO_128_PP 0,0,0,1,1 0 1,0,0,1,0 0
> XO_RO_SHI_RO_128_PP 0,0,0,0,0 0 0,1,0,1,0 0
> XO_SHI_RO_256_PP 0,1,1,0,1 0 0,0,0,0,1 0
> XO_SHI_RO_512_PP 0,0,1,0,0 0 0,0,2,1,0 0
> XO_RO_SHI_RO_1024_PP 0,0,3,0,1 0 0,0,0,0,0 0
> XO_RO_SHI_RO_1024_S 0,1,0,0,0 0 0,0,0,1,0 0
> XO_RO_SHI_RO_1024_SS 0,1,0,0,0 0 0,1,0,0,1 0
>
> Notable changes:
>
> JDK : 74,72,74,73,74 -> 50,51,52,49,51
> TWO_CMRES : 4,3,3,5,4 -> 0,1,0,0,0
> XO_SHI_RO_128_PLUS : 1,2,2,1,1 -> 0,0,1,0,0
>
>
> Changes of systematic failures:
>
> JDK : 70 -> 48
> TWO_CMRES : 3 -> 0
> XO_SHI_RO_128_PLUS : 1 -> 0
>
>
> Unchanged systematic failures:
>
> WELL_* and MT variants systematically fail linear complexity. WELL_512
> and WELL_1024 also fail matrix rank.
>
> The other generators have similar results before and after, i.e. no
> systematic failures.
>
> Of note the PCG RR variants never fail. This was true in the old results
> too.
>
>
> I've added an option to the results command that generates the table for
> the user guide to add the systematic failures in parentheses after the
> failure counts, e.g.
>
> XO_SHI_RO_512_PP 0,0,1,0,0 (0)
>
> I am not sure if this is worthwhile. There are no cases amongst the good
> generators (no systematic failures) where there are sporadic failures in
> all 5 test runs. Thus all the good generators have at least one run with
> no failures.
Perhaps, you could put in bold the name of good generators
(those with no systematic failure). That would literally highlight
the recommendation to avoid "java.util.Random".
Regards,
Gilles
>
> So the value would be mainly zero except for the generators noted above.
>
> Alex
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [rng] BigCrush results
Posted by Alex Herbert <al...@gmail.com>.
On 07/10/2019 15:00, Gilles Sadowski wrote:
> Hi.
>
> Le lun. 7 oct. 2019 à 15:54, Alex Herbert <al...@gmail.com> a écrit :
>> I have found a bug in the stress test application that affects all the
>> BigCrush results.
>>
>> This was found when updating the stress test application to support
>> 64-bit testing within PractRand, and testing of upper and lower 32-bits
>> from 64-bit out. An initial attempt to show that the lower 32-bits, bit
>> reversed, of the XOR_SHIFT_1024_S systematically fail Linear Complexity
>> tests did not work.
>>
>> The bug was a dual reversal of the platform byte-order. Code was added
>> to reverse the byte order in the link c application that read from stdin
>> and passes numbers to BigCrush. Later code was added to reverse the byte
>> order in the Java application so allowing Dieharder to work. The
>> original c application should have been updated but was not (despite the
>> fact I thought I had done this). So the byte order was not correct after
>> a dual reversal and the BigCrush results refer to a byte reversed output
>> sequence.
>>
>> I fixed the code and have verified that the stress test application can
>> identify Linear Complexity failures in XOR_SHIFT_1024_S.
>>
>> I am rerunning BigCrush for the other generators.
>>
>> Testing of generators with PractRand is still under trial and will
>> probably not make it for the 1.3 release. The only outstanding item for
>> 1.3 is continued work on updating the user guide.
> The userguide (and web site) can be updated at any time (between
> releases); it is not be a blocker.
BigCrush has finished. Not much changed.
I've added an option to look for systematic failures to the stress test
application. It is shown in the below table using the intersection
symbol (view these tables using a fixed width font):
RNG New ∩ Old ∩
JDK 50,51,52,49,51 48 74,72,74,73,74 70
WELL_512_A 6,7,8,6,6 6 7,6,6,6,6 6
WELL_1024_A 5,4,5,5,4 4 4,4,5,4,4 4
WELL_19937_A 2,2,3,3,3 2 3,3,2,2,2 2
WELL_19937_C 4,2,2,2,2 2 2,2,3,2,2 2
WELL_44497_A 3,2,2,2,3 2 2,2,2,2,3 2
WELL_44497_B 2,2,2,2,2 2 2,3,2,2,2 2
MT 2,3,2,2,3 2 2,3,2,2,2 2
ISAAC 1,1,0,0,1 0 0,1,2,0,0 0
SPLIT_MIX_64 0,0,0,1,0 0 1,0,0,0,0 0
XOR_SHIFT_1024_S 1,0,0,0,2 0 0,0,0,0,0 0
TWO_CMRES 0,1,0,0,0 0 4,3,3,5,4 3
MT_64 2,2,2,2,2 2 2,3,2,2,2 2
MWC_256 0,1,1,1,0 0 0,0,0,2,0 0
KISS 1,1,0,0,0 0 0,0,0,0,0 0
XOR_SHIFT_1024_S_PHI 0,2,0,0,1 0 0,0,0,0,0 0
XO_RO_SHI_RO_64_S 1,2,3,1,1 1 1,1,2,1,3 1
XO_RO_SHI_RO_64_SS 0,1,0,0,0 0 0,0,0,0,0 0
XO_SHI_RO_128_PLUS 0,0,1,0,0 0 1,2,2,1,1 1
XO_SHI_RO_128_SS 1,0,1,0,0 0 0,1,0,0,0 0
XO_RO_SHI_RO_128_PLUS 1,0,0,0,0 0 0,1,0,0,0 0
XO_RO_SHI_RO_128_SS 1,1,1,0,0 0 1,0,1,0,0 0
XO_SHI_RO_256_PLUS 1,0,0,0,0 0 0,0,0,0,0 0
XO_SHI_RO_256_SS 0,0,0,0,1 0 0,1,0,2,1 0
XO_SHI_RO_512_PLUS 0,2,0,0,0 0 0,0,0,2,2 0
XO_SHI_RO_512_SS 0,0,0,0,0 0 0,1,0,1,0 0
PCG_XSH_RR_32 0,0,0,0,0 0 0,0,0,0,0 0
PCG_XSH_RS_32 0,1,2,1,0 0 1,0,1,1,0 0
PCG_RXS_M_XS_64 0,1,0,0,0 0 0,0,1,0,0 0
PCG_MCG_XSH_RR_32 0,0,0,0,0 0 0,0,0,0,0 0
PCG_MCG_XSH_RS_32 2,1,0,1,0 0 1,1,0,0,0 0
MSWS 0,0,0,1,2 0 0,0,0,1,0 0
SFC_32 0,0,0,1,1 0 0,1,0,0,2 0
SFC_64 0,0,0,1,2 0 0,1,0,0,1 0
JSF_32 0,0,0,1,2 0 0,0,0,1,1 0
JSF_64 0,0,2,0,0 0 1,1,0,0,0 0
XO_SHI_RO_128_PP 0,0,0,1,1 0 1,0,0,1,0 0
XO_RO_SHI_RO_128_PP 0,0,0,0,0 0 0,1,0,1,0 0
XO_SHI_RO_256_PP 0,1,1,0,1 0 0,0,0,0,1 0
XO_SHI_RO_512_PP 0,0,1,0,0 0 0,0,2,1,0 0
XO_RO_SHI_RO_1024_PP 0,0,3,0,1 0 0,0,0,0,0 0
XO_RO_SHI_RO_1024_S 0,1,0,0,0 0 0,0,0,1,0 0
XO_RO_SHI_RO_1024_SS 0,1,0,0,0 0 0,1,0,0,1 0
Notable changes:
JDK : 74,72,74,73,74 -> 50,51,52,49,51
TWO_CMRES : 4,3,3,5,4 -> 0,1,0,0,0
XO_SHI_RO_128_PLUS : 1,2,2,1,1 -> 0,0,1,0,0
Changes of systematic failures:
JDK : 70 -> 48
TWO_CMRES : 3 -> 0
XO_SHI_RO_128_PLUS : 1 -> 0
Unchanged systematic failures:
WELL_* and MT variants systematically fail linear complexity. WELL_512
and WELL_1024 also fail matrix rank.
The other generators have similar results before and after, i.e. no
systematic failures.
Of note the PCG RR variants never fail. This was true in the old results
too.
I've added an option to the results command that generates the table for
the user guide to add the systematic failures in parentheses after the
failure counts, e.g.
XO_SHI_RO_512_PP 0,0,1,0,0 (0)
I am not sure if this is worthwhile. There are no cases amongst the good
generators (no systematic failures) where there are sporadic failures in
all 5 test runs. Thus all the good generators have at least one run with
no failures.
So the value would be mainly zero except for the generators noted above.
Alex
>
> Best regards,
> Gilles
>
>> Alex
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
Re: [rng] BigCrush results
Posted by Alex Herbert <al...@gmail.com>.
On 07/10/2019 15:00, Gilles Sadowski wrote:
> Hi.
>
> Le lun. 7 oct. 2019 à 15:54, Alex Herbert <al...@gmail.com> a écrit :
>> I have found a bug in the stress test application that affects all the
>> BigCrush results.
>>
>> This was found when updating the stress test application to support
>> 64-bit testing within PractRand, and testing of upper and lower 32-bits
>> from 64-bit out. An initial attempt to show that the lower 32-bits, bit
>> reversed, of the XOR_SHIFT_1024_S systematically fail Linear Complexity
>> tests did not work.
>>
>> The bug was a dual reversal of the platform byte-order. Code was added
>> to reverse the byte order in the link c application that read from stdin
>> and passes numbers to BigCrush. Later code was added to reverse the byte
>> order in the Java application so allowing Dieharder to work. The
>> original c application should have been updated but was not (despite the
>> fact I thought I had done this). So the byte order was not correct after
>> a dual reversal and the BigCrush results refer to a byte reversed output
>> sequence.
>>
>> I fixed the code and have verified that the stress test application can
>> identify Linear Complexity failures in XOR_SHIFT_1024_S.
>>
>> I am rerunning BigCrush for the other generators.
>>
>> Testing of generators with PractRand is still under trial and will
>> probably not make it for the 1.3 release. The only outstanding item for
>> 1.3 is continued work on updating the user guide.
> The userguide (and web site) can be updated at any time (between
> releases); it is not be a blocker.
OK.
I am just going through adding the new parts of the code to it. As part
of this process I am finding a few niggles that I am fixing. I'm also
addressing the SonarCloud report. So far I've found a few things that
add value to the release so I'd prefer to finish this. I think in a week
it should be all done. Getting a release out would be good.
> Best regards,
> Gilles
>
>> Alex
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [rng] BigCrush results
Posted by Gilles Sadowski <gi...@gmail.com>.
Hi.
Le lun. 7 oct. 2019 à 15:54, Alex Herbert <al...@gmail.com> a écrit :
>
> I have found a bug in the stress test application that affects all the
> BigCrush results.
>
> This was found when updating the stress test application to support
> 64-bit testing within PractRand, and testing of upper and lower 32-bits
> from 64-bit out. An initial attempt to show that the lower 32-bits, bit
> reversed, of the XOR_SHIFT_1024_S systematically fail Linear Complexity
> tests did not work.
>
> The bug was a dual reversal of the platform byte-order. Code was added
> to reverse the byte order in the link c application that read from stdin
> and passes numbers to BigCrush. Later code was added to reverse the byte
> order in the Java application so allowing Dieharder to work. The
> original c application should have been updated but was not (despite the
> fact I thought I had done this). So the byte order was not correct after
> a dual reversal and the BigCrush results refer to a byte reversed output
> sequence.
>
> I fixed the code and have verified that the stress test application can
> identify Linear Complexity failures in XOR_SHIFT_1024_S.
>
> I am rerunning BigCrush for the other generators.
>
> Testing of generators with PractRand is still under trial and will
> probably not make it for the 1.3 release. The only outstanding item for
> 1.3 is continued work on updating the user guide.
The userguide (and web site) can be updated at any time (between
releases); it is not be a blocker.
Best regards,
Gilles
>
> Alex
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org