[GitHub] asitstands opened a new issue #10369: Proper seeding of the random number generators for parallel CPU threads and multiple GPU devices

[GitBox <gi...@apache.org>]

asitstands opened a new issue #10369: Proper seeding of the random number generators for parallel CPU threads and multiple GPU devices
URL: https://github.com/apache/incubator-mxnet/issues/10369
 
 
   It looks like the random number generators for CPU threads are seeded with consecutive integers. https://github.com/apache/incubator-mxnet/blob/5a480f0237c28231bcf486f39f917b65513ee6a3/src/common/random_generator.h#L92 So if the first generator is seeded with `s`, the `n`-th generator is seeded with `s+n-1`. Is it right? If so, is there a reason? Of course, seeding `std::mt19937` generators with consecutive 32 bit integers does not mean setting their internal states with consecutive integers. However, as far as I know, it is not a good way to seed the generators for parallel use. It could introduce unexpected correlations between random sequences.
   
   The best way may be to divide the entire random number sequence that `mt19937` generates into `N` non-overlapping subsequences (`N` is the number of generators) and seed each generator to produce one of the subsequences. `mt19937` has period of `2^19937`. That is a huge number and, in practice, we can seed the generators to just skip over enough number of random numbers instead of dividing by `N`. C++'s random engines, including `mt19937`, provides `discard(long long n)` method to skip over `n` numbers. We can use this to seed the parallel generators so that the generators safely generate non-overlapping sequence of 2^64 numbers. Or we can call `discard` multiple times for logner sequence. The method is fast enough and usually seeding is not frequent. I think that 2^64 is an enough number for usual use cases. I'm not sure whether the current seeding method provides any guarantee like this.
   
   I think that seeding multiple GPU devices also has the similar problem. cuRAND takes care of the forementioned issue for the threads in a device but we need to care multuple devices. Is the current formula using the magic number and the device id a proven jump function for Philox4_32_10 generator? https://github.com/apache/incubator-mxnet/blob/5a480f0237c28231bcf486f39f917b65513ee6a3/src/resource.cc#L306 If not, I think that we can utilize `skipahead` function which has the same effect of `discard` of c++ standard.
   
   Random number generations could have subtle issues and I'm not aware of the past discussions on the current implementation, so I could miss something. If the above argument looks sound to the community, I'll work on this issue.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services