[GitHub] [incubator-mxnet] szha commented on a change in pull request #17547: Fix cudnn Dropout reproducibility

[GitBox <gi...@apache.org>]

szha commented on a change in pull request #17547:
URL: https://github.com/apache/incubator-mxnet/pull/17547#discussion_r413213237



##########
File path: src/operator/nn/dropout-inl.h
##########
@@ -255,8 +255,13 @@ class DropoutOp {
       Stream<xpu> *s = ctx.get_stream<xpu>();
 
       // set dropout state.
-      ctx.requested[0].get_cudnn_dropout_desc(&dropout_desc_, s, 1.0f - this->pkeep_, seed_);
-
+      Random<xpu, unsigned> *prnd = ctx.requested[1].get_random<xpu, unsigned>(s);
+      uint64_t rng_seed = prnd->GetSeed();
+      // reset dropout descriptor if rng seed changed.
+      bool reset = seed_ != rng_seed;
+      seed_ = rng_seed;
+      ctx.requested[0].get_cudnn_dropout_desc(&dropout_desc_, s, 1.0f - this->pkeep_,
+          seed_, reset);

Review comment:
       @roywei there are more than 1 random number generators in the resources (4 by default) and they could have different seeds. The result of rotating random number generator seeds here is that cudnn dropout state is reinitialized very often. @sxjscience observed that there's significant performance impact because of this.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org