You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/08/05 21:26:48 UTC
[GitHub] [incubator-mxnet] dcmaddix opened a new issue #18865: mx.random.seed does not give deterministic results on multi-cpus
dcmaddix opened a new issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865
## Description
I am trying to fix the random seed across multiple cpus and followed the example to pass `ctx` here : https://mxnet.apache.org/versions/1.6/api/python/docs/api/mxnet/random/index.html and the example code works but it seems order dependent. See the output below. If I call mx.random.seed(128, ctx=mx.cpu(0)) and mx.random.seed(128, ctx=mx.cpu(1)), it does not work. Also is this the only way to loop over all the num_cpus and set each ctx?
## To Reproduce
`mx.random.seed(128, ctx=mx.cpu(0))
print(mx.nd.random.normal(shape=(2,2), ctx=mx.cpu(0)).asnumpy())
[[ 0.47400656 0.20251541]
[ 1.3648157 -1.4962182 ]]
mx.random.seed(128, ctx=mx.cpu(1))
print(mx.nd.random.normal(shape=(2,2), ctx=mx.cpu(1)).asnumpy())
[[ 0.47400656 0.20251541]
[ 1.3648157 -1.4962182 ]]
`
`mx.random.seed(128, ctx=mx.cpu(0))
mx.random.seed(128, ctx=mx.cpu(1))
print(mx.nd.random.normal(shape=(2,2), ctx=mx.cpu(0)).asnumpy())
[[ 0.47400656 0.20251541]
[ 1.3648157 -1.4962182 ]]
print(mx.nd.random.normal(shape=(2,2), ctx=mx.cpu(1)).asnumpy())
[[ 1.0954498 -0.20808885]
[ 1.590508 -0.41777727]]
`
The second arrays do not match and I am not sure why because the seed is set. We also see this when trying to use multi-processing in our algorithms and the numbers are very different even for fixed seed. Above is a simple representative example.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sxjscience commented on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
sxjscience commented on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-670056196
However resetting the seed of cpu(1) should give us deterministic results at CPU(1). The current behavior is not expected.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-669594144
`mx.cpu(0)` is not really different from `mx.cpu(1)`. There is only one random resource for CPUs, whereas there is one per device-id for GPUs:
https://github.com/apache/incubator-mxnet/blob/6bbd53107aa16fc41e8d462cf5dc46fb70d592df/src/resource.cc#L394-L402
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sxjscience commented on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
sxjscience commented on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-669635070
You may see also the issue: https://github.com/apache/incubator-mxnet/issues/15662
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-670057042
It's not expected and should be fixed. I'd just like to point out the apparent root cause.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] joeyism commented on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
joeyism commented on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-825774901
Is there any updates to this problem? It's kind of a big deal because models are producing slightly different results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] github-actions[bot] commented on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-669520996
Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue.
Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly.
If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on [contributing to MXNet](https://mxnet.apache.org/community/contribute) and our [development guides wiki](https://cwiki.apache.org/confluence/display/MXNET/Developments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-670057042
It's not expected and should be fixed. I'd just like to point out the root cause.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] dcmaddix commented on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
dcmaddix commented on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-669645930
> `mx.cpu(0)` is not really different from `mx.cpu(1)`. There is only one random resource for CPUs, whereas there is one per device-id for GPUs:
>
> https://github.com/apache/incubator-mxnet/blob/6bbd53107aa16fc41e8d462cf5dc46fb70d592df/src/resource.cc#L394-L402
Why is the result different than on different cpus? The seed does not seem to be fixed in my example. Thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] joeyism edited a comment on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
joeyism edited a comment on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-825774901
Are there any updates to this problem? It's kind of a big deal because models are producing slightly different results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-670053997
> Why is the result different than on different cpus? The seed does not seem to be fixed in my example. Thank you!
You are running two random operations in sequence. The first operation uses the fixed seed but modifies the internal state of the random resource. My point is that even though you specify different contexts (`mx.cpu(0), mx.cpu(1)`), they rely on the same shared random state.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18865: mx.random.seed does not give deterministic results on multi-cpus
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18865:
URL: https://github.com/apache/incubator-mxnet/issues/18865#issuecomment-670053997
> Why is the result different than on different cpus? The seed does not seem to be fixed in my example. Thank you!
You are running two random operations in sequence. The first operation uses the fixed seed but modifies the internal state of the random resource.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org