You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/09/02 15:31:22 UTC
[GitHub] [incubator-mxnet] bricksdont opened a new issue #19073: Multinomial sampling produces impossible values
bricksdont opened a new issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073
## Description
Multinomial sampling produces impossible values.
If I understand correctly, if `mx.random.multinomial` is given a distribution such as:
dist = mx.nd.array([0.3, 0.2, 0.4, 0.1])
sample = mx.random.multinomial(dist)
Since, [according to the documentation, outputs are 0-indexed,](https://mxnet.incubator.apache.org/versions/1.6/api/python/docs/api/ndarray/random/index.html#mxnet.ndarray.random.multinomial) the sample value can be either 0, 1, 2 or 3 - but never 4. But in Sockeye and in simple tests (see below), very often 5 would be returned.
### Error Message
The actual error appears in Sockeye, please refer to this issue for details:
https://github.com/awslabs/sockeye/issues/866
To summarize, calling `random.multinomial` on some occasions returns impossible values.
## To Reproduce
To reproduce the Sockeye error that is thrown:
https://gist.github.com/bricksdont/58dfa0964201c83961a30f23406baa5d
To reproduce the MXnet behaviour in isolation:
https://gist.github.com/bricksdont/e0304b9cc6ecbcc868368353d6e6262c
### Steps to reproduce
Run the shell scripts as indicated in the gists above, e.g.
python sample_mxnet.py --seed 1 --context cpu --num-trials 10 --dist-size 20
## Environment
We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:
```
curl --retry 10 -s https://raw.githubusercontent.com/apache/incubator-mxnet/master/tools/diagnose.py | python
----------Python Info----------
Version : 3.5.2
Compiler : GCC 5.4.0 20160609
Build : ('default', 'Apr 16 2020 17:47:17')
Arch : ('64bit', 'ELF')
------------Pip Info-----------
Version : 20.2.2
Directory : /net/cephfs/scratch/mathmu/map-volatility/venvs/sockeye3-cpu/lib/python3.5/site-packages/pip
----------MXNet Info-----------
Version : 1.6.0
Directory : /net/cephfs/scratch/mathmu/map-volatility/venvs/sockeye3-cpu/lib/python3.5/site-packages/mxnet
Commit Hash : 6eec9da55c5096079355d1f1a5fa58dcf35d6752
Library : ['/net/cephfs/scratch/mathmu/map-volatility/venvs/sockeye3-cpu/lib/python3.5/site-packages/mxnet/libmxnet.so']
Build features:
✖ CUDA
✖ CUDNN
✖ NCCL
✖ CUDA_RTC
✖ TENSORRT
✔ CPU_SSE
✔ CPU_SSE2
✔ CPU_SSE3
✔ CPU_SSE4_1
✔ CPU_SSE4_2
✖ CPU_SSE4A
✔ CPU_AVX
✖ CPU_AVX2
✔ OPENMP
✖ SSE
✔ F16C
✖ JEMALLOC
✔ BLAS_OPEN
✖ BLAS_ATLAS
✖ BLAS_MKL
✖ BLAS_APPLE
✔ LAPACK
✖ MKLDNN
✔ OPENCV
✖ CAFFE
✖ PROFILER
✔ DIST_KVSTORE
✖ CXX14
✖ INT64_TENSOR_SIZE
✔ SIGNAL_HANDLER
✖ DEBUG
✖ TVM_OP
----------System Info----------
Platform : Linux-4.15.0-101-generic-x86_64-with-Ubuntu-16.04-xenial
system : Linux
node : login0
release : 4.15.0-101-generic
version : #102~16.04.1-Ubuntu SMP Mon May 11 11:38:16 UTC 2020
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 8
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 60
Model name: Intel Core Processor (Haswell, no TSX)
Stepping: 1
CPU MHz: 2394.454
BogoMIPS: 4788.90
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx rdtscp lm constant_tsc rep_good nopl cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti fsgsbase bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat
----------Network Test----------
Setting timeout: 10
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0138 sec, LOAD: 0.4219 sec.
Error open Conda: https://repo.continuum.io/pkgs/free/, HTTP Error 403: Forbidden, DNS finished in 0.003739595413208008 sec.
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0034 sec, LOAD: 0.5792 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0445 sec, LOAD: 0.7186 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0868 sec, LOAD: 0.2290 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.2111 sec, LOAD: 0.7232 sec.
----------Environment----------
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] sxjscience commented on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
sxjscience commented on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-723237552
@bricksdont The binary search here does not seem to be correct: https://github.com/apache/incubator-mxnet/blob/9d440868603ad26b702e12ddd2587e5c4b56e42b/src/operator/random/sample_multinomial_op.h#L128-L143
We may choose to directly use the discrete distribution in std c++ http://www.cplusplus.com/reference/random/discrete_distribution/ @xidulu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bricksdont removed a comment on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
bricksdont removed a comment on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-685823792
Not sure if this really is a bug - closing to investigate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] sxjscience commented on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
sxjscience commented on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-723280696
Also ping @zixuanweeei @pengzhao-intel
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] sxjscience edited a comment on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
sxjscience edited a comment on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-723237552
@bricksdont The binary search here does not seem to be correct: https://github.com/apache/incubator-mxnet/blob/9d440868603ad26b702e12ddd2587e5c4b56e42b/src/operator/random/sample_multinomial_op.h#L128-L143
We may choose to directly use the discrete distribution in std c++ http://www.cplusplus.com/reference/random/discrete_distribution/ or use the std functions for binary search.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] xidulu commented on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
xidulu commented on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-724673978
In short, you could add a check after the while loop (before https://github.com/apache/incubator-mxnet/blob/9d440868603ad26b702e12ddd2587e5c4b56e42b/src/operator/random/sample_multinomial_op.h#L141) to see if left == K (which is not in [0, K) ), if that happens, you should manually call `left--` before so that the result still lies in [0, K).
I would love to help you but unfortunately I currently do not have any device capable of to build MXNet from source.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] sxjscience edited a comment on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
sxjscience edited a comment on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-723242017
@bricksdont Would you be willing to help fix it? To reproduce the wrong behavior, let's consider
```python
import mxnet as mx
dist = mx.nd.array([0.9])
for _ in range(100):
out = mx.random.multinomial(dist).asscalar()
if out == 1:
print('Error!!!!')
```
Output:
```
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
```
We may try to use `std::upper_bound` and `thrust::upper_bound`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bricksdont commented on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
bricksdont commented on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-724521983
@xidulu Thanks! What exactly do you suggest be changed in the code? Could you open a PR for this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bricksdont commented on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
bricksdont commented on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-723260080
Hi @sxjscience, thanks for reviving this issue and providing a **much** simpler way to reproduce!
Unfortunately, I lack the C++ skills to read this reliably and fix it. I hope someone else can take this on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] sxjscience commented on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
sxjscience commented on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-723242017
@bricksdont Would you be willing to help fix it? To reproduce the wrong behavior, let's consider
```python
import mxnet as mx
dist = mx.nd.array([0.9])
for _ in range(100):
out = mx.random.multinomial(dist).asscalar()
if out == 1:
print('Error!!!!')
```
Output:
```
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
Error!!!!
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] xidulu edited a comment on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
xidulu edited a comment on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-724673978
In short, you could add a check after the while loop (before https://github.com/apache/incubator-mxnet/blob/9d440868603ad26b702e12ddd2587e5c4b56e42b/src/operator/random/sample_multinomial_op.h#L141) to see if left == K (which is not in [0, K) ), if that happens, you should manually call `left--` before so that the result still lies in [0, K).
I would love to open a PR for that but unfortunately I currently do not have any device capable of to build MXNet from source.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] xidulu commented on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
xidulu commented on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-723400122
@sxjscience
The implementation of the binary search is correct.
The problem behind is that the binary search would return the index of the first element larger or equal to `value`. When no such element is found, i.e. `value` is larger than all elements in [0, K), index K will be returned, which is invalid.
In such case, one should add a special check after the loop, just like the origin version of multinomial:
https://sourcegraph.com/github.com/apache/incubator-mxnet/-/commit/e6fad30e45e6ec0ddef5c18093e8163cd2a7c62c#diff-dfa5ed462be234eb86a84d01466756cfL140
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] sxjscience edited a comment on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
sxjscience edited a comment on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-723237552
@bricksdont The binary search here does not seem to be correct: https://github.com/apache/incubator-mxnet/blob/9d440868603ad26b702e12ddd2587e5c4b56e42b/src/operator/random/sample_multinomial_op.h#L128-L143
We may choose to directly use the discrete distribution in std c++ http://www.cplusplus.com/reference/random/discrete_distribution/ or use `std::upper_bound` directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bricksdont closed issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
bricksdont closed issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] bricksdont commented on issue #19073: Multinomial sampling produces impossible values
Posted by GitBox <gi...@apache.org>.
bricksdont commented on issue #19073:
URL: https://github.com/apache/incubator-mxnet/issues/19073#issuecomment-685823792
Not sure if this really is a bug - closing to investigate
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org