You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/09/25 22:15:47 UTC
[GitHub] [incubator-mxnet] leezu opened a new issue #19231: opperf memory corruption crash after stack operator
leezu opened a new issue #19231:
URL: https://github.com/apache/incubator-mxnet/issues/19231
## Description
Running the opperf benchmark on the master branch causes memory corruption and crash after the `stack` operator
### Error Message
First try
```
INFO:root:Complete Benchmark - split
INFO:root:Begin Benchmark - stack
INFO:root:Complete Benchmark - stack
invalid fastbin entry (free)
zsh: abort (core dumped) PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 --output-format json
```
Second try
```
INFO:root:Begin Benchmark - split
INFO:root:Complete Benchmark - split
INFO:root:Begin Benchmark - stack
INFO:root:Complete Benchmark - stack
corrupted size vs. prev_size
zsh: abort (core dumped) PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 --output-format json
```
## To Reproduce
On Ubuntu 18.04, compile with `cmake -GNinja -DUSE_CUDA=0 ..; ninja`.
Then run the benchmark with `PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 mxnet-master/benchmark/opperf/opperf.py --output-format json --output-file mxnet_operator_benchmark_results.json`
cc @ChaiBapchya
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19231: opperf memory corruption crash after stack operator
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19231:
URL: https://github.com/apache/incubator-mxnet/issues/19231#issuecomment-699201393
Essentially it crashes for any set of operators:
If I disable anything but `run_mx_unary_operators_benchmarks`, it crashes as follows
```
INFO:root:Running MXNet operator benchmarks with the following options: Namespace(ctx='cpu', dtype='float32', int64_tensor='off', output_file='mxnet_operator_benchmark_results.json', output_format='json', profiler='native', runs=100, warmup=25)
[22:52:46] ../src/storage/storage.cc:199: Using Pooled (Naive) StorageManager for CPU
INFO:root:Begin Benchmark - amp_multicast
INFO:root:Complete Benchmark - amp_multicast
INFO:root:Begin Benchmark - BlockGrad
INFO:root:Complete Benchmark - BlockGrad
INFO:root:Begin Benchmark - abs
INFO:root:Complete Benchmark - abs
INFO:root:Begin Benchmark - amp_cast
INFO:root:Complete Benchmark - amp_cast
INFO:root:Begin Benchmark - arccos
INFO:root:Complete Benchmark - arccos
INFO:root:Begin Benchmark - arccosh
INFO:root:Complete Benchmark - arccosh
INFO:root:Begin Benchmark - arcsin
INFO:root:Complete Benchmark - arcsin
INFO:root:Begin Benchmark - arcsinh
INFO:root:Complete Benchmark - arcsinh
INFO:root:Begin Benchmark - arctan
INFO:root:Complete Benchmark - arctan
INFO:root:Begin Benchmark - arctanh
INFO:root:Complete Benchmark - arctanh
INFO:root:Begin Benchmark - argmax_channel
INFO:root:Complete Benchmark - argmax_channel
INFO:root:Begin Benchmark - cast
INFO:root:Complete Benchmark - cast
INFO:root:Begin Benchmark - cbrt
INFO:root:Complete Benchmark - cbrt
INFO:root:Begin Benchmark - ceil
INFO:root:Complete Benchmark - ceil
INFO:root:Begin Benchmark - cos
INFO:root:Complete Benchmark - cos
INFO:root:Begin Benchmark - cosh
INFO:root:Complete Benchmark - cosh
INFO:root:Begin Benchmark - degrees
INFO:root:Complete Benchmark - degrees
INFO:root:Begin Benchmark - digamma
INFO:root:Complete Benchmark - digamma
INFO:root:Begin Benchmark - erf
INFO:root:Complete Benchmark - erf
INFO:root:Begin Benchmark - erfinv
INFO:root:Complete Benchmark - erfinv
INFO:root:Begin Benchmark - exp
INFO:root:Complete Benchmark - exp
INFO:root:Begin Benchmark - expm1
INFO:root:Complete Benchmark - expm1
INFO:root:Begin Benchmark - fix
INFO:root:Complete Benchmark - fix
INFO:root:Begin Benchmark - flatten
INFO:root:Complete Benchmark - flatten
INFO:root:Begin Benchmark - floor
INFO:root:Complete Benchmark - floor
INFO:root:Begin Benchmark - gamma
INFO:root:Complete Benchmark - gamma
INFO:root:Begin Benchmark - gammaln
INFO:root:Complete Benchmark - gammaln
INFO:root:Begin Benchmark - identity
INFO:root:Complete Benchmark - identity
INFO:root:Begin Benchmark - log
INFO:root:Complete Benchmark - log
INFO:root:Begin Benchmark - log10
INFO:root:Complete Benchmark - log10
INFO:root:Begin Benchmark - log1p
INFO:root:Complete Benchmark - log1p
INFO:root:Begin Benchmark - log2
INFO:root:Complete Benchmark - log2
INFO:root:Begin Benchmark - logical_not
INFO:root:Complete Benchmark - logical_not
INFO:root:Begin Benchmark - make_loss
INFO:root:Complete Benchmark - make_loss
INFO:root:Begin Benchmark - negative
INFO:root:Complete Benchmark - negative
INFO:root:Begin Benchmark - ones_like
INFO:root:Complete Benchmark - ones_like
INFO:root:Begin Benchmark - radians
INFO:root:Complete Benchmark - radians
INFO:root:Begin Benchmark - rcbrt
INFO:root:Complete Benchmark - rcbrt
INFO:root:Begin Benchmark - reciprocal
INFO:root:Complete Benchmark - reciprocal
INFO:root:Begin Benchmark - relu
INFO:root:Complete Benchmark - relu
INFO:root:Begin Benchmark - rint
INFO:root:Complete Benchmark - rint
INFO:root:Begin Benchmark - round
INFO:root:Complete Benchmark - round
INFO:root:Begin Benchmark - rsqrt
INFO:root:Complete Benchmark - rsqrt
INFO:root:Begin Benchmark - shape_array
INFO:root:Complete Benchmark - shape_array
INFO:root:Begin Benchmark - shuffle
INFO:root:Complete Benchmark - shuffle
INFO:root:Begin Benchmark - sigmoid
INFO:root:Complete Benchmark - sigmoid
INFO:root:Begin Benchmark - sign
INFO:root:Complete Benchmark - sign
INFO:root:Begin Benchmark - sin
INFO:root:Complete Benchmark - sin
INFO:root:Begin Benchmark - sinh
INFO:root:Complete Benchmark - sinh
INFO:root:Begin Benchmark - size_array
INFO:root:Complete Benchmark - size_array
INFO:root:Begin Benchmark - softsign
INFO:root:Complete Benchmark - softsign
INFO:root:Begin Benchmark - sqrt
INFO:root:Complete Benchmark - sqrt
INFO:root:Begin Benchmark - square
INFO:root:Complete Benchmark - square
INFO:root:Begin Benchmark - stop_gradient
INFO:root:Complete Benchmark - stop_gradient
INFO:root:Begin Benchmark - tan
INFO:root:Complete Benchmark - tan
INFO:root:Begin Benchmark - tanh
INFO:root:Complete Benchmark - tanh
INFO:root:Begin Benchmark - trunc
INFO:root:Complete Benchmark - trunc
INFO:root:Begin Benchmark - zeros_like
INFO:root:Complete Benchmark - zeros_like
malloc_consolidate(): invalid chunk size
zsh: abort (core dumped) PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 --output-format json
PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 --output-format json 734.74s user 170.63s system 4594% cpu 19.704 total
```
If I disable anything but `run_mx_binary_broadcast_operators_benchmarks`, it crashes as follows
```
time PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 mxnet-master/benchmark/opperf/opperf.py --output-format json --output-file mxnet_operator_benchmark_results.json
INFO:root:Running MXNet operator benchmarks with the following options: Namespace(ctx='cpu', dtype='float32', int64_tensor='off', output_file='mxnet_operator_benchmark_results.json', output_format='json', profiler='native', runs=100, warmup=25)
[22:52:11] ../src/storage/storage.cc:199: Using Pooled (Naive) StorageManager for CPU
INFO:root:Begin Benchmark - broadcast_add
INFO:root:Complete Benchmark - broadcast_add
INFO:root:Begin Benchmark - broadcast_div
INFO:root:Complete Benchmark - broadcast_div
INFO:root:Begin Benchmark - broadcast_equal
INFO:root:Complete Benchmark - broadcast_equal
INFO:root:Begin Benchmark - broadcast_greater
INFO:root:Complete Benchmark - broadcast_greater
INFO:root:Begin Benchmark - broadcast_greater_equal
INFO:root:Complete Benchmark - broadcast_greater_equal
INFO:root:Begin Benchmark - broadcast_hypot
INFO:root:Complete Benchmark - broadcast_hypot
INFO:root:Begin Benchmark - broadcast_lesser
INFO:root:Complete Benchmark - broadcast_lesser
INFO:root:Begin Benchmark - broadcast_lesser_equal
INFO:root:Complete Benchmark - broadcast_lesser_equal
INFO:root:Begin Benchmark - broadcast_logical_and
INFO:root:Complete Benchmark - broadcast_logical_and
INFO:root:Begin Benchmark - broadcast_logical_or
INFO:root:Complete Benchmark - broadcast_logical_or
INFO:root:Begin Benchmark - broadcast_logical_xor
INFO:root:Complete Benchmark - broadcast_logical_xor
INFO:root:Begin Benchmark - broadcast_maximum
INFO:root:Complete Benchmark - broadcast_maximum
INFO:root:Begin Benchmark - broadcast_minimum
INFO:root:Complete Benchmark - broadcast_minimum INFO:root:Begin Benchmark - broadcast_minus INFO:root:Complete Benchmark - broadcast_minus INFO:root:Begin Benchmark - broadcast_mod
INFO:root:Complete Benchmark - broadcast_mod INFO:root:Begin Benchmark - broadcast_mul INFO:root:Complete Benchmark - broadcast_mul
INFO:root:Begin Benchmark - broadcast_not_equal
INFO:root:Complete Benchmark - broadcast_not_equal
INFO:root:Begin Benchmark - broadcast_plus
INFO:root:Complete Benchmark - broadcast_plus
INFO:root:Begin Benchmark - broadcast_power
INFO:root:Complete Benchmark - broadcast_power
INFO:root:Begin Benchmark - broadcast_sub
INFO:root:Complete Benchmark - broadcast_sub
invalid fastbin entry (free)
zsh: abort (core dumped) PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 --output-format json
```
@ChaiBapchya the issue may be with the opperf setup itself. Would you have time to fix it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19231: opperf memory corruption crash after stack operator
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19231:
URL: https://github.com/apache/incubator-mxnet/issues/19231#issuecomment-699189148
Issue persists after removing the stack op
```
INFO:root:Begin Benchmark - concat
INFO:root:Complete Benchmark - concat
INFO:root:Begin Benchmark - split
INFO:root:Complete Benchmark - split
free(): invalid pointer
zsh: abort (core dumped) PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 --output-format json
PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 --output-format json 6194.92s user 1736.49s system 4770% cpu 2:46.26 total
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #19231: opperf memory corruption crash after stack operator
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on issue #19231:
URL: https://github.com/apache/incubator-mxnet/issues/19231#issuecomment-699184395
One thing to be sure its an issue with an op would be to try if opperf terminates successfully w/o stack op
did you try removing the stack op from the mix?
https://github.com/apache/incubator-mxnet/blob/b225fa5d3c9b6d9ba10533fcbf20a20c6aa96aff/benchmark/opperf/nd_operations/array_manipulation_operators.py#L251-L263
this is where it gets tested
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org