You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/09/25 22:15:47 UTC

[GitHub] [incubator-mxnet] leezu opened a new issue #19231: opperf memory corruption crash after stack operator

leezu opened a new issue #19231:
URL: https://github.com/apache/incubator-mxnet/issues/19231


   ## Description
   Running the opperf benchmark on the master branch causes memory corruption and crash after the `stack` operator
   
   ### Error Message
   First try
   ```
   INFO:root:Complete Benchmark - split
   INFO:root:Begin Benchmark - stack
   INFO:root:Complete Benchmark - stack
   invalid fastbin entry (free)
   zsh: abort (core dumped)  PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3  --output-format json
   ```
   
   Second try
   ```
   INFO:root:Begin Benchmark - split
   INFO:root:Complete Benchmark - split
   INFO:root:Begin Benchmark - stack
   INFO:root:Complete Benchmark - stack
   corrupted size vs. prev_size
   zsh: abort (core dumped)  PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3  --output-format json
   ```
   
   ## To Reproduce
   On Ubuntu 18.04, compile with `cmake -GNinja -DUSE_CUDA=0  ..; ninja`.
   Then run the benchmark with `PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 mxnet-master/benchmark/opperf/opperf.py --output-format json --output-file mxnet_operator_benchmark_results.json`
   
   
   cc @ChaiBapchya 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org


[GitHub] [incubator-mxnet] leezu commented on issue #19231: opperf memory corruption crash after stack operator

Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19231:
URL: https://github.com/apache/incubator-mxnet/issues/19231#issuecomment-699201393


   Essentially it crashes for any set of operators:
   
   If I disable anything but `run_mx_unary_operators_benchmarks`, it crashes as follows
   
   ```
   INFO:root:Running MXNet operator benchmarks with the following options: Namespace(ctx='cpu', dtype='float32', int64_tensor='off', output_file='mxnet_operator_benchmark_results.json', output_format='json', profiler='native', runs=100, warmup=25)
   [22:52:46] ../src/storage/storage.cc:199: Using Pooled (Naive) StorageManager for CPU
   INFO:root:Begin Benchmark - amp_multicast
   INFO:root:Complete Benchmark - amp_multicast
   INFO:root:Begin Benchmark - BlockGrad
   INFO:root:Complete Benchmark - BlockGrad
   INFO:root:Begin Benchmark - abs
   INFO:root:Complete Benchmark - abs
   INFO:root:Begin Benchmark - amp_cast
   INFO:root:Complete Benchmark - amp_cast
   INFO:root:Begin Benchmark - arccos
   INFO:root:Complete Benchmark - arccos
   INFO:root:Begin Benchmark - arccosh
   INFO:root:Complete Benchmark - arccosh
   INFO:root:Begin Benchmark - arcsin
   INFO:root:Complete Benchmark - arcsin
   INFO:root:Begin Benchmark - arcsinh
   INFO:root:Complete Benchmark - arcsinh
   INFO:root:Begin Benchmark - arctan
   INFO:root:Complete Benchmark - arctan
   INFO:root:Begin Benchmark - arctanh
   INFO:root:Complete Benchmark - arctanh
   INFO:root:Begin Benchmark - argmax_channel
   INFO:root:Complete Benchmark - argmax_channel
   INFO:root:Begin Benchmark - cast
   INFO:root:Complete Benchmark - cast
   INFO:root:Begin Benchmark - cbrt
   INFO:root:Complete Benchmark - cbrt
   INFO:root:Begin Benchmark - ceil
   INFO:root:Complete Benchmark - ceil
   INFO:root:Begin Benchmark - cos
   INFO:root:Complete Benchmark - cos
   INFO:root:Begin Benchmark - cosh
   INFO:root:Complete Benchmark - cosh
   INFO:root:Begin Benchmark - degrees
   INFO:root:Complete Benchmark - degrees
   INFO:root:Begin Benchmark - digamma
   INFO:root:Complete Benchmark - digamma
   INFO:root:Begin Benchmark - erf
   INFO:root:Complete Benchmark - erf
   INFO:root:Begin Benchmark - erfinv
   INFO:root:Complete Benchmark - erfinv
   INFO:root:Begin Benchmark - exp
   INFO:root:Complete Benchmark - exp
   INFO:root:Begin Benchmark - expm1
   INFO:root:Complete Benchmark - expm1
   INFO:root:Begin Benchmark - fix
   INFO:root:Complete Benchmark - fix
   INFO:root:Begin Benchmark - flatten
   INFO:root:Complete Benchmark - flatten
   INFO:root:Begin Benchmark - floor
   INFO:root:Complete Benchmark - floor
   INFO:root:Begin Benchmark - gamma
   INFO:root:Complete Benchmark - gamma
   INFO:root:Begin Benchmark - gammaln
   INFO:root:Complete Benchmark - gammaln
   INFO:root:Begin Benchmark - identity
   INFO:root:Complete Benchmark - identity
   INFO:root:Begin Benchmark - log
   INFO:root:Complete Benchmark - log
   INFO:root:Begin Benchmark - log10
   INFO:root:Complete Benchmark - log10
   INFO:root:Begin Benchmark - log1p
   INFO:root:Complete Benchmark - log1p
   INFO:root:Begin Benchmark - log2
   INFO:root:Complete Benchmark - log2
   INFO:root:Begin Benchmark - logical_not
   INFO:root:Complete Benchmark - logical_not
   INFO:root:Begin Benchmark - make_loss
   INFO:root:Complete Benchmark - make_loss
   INFO:root:Begin Benchmark - negative
   INFO:root:Complete Benchmark - negative
   INFO:root:Begin Benchmark - ones_like
   INFO:root:Complete Benchmark - ones_like
   INFO:root:Begin Benchmark - radians
   INFO:root:Complete Benchmark - radians
   INFO:root:Begin Benchmark - rcbrt
   INFO:root:Complete Benchmark - rcbrt
   INFO:root:Begin Benchmark - reciprocal
   INFO:root:Complete Benchmark - reciprocal
   INFO:root:Begin Benchmark - relu
   INFO:root:Complete Benchmark - relu
   INFO:root:Begin Benchmark - rint
   INFO:root:Complete Benchmark - rint
   INFO:root:Begin Benchmark - round
   INFO:root:Complete Benchmark - round
   INFO:root:Begin Benchmark - rsqrt
   INFO:root:Complete Benchmark - rsqrt
   INFO:root:Begin Benchmark - shape_array
   INFO:root:Complete Benchmark - shape_array
   INFO:root:Begin Benchmark - shuffle
   INFO:root:Complete Benchmark - shuffle
   INFO:root:Begin Benchmark - sigmoid
   INFO:root:Complete Benchmark - sigmoid
   INFO:root:Begin Benchmark - sign
   INFO:root:Complete Benchmark - sign
   INFO:root:Begin Benchmark - sin
   INFO:root:Complete Benchmark - sin
   INFO:root:Begin Benchmark - sinh
   INFO:root:Complete Benchmark - sinh
   INFO:root:Begin Benchmark - size_array
   INFO:root:Complete Benchmark - size_array
   INFO:root:Begin Benchmark - softsign
   INFO:root:Complete Benchmark - softsign
   INFO:root:Begin Benchmark - sqrt
   INFO:root:Complete Benchmark - sqrt
   INFO:root:Begin Benchmark - square
   INFO:root:Complete Benchmark - square
   INFO:root:Begin Benchmark - stop_gradient
   INFO:root:Complete Benchmark - stop_gradient
   INFO:root:Begin Benchmark - tan
   INFO:root:Complete Benchmark - tan
   INFO:root:Begin Benchmark - tanh
   INFO:root:Complete Benchmark - tanh
   INFO:root:Begin Benchmark - trunc
   INFO:root:Complete Benchmark - trunc
   INFO:root:Begin Benchmark - zeros_like
   INFO:root:Complete Benchmark - zeros_like
   malloc_consolidate(): invalid chunk size
   zsh: abort (core dumped)  PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3  --output-format json
   PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3  --output-format json    734.74s user 170.63s system 4594% cpu 19.704 total
   ```
   
   If I disable anything but `run_mx_binary_broadcast_operators_benchmarks`, it crashes as follows
   
   
   ```
   time PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3 mxnet-master/benchmark/opperf/opperf.py --output-format json --output-file mxnet_operator_benchmark_results.json
   INFO:root:Running MXNet operator benchmarks with the following options: Namespace(ctx='cpu', dtype='float32', int64_tensor='off', output_file='mxnet_operator_benchmark_results.json', output_format='json', profiler='native', runs=100, warmup=25)
   [22:52:11] ../src/storage/storage.cc:199: Using Pooled (Naive) StorageManager for CPU
   INFO:root:Begin Benchmark - broadcast_add
   INFO:root:Complete Benchmark - broadcast_add
   INFO:root:Begin Benchmark - broadcast_div
   INFO:root:Complete Benchmark - broadcast_div
   INFO:root:Begin Benchmark - broadcast_equal
   INFO:root:Complete Benchmark - broadcast_equal
   INFO:root:Begin Benchmark - broadcast_greater
   INFO:root:Complete Benchmark - broadcast_greater
   INFO:root:Begin Benchmark - broadcast_greater_equal
   INFO:root:Complete Benchmark - broadcast_greater_equal
   INFO:root:Begin Benchmark - broadcast_hypot
   INFO:root:Complete Benchmark - broadcast_hypot
   INFO:root:Begin Benchmark - broadcast_lesser
   INFO:root:Complete Benchmark - broadcast_lesser
   INFO:root:Begin Benchmark - broadcast_lesser_equal
   INFO:root:Complete Benchmark - broadcast_lesser_equal
   INFO:root:Begin Benchmark - broadcast_logical_and
   INFO:root:Complete Benchmark - broadcast_logical_and
   INFO:root:Begin Benchmark - broadcast_logical_or
   INFO:root:Complete Benchmark - broadcast_logical_or
   INFO:root:Begin Benchmark - broadcast_logical_xor
   INFO:root:Complete Benchmark - broadcast_logical_xor
   INFO:root:Begin Benchmark - broadcast_maximum
   INFO:root:Complete Benchmark - broadcast_maximum
   INFO:root:Begin Benchmark - broadcast_minimum
   INFO:root:Complete Benchmark - broadcast_minimum                                                                                                                                                                                                                                                                  INFO:root:Begin Benchmark - broadcast_minus                                                                                                                                                                                                                                                                       INFO:root:Complete Benchmark - broadcast_minus                                                                                                                                                                                                                                                                    INFO:root:Begin Benchmark - broadcast_mod                                    
                                                                                                                                                                                                                                      INFO:root:Complete Benchmark - broadcast_mod                                                                                                                                                                                                                                                                      INFO:root:Begin Benchmark - broadcast_mul                                                                                                                                                                                                                                                                         INFO:root:Complete Benchmark - broadcast_mul                                                                                                                
                                                                                                                                                       INFO:root:Begin Benchmark - broadcast_not_equal
   INFO:root:Complete Benchmark - broadcast_not_equal
   INFO:root:Begin Benchmark - broadcast_plus
   INFO:root:Complete Benchmark - broadcast_plus
   INFO:root:Begin Benchmark - broadcast_power
   INFO:root:Complete Benchmark - broadcast_power
   INFO:root:Begin Benchmark - broadcast_sub
   INFO:root:Complete Benchmark - broadcast_sub
   invalid fastbin entry (free)
   zsh: abort (core dumped)  PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3  --output-format json
   ```
   
   
   
   @ChaiBapchya the issue may be with the opperf setup itself. Would you have time to fix it?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org


[GitHub] [incubator-mxnet] leezu commented on issue #19231: opperf memory corruption crash after stack operator

Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19231:
URL: https://github.com/apache/incubator-mxnet/issues/19231#issuecomment-699189148


   Issue persists after removing the stack op
   
   ```
   INFO:root:Begin Benchmark - concat
   INFO:root:Complete Benchmark - concat
   INFO:root:Begin Benchmark - split
   INFO:root:Complete Benchmark - split
   free(): invalid pointer
   zsh: abort (core dumped)  PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3  --output-format json
   PYTHONPATH=$PYTHONPATH:~/src/mxnet-master/ python3  --output-format json    6194.92s user 1736.49s system 4770% cpu 2:46.26 total
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org


[GitHub] [incubator-mxnet] ChaiBapchya commented on issue #19231: opperf memory corruption crash after stack operator

Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on issue #19231:
URL: https://github.com/apache/incubator-mxnet/issues/19231#issuecomment-699184395


   One thing to be sure its an issue with an op would be to try if opperf terminates successfully w/o stack op
   did you try removing the stack op from the mix?
   
   https://github.com/apache/incubator-mxnet/blob/b225fa5d3c9b6d9ba10533fcbf20a20c6aa96aff/benchmark/opperf/nd_operations/array_manipulation_operators.py#L251-L263
   this is where it gets tested


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org