You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/09/25 12:09:27 UTC
[GitHub] Angzz edited a comment on issue #12619: Too large max depth value
in _recursive_fork_recordio
Angzz edited a comment on issue #12619: Too large max depth value in _recursive_fork_recordio
URL: https://github.com/apache/incubator-mxnet/issues/12619#issuecomment-424316685
when update to 1.3.1b20180925, error occurs when train ssd with coco, but voc is normal:
---------------- train log and error log ------------------
INFO:root:Start training from [Epoch 0]
[19:54:19] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
[19:54:28] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:109: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
python: malloc.c:3722: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed.
*** Error in `python': malloc(): memory corruption: 0x00007fe3d29b3690 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7fe5b37c87e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x8213e)[0x7fe5b37d313e]
/lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7fe5b37d5184]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_Znwm+0x18)[0x7fe5af411e78]
/home/liang/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x407eb0)[0x7fe52630beb0]
/home/liang/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x40d7c9)[0x7fe5263117c9]
/home/liang/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2b88458)[0x7fe528a8c458]
/home/liang/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2adcb29)[0x7fe5289e0b29]
/home/liang/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2ae6544)[0x7fe5289ea544]
/home/liang/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2aea6c2)[0x7fe5289ee6c2]
/home/liang/.local/lib/python2.7/site-packages/mxnet/libmxnet.so(+0x2ae6c64)[0x7fe5289eac64]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7fe5af43cc80]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7fe5b3b226ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe5b385841d]
======= Memory map: ========
00400000-006de000 r-xp 00000000 103:02 16254177 /usr/bin/python2.7
008dd000-008de000 r--p 002dd000 103:02 16254177 /usr/bin/python2.7
008de000-00955000 rw-p 002de000 103:02 16254177 /usr/bin/python2.7
00955000-00978000 rw-p 00000000 00:00 0
00c8d000-a94d5000 rw-p 00000000 00:00 0 [heap]
a94d5000-a9806000 rw-p 00000000 00:00 0 [heap]
200000000-200200000 rw-s 00000000 00:06 456 /dev/nvidiactl
200200000-200400000 ---p 00000000 00:00 0
200400000-200404000 rw-s 00000000 00:06 456 /dev/nvidiactl
200404000-200600000 ---p 00000000 00:00 0
200600000-200a00000 rw-s 00000000 00:06 456 /dev/nvidiactl
200a00000-201800000 ---p 00000000 00:00 0
201800000-201804000 rw-s 00000000 00:06 456 /dev/nvidiactl
201804000-201a00000 ---p 00000000 00:00 0
201a00000-201e00000 rw-s 00000000 00:06 456 /dev/nvidiactl
201e00000-201e04000 rw-s 00000000 00:06 456 /dev/nvidiactl
201e04000-202000000 ---p 00000000 00:00 0
202000000-202400000 rw-s 00000000 00:06 456 /dev/nvidiactl
202400000-202404000 rw-s 00000000 00:06 456 /dev/nvidiactl
202404000-202600000 ---p 00000000 00:00 0
202600000-202a00000 rw-s 00000000 00:06 456 /dev/nvidiactl
202a00000-202a04000 rw-s 00000000 00:06 456 /dev/nvidiactl
202a04000-202c00000 ---p 00000000 00:00 0
202c00000-203000000 rw-s 00000000 00:06 456 /dev/nvidiactl
203000000-203004000 rw-s 00000000 00:06 456 /dev/nvidiactl
203004000-203200000 ---p 00000000 00:00 0
203200000-203600000 rw-s 00000000 00:06 456 /dev/nvidiactl
203600000-203604000 rw-s 00000000 00:06 456 /dev/nvidiactl
203604000-203800000 ---p 00000000 00:00 0
203800000-203c00000 rw-s 00000000 00:06 456 /dev/nvidiactl
203c00000-203c04000 rw-s 00000000 00:06 456 /dev/nvidiactl
203c04000-203e00000 ---p 00000000 00:00 0
203e00000-204200000 rw-s 00000000 00:06 456 /dev/nvidiactl
204200000-204204000 rw-s 00000000 00:06 456 /dev/nvidiactl
204204000-204400000 ---p 00000000 00:00 0
204400000-204800000 rw-s 00000000 00:06 456 /dev/nvidiactl
204800000-204804000 rw-s 00000000 00:06 456 /dev/nvidiactl
204804000-204a00000 ---p 00000000 00:00 0
204a00000-204e00000 rw-s 00000000 00:06 456 /dev/nvidiactl
204e00000-204e04000 rw-s 00000000 00:06 456 /dev/nvidiactl
204e04000-205000000 ---p 00000000 00:00 0
205000000-205400000 rw-s 00000000 00:06 456 /dev/nvidiactl
205400000-205404000 rw-s 00000000 00:06 456 /dev/nvidiactl
205404000-205600000 ---p 00000000 00:00 0
205600000-205a00000 rw-s 00000000 00:06 456 /dev/nvidiactl
205a00000-205a04000 rw-s 00000000 00:06 456 /dev/nvidiactl
205a04000-205c00000 ---p 00000000 00:00 0
205c00000-206000000 rw-s 00000000 00:06 456 /dev/nvidiactl
206000000-206004000 rw-s 00000000 00:06 456 /dev/nvidiactl
206004000-206200000 ---p 00000000 00:00 0
206200000-206600000 rw-s 00000000 00:06 456 /dev/nvidiactl
206600000-206604000 rw-s 00000000 00:06 456 /dev/nvidiactl
206604000-206800000 ---p 00000000 00:00 0
206800000-206c00000 rw-s 00000000 00:06 456 /dev/nvidiactl
206c00000-206c04000 rw-s 00000000 00:06 456 /dev/nvidiactl
206c04000-206e00000 ---p 00000000 00:00 0
206e00000-207200000 rw-s 00000000 00:06 456 /dev/nvidiactl
207200000-207400000 ---p 00000000 00:00 0
207400000-207600000 rw-s 00000000 00:06 456 /dev/nvidiactl
207600000-207800000 rw-s 00000000 00:06 456 /dev/nvidiactl
207800000-207a00000 ---p 00000000 00:00 0
207a00000-207a04000 rw-s 00000000 00:06 456 /dev/nvidiactl
207a04000-207c00000 ---p 00000000 00:00 0
207c00000-208000000 rw-s 00000000 00:06 456 /dev/nvidiactl
208000000-208e00000 ---p 00000000 00:00 0
208e00000-208e04000 rw-s 00000000 00:06 456 /dev/nvidiactl
208e04000-209000000 ---p 00000000 00:00 0
209000000-209400000 rw-s 00000000 00:06 456 /dev/nvidiactl
209400000-209404000 rw-s 00000000 00:06 456 /dev/nvidiactl
209404000-209600000 ---p 00000000 00:00 0
209600000-209a00000 rw-s 00000000 00:06 456 /dev/nvidiactl
209a00000-209a04000 rw-s 00000000 00:06 456 /dev/nvidiactl
209a04000-209c00000 ---p 00000000 00:00 0
209c00000-20a000000 rw-s 00000000 00:06 456 /dev/nvidiactl
20a000000-20a004000 rw-s 00000000 00:06 456 /dev/nvidiactl
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services