You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/05/14 05:33:10 UTC
[GitHub] [incubator-mxnet] leezu opened a new issue #18314: ThreadedEngine: Segfault
leezu opened a new issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314
## Description
The following code shouldn't crash, but can reliably trigger segfault in ThreadedEngine. (No segfault with NaiveEngine).
```
import gc
gc.set_debug(gc.DEBUG_SAVEALL)
import mxnet as mx
net = mx.gluon.nn.Dense(10, in_units=10)
net.initialize()
del net
print(gc.collect())
for i, x in enumerate(gc.garbage):
print(i, type(x), id(x), end=' ')
try:
print('\t', x)
except Exception as e:
print('failed', e)
```
cc @eric-haibin-lin
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] anirudh2290 commented on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
anirudh2290 commented on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628845269
> We see that after `del net` is called, two ndarrays are finalized thanks to reference counting. 4 arrays are only finalized after a full garbage collection.
aah okay, maybe make the __del__ a no op for testing and try to see which objects are uncollectable ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628407634
To get the backtrace in gdb, use the example without the for loop in the end
```
import gc
gc.set_debug(gc.DEBUG_SAVEALL)
import mxnet as mx
net = mx.gluon.nn.Dense(10, in_units=10)
net.initialize()
del net
print(gc.collect())
print(gc.garbage)
```
```
#0 __GI___pthread_mutex_lock (mutex=0x2a) at ../nptl/pthread_mutex_lock.c:65
#1 0x00007fff40d76349 in __gthread_mutex_lock (__mutex=0x2a) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:748
#2 0x00007fff40d9defe in std::mutex::lock (this=0x2a) at /usr/include/c++/7/bits/std_mutex.h:103
#3 0x00007fff40db701e in std::lock_guard<std::mutex>::lock_guard (this=0x7fffffff99b0, __m=...) at /usr/include/c++/7/bits/std_mutex.h:162
#4 0x00007fff40fa3f65 in mxnet::engine::ThreadedVar::ready_to_read (this=0xa) at ../src/engine/threaded_engine.cc:198
#5 0x00007fff40fa1109 in mxnet::engine::ThreadedEngine::WaitForVar (this=0x55555752d8f0, var=0xa) at ../src/engine/threaded_engine.cc:382
#6 0x00007fff412a6c03 in mxnet::NDArray::WaitToRead (this=0x555556b508b0) at ../src/ndarray/ndarray.cc:2178
#7 0x00007fff412a5eef in mxnet::NDArray::SyncCopyToCPU (this=0x555556b508b0, data=0x555556b465e0, size=100) at ../src/ndarray/ndarray.cc:2112
#8 0x00007fff40d8bd27 in MXNDArraySyncCopyToCPU (handle=0x555556b508b0, data=0x555556b465e0, size=100) at ../src/c_api/c_api.cc:1575
#9 0x00007ffff6965dae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#10 0x00007ffff696571f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#11 0x00007ffff6ba7415 in _call_function_pointer (flags=4353, pProc=0x7fff40d8bcef <MXNDArraySyncCopyToCPU(NDArrayHandle, void*, size_t)>, avalues=0x7fffffffa0c0, atypes=0x7fffffffa090,
restype=0x7ffff6e27908, resmem=0x7fffffffa0f0, argcount=3) at /tmp/python-build.20200514035455.63369/Python-3.8.2/Modules/_ctypes/callproc.c:871
#12 0x00007ffff6ba7e19 in _ctypes_callproc (pProc=0x7fff40d8bcef <MXNDArraySyncCopyToCPU(NDArrayHandle, void*, size_t)>,
argtuple=(<c_void_p at remote 0x7fff9631ec30>, <c_void_p(_arr=<numpy.ndarray at remote 0x7fff96371820>) at remote 0x7fff96309190>, <c_ulong at remote 0x7fff9631e410>), flags=4353, argtypes=0x0,
restype=<_ctypes.PyCSimpleType at remote 0x555555c491b0>, checker=0x0) at /tmp/python-build.20200514035455.63369/Python-3.8.2/Modules/_ctypes/callproc.c:1199
#13 0x00007ffff6ba2169 in PyCFuncPtr_call (self=0x7fff96363130,
inargs=(<c_void_p at remote 0x7fff9631ec30>, <c_void_p(_arr=<numpy.ndarray at remote 0x7fff96371820>) at remote 0x7fff96309190>, <c_ulong at remote 0x7fff9631e410>), kwds=0x0)
at /tmp/python-build.20200514035455.63369/Python-3.8.2/Modules/_ctypes/_ctypes.c:4201
#14 0x00005555555c5076 in _PyObject_MakeTpCall (callable=<_FuncPtr(__name__='MXNDArraySyncCopyToCPU') at remote 0x7fff96363130>, args=0x7fff9632ed80, nargs=3, keywords=0x0) at Objects/call.c:159
#15 0x00005555556acdb1 in _PyObject_Vectorcall (callable=<_FuncPtr(__name__='MXNDArraySyncCopyToCPU') at remote 0x7fff96363130>, args=0x7fff9632ed80, nargsf=9223372036854775811, kwnames=0x0)
at ./Include/cpython/abstract.h:125
#16 0x00005555556c4ab4 in call_function (tstate=0x555555bb4670, pp_stack=0x7fffffffa5f8, oparg=3, kwnames=0x0) at Python/ceval.c:4987
#17 0x00005555556bec66 in _PyEval_EvalFrameDefault (
f=Frame 0x7fff9632ebf0, for file /home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py, line 2819, in asnumpy (self=<NDArray at remote 0x7fff9631b5a0>, data=<numpy.ndarray at remote 0x7fff96371820>), throwflag=0) at Python/ceval.c:3469
#18 0x00005555556aeb45 in PyEval_EvalFrameEx (
f=Frame 0x7fff9632ebf0, for file /home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py, line 2819, in asnumpy (self=<NDArray at remote 0x7fff9631b5a0>, data=<numpy.ndarray at remote 0x7fff96371820>), throwflag=0) at Python/ceval.c:741
#19 0x00005555555c5758 in function_code_fastcall (co=0x7fffebbb9860, args=0x7fff8f456e98, nargs=1,
globals={'__name__': 'mxnet.ndarray.ndarray', '__doc__': 'NDArray API of MXNet.', '__package__': 'mxnet.ndarray', '__loader__': <SourceFileLoader(name='mxnet.ndarray.ndarray', path='/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py') at remote 0x7fffebb8a1e0>, '__spec__': <ModuleSpec(name='mxnet.ndarray.ndarray', loader=<...>, origin='/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/__pycache__/ndarray.cpython-38.pyc', _initializing=False) at remote 0x7fffebb8a230>, '__file__': '/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py', '__cached__': '/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/__pycache__/ndarray.cpython-38.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <type...(truncated)) at Objects/call.c:283
#20 0x00005555555c5fc8 in _PyFunction_Vectorcall (func=<function at remote 0x7fffe6bd1870>, stack=0x7fff8f456e90, nargsf=9223372036854775809, kwnames=0x0) at Objects/call.c:410
#21 0x00005555556acdc9 in _PyObject_Vectorcall (callable=<function at remote 0x7fffe6bd1870>, args=0x7fff8f456e90, nargsf=9223372036854775809, kwnames=0x0) at ./Include/cpython/abstract.h:127
#22 0x00005555556c4ab4 in call_function (tstate=0x555555bb4670, pp_stack=0x7fffffffb838, oparg=1, kwnames=0x0) at Python/ceval.c:4987
#23 0x00005555556bece0 in _PyEval_EvalFrameDefault (
f=Frame 0x7fff8f456d00, for file /home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py, line 257, in __repr__ (self=<NDArray at remote 0x7fff9631b5a0>, shape_info='10x10'), throwflag=0)
at Python/ceval.c:3486
#24 0x00005555556aeb45 in PyEval_EvalFrameEx (
f=Frame 0x7fff8f456d00, for file /home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py, line 257, in __repr__ (self=<NDArray at remote 0x7fff9631b5a0>, shape_info='10x10'), throwflag=0)
at Python/ceval.c:741
#25 0x00005555555c5758 in function_code_fastcall (co=0x7fffebb901e0, args=0x7fffffffc888, nargs=1,
globals={'__name__': 'mxnet.ndarray.ndarray', '__doc__': 'NDArray API of MXNet.', '__package__': 'mxnet.ndarray', '__loader__': <SourceFileLoader(name='mxnet.ndarray.ndarray', path='/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py') at remote 0x7fffebb8a1e0>, '__spec__': <ModuleSpec(name='mxnet.ndarray.ndarray', loader=<...>, origin='/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py', loader_state=None, submodule_search_locations=None, _set_fileattr=True, _cached='/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/__pycache__/ndarray.cpython-38.pyc', _initializing=False) at remote 0x7fffebb8a230>, '__file__': '/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/ndarray.py', '__cached__': '/home/ubuntu/src/mxnet-master/python/mxnet/ndarray/__pycache__/ndarray.cpython-38.pyc', '__builtins__': {'__name__': 'builtins', '__doc__': "Built-in functions, exceptions, and other objects.\n\nNoteworthy: None is the `nil' object; Ellipsis represents `...' in slices.", '__package__': '', '__loader__': <type...(truncated)) at Objects/call.c:283
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628829305
I can confirm that __del__ is called on the object, but it's still kept around in Python afterwards.
In practice that may only make problems when debugging memory leaks (as is the case for how I ran into these segfaults)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu closed issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu closed issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628821443
The mutex in above core dump is actually non existant. The problem here is that the ffi call to MXNDArraySyncCopyToCPU passes a handle whose Chunk was cleaned up. So the segfault is just a symptom of memory and reference management issues.
the array
```
(gdb) p *this
$6 = {ptr_ = std::shared_ptr<mxnet::NDArray::Chunk> (use count 1645762192, weak count 22028) = {get() = 0x560d62a70cb0}, shape_ = {<mxnet::Tuple<long>> = {static kStackCache = 4, ndim_ = 2,
num_heap_allocated_ = 0, data_stack_ = {10, 10, -3617008641903833651, -3617008641903833651}, data_heap_ = 0x0}, <No data fields>}, byte_offset_ = 0, dtype_ = 0, reuse_ = false,
storage_type_ = mxnet::kDefaultStorage, autograd_entry_ = {node = std::shared_ptr<nnvm::Node> (use count 1647685840, weak count 22028) = {get() = 0x560d622fe220}, index = 0, version = 0},
deferredcompute_entry_ = {node = std::shared_ptr<nnvm::Node> (empty) = {get() = 0x0}, index = 0, version = 0}, tblob_ = {dptr_ = 0x0, shape_ = {<mxnet::Tuple<long>> = {static kStackCache = 4,
ndim_ = -1, num_heap_allocated_ = 0, data_stack_ = {-3617008641903833651, -2459565876494606899, -2459565876494606883, -2459565876494606883}, data_heap_ = 0x0}, <No data fields>}, type_flag_ = 0,
dltensor_ = {data = 0x0, ctx = {device_type = kDLCPU, device_id = 0}, ndim = -1, dtype = {code = 2 '\002', bits = 32 ' ', lanes = 1}, shape = 0x560d62188f78, strides = 0x0, byte_offset = 0}}}
```
and the chunk
```
p *ptr_
$4 = {shandle = {dptr = 0xdddddddddddddddd, size = 481, ctx = {dev_type = 1655088448, dev_id = 22029, static kMaxDevType = 6, static kMaxDevID = 16}, shared_pid = 1645776592, shared_id = 22029,
profiler_scope = Python Exception <class 'OverflowError'> int too big to convert:
, Python Exception <class 'OverflowError'> int too big to convert:
name = }, aux_handles = std::vector of length 0, capacity 0, mkl_mem_ = <error reading variable: Cannot access memory at address 0xdddddddddddddde5>, var = 0xdddddddddddddddd,
static_data = 221, delay_alloc = 221, storage_type = -572662307, aux_types = std::vector of length 0, capacity 0, ctx = {dev_type = 3722304989, dev_id = -572662307, static kMaxDevType = 6,
static kMaxDevID = 16}, storage_shape = {<mxnet::Tuple<long>> = {static kStackCache = 4, ndim_ = -572662307, num_heap_allocated_ = -572662307, data_stack_ = {-2459565876494606883,
-2459565876494606883, -2459565876494606883, -2459565876494606883}, data_heap_ = 0xdddddddddddddddd}, <No data fields>}, aux_shapes = std::vector of length 0, capacity 0,
storage_ref_ = <error reading variable: Cannot access memory at address 0xdddddddddddddde5>, engine_ref_ = <error reading variable: Cannot access memory at address 0xdddddddddddddde5>}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628398197
On 1.6 it also triggers segfault with NaiveEngine. But not on master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628969701
Let's close this issue as it's specific to the use of gc.DEBUG_SAVEALL. The leak is fixed in https://github.com/apache/incubator-mxnet/pull/18328
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628407634
Use https://github.com/apache/incubator-mxnet/pull/18315 to get the coredump associated with the segfault
```
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007f144d380801 in __GI_abort () at abort.c:79
#2 0x00007f13973c94c2 in mxnet::SegfaultLogger (sig=11) at ../src/initialize.cc:66
#3 <signal handler called>
#4 __GI___pthread_mutex_lock (mutex=0x7ffc00000020) at ../nptl/pthread_mutex_lock.c:65
#5 0x00007f1397096b3d in __gthread_mutex_lock (__mutex=0x7ffc00000020) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:748
#6 0x00007f13970be6f2 in std::mutex::lock (this=0x7ffc00000020) at /usr/include/c++/7/bits/std_mutex.h:103
#7 0x00007f13970d7812 in std::lock_guard<std::mutex>::lock_guard (this=0x7ffc08a27cb0, __m=...) at /usr/include/c++/7/bits/std_mutex.h:162
#8 0x00007f13972c4759 in mxnet::engine::ThreadedVar::ready_to_read (this=0x7ffc00000000) at ../src/engine/threaded_engine.cc:198
#9 0x00007f13972c18fd in mxnet::engine::ThreadedEngine::WaitForVar (this=0x560344701760, var=0x7ffc00000000) at ../src/engine/threaded_engine.cc:382
#10 0x00007f13975c73f1 in mxnet::NDArray::WaitToRead (this=0x560343f7e750) at ../src/ndarray/ndarray.cc:2178
#11 0x00007f13975c66dd in mxnet::NDArray::SyncCopyToCPU (this=0x560343f7e750, data=0x7f1218001b00, size=10) at ../src/ndarray/ndarray.cc:2112
#12 0x00007f13970ac51b in MXNDArraySyncCopyToCPU (handle=0x560343f7e750, data=0x7f1218001b00, size=10) at ../src/c_api/c_api.cc:1575
#13 0x00007f144cc85dae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#14 0x00007f144cc8571f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#15 0x00007f144cec7415 in _call_function_pointer (flags=4353, pProc=0x7f13970ac4e3 <MXNDArraySyncCopyToCPU(NDArrayHandle, void*, size_t)>, avalues=0x7ffc08a283c0, atypes=0x7ffc08a28390,
restype=0x7f144d14bdb8, resmem=0x7ffc08a283f0, argcount=3) at /tmp/python-build.20200514035455.63369/Python-3.8.2/Modules/_ctypes/callproc.c:871
#16 0x00007f144cec7e19 in _ctypes_callproc (pProc=0x7f13970ac4e3 <MXNDArraySyncCopyToCPU(NDArrayHandle, void*, size_t)>, argtuple=0x7f12f4166ef0, flags=4353, argtypes=0x0, restype=0x560342e1b8e0,
checker=0x0) at /tmp/python-build.20200514035455.63369/Python-3.8.2/Modules/_ctypes/callproc.c:1199
#17 0x00007f144cec2169 in PyCFuncPtr_call (self=0x7f12f4145050, inargs=0x7f12f4166ef0, kwds=0x0) at /tmp/python-build.20200514035455.63369/Python-3.8.2/Modules/_ctypes/_ctypes.c:4201
#18 0x0000560342627076 in ?? ()
#19 0x0000000000000000 in ?? ()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628969701
Let's close this issue as it's specific to the use of gc.DEBUG_SAVEALL
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628407634
Use https://github.com/apache/incubator-mxnet/pull/18315 to get the coredump associated with the segfault. Loading it with gdb gives this backtrace
```
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007f144d380801 in __GI_abort () at abort.c:79
#2 0x00007f13973c94c2 in mxnet::SegfaultLogger (sig=11) at ../src/initialize.cc:66
#3 <signal handler called>
#4 __GI___pthread_mutex_lock (mutex=0x7ffc00000020) at ../nptl/pthread_mutex_lock.c:65
#5 0x00007f1397096b3d in __gthread_mutex_lock (__mutex=0x7ffc00000020) at /usr/include/x86_64-linux-gnu/c++/7/bits/gthr-default.h:748
#6 0x00007f13970be6f2 in std::mutex::lock (this=0x7ffc00000020) at /usr/include/c++/7/bits/std_mutex.h:103
#7 0x00007f13970d7812 in std::lock_guard<std::mutex>::lock_guard (this=0x7ffc08a27cb0, __m=...) at /usr/include/c++/7/bits/std_mutex.h:162
#8 0x00007f13972c4759 in mxnet::engine::ThreadedVar::ready_to_read (this=0x7ffc00000000) at ../src/engine/threaded_engine.cc:198
#9 0x00007f13972c18fd in mxnet::engine::ThreadedEngine::WaitForVar (this=0x560344701760, var=0x7ffc00000000) at ../src/engine/threaded_engine.cc:382
#10 0x00007f13975c73f1 in mxnet::NDArray::WaitToRead (this=0x560343f7e750) at ../src/ndarray/ndarray.cc:2178
#11 0x00007f13975c66dd in mxnet::NDArray::SyncCopyToCPU (this=0x560343f7e750, data=0x7f1218001b00, size=10) at ../src/ndarray/ndarray.cc:2112
#12 0x00007f13970ac51b in MXNDArraySyncCopyToCPU (handle=0x560343f7e750, data=0x7f1218001b00, size=10) at ../src/c_api/c_api.cc:1575
#13 0x00007f144cc85dae in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#14 0x00007f144cc8571f in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#15 0x00007f144cec7415 in _call_function_pointer (flags=4353, pProc=0x7f13970ac4e3 <MXNDArraySyncCopyToCPU(NDArrayHandle, void*, size_t)>, avalues=0x7ffc08a283c0, atypes=0x7ffc08a28390,
restype=0x7f144d14bdb8, resmem=0x7ffc08a283f0, argcount=3) at /tmp/python-build.20200514035455.63369/Python-3.8.2/Modules/_ctypes/callproc.c:871
#16 0x00007f144cec7e19 in _ctypes_callproc (pProc=0x7f13970ac4e3 <MXNDArraySyncCopyToCPU(NDArrayHandle, void*, size_t)>, argtuple=0x7f12f4166ef0, flags=4353, argtypes=0x0, restype=0x560342e1b8e0,
checker=0x0) at /tmp/python-build.20200514035455.63369/Python-3.8.2/Modules/_ctypes/callproc.c:1199
#17 0x00007f144cec2169 in PyCFuncPtr_call (self=0x7f12f4145050, inargs=0x7f12f4166ef0, kwds=0x0) at /tmp/python-build.20200514035455.63369/Python-3.8.2/Modules/_ctypes/_ctypes.c:4201
#18 0x0000560342627076 in ?? ()
#19 0x0000000000000000 in ?? ()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628833322
That's because I set gc.set_debug(gc.DEBUG_SAVEALL). I want to preserve all objects found by gc, because I need to figure out where the reference cycle that prevented them from being freed by reference counting, came from.
Howeve, I find that despite gc.set_debug(gc.DEBUG_SAVEALL), the gc will still call the finalizer on the objects it finds. That's why we end up with zombie objects in gc.garbage:
```
Calling finalizer for c_void_p(94420426532208)
Calling finalizer for c_void_p(94420426532208)
Calling collect
Calling finalizer for c_void_p(94420426437968)
Calling finalizer for c_void_p(94420427373968)
Calling finalizer for c_void_p(94420428079184)
Calling finalizer for c_void_p(94420426198000)
```
We see that after `del net` is called, two ndarrays are finalized thanks to reference counting. 4 arrays are only finalized after a full garbage collection.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] anirudh2290 commented on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
anirudh2290 commented on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628821250
not sure completely what it is, but it could be that the one of the objects in garbage is a ndarray and when you call print it calls its __repr__ and asnumpy, the dependency engine tries to figure if the underlying threadedvar is present but looks like it may have been deleted. Maybe it is worth to figure that for all the objects appended to garbage which were ndarray, if delete was called on them already ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] anirudh2290 commented on issue #18314: ThreadedEngine: Segfault
Posted by GitBox <gi...@apache.org>.
anirudh2290 commented on issue #18314:
URL: https://github.com/apache/incubator-mxnet/issues/18314#issuecomment-628831199
From https://docs.python.org/3/library/gc.html#gc.garbage , "Changed in version 3.4: Following PEP 442, objects with a __del__() method don’t end up in gc.garbage anymore."
but this statement doesnt seem to be true here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org