You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2017/11/14 01:53:38 UTC
[GitHub] analog-cbarber opened a new issue #8640: CustomOp with more than one auxiliary state segfaults
analog-cbarber opened a new issue #8640: CustomOp with more than one auxiliary state segfaults
URL: https://github.com/apache/incubator-mxnet/issues/8640
## Description
A trivial CustomOp in python that uses more than one auxiliary state variable often causes
a segfault, typically on process shutdown.
## Environment info (Required)
```
----------Python Info----------
Version : 3.6.0
Compiler : GCC 4.2.1 (Apple Inc. build 5666) (dot 3)
Build : ('v3.6.0:41df79263a11', 'Dec 22 2016 17:23:13')
Arch : ('64bit', '')
------------Pip Info-----------
Version : 9.0.1
Directory : /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version : 0.12.1
Directory : /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/mxnet
Commit Hash : c11a608310116678511d9bcad96f372309fbee4e
----------System Info----------
Platform : Darwin-16.7.0-x86_64-i386-64bit
system : Darwin
node : cbarber-l02.fios-router.home
release : 16.7.0
version : Darwin Kernel Version 16.7.0: Wed Oct 4 00:17:00 PDT 2017; root:xnu-3789.71.6~1/RELEASE_X86_64
----------Hardware Info----------
machine : x86_64
processor : i386
b'machdep.cpu.extfeatures: SYSCALL XD 1GBPAGE EM64T LAHF LZCNT RDTSCP TSCI'
b'machdep.cpu.leaf7_features: SMEP ERMS RDWRFSGS TSC_THREAD_OFFSET BMI1 AVX2 BMI2 INVPCID FPU_CSDS'
b'machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C'
b'machdep.cpu.brand_string: Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz'
----------Network Test----------
Setting timeout: 10
Error open MXNet: https://github.com/apache/incubator-mxnet, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>, DNS finished in 0.023125886917114258 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.1197 sec, LOAD: 0.2253 sec.
Error open Gluon Tutorial(cn): https://zh.gluon.ai, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>, DNS finished in 0.16897797584533691 sec.
Error open FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>, DNS finished in 0.03162503242492676 sec.
Error open PYPI: https://pypi.python.org/pypi/pip, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>, DNS finished in 0.017091989517211914 sec.
Error open Conda: https://repo.continuum.io/pkgs/free/, <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)>, DNS finished in 0.06839799880981445 sec
```
Package used (Python/R/Scala/Julia):
Python 3.6
## Error Message:
Usually see:
```
Segmentation fault: 11
Stack trace returned 9 entries:
[bt] (0) 0 libmxnet.so 0x0000000104e5c0ef _ZN5mxnet15segfault_loggerEi + 63
[bt] (1) 1 libsystem_platform.dylib 0x00007fff94a38b3a _sigtramp + 26
[bt] (2) 2 Python 0x00000001001a827c _Py_zfill__doc__ + 31044
[bt] (3) 3 Python 0x00000001001493e3 _PyGC_CollectIfEnabled + 99
[bt] (4) 4 Python 0x0000000100128b17 Py_FinalizeEx + 119
[bt] (5) 5 Python 0x0000000100146e3e Py_Main + 1790
[ 1.]
[bt] (6) 6 Python 0x0000000100000dfe Python + 3582
[bt] (7) 7 Python 0x0000000100000c34 Python + 3124
[bt] (8) 8 ??? 0x0000000000000002 0x0 + 2
```
Also saw:
```
Python(50710,0x7fff9d7463c0) malloc: *** error for object 0x10acee5e8: pointer being freed was not allocated
*** set a breakpoint in malloc_error_break to debug
```
## Minimum reproducible example
```
import mxnet as mx
class BadAux(mx.operator.CustomOp):
def forward(self, is_train, req, in_data, out_data, aux):
self.assign(out_data[0], req[0], in_data[0][0])
@mx.operator.register('bad-aux')
class BadAuxProp(mx.operator.CustomOpProp):
def __init__(self):
super().__init__(need_top_grad=False)
def infer_shape(self, in_shape):
out_shape = [(1,)]
aux_shape = [(1,), (1,)]
return in_shape, out_shape, aux_shape
def list_arguments(self):
return ['data']
def list_auxiliary_states(self):
return ['a', 'b']
def declare_backward_dependency(self, out_grad, in_data, out_data):
return []
def create_operator(self, ctx, in_shapes, in_dtypes):
return BadAux()
q = mx.nd.Custom(mx.nd.ones((10000,)),
mx.nd.zeros((1,)),
mx.nd.zeros((1,)),
op_type='bad-aux')
print(q)
```
## Steps to reproduce
Run the included script. It fails for me about 2 out of 10 tries.
## What have you tried to solve it?
Haven't seen the problem when using only one auxiliary state.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services