You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/07/11 23:25:39 UTC
[GitHub] OneRaynyDay opened a new issue #11659: Segmentation fault upon
using onnx
OneRaynyDay opened a new issue #11659: Segmentation fault upon using onnx
URL: https://github.com/apache/incubator-mxnet/issues/11659
## Description
Hi, Airbnb is currently using ONNX for our model applications. We ran into this error while trying to serialize mxnet with onnx. It appears that trying to import onnx at some stage of `export_model` in `mx2onnx` gives us a segmentation fault.
NOTE: The problem is not reproducible in mac OS X. The code is in a nosetest and the tests all pass in High Sierra. Specific OS information is enumerated below.
# Error:
```
Segmentation fault: 11
Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x19e004) [0x7f9952c71004]
[bt] (1) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x2c42bc6) [0x7f9955715bc6]
[bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f9975bfef20]
[bt] (3) /usr/local/lib/python3.6/dist-packages/onnx/onnx_cpp2py_export.cpython-36m-x86_64-linux-gnu.so(+0x63bfd) [0x7f99388cabfd]
[bt] (4) /usr/local/lib/python3.6/dist-packages/onnx/onnx_cpp2py_export.cpython-36m-x86_64-linux-gnu.so(+0x6f82f) [0x7f99388d682f]
[bt] (5) /usr/local/lib/python3.6/dist-packages/onnx/onnx_cpp2py_export.cpython-36m-x86_64-linux-gnu.so(+0x54019) [0x7f99388bb019]
[bt] (6) /usr/local/lib/python3.6/dist-packages/onnx/onnx_cpp2py_export.cpython-36m-x86_64-linux-gnu.so(PyInit_onnx_cpp2py_export+0x10f) [0x7f99388bea1f]
[bt] (7) /usr/bin/python3.6(_PyImport_LoadDynamicModuleWithSpec+0x17b) [0x576fab]
[bt] (8) /usr/bin/python3.6() [0x574825]
[bt] (9) /usr/bin/python3.6(PyCFunction_Call+0xbd) [0x4c517d]
```
# Operating system information:
```
root@ec34adf6c4e7:/mnt/bighead# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04 LTS
Release: 18.04
Codename: bionic
```
# PDB stack trace(sensitive details redacted):
```
-> self.serialize_transform(model, X, serializer_key='onnx')
(Pdb) s
--Call--
> /mnt/xxx/python/xxx/tests/test_utils.py(16)serialize_transform()
-> def serialize_transform(self, model, data, check=True, serializer_key=None):
(Pdb) s
> /mnt/xxx/python/xxx/tests/test_utils.py(28)serialize_transform()
-> srz = get_serializer(model.serializer, key=serializer_key)
(Pdb) n
> /mnt/xxx/python/xxx/tests/test_utils.py(29)serialize_transform()
-> desrz = get_deserializer(model.deserializer, key=serializer_key)
(Pdb) n
> /mnt/xxx/python/xxx/tests/test_utils.py(30)serialize_transform()
-> with tempfile.TemporaryDirectory() as temp_dir:
(Pdb) n
> /mnt/xxx/python/xxx/tests/test_utils.py(31)serialize_transform()
-> srz(model, temp_dir)
(Pdb) s
--Call--
> /mnt/xxx/python/xxx/ml_frameworks/mxnet/gluon.py(393)onnx_serialize_fn()
-> @classmethod
(Pdb) n
> /mnt/xxx/python/xxx/ml_frameworks/mxnet/gluon.py(406)onnx_serialize_fn()
-> try:
(Pdb) n
> /mnt/xxx/python/xxx/ml_frameworks/mxnet/gluon.py(407)onnx_serialize_fn()
-> import onnx # noqa
(Pdb) n
Segmentation fault: 11
Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x19e004) [0x7faa1b0d0004]
[bt] (1) /usr/local/lib/python3.6/dist-packages/mxnet/libmxnet.so(+0x2c42bc6) [0x7faa1db74bc6]
[bt] (2) /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7faa3e05df20]
[bt] (3) /usr/local/lib/python3.6/dist-packages/onnx/onnx_cpp2py_export.cpython-36m-x86_64-linux-gnu.so(+0x63bfd) [0x7fa9f4c0bbfd]
[bt] (4) /usr/local/lib/python3.6/dist-packages/onnx/onnx_cpp2py_export.cpython-36m-x86_64-linux-gnu.so(+0x6f82f) [0x7fa9f4c1782f]
[bt] (5) /usr/local/lib/python3.6/dist-packages/onnx/onnx_cpp2py_export.cpython-36m-x86_64-linux-gnu.so(+0x54019) [0x7fa9f4bfc019]
[bt] (6) /usr/local/lib/python3.6/dist-packages/onnx/onnx_cpp2py_export.cpython-36m-x86_64-linux-gnu.so(PyInit_onnx_cpp2py_export+0x10f) [0x7fa9f4bffa1f]
[bt] (7) /usr/bin/python3.6(_PyImport_LoadDynamicModuleWithSpec+0x17b) [0x576fab]
[bt] (8) /usr/bin/python3.6() [0x574825]
[bt] (9) /usr/bin/python3.6(PyCFunction_Call+0xbd) [0x4c517d]
```
### A couple lines around the segfault code
```
@classmethod
def onnx_serialize_fn(cls, t, path):
"""
...
"""
try:
import onnx # noqa
except ImportError:
raise ImportError('Onnx and protobuf need to be installed. Instructions to'
+ ' install - https://github.com/onnx/onnx#installation')
...
```
# ONNX and protobuf versions
```
root@ec34adf6c4e7:/mnt# pip3 freeze | grep onnx
onnx==1.2.2
root@ec34adf6c4e7:/mnt# pip3 freeze | grep protobuf
protobuf==3.6.0
root@ec34adf6c4e7:/mnt/bighead# pip freeze | grep mxnet
keras-mxnet==2.2.0
mxnet-mkl==1.3.0b20180711
```
# Normal imports do not cause segfault
Executing the script below does not cause a segmentation fault.
```
import mxnet as mx
import mxnet.contrib.onnx.mx2onnx as mx2onnx
try:
import onnx
except:
print("Onnx is not installed. The function below will not work.")
mx2onnx.export_model.export_model('gluon/mxnet-symbol.json', 'gluon/mxnet-0000.params', [(1, 1, 28, 28)], onnx_file_path='/mnt/model.onnx', verbose=True)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services