You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/02/20 01:52:07 UTC
[GitHub] [incubator-mxnet] larroy opened a new issue #17635: Windows
segmentation faults in GPU tests
larroy opened a new issue #17635: Windows segmentation faults in GPU tests
URL: https://github.com/apache/incubator-mxnet/issues/17635
## Description
Windows GPU tests from the updated environment in https://github.com/aiengines/ci fails with the following:
```
======================================================================
ERROR: Failure: OSError (exception: access violation writing 0x0000000000000000)
----------------------------------------------------------------------
Traceback (most recent call last):
File "C:\Python37\lib\site-packages\nose\failure.py", line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
File "C:\Python37\lib\site-packages\nose\loader.py", line 418, in loadTestsFromName
addr.filename, addr.module)
File "C:\Python37\lib\site-packages\nose\importer.py", line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
File "C:\Python37\lib\site-packages\nose\importer.py", line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
File "C:\Python37\lib\imp.py", line 235, in load_module
return load_source(name, filename, file)
File "C:\Python37\lib\imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "C:\Users\Administrator\mxnet\tests\python\unittest\test_test_utils.py", line 21, in <module>
import mxnet as mx
File "C:\Users\Administrator\mxnet\windows_package\python\mxnet\__init__.py", line 33, in <module>
from . import contrib
File "C:\Users\Administrator\mxnet\windows_package\python\mxnet\contrib\__init__.py", line 27, in <module>
from . import autograd
File "C:\Users\Administrator\mxnet\windows_package\python\mxnet\contrib\autograd.py", line 27, in <module>
from ..ndarray import NDArray, zeros_like, _GRAD_REQ_MAP
File "C:\Users\Administrator\mxnet\windows_package\python\mxnet\ndarray\__init__.py", line 20, in <module>
from . import _internal, contrib, linalg, op, random, sparse, utils, image, ndarray, numpy
File "C:\Users\Administrator\mxnet\windows_package\python\mxnet\ndarray\numpy\__init__.py", line 23, in <module>
from . import _register
File "C:\Users\Administrator\mxnet\windows_package\python\mxnet\ndarray\numpy\_register.py", line 21, in <module>
from ..register import _make_ndarray_function
File "C:\Users\Administrator\mxnet\windows_package\python\mxnet\ndarray\register.py", line 277, in <module>
_init_op_module('mxnet', 'ndarray', _make_ndarray_function)
File "C:\Users\Administrator\mxnet\windows_package\python\mxnet\base.py", line 682, in _init_op_module
ctypes.byref(plist)))
OSError: exception: access violation writing 0x0000000000000000
======================================================================
ERROR: Failure: OSError (exception: access violation writing 0x0000000000000000)
----------------------------------------------------------------------
Traceback (most recent call last):
```
### Error Message
(Paste the complete error message. Please also include stack trace by setting environment variable `DMLC_LOG_STACK_TRACE_DEPTH=10` before running your script.)
## To Reproduce
(If you developed your own code, please provide a short script that reproduces the error. For existing examples, please provide link.)
### Steps to reproduce
(Paste the commands you ran that produced the error.)
1.
2.
## What have you tried to solve it?
1.
2.
## Environment
We recommend using our script for collecting the diagnositc information. Run the following command and paste the outputs below:
```
curl --retry 10 -s https://raw.githubusercontent.com/dmlc/gluon-nlp/master/tools/diagnose.py | python
# paste outputs here
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] vexilligera commented on issue #17635: Windows
segmentation faults in GPU tests
Posted by GitBox <gi...@apache.org>.
vexilligera commented on issue #17635: Windows segmentation faults in GPU tests
URL: https://github.com/apache/incubator-mxnet/issues/17635#issuecomment-588794027
I was able to run the script ci/windows/test_py3_gpu.ps1 properly and did not run into this error with the setup script provided on a local Windows machine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] larroy commented on issue #17635: Windows
segmentation faults in GPU tests
Posted by GitBox <gi...@apache.org>.
larroy commented on issue #17635: Windows segmentation faults in GPU tests
URL: https://github.com/apache/incubator-mxnet/issues/17635#issuecomment-589304572
Ok, that's surprising. what build did you do before running the tests?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] vexilligera commented on issue #17635: Windows
segmentation faults in GPU tests
Posted by GitBox <gi...@apache.org>.
vexilligera commented on issue #17635: Windows segmentation faults in GPU tests
URL: https://github.com/apache/incubator-mxnet/issues/17635#issuecomment-589456636
> Ok, that's surprising. what build did you do before running the tests?
I simply ran pre_setup.ps1 and setup.ps1. Then I cloned MXNet and run `py -3 ci/build_windows.py -f WIN_GPU` and went into the python folder to install with `python setup.py install`.
I was able to import mxnet properly and the test script worked fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] vexilligera commented on issue #17635: Windows
segmentation faults in GPU tests
Posted by GitBox <gi...@apache.org>.
vexilligera commented on issue #17635: Windows segmentation faults in GPU tests
URL: https://github.com/apache/incubator-mxnet/issues/17635#issuecomment-590804348
It turns out that the CUDA architecture detection probably failed, if you go with the 5.2 in the script it does give you mxnet_70.dll on P3 instance, but the binary might be wrong. When I set '-DMXNET_CUDA_ARCH="7.0” ‘ things work properly.
Still pretty weird since yesterday when I was trying this with Clang it gives me the same segmentation fault. I'll probably dig deeper into this if this happens again on a fresh P3 instance when I try to integrate TVM ops.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services