You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/06/22 21:28:02 UTC
[GitHub] [incubator-mxnet] DickJC123 opened a new issue #20374: test_autograd.py very slow (hangs?) on single-cpu-socket system. Meaning of mx.cpu(1) in that case?
DickJC123 opened a new issue #20374:
URL: https://github.com/apache/incubator-mxnet/issues/20374
## Description
I'm seeing the command:
```
pytest --verbose -s tests/python/train/test_autograd.py::test_autograd
```
timeout after 20 minutes on a system with 1 cpu socket, where the test runs in under a minute on a 2 cpu system. The test is a cpu test that references mx.cpu(1).
What is the meaning of mx.cpu(1), in particular on a single-socket system? Should this throw an exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] szha commented on issue #20374: test_autograd.py very slow (hangs?) on single-cpu-socket system. Meaning of mx.cpu(1) in that case?
Posted by GitBox <gi...@apache.org>.
szha commented on issue #20374:
URL: https://github.com/apache/incubator-mxnet/issues/20374#issuecomment-867112026
All CPUs will be used for compute and the device index doesn't have special meaning with respect to compute, threads, or locality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 commented on issue #20374: test_autograd.py very slow (hangs?) on single-cpu-socket system. Meaning of mx.cpu(1) in that case?
Posted by GitBox <gi...@apache.org>.
DickJC123 commented on issue #20374:
URL: https://github.com/apache/incubator-mxnet/issues/20374#issuecomment-867001581
Turns out the 'slow' system with one socketed cpu was an AMD arch, and the 2-cpu system had Intel. Exploring further, a 1-cpu Intel system is fast, and a two-cpu AMD arch system is slow. In other words, the slowness appears tied to the AMD architecture. Perhaps a misconfiguration of openmp for AMD arch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] DickJC123 commented on issue #20374: test_autograd.py very slow (hangs?) on single-cpu-socket system. Meaning of mx.cpu(1) in that case?
Posted by GitBox <gi...@apache.org>.
DickJC123 commented on issue #20374:
URL: https://github.com/apache/incubator-mxnet/issues/20374#issuecomment-868014011
I developed a work-around for myself for the tests/python/train/test_autograd.py test:
- If the cpu type is AMD, set OMP_NUM_THREADS=min(32, nproc)
- If the cpu type is AMD **and** there's only one cpu socket, omit the portion of the test that references mx.cpu(1)
These rules get the test to pass in a reasonable time on the machines I have access to. I'm reluctant to make a PR of these changes, since clearly our upstream CI does not exercise the failure mode. Also, this does not represent a proper tuning of openmp on the AMD arch. Perhaps someone with a greater stake in training on AMD cpus can pick up this issue and run with it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org