You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/06/14 07:23:50 UTC
[GitHub] [incubator-mxnet] nttstar opened a new issue #20346: How to run mxnet on NVidia Ampere GPUs?
nttstar opened a new issue #20346:
URL: https://github.com/apache/incubator-mxnet/issues/20346
## Description
(A clear and concise description of what the bug is.)
The pip installed and built from source package are both failed while training [arcface](https://github.com/deepinsight/insightface/tree/master/recognition/ArcFace), with 1.8.0 branch(tag).
### Error Message
(Paste the complete error message. Please also include stack trace by setting environment variable `DMLC_LOG_STACK_TRACE_DEPTH=100` before running your script.)
For ``pip install mxnet-cu112``, the training process will hung on after the log of
```
[15:05:53] ../src/base.cc:80: cuDNN lib mismatch: linked-against version 8101 != compiled-against version 8100. Set MXNET_CUDNN_LIB_CHECKING=0 to quiet this warning.
[15:06:39] ../src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (set the environment variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
```
For building from source:
Firstly, I made some changes at ``cmake/upstream/select_compute_arch.cmake`` to add Ampere arch:
```
if(CUDA_VERSION VERSION_GREATER_EQUAL "11.0")
list(APPEND CUDA_KNOWN_GPU_ARCHITECTURES "Ampere")
list(APPEND CUDA_COMMON_GPU_ARCHITECTURES "8.0")
list(APPEND CUDA_ALL_GPU_ARCHITECTURES "8.0")
set(_CUDA_MAX_COMMON_ARCHITECTURE "8.0+PTX")
set(CUDA_LIMIT_GPU_ARCHITECTURE "8.6")
list(REMOVE_ITEM CUDA_COMMON_GPU_ARCHITECTURES "3.5" "5.0")
list(REMOVE_ITEM CUDA_ALL_GPU_ARCHITECTURES "3.0" "3.2")
endif()
if(CUDA_VERSION VERSION_GREATER_EQUAL "11.1")
list(APPEND CUDA_COMMON_GPU_ARCHITECTURES "8.6")
list(APPEND CUDA_ALL_GPU_ARCHITECTURES "8.6")
set(_CUDA_MAX_COMMON_ARCHITECTURE "8.6+PTX")
set(CUDA_LIMIT_GPU_ARCHITECTURE "9.0")
endif()
```
and
```
elseif(${arch_name} STREQUAL "Ampere")
set(arch_bin 8.0)
set(arch_ptx 8.0)
```
The training process will exit with the error code:
```
cuDNN: Check failed: e == CUDNN_STATUS_SUCCESS (14 vs. 0) : CUDNN_STATUS_VERSION_MISMATCH
```
(My CUDNN is 8.1.0)
Can anyone give me some advice?
## Environment
cuda 11.2(in docker)
cudnn 8.1.0
mxnet-cu112==1.8.0
mxnet source on 1.8.0 tag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] ptrendx commented on issue #20346: How to run mxnet on NVidia Ampere GPUs?
Posted by GitBox <gi...@apache.org>.
ptrendx commented on issue #20346:
URL: https://github.com/apache/incubator-mxnet/issues/20346#issuecomment-860859452
Hi @nttstar, could you give more detailed repro instructions for the hang you are seeing? Generally speaking MXNet 1.8 pip wheel should work on Ampere GPUs so this sounds like a bug somewhere that we should investigate.
@DickJC123 FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #20346: How to run mxnet on NVidia Ampere GPUs?
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #20346:
URL: https://github.com/apache/incubator-mxnet/issues/20346#issuecomment-859782746
select_compute_arch.cmake is taken from https://gitlab.kitware.com/cmake/cmake/blob/master/Modules/FindCUDA/select_compute_arch.cmake but hasn't been updated in a year in mxnet. Would you like to open a PR to update to the latest select_compute_arch.cmake based on the upstream file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #20346: How to run mxnet on NVidia Ampere GPUs?
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #20346:
URL: https://github.com/apache/incubator-mxnet/issues/20346#issuecomment-859782746
select_compute_arch.cmake is taken from https://gitlab.kitware.com/cmake/cmake/blob/master/Modules/FindCUDA/select_compute_arch.cmake but hasn't been updated in a year. Would you like to open a PR to update to the latest select_compute_arch.cmake based on the upstream file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org