You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@singa.apache.org by GitBox <gi...@apache.org> on 2020/05/25 04:52:38 UTC
[GitHub] [singa] chrishkchris opened a new pull request #709: Update CUDNN from 7.3.1 to 7.6.5
chrishkchris opened a new pull request #709:
URL: https://github.com/apache/singa/pull/709
I see some other frameworks have updated their cudnn to 7.6.
Our cudnn version is quite old, so I modify the version of cudnn from 7.3.1 to 7.6.5 (in conda env as well as docker env). Therefore, when we do performance analysis, we can base on same/similar version of cudnn.
Meanswhile, the speed seems does not change, compared in panda13 (cuda 10.2).
(i) cudnn 7.3.1
```
(singapy36) dcsysh@panda13:~/singa/examples/cnn$ python3 benchmark.py
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:13<00:00, 7.56it/s]
Throughput = 241.5528073484408 per second
TotalTime=13.247620820999146
Total=0.13247620820999145
```
(ii) cudnn 7.6.5
```
(singapy36cudnn765) dcsysh@panda13:~/singa/examples/cnn$ python benchmark.py
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:13<00:00, 7.55it/s]
Throughput = 241.11137013487107 per second
TotalTime=13.271875143051147
Total=0.13271875143051148
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris edited a comment on pull request #709: Upgrade CUDNN version from 7.3.1 to 7.6.5
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on pull request #709:
URL: https://github.com/apache/singa/pull/709#issuecomment-635948985
I found the problem:
in nvidia/cuda:10.2-devel-ubuntu16.04, the location of cublas header moved to another directories,
To simply solve it, I provided a new docker container of cuda10.2 specifically for the conda build using cuda10.2. at tool/conda/docker/cuda10.2/Dockerfile
Also, in order to upgrade to cuda 10.2 the nccl needed to be upgraded to 2.6.4. But I keep locking nccl 2.4.8 using cuda 10.0 for performance reason.
I think PR is ready for merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on pull request #709: Upgrade CUDNN version from 7.3.1 to 7.6.5
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on pull request #709:
URL: https://github.com/apache/singa/pull/709#issuecomment-635948985
I found the problem:
in nvidia/cuda:10.2-devel-ubuntu16.04, the location of cublas header moved to another directories,
To simply solve it, I provided a new docker container of cuda10.2 specifically for the conda build using cuda10.2. at tool/conda/docker/cuda10.2/Dockerfile
Then, in order to upgrade to cuda 10.2 the nccl needed to be upgraded to 2.6.4. But I keep locking nccl 2.4.8 using cuda 10.0 for performance reason.
I think PR is ready for merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris edited a comment on pull request #709: Upgrade CUDNN version from 7.3.1 to 7.6.5
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on pull request #709:
URL: https://github.com/apache/singa/pull/709#issuecomment-635215622
Concerning my conda build test in our server:
Firstly, the cuda runtime library version at each node is:
panda 1-4 cuda 9.0
panda 5-8 cuda 10.0
panda 5-16 cuda 10.1
panda 17-18 cuda 10.2
1. When I use the nvidia docker nvidia/cuda:10.0-devel-ubuntu16.04 to build the conda package on panda 18 using cudatoolkit 10.2 and cudnn 7.6.5. It can build the package and pass the test case. However, when I install the conda package and run, it returns error message:
"ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory"
2. When I use the nvidia docker nvidia/cuda:10.2-devel-ubuntu16.04 to build conda package on panda 18 using cudatoolkit 10.2 and cudnn 7.6.5. It can not compile with the error message:
"/root/miniconda/conda-bld/singa_1590645399908/work/include/singa/utils/cuda_utils.h:24:23: fatal error: cublas_v2.h: No such file or directory"
3. Last time I built the conda package using cuda 10.0 cudnn 7.6.5, the package can be used sucessfully at panda 7 and panda 13 (seems that runtime cuda library installed at every nodes are backward compatiable)
Therefore, in this PR I cannot update to cuda 10.2 (because I did not success to make it run after building it)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] nudles merged pull request #709: Upgrade CUDNN version from 7.3.1 to 7.6.5
Posted by GitBox <gi...@apache.org>.
nudles merged pull request #709:
URL: https://github.com/apache/singa/pull/709
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [singa] chrishkchris commented on pull request #709: Upgrade CUDNN version from 7.3.1 to 7.6.5
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on pull request #709:
URL: https://github.com/apache/singa/pull/709#issuecomment-635215622
Concerning my conda build test in our server:
Firstly, the cuda runtime library version at each node is:
panda 1-4 cuda 9.0
panda 5-8 cuda 10.0
panda 5-16 cuda 10.1
panda 17-18 cuda 10.2
1. When I use the nvidia docker nvidia/cuda:10.0-devel-ubuntu16.04 to build the conda package on panda 18 using cudatoolkit 10.2 and cudnn 7.6.5. It can build the package and pass the test case. However, when I install the conda package and run, it returns error message:
"ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory"
2. When I use the nvidia docker nvidia/cuda:10.2-devel-ubuntu16.04 to build conda package on panda 18 using cudatoolkit 10.2 and cudnn 7.6.5. It can not compile with the error message:
"/root/miniconda/conda-bld/singa_1590645399908/work/include/singa/utils/cuda_utils.h:24:23: fatal error: cublas_v2.h: No such file or directory"
3. Last time I built the conda package using cuda 10.0 cudnn 7.6.5, the package can be used sucessfully at panda 7 and panda 13 (seems that runtime cuda library installed at every nodes are backward compatiable)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org