You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2019/12/23 22:21:19 UTC
[GitHub] [incubator-mxnet] cyrusbehr opened a new issue #17158: Cache CUDA
modules when building Mxnet
cyrusbehr opened a new issue #17158: Cache CUDA modules when building Mxnet
URL: https://github.com/apache/incubator-mxnet/issues/17158
I am trying to build Mxnet with GPU support using the following options:
```
cmake -DUSE_CUDNN=1 -DCUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda -DCMAKE_BUILD_TYPE=Release -DBLAS=open -DUSE_OPENCV=OFF -DUSE_CPP_PACKAGE=ON -DENABLE_CUDA_RTC=ON ..
```
Currently, the build takes quiet long (~3 hours). I've noticed that the majority of the time is spent building the Cuda modules.
For my use case, I'll be rebuilding the library often (as part of my CI pipeline) but would like to reduce the build time. Is there some way to run an incremental build which caches these Cuda modules (as they are relatively unchanging) and re-uses them between builds to speed things up?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] cyrusbehr closed issue #17158: Cache CUDA modules
when building Mxnet
Posted by GitBox <gi...@apache.org>.
cyrusbehr closed issue #17158: Cache CUDA modules when building Mxnet
URL: https://github.com/apache/incubator-mxnet/issues/17158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] yajiedesign commented on issue #17158: Cache CUDA
modules when building Mxnet
Posted by GitBox <gi...@apache.org>.
yajiedesign commented on issue #17158: Cache CUDA modules when building Mxnet
URL: https://github.com/apache/incubator-mxnet/issues/17158#issuecomment-568639478
You can try ccache in linux.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on issue #17158: Cache CUDA
modules when building Mxnet
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #17158: Cache CUDA modules when building Mxnet
URL: https://github.com/apache/incubator-mxnet/issues/17158#issuecomment-569630401
@cyrusbehr FYI, #17031 is merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] cyrusbehr commented on issue #17158: Cache CUDA
modules when building Mxnet
Posted by GitBox <gi...@apache.org>.
cyrusbehr commented on issue #17158: Cache CUDA modules when building Mxnet
URL: https://github.com/apache/incubator-mxnet/issues/17158#issuecomment-568778188
Excellent thank you, I will give these a try
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu edited a comment on issue #17158: Cache
CUDA modules when building Mxnet
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #17158: Cache CUDA modules when building Mxnet
URL: https://github.com/apache/incubator-mxnet/issues/17158#issuecomment-568708432
In addition to ccache, you can also
1) Use ninja instead of make. For that specify `cmake -DNinja [...]` and use `ninja` instead of `make -j`.
2) Use a single a single cuda compute architecture. With https://github.com/apache/incubator-mxnet/pull/17031 you only need to set `-DMXNET_CUDA_ARCH=7.0` (as an example), but until it is merged, you need to set multiple variables. See the commit message at #17031 for the variables you currently need to set. If it's not urgent, you can wait a few days until #17031 is merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-mxnet] leezu commented on issue #17158: Cache CUDA
modules when building Mxnet
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #17158: Cache CUDA modules when building Mxnet
URL: https://github.com/apache/incubator-mxnet/issues/17158#issuecomment-568708432
1) Use ninja instead of make. For that specify `cmake -DNinja [...]` and use `ninja` instead of `make -j`.
2) Use a single a single cuda compute architecture. With https://github.com/apache/incubator-mxnet/pull/17031 you only need to set `-DMXNET_CUDA_ARCH=7.0` (as an example), but until it is merged, you need to set multiple variables. See the commit message at #17031 for the variables you currently need to set. If it's not urgent, you can wait a few days until #17031 is merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services