You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/09/15 03:54:00 UTC
[GitHub] [incubator-mxnet] josephevans opened a new pull request #19149: [WIP] [v1.x] Add new CI pipeline for testing Cuda 11.0 builds
josephevans opened a new pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149
## Description ##
This PR adds a new pipeline to CI for testing builds under Cuda 11.0.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#issuecomment-693137527
Jenkins CI successfully triggered : [windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489087940
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -806,6 +806,24 @@ build_ubuntu_gpu_cuda101_cudnn7() {
make cython PYTHON=python3
}
+build_ubuntu_gpu_cuda110_cudnn8() {
+ set -ex
+ build_ccache_wrappers
+ make \
+ USE_BLAS=openblas \
+ USE_MKLDNN=0 \
+ USE_CUDA=1 \
+ USE_CUDA_PATH=/usr/local/cuda \
+ USE_CUDNN=1 \
+ USE_TVM_OP=0 \
+ USE_CPP_PACKAGE=1 \
+ USE_DIST_KVSTORE=1 \
+ CUDA_ARCH="$CI_CUDA_COMPUTE_CAPABILITIES" \
Review comment:
I think the team just wanted to know it would build and test against cuda 11.0 and libcudnn8, which is what this PR tries to address. We could add arch 80 for just this build.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] samskalicky commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
samskalicky commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489058273
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -806,6 +806,24 @@ build_ubuntu_gpu_cuda101_cudnn7() {
make cython PYTHON=python3
}
+build_ubuntu_gpu_cuda110_cudnn8() {
+ set -ex
+ build_ccache_wrappers
+ make \
+ USE_BLAS=openblas \
+ USE_MKLDNN=0 \
Review comment:
yes, but this particular job compiled with it explicitly disabled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] ChaiBapchya commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
ChaiBapchya commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489057391
##########
File path: ci/docker/Dockerfile.build.ubuntu_build_cuda110
##########
@@ -0,0 +1,46 @@
+# -*- mode: dockerfile -*-
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Dockerfile to build MXNet on Ubuntu 16.04 for GPU but on
+# a CPU-only instance. This restriction is caused by the CPP-
+# package generation, requiring the actual CUDA library to be
+# present
+
+FROM nvidia/cuda:11.0-cudnn8-devel-ubuntu16.04
+
+WORKDIR /work/deps
+
+COPY install/ubuntu_core.sh /work/
+RUN /work/ubuntu_core.sh
+COPY install/deb_ubuntu_ccache.sh /work/
+RUN /work/deb_ubuntu_ccache.sh
+COPY install/ubuntu_python.sh /work/
+COPY install/requirements /work/
+RUN /work/ubuntu_python.sh
+
+# Keep this at the end since this command is not cachable
+ARG USER_ID=0
+ARG GROUP_ID=0
+COPY install/ubuntu_adduser.sh /work/
+RUN /work/ubuntu_adduser.sh
+
+COPY runtime_functions.sh /work/
+
+WORKDIR /work/mxnet
+ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/compat
Review comment:
I remember facing issues with cuda/compat directory while migrating from G3 to G4. Do we need this? @ptrendx @leezu confirm.
It was only TVM OP which needed that right? And since we are disabling it we shouldn't need compat dir.
Correct me if wrong.
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -806,6 +806,24 @@ build_ubuntu_gpu_cuda101_cudnn7() {
make cython PYTHON=python3
}
+build_ubuntu_gpu_cuda110_cudnn8() {
+ set -ex
+ build_ccache_wrappers
+ make \
+ USE_BLAS=openblas \
+ USE_MKLDNN=0 \
Review comment:
MKLDNN is ON by default right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
josephevans commented on pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#issuecomment-692990411
@leezu @ChaiBapchya Can you guys please review?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] samskalicky commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
samskalicky commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489058958
##########
File path: ci/docker/Dockerfile.build.ubuntu_build_cuda110
##########
@@ -0,0 +1,46 @@
+# -*- mode: dockerfile -*-
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Dockerfile to build MXNet on Ubuntu 16.04 for GPU but on
+# a CPU-only instance. This restriction is caused by the CPP-
+# package generation, requiring the actual CUDA library to be
+# present
+
+FROM nvidia/cuda:11.0-cudnn8-devel-ubuntu16.04
+
+WORKDIR /work/deps
+
+COPY install/ubuntu_core.sh /work/
+RUN /work/ubuntu_core.sh
+COPY install/deb_ubuntu_ccache.sh /work/
+RUN /work/deb_ubuntu_ccache.sh
+COPY install/ubuntu_python.sh /work/
+COPY install/requirements /work/
+RUN /work/ubuntu_python.sh
+
+# Keep this at the end since this command is not cachable
+ARG USER_ID=0
+ARG GROUP_ID=0
+COPY install/ubuntu_adduser.sh /work/
+RUN /work/ubuntu_adduser.sh
+
+COPY runtime_functions.sh /work/
+
+WORKDIR /work/mxnet
+ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/compat
Review comment:
Pretty sure we need this:
```
/usr/local/cuda-10.1/compat
[root@514f7b276090 compat]# ls
libcuda.so libcuda.so.418.152.00 libnvidia-ptxjitcompiler.so.1
libcuda.so.1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
josephevans commented on pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#issuecomment-693137491
@mxnet-bot run ci [windows-gpu]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489059155
##########
File path: ci/docker/Dockerfile.build.ubuntu_build_cuda110
##########
@@ -0,0 +1,46 @@
+# -*- mode: dockerfile -*-
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Dockerfile to build MXNet on Ubuntu 16.04 for GPU but on
+# a CPU-only instance. This restriction is caused by the CPP-
+# package generation, requiring the actual CUDA library to be
+# present
+
+FROM nvidia/cuda:11.0-cudnn8-devel-ubuntu16.04
+
+WORKDIR /work/deps
+
+COPY install/ubuntu_core.sh /work/
+RUN /work/ubuntu_core.sh
+COPY install/deb_ubuntu_ccache.sh /work/
+RUN /work/deb_ubuntu_ccache.sh
+COPY install/ubuntu_python.sh /work/
+COPY install/requirements /work/
+RUN /work/ubuntu_python.sh
+
+# Keep this at the end since this command is not cachable
+ARG USER_ID=0
+ARG GROUP_ID=0
+COPY install/ubuntu_adduser.sh /work/
+RUN /work/ubuntu_adduser.sh
+
+COPY runtime_functions.sh /work/
+
+WORKDIR /work/mxnet
+ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/compat
Review comment:
Without this, the cpp-package fails to build (due to unable to find libcuda.so.1, which libmxnet.so is linked against.)
I could also disable the cpp-package portion of the build, since it's actually not being used in the test pipeline steps.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] samskalicky merged pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
samskalicky merged pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] sandeep-krishnamurthy commented on pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
sandeep-krishnamurthy commented on pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#issuecomment-694391672
Thank you very much @josephevans :-) :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489061538
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -806,6 +806,24 @@ build_ubuntu_gpu_cuda101_cudnn7() {
make cython PYTHON=python3
}
+build_ubuntu_gpu_cuda110_cudnn8() {
+ set -ex
+ build_ccache_wrappers
+ make \
+ USE_BLAS=openblas \
+ USE_MKLDNN=0 \
Review comment:
I copied the build function from the main full-gpu build (see https://github.com/apache/incubator-mxnet/blob/v1.x/ci/jenkins/Jenkinsfile_unix_gpu#L39) since it seems like we wanted a slimmed-down pipeline for cuda 11.0 for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #19149: [WIP] [v1.x] Add new CI pipeline for testing Cuda 11.0 builds
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#issuecomment-692447060
Hey @josephevans , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [miscellaneous, sanity, unix-gpu, centos-gpu, clang, website, windows-cpu, unix-cpu, edge, windows-gpu, centos-cpu]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] josephevans commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
josephevans commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489094418
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -806,6 +806,24 @@ build_ubuntu_gpu_cuda101_cudnn7() {
make cython PYTHON=python3
}
+build_ubuntu_gpu_cuda110_cudnn8() {
+ set -ex
+ build_ccache_wrappers
+ make \
+ USE_BLAS=openblas \
+ USE_MKLDNN=0 \
+ USE_CUDA=1 \
+ USE_CUDA_PATH=/usr/local/cuda \
+ USE_CUDNN=1 \
+ USE_TVM_OP=0 \
+ USE_CPP_PACKAGE=1 \
+ USE_DIST_KVSTORE=1 \
+ CUDA_ARCH="$CI_CUDA_COMPUTE_CAPABILITIES" \
Review comment:
Added compute_80 and sm_80 to CUDA_ARCH.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] samskalicky commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
samskalicky commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489086363
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -806,6 +806,24 @@ build_ubuntu_gpu_cuda101_cudnn7() {
make cython PYTHON=python3
}
+build_ubuntu_gpu_cuda110_cudnn8() {
+ set -ex
+ build_ccache_wrappers
+ make \
+ USE_BLAS=openblas \
+ USE_MKLDNN=0 \
+ USE_CUDA=1 \
+ USE_CUDA_PATH=/usr/local/cuda \
+ USE_CUDNN=1 \
+ USE_TVM_OP=0 \
+ USE_CPP_PACKAGE=1 \
+ USE_DIST_KVSTORE=1 \
+ CUDA_ARCH="$CI_CUDA_COMPUTE_CAPABILITIES" \
Review comment:
👍 can we do this @josephevans ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] DickJC123 commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
DickJC123 commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489085353
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -806,6 +806,24 @@ build_ubuntu_gpu_cuda101_cudnn7() {
make cython PYTHON=python3
}
+build_ubuntu_gpu_cuda110_cudnn8() {
+ set -ex
+ build_ccache_wrappers
+ make \
+ USE_BLAS=openblas \
+ USE_MKLDNN=0 \
+ USE_CUDA=1 \
+ USE_CUDA_PATH=/usr/local/cuda \
+ USE_CUDNN=1 \
+ USE_TVM_OP=0 \
+ USE_CPP_PACKAGE=1 \
+ USE_DIST_KVSTORE=1 \
+ CUDA_ARCH="$CI_CUDA_COMPUTE_CAPABILITIES" \
Review comment:
I notice that earlier in this file, we have the lines
```
CI_CUDA_COMPUTE_CAPABILITIES="-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70"
CI_CMAKE_CUDA_ARCH="5.2 7.0"
```
Not sure what we're trying to prove exactly with this PR, but do we gain any confidence by adding arch 80, even though there are no A100's in the CI?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] samskalicky commented on a change in pull request #19149: [v1.x] Add new CI pipeline for building and testing with cuda 11.0.
Posted by GitBox <gi...@apache.org>.
samskalicky commented on a change in pull request #19149:
URL: https://github.com/apache/incubator-mxnet/pull/19149#discussion_r489058958
##########
File path: ci/docker/Dockerfile.build.ubuntu_build_cuda110
##########
@@ -0,0 +1,46 @@
+# -*- mode: dockerfile -*-
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# Dockerfile to build MXNet on Ubuntu 16.04 for GPU but on
+# a CPU-only instance. This restriction is caused by the CPP-
+# package generation, requiring the actual CUDA library to be
+# present
+
+FROM nvidia/cuda:11.0-cudnn8-devel-ubuntu16.04
+
+WORKDIR /work/deps
+
+COPY install/ubuntu_core.sh /work/
+RUN /work/ubuntu_core.sh
+COPY install/deb_ubuntu_ccache.sh /work/
+RUN /work/deb_ubuntu_ccache.sh
+COPY install/ubuntu_python.sh /work/
+COPY install/requirements /work/
+RUN /work/ubuntu_python.sh
+
+# Keep this at the end since this command is not cachable
+ARG USER_ID=0
+ARG GROUP_ID=0
+COPY install/ubuntu_adduser.sh /work/
+RUN /work/ubuntu_adduser.sh
+
+COPY runtime_functions.sh /work/
+
+WORKDIR /work/mxnet
+ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/cuda/compat
Review comment:
Pretty sure we need this:
```
/usr/local/cuda-11.0/compat/libcuda.so.450.51.06
/usr/local/cuda-11.0/compat/libcuda.so
/usr/local/cuda-11.0/compat/libcuda.so.1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org