You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@mxnet.apache.org by GitBox <gi...@apache.org> on 2020/11/09 14:36:53 UTC
[GitHub] [incubator-mxnet] kpuatamazon opened a new issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
kpuatamazon opened a new issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502
## Description
intgemm is a 3rd-party library written by me and included as a submodule. Unrelated continuous integration tests were going slower afterwards on CentOS 7 CPU. This has been discussed in comments on
https://github.com/apache/incubator-mxnet/commit/13936020d4cc3ccb2d4192adccaa282cef509193
After losing several hairs, the issue appears to be OpenMP support in intgemm's `CMakeLists.txt`:
https://github.com/kpu/intgemm/blob/8f28282c3bd854922da638024d2659be52e892e9/CMakeLists.txt#L47-L56
I think this is causing MXNet to use the slow CentOS OpenMP instead of the bundled support.
## Question
What's the best practice for a standalone library that has its own OpenMP support to not step on MXNet's internal support?
## To Reproduce
Started with c5.18xlarge with the latest AL2 machine learning image.
To build:
```
# Always delete the build directory. This is sneaky and appears to survive.
rm -rf build; ci/build.py --docker-registry mxnetci --platform centos7_cpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh build_centos7_cpu
```
To run:
```
#Running
docker run --cap-add SYS_PTRACE --rm --shm-size=500m -v $HOME/incubator-mxnet:/work/mxnet -v $HOME/incubator-mxnet/build:/work/build -v $HOME/.ccache:/work/ccache -u 1001:1001 -e CCACHE_MAXSIZE=500G -e CCACHE_TEMPDIR=/tmp/ccache -e CCACHE_DIR=/work/ccache -e CCACHE_LOGFILE=/tmp/ccache.log -ti mxnetci/build.centos7_cpu:latest bash
CI_CUDA_COMPUTE_CAPABILITIES='-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70'
CI_CMAKE_CUDA_ARCH='5.2 7.0'
set +x
source /opt/rh/rh-python36/enable
export PATH=/opt/rh/rh-python36/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PATH=/opt/rh/rh-python36/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
export LD_LIBRARY_PATH=/opt/rh/rh-python36/root/usr/lib64
LD_LIBRARY_PATH=/opt/rh/rh-python36/root/usr/lib64
export MANPATH=/opt/rh/rh-python36/root/usr/share/man:
MANPATH=/opt/rh/rh-python36/root/usr/share/man:
export PKG_CONFIG_PATH=/opt/rh/rh-python36/root/usr/lib64/pkgconfig
PKG_CONFIG_PATH=/opt/rh/rh-python36/root/usr/lib64/pkgconfig
export XDG_DATA_DIRS=/opt/rh/rh-python36/root/usr/share:/usr/local/share:/usr/share
XDG_DATA_DIRS=/opt/rh/rh-python36/root/usr/share:/usr/local/share:/usr/share
cd /work/mxnet
nproc
expr 72 / 4
OMP_NUM_THREADS=18
python -m pytest --verbose tests/python/unittest/test_gluon.py::test_slice_pooling2d_slice_pooling2d
```
Repeat the above steps for master and again with these lines commented out in `3rdparty/intgemm/CMakeLists.txt`
```
#option(USE_OPENMP "Use OpenMP" OFF)
#if (USE_OPENMP)
# message(STATUS "Compiling with OpenMP")
# find_package(OpenMP)
# if (NOT ${OpenMP_CXX_FOUND})
# message(SEND_ERROR "OpenMP requested but C++ support not found")
# endif()
# add_compile_options(${OpenMP_CXX_FLAGS})
# target_link_libraries(intgemm PUBLIC OpenMP::OpenMP_CXX)
#endif()
```
The master version takes about 548.22s on a c5.18xlarge, while the commented version takes about 58.87s.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724128401
> I think this is causing MXNet to use the slow CentOS OpenMP instead of the bundled support.
Another hypothesis is that the slowdown results from an interaction of the llvm openmp linked by mxnet + the gcc 7 openmp (?) provided by devtoolset 7 on centos and used by intgemm. Let's try if the issue persists when only linking one openmp implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724189109
On the CD test pipeline (which uses the system provided openmp), I observe on Aug 30 2020:
```
[2020-08-30T20:36:17.326Z] 20.44s call tests/python/unittest/test_gluon.py::test_slice_pooling2d_slice_pooling2d
```
https://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/restricted-mxnet-cd/pipelines/mxnet-cd-release-job/runs/1633/nodes/250/steps/288/log/?start=0
and on Sep 1 (after intgemm was merged)
```
[2020-09-01T20:22:31.173Z] 19.14s call tests/python/unittest/test_gluon.py::test_slice_pooling2d_slice_pooling2d
```
https://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/restricted-mxnet-cd/pipelines/mxnet-cd-release-job/runs/1643/nodes/253/steps/288/log/?start=0
I checked the [respective build logs](https://jenkins.mxnet-ci.amazon-ml.com/blue/rest/organizations/jenkins/pipelines/restricted-mxnet-cd/pipelines/mxnet-cd-release-job/runs/1633/nodes/66/steps/173/log/?start=0), and they do not contain the string `3rdparty/openmp`. They only list
```
[2020-08-30T19:58:43.183Z] -- Found OpenMP_C: -fopenmp (found version "4.5")
[2020-08-30T19:58:43.183Z] -- Found OpenMP_CXX: -fopenmp (found version "4.5")
[2020-08-30T19:58:43.183Z] -- Found OpenMP: TRUE (found version "4.5")
```
meaning that the CentOS 7 system openmp is used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu closed issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu closed issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu removed a comment on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu removed a comment on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724191990
We can compare the test commands on the CD
https://github.com/apache/incubator-mxnet/blob/3042df0b77049257a0b853606e69778cec24fd4b/ci/docker/runtime_functions.sh#L747-L785
to the ones on the CI (where the slowdown is observed)
https://github.com/apache/incubator-mxnet/blob/3042df0b77049257a0b853606e69778cec24fd4b/ci/docker/runtime_functions.sh#L873-L882
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724179407
Thank you for testing this @kpuatamazon! If the slowdown is solely due to the CentOS 7 OpenMP, it implies that we should also enable 3rdparty/openmp in the static builds distributed by others on pypi etc. It was previously disabled for consistency with the old Makefile staticbuild.
https://github.com/apache/incubator-mxnet/blob/564c6d307e3439c1e5bb9bbd7e82d6744bea6a83/CMakeLists.txt#L412
Further, if the slowdown is due to the CentOS 7 OpenMP and unrelated to intgemm, I would expect the "slow test" issue to be present in the Jenkins CD pipeline both before and after intgemm was merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724212516
I can reproduce the difference in perforamce between CD CentOS setup and CI CentOS setup locally.
The test runs in ~20 seconds in the CD setup and takes much longer on the CI setup.
Both use the system provided openmp.
For CD setup, I use
```
rm -rf lib build; python ci/build.py --cache-intermediate --platform centos7_cpu /work/runtime_functions.sh build_static_libmxnet cpu
```
and for CI setup I use
```
rm -rf lib build; ci/build.py --docker-registry mxnetci --platform centos7_cpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh build_centos7_cpu
```
In both cases I have modified the cmake file to remove intgemm:
``` diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 07075d752..8bfd03f53 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -295,12 +295,6 @@ if(USE_MKLDNN)
set_target_properties(dnnl PROPERTIES CXX_CLANG_TIDY "") # don't lint 3rdparty dependency
endif()
-if(USE_INTGEMM)
- message(STATUS "Using intgemm")
- add_subdirectory(3rdparty/intgemm EXCLUDE_FROM_ALL)
- add_definitions(-DMXNET_USE_INTGEMM=1)
-endif()
-
# Allow Cuda compiles outside of src tree to find things in 'src' and 'include'
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/include)
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src)
@@ -509,10 +503,8 @@ endif()
FILE(GLOB_RECURSE SOURCE "src/*.cc" "src/*.h" "include/*.h")
FILE(GLOB_RECURSE CUDA "src/*.cu" "src/*.cuh")
-if(NOT USE_INTGEMM)
FILE(GLOB_RECURSE INTGEMM_OPERATOR_SOURCE "src/operator/contrib/intgemm/*.cc" "src/operator/contrib/intgemm/*.h")
list(REMOVE_ITEM SOURCE ${INTGEMM_OPERATOR_SOURCE})
-endif()
# add nnvm to source
FILE(GLOB_RECURSE NNVMSOURCE
@@ -840,9 +832,6 @@ if(USE_MKLDNN)
${CMAKE_BINARY_DIR}/3rdparty/mkldnn/include/dnnl_version.h ${CMAKE_SOURCE_DIR}/include/mkldnn/)
endif()
-if(USE_INTGEMM)
- target_link_libraries(mxnet PRIVATE intgemm)
-endif()
function(BuildTVMOP)
# scope the variables in BuildTVM.cmake to avoid conflict
```
and deleted the `3rdparty/openmp` folder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] kpuatamazon commented on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
kpuatamazon commented on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724176697
The Centos 7 OpenMP performance is terrible.
To test @leezu 's hypothesis, I created a branch https://github.com/kpuatamazon/incubator-mxnet/tree/removeopenmp without the 3rdparty OpenMP. This means there should only be the OS-/compiler-provided OpenMP.
The same test above took 542.95s with intgemm's CMakeLists.txt as is, and 539.99s with my OpenMP commented out like below. No real difference here.
```
#option(USE_OPENMP "Use OpenMP" OFF)
#if (USE_OPENMP)
# message(STATUS "Compiling with OpenMP")
# find_package(OpenMP)
# if (NOT ${OpenMP_CXX_FOUND})
# message(SEND_ERROR "OpenMP requested but C++ support not found")
# endif()
# add_compile_options(${OpenMP_CXX_FLAGS})
# target_link_libraries(intgemm PUBLIC OpenMP::OpenMP_CXX)
#endif()
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724191990
We can compare the test commands on the CD
https://github.com/apache/incubator-mxnet/blob/3042df0b77049257a0b853606e69778cec24fd4b/ci/docker/runtime_functions.sh#L747-L785
to the ones on the CI (where the slowdown is observed)
https://github.com/apache/incubator-mxnet/blob/3042df0b77049257a0b853606e69778cec24fd4b/ci/docker/runtime_functions.sh#L873-L882
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu edited a comment on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu edited a comment on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724212516
I can reproduce the difference in perforamce between CD CentOS setup and CI CentOS setup locally.
The test runs in ~20 seconds in the CD setup and takes much longer on the CI setup.
For CD setup, I use
```
rm -rf lib build; python ci/build.py --cache-intermediate --platform centos7_cpu /work/runtime_functions.sh build_static_libmxnet cpu
```
and for CI setup I use
```
rm -rf lib build; ci/build.py --docker-registry mxnetci --platform centos7_cpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh build_centos7_cpu
```
In both cases I have modified the cmake file to remove intgemm:
``` diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 07075d752..8bfd03f53 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -295,12 +295,6 @@ if(USE_MKLDNN)
set_target_properties(dnnl PROPERTIES CXX_CLANG_TIDY "") # don't lint 3rdparty dependency
endif()
-if(USE_INTGEMM)
- message(STATUS "Using intgemm")
- add_subdirectory(3rdparty/intgemm EXCLUDE_FROM_ALL)
- add_definitions(-DMXNET_USE_INTGEMM=1)
-endif()
-
# Allow Cuda compiles outside of src tree to find things in 'src' and 'include'
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/include)
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src)
@@ -509,10 +503,8 @@ endif()
FILE(GLOB_RECURSE SOURCE "src/*.cc" "src/*.h" "include/*.h")
FILE(GLOB_RECURSE CUDA "src/*.cu" "src/*.cuh")
-if(NOT USE_INTGEMM)
FILE(GLOB_RECURSE INTGEMM_OPERATOR_SOURCE "src/operator/contrib/intgemm/*.cc" "src/operator/contrib/intgemm/*.h")
list(REMOVE_ITEM SOURCE ${INTGEMM_OPERATOR_SOURCE})
-endif()
# add nnvm to source
FILE(GLOB_RECURSE NNVMSOURCE
@@ -840,9 +832,6 @@ if(USE_MKLDNN)
${CMAKE_BINARY_DIR}/3rdparty/mkldnn/include/dnnl_version.h ${CMAKE_SOURCE_DIR}/include/mkldnn/)
endif()
-if(USE_INTGEMM)
- target_link_libraries(mxnet PRIVATE intgemm)
-endif()
function(BuildTVMOP)
# scope the variables in BuildTVM.cmake to avoid conflict
```
and deleted the `3rdparty/openmp` folder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724128401
> I think this is causing MXNet to use the slow CentOS OpenMP instead of the bundled support.
Another hypothesis is that the slowdown results from an interaction of the llvm openmp linked by mxnet + the gcc 7 openmp (?) provided by devtoolset 7 on centos. Let's try if the issue persists when only linking one openmp implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724316393
@kpuatamazon removing the test coverage instrumentations from the centos build brings the test back to `24.18s`.
I suspect that the test coverage instrumentation would never be part of llvm OpenMP (3rdparty/openmp) due to
https://github.com/apache/incubator-mxnet/blob/551a8d3e94cfa033a933a5aaafafcc0b23ecedcf/CMakeLists.txt#L401
However, as intgemm pulls in the system openmp the coverage instrumentations starts slowing down the tests.
I'm not yet sure why the same doesn't apply to the ubuntu cpu tests, which also include intgemm and enable test coverage instrumentations.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org
[GitHub] [incubator-mxnet] leezu commented on issue #19502: CI slowdown on CentOS CPU related to OpenMP and intgemm
Posted by GitBox <gi...@apache.org>.
leezu commented on issue #19502:
URL: https://github.com/apache/incubator-mxnet/issues/19502#issuecomment-724212516
I can reproduce the difference in perforamce between CD CentOS setup and CI CentOS setup locally.
For CD setup, I use
```
rm -rf lib build; python ci/build.py --cache-intermediate --platform centos7_cpu /work/runtime_functions.sh build_static_libmxnet cpu
```
and for CI setup I use
```
rm -rf lib build; ci/build.py --docker-registry mxnetci --platform centos7_cpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh build_centos7_cpu
```
In both cases I have modified the cmake file to remove intgemm:
``` diff
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 07075d752..8bfd03f53 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -295,12 +295,6 @@ if(USE_MKLDNN)
set_target_properties(dnnl PROPERTIES CXX_CLANG_TIDY "") # don't lint 3rdparty dependency
endif()
-if(USE_INTGEMM)
- message(STATUS "Using intgemm")
- add_subdirectory(3rdparty/intgemm EXCLUDE_FROM_ALL)
- add_definitions(-DMXNET_USE_INTGEMM=1)
-endif()
-
# Allow Cuda compiles outside of src tree to find things in 'src' and 'include'
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/include)
include_directories(${CMAKE_CURRENT_SOURCE_DIR}/src)
@@ -509,10 +503,8 @@ endif()
FILE(GLOB_RECURSE SOURCE "src/*.cc" "src/*.h" "include/*.h")
FILE(GLOB_RECURSE CUDA "src/*.cu" "src/*.cuh")
-if(NOT USE_INTGEMM)
FILE(GLOB_RECURSE INTGEMM_OPERATOR_SOURCE "src/operator/contrib/intgemm/*.cc" "src/operator/contrib/intgemm/*.h")
list(REMOVE_ITEM SOURCE ${INTGEMM_OPERATOR_SOURCE})
-endif()
# add nnvm to source
FILE(GLOB_RECURSE NNVMSOURCE
@@ -840,9 +832,6 @@ if(USE_MKLDNN)
${CMAKE_BINARY_DIR}/3rdparty/mkldnn/include/dnnl_version.h ${CMAKE_SOURCE_DIR}/include/mkldnn/)
endif()
-if(USE_INTGEMM)
- target_link_libraries(mxnet PRIVATE intgemm)
-endif()
function(BuildTVMOP)
# scope the variables in BuildTVM.cmake to avoid conflict
```
and deleted the `3rdparty/openmp` folder.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org