You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@singa.apache.org by GitBox <gi...@apache.org> on 2020/03/09 15:55:58 UTC
[GitHub] [singa] chrishkchris opened a new pull request #624: [WIP]
SINGA-487 Include NCCL and MPICH in conda build
chrishkchris opened a new pull request #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624
Seems that adding nccl and mpich is okay in conda build of singa, but need to check further and add other thing such as python "deprecated"
```
root@3c17fd6cb72e:~/dcsysh/singa/tool/conda/singa# conda-build -c conda-forge . --python 3.6
No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.11
Copying /root/dcsysh/singa to /root/miniconda/conda-bld/singa_1583767660420/work/
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from /root/dcsysh/singa/tool/conda/singa/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /root/dcsysh/singa/tool/conda/singa/conda_build_config.yaml
Adding in variants from config.variant
INFO:conda_build.variants:Adding in variants from config.variant
/root/miniconda/lib/python3.7/site-packages/conda_build/environ.py:427: UserWarning: The environment variable 'CUDA' is being passed through with value '10.0'. If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
UserWarning
Attempting to finalize metadata for singa
INFO:conda_build.metadata:Attempting to finalize metadata for singa
Error: Failed to render jinja template in /root/dcsysh/singa/tool/conda/singa/meta.yaml:
'nccl' is undefined
root@3c17fd6cb72e:~/dcsysh/singa/tool/conda/singa# conda-build -c conda-forge . --python 3.6
No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml. Falling back to default numpy value of 1.11
Copying /root/dcsysh/singa to /root/miniconda/conda-bld/singa_1583767754024/work/
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from /root/dcsysh/singa/tool/conda/singa/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /root/dcsysh/singa/tool/conda/singa/conda_build_config.yaml
Adding in variants from config.variant
INFO:conda_build.variants:Adding in variants from config.variant
/root/miniconda/lib/python3.7/site-packages/conda_build/environ.py:427: UserWarning: The environment variable 'CUDA' is being passed through with value '10.0'. If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
UserWarning
Attempting to finalize metadata for singa
INFO:conda_build.metadata:Attempting to finalize metadata for singa
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Adding .* to spec 'libprotobuf 3.6.1' to ensure satisfiability. Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead. See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING:conda_build.utils:Adding .* to spec 'libprotobuf 3.6.1' to ensure satisfiability. Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead. See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING conda_build.utils:ensure_valid_spec(1749): Adding .* to spec 'libprotobuf 3.6.1' to ensure satisfiability. Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead. See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
Adding .* to spec 'libopenblas 0.3.3' to ensure satisfiability. Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead. See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING:conda_build.utils:Adding .* to spec 'libopenblas 0.3.3' to ensure satisfiability. Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead. See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
WARNING conda_build.utils:ensure_valid_spec(1749): Adding .* to spec 'libopenblas 0.3.3' to ensure satisfiability. Please consider putting {{ var_name }}.* or some relational operator (>/</>=/<=) on this spec in meta.yaml, or if req is also a build req, using {{ pin_compatible() }} jinja2 function instead. See https://conda.io/docs/user-guide/tasks/build-packages/variants.html#pinning-at-the-variant-level
BUILD START: ['singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36.tar.bz2']
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
## Package Plan ##
environment location: /root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl
The following NEW packages will be INSTALLED:
_libgcc_mutex: 0.1-conda_forge conda-forge
_openmp_mutex: 4.5-0_gnu conda-forge
blas: 1.1-openblas conda-forge
ca-certificates: 2019.11.28-hecc5488_0 conda-forge
certifi: 2019.11.28-py36_0 conda-forge
cudatoolkit: 10.0.130-0
cudnn: 7.3.1-cuda10.0_0
glog: 0.3.5-hf484d3e_1001 conda-forge
ld_impl_linux-64: 2.33.1-h53a641e_8 conda-forge
libffi: 3.2.1-he1b5a44_1006 conda-forge
libgcc-ng: 9.2.0-h24d8f2e_2 conda-forge
libgfortran-ng: 7.3.0-hdf63c60_5 conda-forge
libgomp: 9.2.0-h24d8f2e_2 conda-forge
libprotobuf: 3.6.1-hdbcaa40_1001 conda-forge
libstdcxx-ng: 9.2.0-hdf63c60_2 conda-forge
mpi: 1.0-mpich conda-forge
mpich: 3.3.2-hc856adb_0 conda-forge
nccl: 2.5.7.1-hd6f8bf8_0 conda-forge
ncurses: 6.1-hf484d3e_1002 conda-forge
numpy: 1.16.0-py36_blas_openblash1522bff_1000 conda-forge [blas_openblas]
openblas: 0.3.3-h9ac9557_1001 conda-forge
openssl: 1.1.1d-h516909a_0 conda-forge
pcre: 8.44-he1b5a44_0 conda-forge
pip: 20.0.2-py_2 conda-forge
protobuf: 3.6.1-py36hf484d3e_1001 conda-forge
python: 3.6.10-h9d8adfe_1009_cpython conda-forge
python_abi: 3.6-1_cp36m conda-forge
readline: 8.0-hf8c457e_0 conda-forge
setuptools: 46.0.0-py36_0 conda-forge
six: 1.14.0-py36_0 conda-forge
sqlite: 3.30.1-hcee41ef_0 conda-forge
swig: 3.0.12-hf484d3e_1003 conda-forge
tk: 8.6.10-hed695b0_0 conda-forge
wheel: 0.34.2-py_1 conda-forge
xz: 5.2.4-h14c3975_1001 conda-forge
zlib: 1.2.11-h516909a_1006 conda-forge
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
## Package Plan ##
environment location: /root/miniconda/conda-bld/singa_1583767754024/_build_env
The following NEW packages will be INSTALLED:
_libgcc_mutex: 0.1-conda_forge conda-forge
_openmp_mutex: 4.5-0_gnu conda-forge
binutils_impl_linux-64: 2.28.1-had2808c_3
binutils_linux-64: 5.4.0-had2808c_24
bzip2: 1.0.8-h516909a_2 conda-forge
ca-certificates: 2019.11.28-hecc5488_0 conda-forge
cmake: 3.16.4-h28c56e5_0 conda-forge
expat: 2.2.9-he1b5a44_2 conda-forge
gcc_impl_linux-64: 5.4.0-habb00fd_3
gcc_linux-64: 5.4.0-h98af8de_24
gxx_impl_linux-64: 5.4.0-hdf63c60_3
gxx_linux-64: 5.4.0-h98af8de_24
krb5: 1.16.4-h2fd8d38_0 conda-forge
libcurl: 7.68.0-hda55be3_0 conda-forge
libedit: 3.1.20170329-hf8c457e_1001 conda-forge
libgcc-ng: 9.2.0-h24d8f2e_2 conda-forge
libgomp: 9.2.0-h24d8f2e_2 conda-forge
libssh2: 1.8.2-h22169c7_2 conda-forge
libstdcxx-ng: 9.2.0-hdf63c60_2 conda-forge
libuv: 1.34.0-h516909a_0 conda-forge
make: 4.3-h516909a_0 conda-forge
ncurses: 6.1-hf484d3e_1002 conda-forge
openssl: 1.1.1d-h516909a_0 conda-forge
rhash: 1.3.6-h14c3975_1001 conda-forge
tk: 8.6.10-hed695b0_0 conda-forge
xz: 5.2.4-h14c3975_1001 conda-forge
zlib: 1.2.11-h516909a_1006 conda-forge
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
source tree in: /root/miniconda/conda-bld/singa_1583767754024/work
export PREFIX=/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl
export BUILD_PREFIX=/root/miniconda/conda-bld/singa_1583767754024/_build_env
export SRC_DIR=/root/miniconda/conda-bld/singa_1583767754024/work
INFO: activate-binutils_linux-64.sh made the following environmental changes:
+ADDR2LINE=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-addr2line
+AR=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-ar
+AS=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-as
+CXXFILT=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++filt
+ELFEDIT=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-elfedit
+GPROF=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-gprof
+HOST=x86_64-conda_cos6-linux-gnu
+LD=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-ld
+LD_GOLD=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-ld.gold
+NM=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-nm
+OBJCOPY=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-objcopy
+OBJDUMP=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-objdump
+RANLIB=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-ranlib
+READELF=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-readelf
+SIZE=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-size
+STRINGS=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-strings
+STRIP=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-strip
INFO: activate-gcc_linux-64.sh made the following environmental changes:
+CC=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-cc
+CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -I$PREFIX/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix
+CPP=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-cpp
+CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2
+DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -Og -g -Wall -Wextra -fvar-tracking-assignments -pipe -I$PREFIX/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix
+DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og
+GCC=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-gcc
+GCC_AR=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-gcc-ar
+GCC_NM=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-gcc-nm
+GCC_RANLIB=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-gcc-ranlib
+LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,-rpath,$PREFIX/lib -L$PREFIX/lib
+_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_cos6_linux_gnu
INFO: activate-gxx_linux-64.sh made the following environmental changes:
+CXX=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++
+CXXFLAGS=-fvisibility-inlines-hidden -std=c++11 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -I$PREFIX/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix
+DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -std=c++11 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -Og -g -Wall -Wextra -fvar-tracking-assignments -pipe -I$PREFIX/include -fdebug-prefix-map=${SRC_DIR}=/usr/local/src/conda/${PKG_NAME}-${PKG_VERSION} -fdebug-prefix-map=${PREFIX}=/usr/local/src/conda-prefix
+GXX=$BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-g++
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-cc
-- Check for working C compiler: $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++
-- Check for working CXX compiler: $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Protobuf: $PREFIX/lib/libprotobuf.so;-lpthread (found suitable version "3.6.1", minimum required is "3.0")
-- Found CBLAS: $PREFIX/include
-- Found GLOG: $PREFIX/include
-- Found Threads: TRUE
CMake Warning at /root/miniconda/conda-bld/singa_1583767754024/_build_env/share/cmake-3.16/Modules/FindCUDA.cmake:900 (message):
Expecting to find librt for libcudart_static, but didn't find it.
Call Stack (most recent call first):
cmake/Cuda.cmake:20 (FIND_PACKAGE)
cmake/Dependencies.cmake:85 (INCLUDE)
CMakeLists.txt:75 (INCLUDE)
-- Found cuda_v10.0
-- Found CUDNN: $PREFIX/include
-- Found Cudnn_7301 at $PREFIX/include $PREFIX/lib/libcudnn.so
-- Found PythonInterp: $PREFIX/bin/python3 (found suitable version "3.6.10", minimum required is "3")
-- Found PythonLibs: $PREFIX/lib/libpython3.6m.so (found suitable version "3.6.10", minimum required is "3")
-- Found SWIG: $PREFIX/bin/swig (found suitable version "3.0.12", minimum required is "3.0.10")
-- Found MPI at $PREFIX/include
-- Found MPI lib at $PREFIX/lib/libmpi.so
-- Found all lib at $PREFIX/lib/libprotobuf.so;$PREFIX/lib/libopenblas.so;$PREFIX/lib/libglog.so;$PREFIX/lib/libcudnn.so;/usr/local/cuda/lib64/libcudart.so;/usr/local/cuda/lib64/libcurand.so;/usr/local/cuda/lib64/libcusparse.so;/usr/local/cuda/lib64/libcublas.so;$SRC_DIR/build/lib/libcnmem.a;$PREFIX/lib/libmpi.so;$PREFIX/lib/libmpicxx.so
-- Found NCCL at $PREFIX/include
-- Found NCCL lib at $PREFIX/lib/libnccl.so
-- Configuring done
-- Generating done
-- Build files have been written to: $SRC_DIR/build
make[1]: Entering directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
Scanning dependencies of target cnmem
make[2]: Leaving directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
[ 1%] Creating directories for 'cnmem'
[ 2%] Performing download step (git clone) for 'cnmem'
Cloning into 'cnmem'...
Already on 'master'
Your branch is up-to-date with 'origin/master'.
[ 3%] No patch step for 'cnmem'
[ 4%] Performing update step for 'cnmem'
Current branch master is up to date.
[ 5%] Performing configure step for 'cnmem'
-- The C compiler identification is GNU 5.4.0
-- The CXX compiler identification is GNU 5.4.0
-- Check for working C compiler: $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-cc
-- Check for working C compiler: $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++
-- Check for working CXX compiler: $BUILD_PREFIX/bin/x86_64-conda_cos6-linux-gnu-c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
CMake Warning at /root/miniconda/conda-bld/singa_1583767754024/_build_env/share/cmake-3.16/Modules/FindCUDA.cmake:900 (message):
Expecting to find librt for libcudart_static, but didn't find it.
Call Stack (most recent call first):
CMakeLists.txt:6 (find_package)
-- Configuring done
-- Generating done
-- Build files have been written to: $SRC_DIR/build/cnmem-prefix/src/cnmem-build
[ 6%] Performing build step for 'cnmem'
make[3]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[4]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[5]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
Scanning dependencies of target cnmem
make[5]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[5]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
[ 50%] Building CXX object CMakeFiles/cnmem.dir/src/cnmem.cpp.o
[100%] Linking CXX static library libcnmem.a
make[5]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
[100%] Built target cnmem
make[4]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[3]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
[ 7%] Performing install step for 'cnmem'
make[3]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[4]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[5]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[5]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
[100%] Built target cnmem
make[4]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
Install the project...
-- Install configuration: ""
-- Installing: $SRC_DIR/build/lib/libcnmem.a
-- Installing: $SRC_DIR/build/include/cnmem.h
make[3]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
[ 8%] Completed 'cnmem'
make[2]: Leaving directory '$SRC_DIR/build'
[ 8%] Built target cnmem
make[2]: Entering directory '$SRC_DIR/build'
Scanning dependencies of target copy_protobuf
make[2]: Leaving directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
[ 9%] Running cpp protocol buffer compiler on $SRC_DIR/src/proto/model.proto
[libprotobuf WARNING google/protobuf/compiler/parser.cc:562] No syntax specified for the proto file: model.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
[ 10%] Running cpp protocol buffer compiler on $SRC_DIR/src/proto/caffe.proto
[ 11%] Running cpp protocol buffer compiler on $SRC_DIR/src/proto/core.proto
[libprotobuf WARNING google/protobuf/compiler/parser.cc:562] No syntax specified for the proto file: core.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
[ 12%] Running cpp protocol buffer compiler on $SRC_DIR/src/proto/io.proto
[libprotobuf WARNING google/protobuf/compiler/parser.cc:562] No syntax specified for the proto file: io.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
[ 13%] Copying Protobuf headers
make[2]: Leaving directory '$SRC_DIR/build'
[ 13%] Built target copy_protobuf
make[2]: Entering directory '$SRC_DIR/build'
[ 14%] Building NVCC (Device) object src/CMakeFiles/cuda_compile_1.dir/core/tensor/cuda_compile_1_generated_math_kernel.cu.o
Scanning dependencies of target singa_objects
make[2]: Leaving directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
[ 15%] Building CXX object src/CMakeFiles/singa_objects.dir/caffe.pb.cc.o
[ 16%] Building CXX object src/CMakeFiles/singa_objects.dir/core.pb.cc.o
[ 17%] Building CXX object src/CMakeFiles/singa_objects.dir/io.pb.cc.o
[ 18%] Building CXX object src/CMakeFiles/singa_objects.dir/model.pb.cc.o
[ 19%] Building CXX object src/CMakeFiles/singa_objects.dir/utils/channel.cc.o
[ 20%] Building CXX object src/CMakeFiles/singa_objects.dir/utils/logging.cc.o
[ 21%] Building CXX object src/CMakeFiles/singa_objects.dir/io/binfile_reader.cc.o
[ 22%] Building CXX object src/CMakeFiles/singa_objects.dir/io/binfile_writer.cc.o
[ 23%] Building CXX object src/CMakeFiles/singa_objects.dir/io/communicator.cc.o
[ 24%] Building CXX object src/CMakeFiles/singa_objects.dir/io/csv_decoder.cc.o
[ 25%] Building CXX object src/CMakeFiles/singa_objects.dir/io/csv_encoder.cc.o
[ 26%] Building CXX object src/CMakeFiles/singa_objects.dir/io/image_transformer.cc.o
[ 27%] Building CXX object src/CMakeFiles/singa_objects.dir/io/jpg_decoder.cc.o
[ 28%] Building CXX object src/CMakeFiles/singa_objects.dir/io/jpg_encoder.cc.o
[ 29%] Building CXX object src/CMakeFiles/singa_objects.dir/io/lmdb_reader.cc.o
[ 30%] Building CXX object src/CMakeFiles/singa_objects.dir/io/lmdb_writer.cc.o
[ 31%] Building CXX object src/CMakeFiles/singa_objects.dir/io/snapshot.cc.o
[ 32%] Building CXX object src/CMakeFiles/singa_objects.dir/io/textfile_reader.cc.o
[ 34%] Building CXX object src/CMakeFiles/singa_objects.dir/io/textfile_writer.cc.o
[ 35%] Building CXX object src/CMakeFiles/singa_objects.dir/io/network/endpoint.cc.o
[ 36%] Building CXX object src/CMakeFiles/singa_objects.dir/io/network/message.cc.o
[ 37%] Building CXX object src/CMakeFiles/singa_objects.dir/core/device/cpp_cpu.cc.o
[ 38%] Building CXX object src/CMakeFiles/singa_objects.dir/core/device/cuda_gpu.cc.o
[ 39%] Building CXX object src/CMakeFiles/singa_objects.dir/core/device/device.cc.o
[ 40%] Building CXX object src/CMakeFiles/singa_objects.dir/core/device/opencl_device.cc.o
[ 41%] Building CXX object src/CMakeFiles/singa_objects.dir/core/device/platform.cc.o
[ 42%] Building CXX object src/CMakeFiles/singa_objects.dir/core/memory/memory.cc.o
[ 43%] Building CXX object src/CMakeFiles/singa_objects.dir/core/scheduler/scheduler.cc.o
[ 44%] Building CXX object src/CMakeFiles/singa_objects.dir/core/tensor/sparse_tensor.cc.o
[ 45%] Building CXX object src/CMakeFiles/singa_objects.dir/core/tensor/tensor.cc.o
/root/miniconda/conda-bld/singa_1583767754024/work/src/core/tensor/tensor.cc: In function 'void singa::SoftMax(const singa::Tensor&, singa::Tensor*, int)':
/root/miniconda/conda-bld/singa_1583767754024/work/src/core/tensor/tensor.cc:763:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (i < axis)
^
/root/miniconda/conda-bld/singa_1583767754024/work/src/core/tensor/tensor.cc: In function 'void singa::SoftMaxBackward(const singa::Tensor&, singa::Tensor*, int, const singa::Tensor&)':
/root/miniconda/conda-bld/singa_1583767754024/work/src/core/tensor/tensor.cc:802:11: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (i < axis)
^
In file included from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/utils/logging.h:32:0,
from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/core/common.h:27,
from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/core/tensor.h:26,
from /root/miniconda/conda-bld/singa_1583767754024/work/src/core/tensor/tensor.cc:18:
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h: In instantiation of 'std::__cxx11::string* google::Check_LEImpl(const T1&, const T2&, const char*) [with T1 = long unsigned int; T2 = int; std::__cxx11::string = std::__cxx11::basic_string<char>]':
/root/miniconda/conda-bld/singa_1583767754024/work/src/core/tensor/./tensor_math_cuda.h:61:3: required from here
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:719:32: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
DEFINE_CHECK_OP_IMPL(Check_LE, <=)
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:148:53: note: in definition of macro 'GOOGLE_PREDICT_TRUE'
#define GOOGLE_PREDICT_TRUE(x) (__builtin_expect(!!(x), 1))
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:719:1: note: in expansion of macro 'DEFINE_CHECK_OP_IMPL'
DEFINE_CHECK_OP_IMPL(Check_LE, <=)
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h: In instantiation of 'std::__cxx11::string* google::Check_EQImpl(const T1&, const T2&, const char*) [with T1 = long unsigned int; T2 = int; std::__cxx11::string = std::__cxx11::basic_string<char>]':
/root/miniconda/conda-bld/singa_1583767754024/work/src/core/tensor/tensor.cc:355:7: required from here
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:717:32: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
DEFINE_CHECK_OP_IMPL(Check_EQ, ==) // Compilation error with CHECK_EQ(NULL, x)?
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:148:53: note: in definition of macro 'GOOGLE_PREDICT_TRUE'
#define GOOGLE_PREDICT_TRUE(x) (__builtin_expect(!!(x), 1))
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:717:1: note: in expansion of macro 'DEFINE_CHECK_OP_IMPL'
DEFINE_CHECK_OP_IMPL(Check_EQ, ==) // Compilation error with CHECK_EQ(NULL, x)?
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h: In instantiation of 'std::__cxx11::string* google::Check_GEImpl(const T1&, const T2&, const char*) [with T1 = long unsigned int; T2 = int; std::__cxx11::string = std::__cxx11::basic_string<char>]':
/root/miniconda/conda-bld/singa_1583767754024/work/src/core/tensor/tensor.cc:508:5: required from here
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:721:32: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
DEFINE_CHECK_OP_IMPL(Check_GE, >=)
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:148:53: note: in definition of macro 'GOOGLE_PREDICT_TRUE'
#define GOOGLE_PREDICT_TRUE(x) (__builtin_expect(!!(x), 1))
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:721:1: note: in expansion of macro 'DEFINE_CHECK_OP_IMPL'
DEFINE_CHECK_OP_IMPL(Check_GE, >=)
^
[ 46%] Building CXX object src/CMakeFiles/singa_objects.dir/model/feed_forward_net.cc.o
[ 47%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/activation.cc.o
[ 48%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/batchnorm.cc.o
[ 49%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/concat.cc.o
In file included from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/utils/logging.h:32:0,
from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/core/common.h:27,
from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/core/tensor.h:26,
from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/model/layer.h:27,
from /root/miniconda/conda-bld/singa_1583767754024/work/src/model/layer/concat.cc:19:
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h: In instantiation of 'std::__cxx11::string* google::Check_GEImpl(const T1&, const T2&, const char*) [with T1 = long unsigned int; T2 = int; std::__cxx11::string = std::__cxx11::basic_string<char>]':
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/layer/concat.cc:47:8: required from here
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:721:32: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
DEFINE_CHECK_OP_IMPL(Check_GE, >=)
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:148:53: note: in definition of macro 'GOOGLE_PREDICT_TRUE'
#define GOOGLE_PREDICT_TRUE(x) (__builtin_expect(!!(x), 1))
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:721:1: note: in expansion of macro 'DEFINE_CHECK_OP_IMPL'
DEFINE_CHECK_OP_IMPL(Check_GE, >=)
^
[ 50%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/convolution.cc.o
[ 51%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/cudnn_activation.cc.o
[ 52%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/cudnn_batchnorm.cc.o
[ 53%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/cudnn_convolution.cc.o
[ 54%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/cudnn_dropout.cc.o
[ 55%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/cudnn_lrn.cc.o
[ 56%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/cudnn_pooling.cc.o
[ 57%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/cudnn_rnn.cc.o
[ 58%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/cudnn_softmax.cc.o
[ 59%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/dense.cc.o
[ 60%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/dropout.cc.o
[ 61%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/flatten.cc.o
[ 62%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/lrn.cc.o
[ 63%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/merge.cc.o
[ 64%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/opencl_convolution.cc.o
[ 65%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/opencl_pooling.cc.o
[ 67%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/pooling.cc.o
[ 68%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/prelu.cc.o
[ 69%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/rnn.cc.o
[ 70%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/slice.cc.o
In file included from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/utils/logging.h:32:0,
from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/core/common.h:27,
from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/core/tensor.h:26,
from /root/miniconda/conda-bld/singa_1583767754024/work/include/singa/model/layer.h:27,
from /root/miniconda/conda-bld/singa_1583767754024/work/src/model/layer/slice.cc:19:
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h: In instantiation of 'std::__cxx11::string* google::Check_GEImpl(const T1&, const T2&, const char*) [with T1 = int; T2 = unsigned int; std::__cxx11::string = std::__cxx11::basic_string<char>]':
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/layer/slice.cc:33:3: required from here
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:721:32: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
DEFINE_CHECK_OP_IMPL(Check_GE, >=)
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:148:53: note: in definition of macro 'GOOGLE_PREDICT_TRUE'
#define GOOGLE_PREDICT_TRUE(x) (__builtin_expect(!!(x), 1))
^
/root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/include/glog/logging.h:721:1: note: in expansion of macro 'DEFINE_CHECK_OP_IMPL'
DEFINE_CHECK_OP_IMPL(Check_GE, >=)
^
[ 71%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/softmax.cc.o
[ 72%] Building CXX object src/CMakeFiles/singa_objects.dir/model/layer/split.cc.o
[ 73%] Building CXX object src/CMakeFiles/singa_objects.dir/model/optimizer/adagrad.cc.o
[ 74%] Building CXX object src/CMakeFiles/singa_objects.dir/model/optimizer/local_all_reduce.cc.o
[ 75%] Building CXX object src/CMakeFiles/singa_objects.dir/model/optimizer/nesterov.cc.o
[ 76%] Building CXX object src/CMakeFiles/singa_objects.dir/model/optimizer/optimizer.cc.o
[ 77%] Building CXX object src/CMakeFiles/singa_objects.dir/model/optimizer/rmsprop.cc.o
[ 78%] Building CXX object src/CMakeFiles/singa_objects.dir/model/optimizer/sgd.cc.o
[ 79%] Building CXX object src/CMakeFiles/singa_objects.dir/model/loss/mse.cc.o
[ 80%] Building CXX object src/CMakeFiles/singa_objects.dir/model/loss/softmax_cross_entropy.cc.o
[ 81%] Building CXX object src/CMakeFiles/singa_objects.dir/model/metric/accuracy.cc.o
[ 82%] Building CXX object src/CMakeFiles/singa_objects.dir/model/updater/local_updater.cc.o
[ 83%] Building CXX object src/CMakeFiles/singa_objects.dir/model/updater/updater.cc.o
[ 84%] Building CXX object src/CMakeFiles/singa_objects.dir/model/operation/batchnorm.cc.o
[ 85%] Building CXX object src/CMakeFiles/singa_objects.dir/model/operation/convolution.cc.o
[ 86%] Building CXX object src/CMakeFiles/singa_objects.dir/model/operation/pooling.cc.o
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/operation/pooling.cc: In function 'singa::Tensor singa::GpuPoolingForward(const singa::CudnnPoolingHandle&, const singa::Tensor&)':
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/operation/pooling.cc:193:39: warning: narrowing conversion of '(int)(& cph)->singa::CudnnPoolingHandle::<anonymous>.singa::PoolingHandle::batchsize' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
x.device(), x.data_type());
^
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/operation/pooling.cc:192:19: warning: narrowing conversion of '(& cph)->singa::CudnnPoolingHandle::<anonymous>.singa::PoolingHandle::batchsize' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
Tensor({cph.batchsize, cph.channels, cph.pooled_height, cph.pooled_width},
^
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/operation/pooling.cc:193:39: warning: narrowing conversion of '(int)(& cph)->singa::CudnnPoolingHandle::<anonymous>.singa::PoolingHandle::channels' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
x.device(), x.data_type());
^
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/operation/pooling.cc:192:34: warning: narrowing conversion of '(& cph)->singa::CudnnPoolingHandle::<anonymous>.singa::PoolingHandle::channels' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
Tensor({cph.batchsize, cph.channels, cph.pooled_height, cph.pooled_width},
^
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/operation/pooling.cc:193:39: warning: narrowing conversion of '(int)(& cph)->singa::CudnnPoolingHandle::<anonymous>.singa::PoolingHandle::pooled_height' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
x.device(), x.data_type());
^
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/operation/pooling.cc:192:48: warning: narrowing conversion of '(& cph)->singa::CudnnPoolingHandle::<anonymous>.singa::PoolingHandle::pooled_height' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
Tensor({cph.batchsize, cph.channels, cph.pooled_height, cph.pooled_width},
^
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/operation/pooling.cc:193:39: warning: narrowing conversion of '(int)(& cph)->singa::CudnnPoolingHandle::<anonymous>.singa::PoolingHandle::pooled_width' from 'int' to 'long unsigned int' inside { } [-Wnarrowing]
x.device(), x.data_type());
^
/root/miniconda/conda-bld/singa_1583767754024/work/src/model/operation/pooling.cc:192:67: warning: narrowing conversion of '(& cph)->singa::CudnnPoolingHandle::<anonymous>.singa::PoolingHandle::pooled_width' from 'const int' to 'long unsigned int' inside { } [-Wnarrowing]
Tensor({cph.batchsize, cph.channels, cph.pooled_height, cph.pooled_width},
^
make[2]: Leaving directory '$SRC_DIR/build'
[ 90%] Built target singa_objects
make[2]: Entering directory '$SRC_DIR/build'
Scanning dependencies of target singa
make[2]: Leaving directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
[ 91%] Linking CXX shared library ../lib/libsinga.so
make[2]: Leaving directory '$SRC_DIR/build'
[ 92%] Built target singa
make[2]: Entering directory '$SRC_DIR/build'
[ 93%] Running Python protocol buffer compiler on $SRC_DIR/src/proto/caffe.proto
[ 94%] Running Python protocol buffer compiler on $SRC_DIR/src/proto/core.proto
[libprotobuf WARNING google/protobuf/compiler/parser.cc:562] No syntax specified for the proto file: core.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
[ 95%] Running Python protocol buffer compiler on $SRC_DIR/src/proto/io.proto
[libprotobuf WARNING google/protobuf/compiler/parser.cc:562] No syntax specified for the proto file: io.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
[ 96%] Running Python protocol buffer compiler on $SRC_DIR/src/proto/model.proto
[libprotobuf WARNING google/protobuf/compiler/parser.cc:562] No syntax specified for the proto file: model.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
Scanning dependencies of target _singa_wrap
make[2]: Leaving directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
[ 97%] Building CXX object python/CMakeFiles/_singa_wrap.dir/__/src/api/singa_wrap.cxx.o
[ 98%] Linking CXX shared library singa/_singa_wrap.so
make[2]: Leaving directory '$SRC_DIR/build'
[100%] Built target _singa_wrap
make[1]: Leaving directory '$SRC_DIR/build'
CMake Warning at /root/miniconda/conda-bld/singa_1583767754024/_build_env/share/cmake-3.16/Modules/FindCUDA.cmake:900 (message):
Expecting to find librt for libcudart_static, but didn't find it.
Call Stack (most recent call first):
cmake/Cuda.cmake:20 (FIND_PACKAGE)
cmake/Dependencies.cmake:85 (INCLUDE)
CMakeLists.txt:75 (INCLUDE)
-- Found cuda_v10.0
-- Found Cudnn_7301 at $PREFIX/include $PREFIX/lib/libcudnn.so
-- Found MPI at $PREFIX/include
-- Found MPI lib at $PREFIX/lib/libmpi.so
-- Found all lib at $PREFIX/lib/libprotobuf.so;$PREFIX/lib/libopenblas.so;$PREFIX/lib/libglog.so;$PREFIX/lib/libcudnn.so;/usr/local/cuda/lib64/libcudart.so;/usr/local/cuda/lib64/libcurand.so;/usr/local/cuda/lib64/libcusparse.so;/usr/local/cuda/lib64/libcublas.so;$SRC_DIR/build/lib/libcnmem.a;$PREFIX/lib/libmpi.so;$PREFIX/lib/libmpicxx.so
-- Found NCCL at $PREFIX/include
-- Found NCCL lib at $PREFIX/lib/libnccl.so
-- Configuring done
-- Generating done
-- Build files have been written to: $SRC_DIR/build
make[1]: Entering directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
make[2]: Leaving directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
[ 1%] Performing update step for 'cnmem'
Current branch master is up to date.
[ 2%] Performing configure step for 'cnmem'
CMake Warning at /root/miniconda/conda-bld/singa_1583767754024/_build_env/share/cmake-3.16/Modules/FindCUDA.cmake:900 (message):
Expecting to find librt for libcudart_static, but didn't find it.
Call Stack (most recent call first):
CMakeLists.txt:6 (find_package)
-- Configuring done
-- Generating done
-- Build files have been written to: $SRC_DIR/build/cnmem-prefix/src/cnmem-build
[ 3%] Performing build step for 'cnmem'
make[3]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[4]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[5]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[5]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
[100%] Built target cnmem
make[4]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[3]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
[ 4%] Performing install step for 'cnmem'
make[3]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[4]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[5]: Entering directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
make[5]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
[100%] Built target cnmem
make[4]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
Install the project...
-- Install configuration: ""
-- Up-to-date: $SRC_DIR/build/lib/libcnmem.a
-- Up-to-date: $SRC_DIR/build/include/cnmem.h
make[3]: Leaving directory '$SRC_DIR/build/cnmem-prefix/src/cnmem-build'
[ 5%] Completed 'cnmem'
make[2]: Leaving directory '$SRC_DIR/build'
[ 8%] Built target cnmem
make[2]: Entering directory '$SRC_DIR/build'
make[2]: Leaving directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
[ 9%] Copying Protobuf headers
make[2]: Leaving directory '$SRC_DIR/build'
[ 13%] Built target copy_protobuf
make[2]: Entering directory '$SRC_DIR/build'
make[2]: Leaving directory '$SRC_DIR/build'
[ 90%] Built target singa_objects
make[2]: Entering directory '$SRC_DIR/build'
make[2]: Leaving directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
[ 91%] Linking CXX shared library ../lib/libsinga.so
make[2]: Leaving directory '$SRC_DIR/build'
[ 92%] Built target singa
make[2]: Entering directory '$SRC_DIR/build'
Scanning dependencies of target _singa_wrap
make[2]: Leaving directory '$SRC_DIR/build'
make[2]: Entering directory '$SRC_DIR/build'
[ 93%] Building CXX object python/CMakeFiles/_singa_wrap.dir/__/src/api/singa_wrap.cxx.o
[ 94%] Linking CXX shared library singa/_singa_wrap.so
make[2]: Leaving directory '$SRC_DIR/build'
[100%] Built target _singa_wrap
make[1]: Leaving directory '$SRC_DIR/build'
Install the project...
-- Install configuration: ""
-- Up-to-date: $PREFIX//include
-- Installing: $PREFIX//include/singa
-- Installing: $PREFIX//include/singa/singa_config.h
-- Installing: $PREFIX//include/singa/proto
-- Installing: $PREFIX//include/singa/proto/model.pb.h
-- Installing: $PREFIX//include/singa/proto/caffe.pb.h
-- Installing: $PREFIX//include/singa/proto/core.pb.h
-- Installing: $PREFIX//include/singa/proto/io.pb.h
-- Installing: $PREFIX//include/cnmem.h
-- Up-to-date: $PREFIX/include//singa
-- Installing: $PREFIX/include//singa/core
-- Installing: $PREFIX/include//singa/core/device.h
-- Installing: $PREFIX/include//singa/core/tensor.h
-- Installing: $PREFIX/include//singa/core/memory.h
-- Installing: $PREFIX/include//singa/core/scheduler.h
-- Installing: $PREFIX/include//singa/core/common.h
-- Installing: $PREFIX/include//singa/utils
-- Installing: $PREFIX/include//singa/utils/tinydir.h
-- Installing: $PREFIX/include//singa/utils/factory.h
-- Installing: $PREFIX/include//singa/utils/logging.h
-- Installing: $PREFIX/include//singa/utils/safe_queue.h
-- Installing: $PREFIX/include//singa/utils/singleton.h
-- Installing: $PREFIX/include//singa/utils/cuda_utils.h
-- Installing: $PREFIX/include//singa/utils/dnnl_utils.h
-- Installing: $PREFIX/include//singa/utils/channel.h
-- Installing: $PREFIX/include//singa/utils/timer.h
-- Installing: $PREFIX/include//singa/utils/integer.h
-- Installing: $PREFIX/include//singa/utils/opencl_utils.h
-- Installing: $PREFIX/include//singa/utils/string.h
-- Installing: $PREFIX/include//singa/model
-- Installing: $PREFIX/include//singa/model/loss.h
-- Installing: $PREFIX/include//singa/model/updater.h
-- Installing: $PREFIX/include//singa/model/layer.h
-- Installing: $PREFIX/include//singa/model/initializer.h
-- Installing: $PREFIX/include//singa/model/optimizer.h
-- Installing: $PREFIX/include//singa/model/metric.h
-- Installing: $PREFIX/include//singa/model/feed_forward_net.h
-- Installing: $PREFIX/include//singa/io
-- Installing: $PREFIX/include//singa/io/encoder.h
-- Installing: $PREFIX/include//singa/io/reader.h
-- Installing: $PREFIX/include//singa/io/writer.h
-- Installing: $PREFIX/include//singa/io/integer.h
-- Installing: $PREFIX/include//singa/io/network.h
-- Installing: $PREFIX/include//singa/io/snapshot.h
-- Installing: $PREFIX/include//singa/io/communicator.h
-- Installing: $PREFIX/include//singa/io/transformer.h
-- Installing: $PREFIX/include//singa/io/decoder.h
-- Up-to-date: $PREFIX//lib
-- Installing: $PREFIX//lib/libcnmem.a
-- Installing: $PREFIX//lib/libsinga.so
running install
running bdist_egg
running egg_info
creating singa.egg-info
writing singa.egg-info/PKG-INFO
writing dependency_links to singa.egg-info/dependency_links.txt
writing entry points to singa.egg-info/entry_points.txt
writing top-level names to singa.egg-info/top_level.txt
writing manifest file 'singa.egg-info/SOURCES.txt'
reading manifest file 'singa.egg-info/SOURCES.txt'
writing manifest file 'singa.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/rafiki
copying rafiki/agent.py -> build/lib/rafiki
copying rafiki/__init__.py -> build/lib/rafiki
creating build/lib/singa
copying singa/model.py -> build/lib/singa
copying singa/tensor.py -> build/lib/singa
copying singa/singa_wrap.py -> build/lib/singa
copying singa/image_tool.py -> build/lib/singa
copying singa/device.py -> build/lib/singa
copying singa/opt.py -> build/lib/singa
copying singa/__init__.py -> build/lib/singa
copying singa/sonnx.py -> build/lib/singa
copying singa/net.py -> build/lib/singa
copying singa/autograd.py -> build/lib/singa
copying singa/metric.py -> build/lib/singa
copying singa/layer.py -> build/lib/singa
copying singa/optimizer.py -> build/lib/singa
copying singa/utils.py -> build/lib/singa
copying singa/data.py -> build/lib/singa
copying singa/converter.py -> build/lib/singa
copying singa/loss.py -> build/lib/singa
copying singa/snapshot.py -> build/lib/singa
copying singa/initializer.py -> build/lib/singa
creating build/lib/singa/proto
copying singa/proto/core_pb2.py -> build/lib/singa/proto
copying singa/proto/caffe_pb2.py -> build/lib/singa/proto
copying singa/proto/__init__.py -> build/lib/singa/proto
copying singa/proto/model_pb2.py -> build/lib/singa/proto
copying singa/proto/io_pb2.py -> build/lib/singa/proto
copying singa/_singa_wrap.so -> build/lib/singa
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/model.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/tensor.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/singa_wrap.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/image_tool.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/device.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/_singa_wrap.so -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/opt.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/__init__.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/sonnx.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/net.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/autograd.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/metric.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/layer.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/optimizer.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/utils.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/data.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/converter.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/loss.py -> build/bdist.linux-x86_64/egg/singa
copying build/lib/singa/snapshot.py -> build/bdist.linux-x86_64/egg/singa
creating build/bdist.linux-x86_64/egg/singa/proto
copying build/lib/singa/proto/core_pb2.py -> build/bdist.linux-x86_64/egg/singa/proto
copying build/lib/singa/proto/caffe_pb2.py -> build/bdist.linux-x86_64/egg/singa/proto
copying build/lib/singa/proto/__init__.py -> build/bdist.linux-x86_64/egg/singa/proto
copying build/lib/singa/proto/model_pb2.py -> build/bdist.linux-x86_64/egg/singa/proto
copying build/lib/singa/proto/io_pb2.py -> build/bdist.linux-x86_64/egg/singa/proto
copying build/lib/singa/initializer.py -> build/bdist.linux-x86_64/egg/singa
creating build/bdist.linux-x86_64/egg/rafiki
copying build/lib/rafiki/agent.py -> build/bdist.linux-x86_64/egg/rafiki
copying build/lib/rafiki/__init__.py -> build/bdist.linux-x86_64/egg/rafiki
byte-compiling build/bdist.linux-x86_64/egg/singa/model.py to model.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/tensor.py to tensor.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/singa_wrap.py to singa_wrap.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/image_tool.py to image_tool.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/device.py to device.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/opt.py to opt.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/sonnx.py to sonnx.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/net.py to net.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/autograd.py to autograd.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/metric.py to metric.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/layer.py to layer.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/optimizer.py to optimizer.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/utils.py to utils.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/data.py to data.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/converter.py to converter.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/loss.py to loss.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/snapshot.py to snapshot.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/proto/core_pb2.py to core_pb2.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/proto/caffe_pb2.py to caffe_pb2.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/proto/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/proto/model_pb2.py to model_pb2.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/proto/io_pb2.py to io_pb2.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/singa/initializer.py to initializer.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/rafiki/agent.py to agent.cpython-36.pyc
byte-compiling build/bdist.linux-x86_64/egg/rafiki/__init__.py to __init__.cpython-36.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying singa.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying singa.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying singa.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying singa.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying singa.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
singa.__pycache__.singa_wrap.cpython-36: module references __file__
creating dist
creating 'dist/singa-2.1.0-py3.6.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing singa-2.1.0-py3.6.egg
creating $PREFIX/lib/python3.6/site-packages/singa-2.1.0-py3.6.egg
Extracting singa-2.1.0-py3.6.egg to $PREFIX/lib/python3.6/site-packages
Adding singa 2.1.0 to easy-install.pth file
Installing singa script to $PREFIX/bin
Installed $PREFIX/lib/python3.6/site-packages/singa-2.1.0-py3.6.egg
Processing dependencies for singa==2.1.0
Finished processing dependencies for singa==2.1.0
Resource usage statistics from building singa:
Process count: 10
CPU time: Sys=0:00:06.5, User=0:02:17.0
Memory: 777.8M
Disk usage: 251.7K
Time elapsed: 0:03:58.8
Packaging singa
INFO:conda_build.build:Packaging singa
INFO conda_build.build:build(1571): Packaging singa
/root/miniconda/lib/python3.7/site-packages/conda_build/environ.py:427: UserWarning: The environment variable 'CUDA' is being passed through with value '10.0'. If you are splitting build and test phases with --no-test, please ensure that this value is also set similarly at test time.
UserWarning
Packaging singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36
INFO:conda_build.build:Packaging singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36
INFO conda_build.build:bundle_conda(891): Packaging singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36
compiling .pyc files...
found egg dir: /root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl/lib/python3.6/site-packages/singa-2.1.0-py3.6.egg
number of files: 102
Warning: rpath /root/miniconda/conda-bld/singa_1583767754024/_build_env/lib is outside prefix /root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl (removing it)
Warning: rpath /usr/local/cuda/lib64 is outside prefix /root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl (removing it)
Warning: rpath /root/miniconda/conda-bld/singa_1583767754024/_build_env/lib is outside prefix /root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl (removing it)
Warning: rpath /usr/local/cuda/lib64 is outside prefix /root/miniconda/conda-bld/singa_1583767754024/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl (removing it)
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libglog.so.0 found in conda-forge::glog-0.3.5-hf484d3e_1001
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libstdc++.so.6 found in conda-forge::libstdcxx-ng-9.2.0-hdf63c60_2
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libgcc_s.so.1 found in conda-forge::libgcc-ng-9.2.0-h24d8f2e_2
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO x86_64-conda_cos6-linux-gnu/sysroot/lib/ld-linux-x86-64.so.2 found in CDT/compiler package defaults::gcc_impl_linux-64-5.4.0-habb00fd_3
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libmpi.so.12 found in conda-forge::mpich-3.3.2-hc856adb_0
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libnccl.so.2 found in conda-forge::nccl-2.5.7.1-hd6f8bf8_0
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libcurand.so.10.0 found in ['cudatoolkit']
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): .. but ['cudatoolkit'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libopenblas.so.0 found in ['openblas']
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): .. but ['openblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO x86_64-conda_cos6-linux-gnu/sysroot/lib/libm.so.6 found in CDT/compiler package defaults::gcc_impl_linux-64-5.4.0-habb00fd_3
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libpython3.6m.so.1.0 found in conda-forge::python-3.6.10-h9d8adfe_1009_cpython
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libcudart.so.10.0 found in ['cudatoolkit']
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): .. but ['cudatoolkit'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libprotobuf.so.17 found in conda-forge::libprotobuf-3.6.1-hdbcaa40_1001
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libcublas.so.10.0 found in ['cudatoolkit']
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): .. but ['cudatoolkit'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO x86_64-conda_cos6-linux-gnu/sysroot/lib/libc.so.6 found in CDT/compiler package defaults::gcc_impl_linux-64-5.4.0-habb00fd_3
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libcudnn.so.7 found in defaults::cudnn-7.3.1-cuda10.0_0
INFO (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO x86_64-conda_cos6-linux-gnu/sysroot/lib/libpthread.so.0 found in CDT/compiler package defaults::gcc_impl_linux-64-5.4.0-habb00fd_3
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): Needed DSO lib/libcusparse.so.10.0 found in ['cudatoolkit']
WARNING (singa,lib/python3.6/site-packages/singa/_singa_wrap.so): .. but ['cudatoolkit'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (singa,lib/libsinga.so): Needed DSO lib/libcurand.so.10.0 found in ['cudatoolkit']
WARNING (singa,lib/libsinga.so): .. but ['cudatoolkit'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
INFO (singa,lib/libsinga.so): Needed DSO lib/libmpi.so.12 found in conda-forge::mpich-3.3.2-hc856adb_0
INFO (singa,lib/libsinga.so): Needed DSO lib/libstdc++.so.6 found in conda-forge::libstdcxx-ng-9.2.0-hdf63c60_2
INFO (singa,lib/libsinga.so): Needed DSO lib/libcudnn.so.7 found in defaults::cudnn-7.3.1-cuda10.0_0
INFO (singa,lib/libsinga.so): Needed DSO lib/libgcc_s.so.1 found in conda-forge::libgcc-ng-9.2.0-h24d8f2e_2
INFO (singa,lib/libsinga.so): Needed DSO x86_64-conda_cos6-linux-gnu/sysroot/lib/ld-linux-x86-64.so.2 found in CDT/compiler package defaults::gcc_impl_linux-64-5.4.0-habb00fd_3
INFO (singa,lib/libsinga.so): Needed DSO lib/libprotobuf.so.17 found in conda-forge::libprotobuf-3.6.1-hdbcaa40_1001
INFO (singa,lib/libsinga.so): Needed DSO lib/libnccl.so.2 found in conda-forge::nccl-2.5.7.1-hd6f8bf8_0
INFO (singa,lib/libsinga.so): Needed DSO x86_64-conda_cos6-linux-gnu/sysroot/lib/libm.so.6 found in CDT/compiler package defaults::gcc_impl_linux-64-5.4.0-habb00fd_3
WARNING (singa,lib/libsinga.so): Needed DSO lib/libcublas.so.10.0 found in ['cudatoolkit']
WARNING (singa,lib/libsinga.so): .. but ['cudatoolkit'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (singa,lib/libsinga.so): Needed DSO lib/libcudart.so.10.0 found in ['cudatoolkit']
WARNING (singa,lib/libsinga.so): .. but ['cudatoolkit'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
INFO (singa,lib/libsinga.so): Needed DSO x86_64-conda_cos6-linux-gnu/sysroot/lib/libpthread.so.0 found in CDT/compiler package defaults::gcc_impl_linux-64-5.4.0-habb00fd_3
INFO (singa,lib/libsinga.so): Needed DSO lib/libglog.so.0 found in conda-forge::glog-0.3.5-hf484d3e_1001
WARNING (singa,lib/libsinga.so): Needed DSO lib/libcusparse.so.10.0 found in ['cudatoolkit']
WARNING (singa,lib/libsinga.so): .. but ['cudatoolkit'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
INFO (singa,lib/libsinga.so): Needed DSO x86_64-conda_cos6-linux-gnu/sysroot/lib/libc.so.6 found in CDT/compiler package defaults::gcc_impl_linux-64-5.4.0-habb00fd_3
WARNING (singa,lib/libsinga.so): Needed DSO lib/libopenblas.so.0 found in ['openblas']
WARNING (singa,lib/libsinga.so): .. but ['openblas'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
INFO (singa): plugin library package conda-forge::numpy-1.16.0-py36_blas_openblash1522bff_1000 in requirements/run but it is not used (i.e. it is overdepending or perhaps statically linked? If that is what you want then add it to `build/ignore_run_exports`)
Fixing permissions
Packaged license file/s.
Detected hard-coded path in text file bin/singa
Detected hard-coded path in binary file lib/libsinga.so
Detected hard-coded path in binary file lib/python3.6/site-packages/singa/_singa_wrap.so
Importing conda-verify failed. Please be sure to test your packages. conda install conda-verify to make this message go away.
WARNING:conda_build.build:Importing conda-verify failed. Please be sure to test your packages. conda install conda-verify to make this message go away.
WARNING conda_build.build:bundle_conda(1042): Importing conda-verify failed. Please be sure to test your packages. conda install conda-verify to make this message go away.
TEST START: /root/miniconda/conda-bld/linux-64/singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36.tar.bz2
Renaming work directory, /root/miniconda/conda-bld/singa_1583767754024/work to /root/miniconda/conda-bld/singa_1583767754024/work_moved_singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36_linux-64
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
## Package Plan ##
environment location: /root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho
The following NEW packages will be INSTALLED:
_libgcc_mutex: 0.1-conda_forge conda-forge
_openmp_mutex: 4.5-1_llvm conda-forge
ca-certificates: 2019.11.28-hecc5488_0 conda-forge
certifi: 2019.11.28-py36_0 conda-forge
cudatoolkit: 10.0.130-0
cudnn: 7.3.1-cuda10.0_0
freetype: 2.10.0-he983fc9_1 conda-forge
future: 0.18.2-py36_0 conda-forge
glog: 0.3.5-hf484d3e_1001 conda-forge
intel-openmp: 2018.0.3-0
jpeg: 9c-h14c3975_1001 conda-forge
ld_impl_linux-64: 2.33.1-h53a641e_8 conda-forge
libblas: 3.8.0-15_mkl conda-forge
libcblas: 3.8.0-15_mkl conda-forge
libffi: 3.2.1-he1b5a44_1006 conda-forge
libgcc-ng: 9.2.0-h24d8f2e_2 conda-forge
libgfortran-ng: 7.3.0-hdf63c60_5 conda-forge
liblapack: 3.8.0-15_mkl conda-forge
libmklml: 2018.0.3-0
libopenblas: 0.3.3-h5a2b251_3
libpng: 1.6.37-hed695b0_0 conda-forge
libprotobuf: 3.6.1-hdbcaa40_1001 conda-forge
libstdcxx-ng: 9.2.0-hdf63c60_2 conda-forge
libtiff: 4.1.0-hc3755c2_3 conda-forge
llvm-openmp: 9.0.1-hc9558a2_2 conda-forge
lz4-c: 1.8.3-he1b5a44_1001 conda-forge
mkl: 2020.0-166 conda-forge
mkl-dnn: 0.14-h6bb024c_0
mpi: 1.0-mpich conda-forge
mpich: 3.3.2-hc856adb_0 conda-forge
nccl: 2.5.7.1-hd6f8bf8_0 conda-forge
ncurses: 6.1-hf484d3e_1002 conda-forge
numpy: 1.16.5-py36h95a1406_0 conda-forge
olefile: 0.46-py_0 conda-forge
onnx: 1.6.0-py36he1b5a44_0 conda-forge
openssl: 1.1.1d-h516909a_0 conda-forge
pillow: 7.0.0-py36hefe7db6_0 conda-forge
pip: 20.0.2-py_2 conda-forge
protobuf: 3.6.1-py36hf484d3e_1001 conda-forge
python: 3.6.10-h9d8adfe_1009_cpython conda-forge
python_abi: 3.6-1_cp36m conda-forge
readline: 8.0-hf8c457e_0 conda-forge
setuptools: 46.0.0-py36_0 conda-forge
singa: 2.1.0.dev-cudnn7.3.1_cuda10.0_py36 local
six: 1.14.0-py36_0 conda-forge
sqlite: 3.30.1-hcee41ef_0 conda-forge
tk: 8.6.10-hed695b0_0 conda-forge
tqdm: 4.43.0-py_0 conda-forge
wheel: 0.34.2-py_1 conda-forge
xz: 5.2.4-h14c3975_1001 conda-forge
zlib: 1.2.11-h516909a_1006 conda-forge
zstd: 1.4.4-h3b9ef0a_1 conda-forge
Preparing transaction: ...working... done
Verifying transaction: ...working...
ClobberWarning: This transaction has incompatible packages due to a shared path.
packages: defaults/linux-64::intel-openmp-2018.0.3-0, conda-forge/linux-64::llvm-openmp-9.0.1-hc9558a2_2
path: 'lib/libiomp5.so'
ClobberWarning: This transaction has incompatible packages due to a shared path.
packages: conda-forge/linux-64::libblas-3.8.0-15_mkl, defaults/linux-64::libopenblas-0.3.3-h5a2b251_3
path: 'lib/libblas.so'
ClobberWarning: This transaction has incompatible packages due to a shared path.
packages: defaults/linux-64::libopenblas-0.3.3-h5a2b251_3, conda-forge/linux-64::libcblas-3.8.0-15_mkl
path: 'lib/libcblas.so'
ClobberWarning: This transaction has incompatible packages due to a shared path.
packages: defaults/linux-64::libopenblas-0.3.3-h5a2b251_3, conda-forge/linux-64::liblapack-3.8.0-15_mkl
path: 'lib/liblapack.so'
done
Executing transaction: ...working...
ClobberWarning: Conda was asked to clobber an existing path.
source path: /root/miniconda/pkgs/llvm-openmp-9.0.1-hc9558a2_2/lib/libiomp5.so
target path: /root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/libiomp5.so
ClobberWarning: Conda was asked to clobber an existing path.
source path: /root/miniconda/pkgs/libopenblas-0.3.3-h5a2b251_3/lib/libblas.so
target path: /root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/libblas.so
ClobberWarning: Conda was asked to clobber an existing path.
source path: /root/miniconda/pkgs/libcblas-3.8.0-15_mkl/lib/libcblas.so
target path: /root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/libcblas.so
ClobberWarning: Conda was asked to clobber an existing path.
source path: /root/miniconda/pkgs/liblapack-3.8.0-15_mkl/lib/liblapack.so
target path: /root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/liblapack.so
done
export PREFIX=/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho
export SRC_DIR=/root/miniconda/conda-bld/singa_1583767754024/test_tmp
+ cd test/python
+ python run.py
EEEEEFEEEEEEE
======================================================================
ERROR: test_api (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_api
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_api.py", line 27, in <module>
from singa import tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'
======================================================================
ERROR: test_layer (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_layer
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_layer.py", line 23, in <module>
from singa import layer
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/layer.py", line 57, in <module>
from . import tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'
======================================================================
ERROR: test_loss (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_loss
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_loss.py", line 23, in <module>
from singa import loss
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/loss.py", line 42, in <module>
from . import tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'
======================================================================
ERROR: test_metric (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_metric
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_metric.py", line 23, in <module>
from singa import metric
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/metric.py", line 45, in <module>
from . import tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'
======================================================================
ERROR: test_batch_norm (test_mkldnn.TestPythonOperation)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_mkldnn.py", line 150, in test_batch_norm
y = singa_wrap.CpuBatchNormForwardInference(handle, x, scale, bias,
AttributeError: module 'singa.singa_wrap' has no attribute 'CpuBatchNormForwardInference'
======================================================================
ERROR: test_pooling (test_mkldnn.TestPythonOperation)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_mkldnn.py", line 112, in test_pooling
y = singa_wrap.CpuPoolingForward(handle, x)
AttributeError: module 'singa.singa_wrap' has no attribute 'CpuPoolingForward'
======================================================================
ERROR: test_net (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_net
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_net.py", line 25, in <module>
from singa import net
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/net.py", line 68, in <module>
from . import tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'
======================================================================
ERROR: test_onnx (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_onnx
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_onnx.py", line 21, in <module>
from singa import tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'
======================================================================
ERROR: test_onnx_backend (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_onnx_backend
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_onnx_backend.py", line 21, in <module>
from singa import tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'd'
======================================================================
ERROR: test_operation (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_operation
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_operation.py", line 21, in <module>
from singa import tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'
======================================================================
ERROR: test_optimizer (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_optimizer
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_optimizer.py", line 26, in <module>
import singa.tensor as tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'
======================================================================
ERROR: test_tensor (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_tensor
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 428, in _find_test_path
module = self._get_module_from_name(name)
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/unittest/loader.py", line 369, in _get_module_from_name
__import__(name)
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_tensor.py", line 24, in <module>
from singa import tensor
File "/root/miniconda/conda-bld/singa_1583767754024/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placeho/lib/python3.6/site-packages/singa/tensor.py", line 58, in <module>
from deprecated import deprecated
ModuleNotFoundError: No module named 'deprecated'
======================================================================
FAIL: test_conv2d (test_mkldnn.TestPythonOperation)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/miniconda/conda-bld/singa_1583767754024/test_tmp/test/python/test_mkldnn.py", line 78, in test_conv2d
self.assertAlmostEqual(4.0, _dW[0], places=5)
AssertionError: 4.0 != 0.040000003 within 5 places
----------------------------------------------------------------------
Ran 13 tests in 0.003s
FAILED (failures=1, errors=12)
TEST CONV2D FORWARD
TEST CONV2D DATA BACKWARD
TEST CONV2D WEIGHT BACKWARD
+ exit 0
Resource usage statistics from testing singa:
Process count: 1
CPU time: unavailable
Memory: 3.0M
Disk usage: 656B
Time elapsed: 0:00:02.1
TEST END: /root/miniconda/conda-bld/linux-64/singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36.tar.bz2
Renaming work directory, /root/miniconda/conda-bld/singa_1583767754024/work to /root/miniconda/conda-bld/singa_1583767754024/work_moved_singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36_linux-64_main_build_loop
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:
anaconda upload /root/miniconda/conda-bld/linux-64/singa-2.1.0.dev-cudnn7.3.1_cuda10.0_py36.tar.bz2
# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes
anaconda_upload is not set. Not uploading wheels: []
####################################################################################
Resource usage summary:
Total time: 0:07:23.3
CPU usage: sys=0:00:06.5, user=0:02:17.0
Maximum memory usage observed: 777.8M
Total disk usage observed (not including envs): 252.4K
####################################################################################
Source and build intermediates have been left in /root/miniconda/conda-bld.
There are currently 4 accumulated.
To remove them, you can run the ```conda build purge``` command
root@3c17fd6cb72e:~/dcsysh/singa/tool/conda/singa#
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598549734
thanks, I try first
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
Re: [GitHub] [singa] chrishkchris edited a comment on issue #624:
[WIP] SINGA-487 Include NCCL and MPICH in conda build
Posted by Wang Wei <wa...@comp.nus.edu.sg>.
how about using two env variables: CUDA=9.0 or 10.0; DIST (set or not set)
for nccl and mpich.
instead of enumerating all combinations, we can only compile SINGA for the
following 3 cases
CUDA=9.0
CUDA=10.0
CUDA=10.0 with DIST
On Fri, Mar 13, 2020 at 10:06 PM GitBox <gi...@apache.org> wrote:
> chrishkchris edited a comment on issue #624: [WIP] SINGA-487 Include NCCL
> and MPICH in conda build
> URL: https://github.com/apache/singa/pull/624#issuecomment-598734929
>
>
> I simplified the selection logic to use only one environment variable
> CUDA:
> 1. export CUDA = 9.0, then use CUDA9.0 without nccl and moich
> 2. export CUDA = 10.0, then use CUDA10.0 without nccl and mpich
> 3. export CUDA = DIST, then use CUDA10.0 WITH nccl and mpich
> 4. without export CUDA, then use CPU only
>
> If we need two environment variables (CUDA and DIST) to determine the
> selection logic, I can change it.
>
> ----------------------------------------------------------------
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
>
> For queries about this service, please contact Infrastructure at:
> users@infra.apache.org
>
>
> With regards,
> Apache Git Services
>
[GitHub] [singa] chrishkchris edited a comment on issue #624: [WIP]
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598734929
I simplified the selection logic to use only one environment variable CUDA:
1. export CUDA = 9.0, then use CUDA9.0 without nccl and moich
2. export CUDA = 10.0, then use CUDA10.0 without nccl and mpich
3. export CUDA = DIST, then use CUDA10.0 WITH nccl and mpich
4. without export CUDA, then use CPU only
If we need two environment variables (CUDA and DIST) to determine the selection logic, I can change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles edited a comment on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles edited a comment on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598186936
travis build is always cpu-only as there is no GPU in travis.
I suggest to have a separate package for distributed training on GPUs like singa-dist. There is no need to compile singa-dist on CPU-only machines. The default singa packages should be compiled without mpich and nccl.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598632866
I combined all commits into one
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris edited a comment on issue #624: SINGA-487
Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598193989
Sorry, please let me clarify:
1. yes, travis is for cpu only, that’s why I created a separate folder tool/conda/singa/travis to NOT include the gpu related library cudnn, nccl, and mpich. This folder is only used by travis for cpu version build.
2. I added the nccl and mpich in the meta file of original folder tool/conda/singa, which is not for travis and used for gpu version of singa conda build
3. Currently this PR should be adding nccl into singa-gpu. Meanwhile, it is also good to separate singa-dist and singa-gpu
Thanks a lot!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r394064996
##########
File path: tool/conda/dist/meta.yaml
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+{% set version = "2.1.0.dev" %}
Review comment:
@dcslin need to remove the hardcode version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r394129951
##########
File path: tool/conda/dist/README.md
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+The conda package specification includes the package name (i.e. singa), version and build string (could be very long).
+To install a certain Singa package we run
+
+ conda install -c nusdbsystem singa=<version>=<build string>
+
+It is inconvenient to type all 3 parts when running the installation commands.
+The meta.yml file in this folder is to create a conda package `singa-dist` as
+an alias of one specific Singa package.
Review comment:
updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: [WIP] SINGA-487
Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598139326
@nudles @dcslin
Additionally, I excluded the nccl and mpich for cpu-only travis build, by two steps
1. copying the folder tool/conda/singa to tool/conda/singa/travis
to store the build.sh and conda config file specially for cpu-only travis
2. In the conda config file specially for cpu-only travis build, I removed cudnn, nccl and mpich from conda. It becomes more light-weight for cpu conda.
You can see the change files. If you accept this separate conda setting specially for cpu-only travis, this PR is ready for merge after rebasing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598558619
> if there is no CUDA, the condition check # will fail, then the corresponding build will not happen.
>
> Yes, I have committed the change as suggested, and waiting for travis to build cpu version again
Thanks! In Travis CI, now the CPU version of SINGA can build successfully.
The condition check # works properly as a switch, so now the Travis CI CPU version can build successfully by switched off GPU and distributed computing related libraries
Meanwhile, as GPU version was done by Jenkins, don't know if need to config it to release the singa-dist
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-597402533
sorry, the travis CI build is not ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: [WIP] SINGA-487
Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-596943000
@dcslin FYI
One error should be solved after adding python deprecated module in conda
Another error is due to without the conda dnnl
i am not sure if there are other errors
by the way i success build the conda but only test error
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598193989
Sorry, please let me clarify:
1. yes, travis is for cpu only, that’s why I created a separate folder tool/conda/singa/travis to NOT include the gpu related library cudnn, nccl, and mpich
2. I added the nccl and mpich in the meta file of original folder tool/conda/singa, which is not for travis and used for gpu version of singa conda build
3. Currently this PR should be adding nccl into singa-gpu. Meanwhile, it is also good to separate singa-dist and singa-gpu
Thanks a lot!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles merged pull request #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles merged pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-600995313
I will change soon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598208070
>
>
> > travis build is always cpu-only as there is no GPU in travis.
>
> You may be able to build cuda in Travis without GPU, see [this example project](https://github.com/jeremad/cuda-travis). However, travis will not be able to run or test it.
>
> With Github Actions, it is possible to build and test with GPU support with self-hosted runners. So SINGA can be tested on the singa GPU servers at NUS using [Github Actions self-hosted runners](https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners). This means we do not need Travis or Jenkins if we switch to Github Actions which can replace both of them with much easier and better features.
sounds great!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r394065501
##########
File path: tool/conda/singa/conda_build_config.yaml
##########
@@ -27,12 +27,16 @@ CONDA_BUILD_SYSROOT:
cudnn: # [linux]
- "7.3.1 cuda10.0_0" # [environ.get("CUDA")=="10.0"]
- "7.3.1 cuda9.0_0" # [environ.get("CUDA")=="9.0"]
-mkldnn:
- - 0.14
python:
- 3.6
- 3.7
+nccl:
+ - 2.4.8.1
+mpich:
+ - 3.3.2
build_str:
- - "cudnn7.3.1_cuda10.0" # [environ.get("CUDA")=="10.0"]
- - "cudnn7.3.1_cuda9.0" # [environ.get("CUDA")=="9.0"]
- - "cpu" # [environ.get("CUDA", "")== ""]
\ No newline at end of file
+ - "cudnn7.3.1_cuda10.0" # [environ.get("CUDA")=="10.0"] && [environ.get("DIST")=="OFF"]
+ - "cudnn7.3.1_cuda9.0" # [environ.get("CUDA")=="9.0"] && [environ.get("DIST")=="OFF"]
+ - "cpu" # [environ.get("CUDA", "")== ""]
+ - "cudnn7.3.1_cuda10.0_nccl2.4.8.1_mpich3.3.2" # [environ.get("CUDA")=="10.0"] && [environ.get("DIST")=="ON"]
Review comment:
did you do the test on gpu machines?
I mean compile singa dist version using conda on local gpu machine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-603753981
seems that 1.16.5 is okay, I will wait for travis ci
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598550932
as I said, travis has no GPUs, hence it will not buil the dist package..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r392030618
##########
File path: tool/conda/singa/meta.yaml
##########
@@ -50,23 +50,24 @@ requirements:
- glog 0.3.5
- numpy 1.16.0
- cudnn {{ cudnn }} # ['cudnn' in str(build_str)]
- - mkl-dnn {{ mkldnn }}
- python {{ python }}
+ - nccl {{ nccl }}
+ - mpich {{ mpich }}
run:
- {{ pin_compatible('glog', max_pin='x.x') }}
- {{ pin_compatible('numpy', max_pin='x.x') }}
- - {{ pin_compatible('mkl-dnn', max_pin='x.x') }}
- cudnn {{ cudnn }} # ['cudnn' in str(build_str)]
- python {{ python }}
+ - nccl {{ nccl }}
+ - mpich {{ mpich }}
Review comment:
add # ['mpich' in str(build_str)]
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r394132369
##########
File path: tool/conda/dist/meta.yaml
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+{% set version = "2.1.0.dev" %}
Review comment:
I have modified the codes (singa/cpu, singa/gpu, singa/dist) based on @dcslin suggestion:
1. remove `{% set version = "2.1.0.dev" %}`.
2. replace `{{ version }}` with `{{ environ.get('GIT_DESCRIBE_TAG') }}`.
If this is not ok, I can reverse one commit to cancel this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r392030571
##########
File path: tool/conda/singa/meta.yaml
##########
@@ -50,23 +50,24 @@ requirements:
- glog 0.3.5
- numpy 1.16.0
- cudnn {{ cudnn }} # ['cudnn' in str(build_str)]
- - mkl-dnn {{ mkldnn }}
- python {{ python }}
+ - nccl {{ nccl }}
Review comment:
add `# ['nccl' in str(build_str)]`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598552387
> if there is no CUDA, the condition check # will fail, then the corresponding build will not happen.
Yes, I have committed the change as suggested, and waiting for travis to build cpu version again
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-597611771
@dcslin Sorry, I tried and wasted the whole day but still cannot pass the travis build test, while I can build the conda package locally... Can you help me out
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris edited a comment on issue #624: [WIP]
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris edited a comment on issue #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598139326
@nudles @dcslin
Additionally, I excluded the nccl and mpich for cpu-only travis build, by two steps
1. copying the folder tool/conda/singa to tool/conda/singa/travis
to store the build.sh and conda config file specially for cpu-only travis build
2. In the conda config file specially for cpu-only travis build, I removed cudnn, nccl and mpich from conda. It becomes more light-weight for cpu conda built by travis.
You can see the change files. If you accept this separate conda setting specially for cpu-only travis built, this PR is ready for merge after rebasing
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] dcslin commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
dcslin commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-597404760
> @dcslin FYI
> One error should be solved after adding python deprecated module in conda
> Another error is due to without the conda dnnl
> i am not sure if there are other errors
> by the way i success build the conda but only test error
I thought i fixed deprecated module multiple times...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-601543273
Ok, I updated the versions of dependence conda package to make singa and onnx backend coexists #631. This PR is ready for merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] moazreyad commented on issue #624: SINGA-487 Include NCCL
and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
moazreyad commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598195029
> travis build is always cpu-only as there is no GPU in travis.
You may be able to build cuda in Travis without GPU, see [this example project](https://github.com/jeremad/cuda-travis). However, travis will not be able to run or test it.
With Github Actions, it is possible to build and test with GPU support with self-hosted runners. So SINGA can be tested on the singa GPU servers at NUS using [Github Actions self-hosted runners](https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners). This means we do not need Travis or Jenkins if we switch to Github Actions which can replace both of them with much easier and better features.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: [WIP] SINGA-487
Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598034497
@dcslin
Now it is better that travis-ci test pass in ubuntu.
The remaining work is to exclude nccl and mpich for CPU only version
I am thinking how to exclude them in CPU only version
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r394113778
##########
File path: tool/conda/singa/conda_build_config.yaml
##########
@@ -27,12 +27,16 @@ CONDA_BUILD_SYSROOT:
cudnn: # [linux]
- "7.3.1 cuda10.0_0" # [environ.get("CUDA")=="10.0"]
- "7.3.1 cuda9.0_0" # [environ.get("CUDA")=="9.0"]
-mkldnn:
- - 0.14
python:
- 3.6
- 3.7
+nccl:
+ - 2.4.8.1
+mpich:
+ - 3.3.2
build_str:
- - "cudnn7.3.1_cuda10.0" # [environ.get("CUDA")=="10.0"]
- - "cudnn7.3.1_cuda9.0" # [environ.get("CUDA")=="9.0"]
- - "cpu" # [environ.get("CUDA", "")== ""]
\ No newline at end of file
+ - "cudnn7.3.1_cuda10.0" # [environ.get("CUDA")=="10.0"] && [environ.get("DIST")=="OFF"]
+ - "cudnn7.3.1_cuda9.0" # [environ.get("CUDA")=="9.0"] && [environ.get("DIST")=="OFF"]
+ - "cpu" # [environ.get("CUDA", "")== ""]
+ - "cudnn7.3.1_cuda10.0_nccl2.4.8.1_mpich3.3.2" # [environ.get("CUDA")=="10.0"] && [environ.get("DIST")=="ON"]
Review comment:
Yes, did the build test in the docker container for conda-build, on 1.cpu version build, 2.gpu version build, and 3.gpu+nccl version build
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r394064776
##########
File path: tool/conda/dist/README.md
##########
@@ -0,0 +1,51 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+The conda package specification includes the package name (i.e. singa), version and build string (could be very long).
+To install a certain Singa package we run
+
+ conda install -c nusdbsystem singa=<version>=<build string>
+
+It is inconvenient to type all 3 parts when running the installation commands.
+The meta.yml file in this folder is to create a conda package `singa-dist` as
+an alias of one specific Singa package.
Review comment:
Singa --> SINGA
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-597135849
@nudles
I have locked versions of nccl 2.4.8 and mpich 3.3.2, because it is possible that newer versions of dependencies may have API change in the future
After including nccl and mpich, the build can be done with -DUSE_DIST=ON.
There is other problem in conda build due to the inexistence of conda dnnl, but I am not going to add dnnl in this PR. Therefore, I temporary use -DUSE_DNNL=OFF.
So the purpose of this PR is just to add nccl 2.4.8 and mpich 3.3.2 in conda and turn on -DUSE_DIST=ON
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598231433
1. The script for travis building is singa/tool/travis/build.sh:
line 32 is conda build tool/conda/singa --python 3.6
If I understand correctly, both CPU build (by Travis) and GPU build are depended on the same folder tool/conda/singa for the conda config (I saw lots of conda libraries sharing between the CPU and GPU versions in this config file)
Therefore, the reason I create the new folder is to NOT include the GPU related library (cudnn, nccl, mpich) , and this folder is used only by the CPU Travis build. The original folder (tool/conda/singa)
is retained for singa-gpu build.
In this case, I modified the line 32 of singa/tool/travis/build.sh to conda build tool/conda/singa/travis --python 3.6
2. Thanks a lot for this method, I will delete the new folder added above and try your method instead. I suppose conda/cpu and conda/gpu are the additional config added on top of the config file in conda/singa, to distinguish the CPU and GPU versions of singa.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-597682781
I change back the title to [WIP], to indicate that this PR is not ready for merge
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: [WIP] SINGA-487
Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598752299
Yes, sure, I will change to use two env variables: CUDA=9.0 or 10.0; DIST (set or not set)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #624: [WIP] SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-596935250
@dcslin encountered the same error?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-602005178
@joddiy is there any big change from onnx 1.5 to 1.6?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598532099
I have added the folder singa/dist and do the corresponding conda config setting (can view the code change). However, I don't know how to solve the error encountered in travis CI. This time it returned message "cudnn is undefined". Anywhere else I need to change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-597648058
> @chrishkchris is the error due to dnnl missing? if yes, then the solution is to build a dnnl conda package. any other errors?
@nudles, sorry, this time is due to the include path error, some header files are not found by the Travis CI build. Seems that I need to add some include path for the Travis CI build. Meanwhile, when I build the conda in a docker container https://github.com/apache/singa/blob/master/tool/conda/docker/cuda10/Dockerfile it was built successfully.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: [WIP] SINGA-487
Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598734929
I simplified the selection logic to use only one environment variable CUDA:
1. export CUDA = 9.0, then use CUDA9.0 without nccl and moich
2. export CUDA = 10.0, then use CUDA10.0 without nccl and mpich
3. export CUDA = DIST, then use CUDA10.0 WITH nccl and mpich
4. without export CUDA, then use CPU only
If we need two environment variables (CUDA and USE_DIST) to determine the selection logic, I can change it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] joddiy commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
joddiy commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-602371509
> if there are dependent libs compatible with 1.5, then we have to change to 1.6..
> […](#)
> On Mon, Mar 23, 2020 at 11:49 AM Joddiy Zhang ***@***.***> wrote: @joddiy <https://github.com/joddiy> is there any big change from onnx 1.5 to 1.6? The operator's version of onnx 1.5 is 10 and for 1.6 is 11, not big changes but some about operators, some operators attributes have been moved to its inputs, such as clip, the max and min values. It's big trouble for my previous implement, however, I re-constructed the backend and frontend last week, so it's fine for now to change from onnx 1.5 to 1.6. Do we need to? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <[#624 (comment)](https://github.com/apache/singa/pull/624#issuecomment-602370309)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA47DR7WPMPIU5BK35OPCYLRI3L5FANCNFSM4LEL3EFQ> .
Ok, little changes for me, not too much, please change to onnx 1.6.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] joddiy commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
joddiy commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-602370309
> @joddiy is there any big change from onnx 1.5 to 1.6?
The operator's version of onnx 1.5 is 10 and for 1.6 is 11, not big changes but some about operators, some operators attributes have been moved to its inputs, such as clip, the max and min values. It's big trouble for my previous implement, however, I re-constructed the backend and frontend last week, so it's fine for now to change from onnx 1.5 to 1.6. Do we need to?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-603705401
When I restore numpy version, ubuntu build pass, but macos build does not pass.
So I may further search for any better version of numpy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-602371111
if there are dependent libs compatible with 1.5, then we have to change to
1.6..
On Mon, Mar 23, 2020 at 11:49 AM Joddiy Zhang <no...@github.com>
wrote:
> @joddiy <https://github.com/joddiy> is there any big change from onnx 1.5
> to 1.6?
>
> The operator's version of onnx 1.5 is 10 and for 1.6 is 11, not big
> changes but some about operators, some operators attributes have been moved
> to its inputs, such as clip, the max and min values. It's big trouble for
> my previous implement, however, I re-constructed the backend and frontend
> last week, so it's fine for now to change from onnx 1.5 to 1.6. Do we need
> to?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/singa/pull/624#issuecomment-602370309>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AA47DR7WPMPIU5BK35OPCYLRI3L5FANCNFSM4LEL3EFQ>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: [WIP] SINGA-487
Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: [WIP] SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598777002
I changed the selection logic to two env variables:
1. CUDA = 9.0 or 10.0
2. DIST = OFF or ON
If the CUDA is not set, then will use CPU
However, for singa-gpu or singa-dist, both the env variables CUDA and DIST should be defined, otherwise there is error (I tried to change the logic before to let DIST unset for default DIST=OFF, but this is not successful in conda_build_config.yaml)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r394113088
##########
File path: tool/conda/singa/build.sh
##########
@@ -31,9 +31,16 @@ else
USE_CUDA=ON
fi
+
+if [ $DIST == "ON" ]; then
+ USE_DIST=ON
+else
+ USE_DIST=OFF
+fi
+
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$PREFIX -DUSE_CUDA=$USE_CUDA \
- -DUSE_PYTHON3=ON -DUSE_MKLDNN=ON -DCMAKE_OSX_SYSROOT=${CONDA_BUILD_SYSROOT} ..
+ -DUSE_PYTHON3=ON -DUSE_DNNL=OFF -DUSE_DIST=$USE_DIST -DCMAKE_OSX_SYSROOT=${CONDA_BUILD_SYSROOT} ..
Review comment:
DNNL is necessary for CNN, so will change to -DUSE_DNNL=ON after DNNL package is ready. (In the past -DUSE_MKLDNN=ON at default)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] dcslin commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
dcslin commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r394118802
##########
File path: tool/conda/dist/meta.yaml
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+{% set version = "2.1.0.dev" %}
Review comment:
We can remove `{% set version = "2.1.0.dev" %}`.
Assume that git latest tag is always available and reliable, then we could replace all `{{ version }}` with `{{ environ.get('GIT_DESCRIBE_TAG') }}`.
One side effect is in travis build, we need to `git fetch --unshallow` to get all the tags(including latest) from remote to build environment before build starts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-601000847
I just resolved the conflict, and the travis CPU build is successful.
some test case error on test_onnx_backend "ImportError: libprotobuf.so.20: cannot open shared object file: No such file or directory" may be solved later.
So this PR concerning NCCL and MPICH should be ready for merge. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598551579
> as I said, travis has no GPUs, hence it will not buil the dist package..
yes, I will use travis to build CPU version only, so I am learning how to switch off the cudnn, nccl, mpich in the travis cpu build
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598551954
if there is no CUDA, the condition check `# ` will fail, then the corresponding build will not happen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r394065182
##########
File path: tool/conda/singa/build.sh
##########
@@ -31,9 +31,16 @@ else
USE_CUDA=ON
fi
+
+if [ $DIST == "ON" ]; then
+ USE_DIST=ON
+else
+ USE_DIST=OFF
+fi
+
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$PREFIX -DUSE_CUDA=$USE_CUDA \
- -DUSE_PYTHON3=ON -DUSE_MKLDNN=ON -DCMAKE_OSX_SYSROOT=${CONDA_BUILD_SYSROOT} ..
+ -DUSE_PYTHON3=ON -DUSE_DNNL=OFF -DUSE_DIST=$USE_DIST -DCMAKE_OSX_SYSROOT=${CONDA_BUILD_SYSROOT} ..
Review comment:
USE_DNNL is a variable whose value depends on the environment. It should not be hard coded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598186936
travis build is always cpu-only as there is no GPU in travis.
I suggest to have a separate package for distributed training on GPUs like singa-dist. There is no need to do compile singa-dist on CPU-only machines. The default singa packages are compiled without mpich and nccl.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-597644525
@chrishkchris is the error due to dnnl missing? if yes, then the solution is to build a dnnl conda package. any other errors?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on issue #624: SINGA-487 Include NCCL and
MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598212865
>
>
> Sorry, please let me clarify:
>
> 1. yes, travis is for cpu only, that’s why I created a separate folder tool/conda/singa/travis to NOT include the gpu related library cudnn, nccl, and mpich. This folder is only used by travis for cpu version build.
>
> 2. I added the nccl and mpich in the meta file of original folder tool/conda/singa, which is not for travis and used for gpu version of singa conda build
>
> 3. Currently this PR should be adding nccl into singa-gpu. Meanwhile, it is also good to separate singa-dist and singa-gpu
> Thanks a lot!
1. since travis is for cpu-only, why do you create another folder for it? nothing needs to be changed. the scripts for travis building are [here](https://github.com/apache/singa/tree/dev/tool/travis)
2. we only need to add a folder conda/dist like conda/cpu and conda/gpu for building singa-dist, and update the scripts in conda/singa/ accordingly. meta.yml of conda supports `if` like this [line](https://github.com/apache/singa/blob/dev/tool/conda/singa/meta.yaml#L52).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-598200478
> Sorry, please let me clarify:
>
> 1. yes, travis is for cpu only, that’s why I created a separate folder tool/conda/singa/travis to NOT include the gpu related library cudnn, nccl, and mpich. This folder is only used by travis for cpu version build.
> 2. I added the nccl and mpich in the meta file of original folder tool/conda/singa, which is not for travis and used for gpu version of singa conda build
> 3. Currently this PR should be adding nccl into singa-gpu. Meanwhile, it is also good to separate singa-dist and singa-gpu
> Thanks a lot!
so maybe I need to figure out a good way to separate singa-gpu and singa-dist (because currently in this PR nccl is added directly to singa-gpu)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] nudles commented on a change in pull request #624:
SINGA-487 Include NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
nudles commented on a change in pull request #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#discussion_r392030507
##########
File path: tool/conda/singa/conda_build_config.yaml
##########
@@ -27,12 +27,16 @@ CONDA_BUILD_SYSROOT:
cudnn: # [linux]
- "7.3.1 cuda10.0_0" # [environ.get("CUDA")=="10.0"]
- "7.3.1 cuda9.0_0" # [environ.get("CUDA")=="9.0"]
-mkldnn:
- - 0.14
python:
- 3.6
- 3.7
+nccl:
+ - 2.4.8.1
+mpich:
+ - 3.3.2
build_str:
- "cudnn7.3.1_cuda10.0" # [environ.get("CUDA")=="10.0"]
- "cudnn7.3.1_cuda9.0" # [environ.get("CUDA")=="9.0"]
- - "cpu" # [environ.get("CUDA", "")== ""]
\ No newline at end of file
+ - "cpu" # [environ.get("CUDA", "")== ""]
+ - "cudnn7.3.1_cuda10.0_nccl2.4.8.1_mpich3.3.2"
Review comment:
It should be
` "cudnn7.3.1_cuda10.0_nccl2.4.8.1_mpich3.3.2" # [environ.get("CUDA")=="10.0"]`
It means if there is GPU on the machine, then build this package.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-603763969
numpy 1.16.5 is ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [singa] chrishkchris commented on issue #624: SINGA-487 Include
NCCL and MPICH in conda build
Posted by GitBox <gi...@apache.org>.
chrishkchris commented on issue #624: SINGA-487 Include NCCL and MPICH in conda build
URL: https://github.com/apache/singa/pull/624#issuecomment-603817946
Ready for merge
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services