You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@singa.apache.org by wa...@apache.org on 2017/01/13 03:36:51 UTC
[2/3] incubator-singa git commit: SINGA-293 - Bug from compiling
PySINGA on Mac OS X with multiple version of Python
SINGA-293 - Bug from compiling PySINGA on Mac OS X with multiple version of Python
Add solution to the FAQ for this problem and updated the jenkins_wheel.sh to fix the error.
update tools/jenkins/README.md and installation.md for the mac osx wheel generation and download url respectivley.
fix a bug in SINGA-292, PROTOBUF_PROTOC_EXECUTABLE -> Protobuf_PROTOC_EXECUTABLE
Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/8621f58b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/8621f58b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/8621f58b
Branch: refs/heads/master
Commit: 8621f58bc933c5be9374bb8d3dc54e6059a7d577
Parents: 45efd35
Author: wangwei <wa...@comp.nus.edu.sg>
Authored: Wed Jan 11 16:37:21 2017 +0800
Committer: wangwei <wa...@comp.nus.edu.sg>
Committed: Thu Jan 12 15:13:14 2017 +0800
----------------------------------------------------------------------
CMakeLists.txt | 4 +--
doc/en/docs/installation.md | 71 ++++++++++++++++++++++----------------
tool/jenkins/README.md | 44 ++++++++++++++++-------
tool/jenkins/jenkins_wheel.sh | 6 +++-
4 files changed, 80 insertions(+), 45 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/8621f58b/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f3f73b9..9f44fd1 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -127,8 +127,8 @@ IF (USE_MODULES)
"--prefix=${CMAKE_BINARY_DIR}/" "CXXFLAGS=-fPIC"
INSTALL_COMMAND "make" "install"
)
- SET(PROTOBUF_PROTOC_EXECUTABLE "${CMAKE_BINARY_DIR}/bin/protoc")
- message(STATUS "${PROTOBUF_PROTOC_EXECUTABLE}")
+ SET(Protobuf_PROTOC_EXECUTABLE "${CMAKE_BINARY_DIR}/bin/protoc")
+ # message(STATUS "${PROTOBUF_PROTOC_EXECUTABLE}")
ExternalProject_Add(openblas
GIT_REPOSITORY "https://github.com/xianyi/OpenBLAS.git"
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/8621f58b/doc/en/docs/installation.md
----------------------------------------------------------------------
diff --git a/doc/en/docs/installation.md b/doc/en/docs/installation.md
index fedb7b8..ac5bded 100755
--- a/doc/en/docs/installation.md
+++ b/doc/en/docs/installation.md
@@ -3,7 +3,7 @@
## Install PySINGA
-PySINGA has been tested on Linux (Ubunu 14.04 and 16.04) and Mac OS (10.11).
+PySINGA has been tested on Linux (Ubunu 14.04 and 16.04) and Mac OS (10.11 and 10.12).
### Install dependent libraries
@@ -41,16 +41,17 @@ to avoid the conflicts of system path and virtual environment path.
Currently, we have the following wheel files,
-| OS | Lang | CUDNN | Link |
+| OS | Device | CUDA/CUDNN | Link |
|:---- |:------|------|:-------|:--------|
-| Ubuntu14.04|CPP11|-|[latest](http://comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/latest/ubuntu14.04-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/)|
-| Ubuntu14.04|CUDA7.5|CUDNN4|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/latest/ubuntu14.04-cuda7.5-cudnn4/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/)|
-| Ubuntu14.04|CUDA7.5|CUDNN5|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/latest/ubuntu14.04-cuda7.5-cudnn5/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/)|
-| Ubuntu16.04|CPP11|-|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/latest/ubuntu16.04-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/)|
-| Ubuntu16.04|CUDA8|CUDNN5|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/latest/ubuntu16.04-cuda8.0-cudnn5/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/)|
-| MacOSX10.11|CPP11|-|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/latest/macosx10.11-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/)|
+| Ubuntu14.04|CPU|-|[latest](http://comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux/latest/ubuntu14.04-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux)|
+| Ubuntu14.04|GPU|CUDA7.5+CUDNN4|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux/latest/ubuntu14.04-cuda7.5-cudnn4/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux)|
+| Ubuntu14.04|GPU|CUDA7.5+CUDNN5|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux/latest/ubuntu14.04-cuda7.5-cudnn5/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux)|
+| Ubuntu16.04|CPU|-|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux/latest/ubuntu16.04-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux)|
+| Ubuntu16.04|GPU|CUDA8+CUDNN5|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux/latest/ubuntu16.04-cuda8.0-cudnn5/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/linux)|
+| MacOSX10.11|CPU|-|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/macosx/latest/macosx10.11-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/macosx)|
+| MacOSX10.12|CPU|-|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/macosx/latest/macosx10.12-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/wheel/macosx)|
-Please download the whl file and execute the following command to install PySINGA,
+Download the whl file and execute the following command to install PySINGA,
$ pip install --upgrade <path to the whel file>
@@ -74,11 +75,11 @@ They can be installed by
$ sudo apt-get install swig
Note that swig has to be installed from source on Ubuntu 14.04.
-After installing numpy, please export the header path of numpy.i as
+After installing numpy, export the header path of numpy.i as
$ export CPLUS_INCLUDE_PATH=`python -c "import numpy; print numpy.get_include()"`:$CPLUS_INCLUDE_PATH
-Please **compile SINGA from source** (see the next section) with `cmake -DUSE_PYTHON=ON ..`,
+**compile SINGA from source** (see the next section) with `cmake -DUSE_PYTHON=ON ..`,
and then run the following commands,
# under the build directory
@@ -98,15 +99,15 @@ The generated wheel file is under "dist" directory.
We have prepared the Debian packages (on architecture: amd64) for SINGA as listed below,
-| OS | Lang | CUDNN | Link |
+| OS | Device | CUDA/CUDNN | Link |
|:---- |:------|------|:-------|:--------|
-| Ubuntu14.04|CPP11|-|[latest](http://comp.nus.edu.sg/~dbsystem/singa/assets/file/debian/latest/ubuntu14.04-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/debian/)|
-| Ubuntu14.04|CUDA7.5|CUDNN4|coming soon|
-| Ubuntu14.04|CUDA7.5|CUDNN5|coming soon|
-| Ubuntu16.04|CPP11|-|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/debian/latest/ubuntu16.04-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/debian/)|
-| Ubuntu16.04|CUDA8|CUDNN5|coming soon|
+| Ubuntu14.04|CPU|-|[latest](http://comp.nus.edu.sg/~dbsystem/singa/assets/file/debian/latest/ubuntu14.04-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/debian/)|
+| Ubuntu14.04|GPU|CUDA7.5+CUDNN4|coming soon|
+| Ubuntu14.04|GPU|CUDA7.5+CUDNN5|coming soon|
+| Ubuntu16.04|CPU|-|[latest](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/debian/latest/ubuntu16.04-cpp/), [history](http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/debian/)|
+| Ubuntu16.04|GPU|CUDA8+CUDNN5|coming soon|
-Please download the deb file and install it via
+Download the deb file and install it via
apt-get install <path to the deb file, e.g., ./python-singa.deb>
@@ -168,7 +169,7 @@ The following instructions are tested on Ubuntu 14.04 for installing dependent l
$ sudo apt-get install python2.7-dev python-pip python-numpy
$ sudo apt-get install libopencv-dev libgoogle-glog-dev liblmdb-dev
-Please note that PySINGA requires swig >=3.0, which could be installed via
+Note that PySINGA requires swig >=3.0, which could be installed via
apt-get on Ubuntu 16.04; but it has to be installed from source for other Ubuntu versions including 14.04.
The following instructions are tested on Mac OS X Yosemite (10.10.5) for installing dependent libraries.
@@ -185,12 +186,12 @@ The following instructions are tested on Mac OS X Yosemite (10.10.5) for install
$ brew install -vd glog lmdb
By default, openblas is installed into /usr/local/opt/openblas. To let the compiler (and cmake) know the openblas
-path, please export
+path,
$ export CMAKE_INCLUDE_PATH=/usr/local/opt/openblas/include:$CMAKE_INCLUDE_PATH
$ export CMAKE_LIBRARY_PATH=/usr/local/opt/openblas/lib:$CMAKE_LIBRARY_PATH
-To let the runtime know the openblas path, please export
+To let the runtime know the openblas path,
$ export LD_LIBRARY_PATH=/usr/local/opt/openblas/library:$LD_LIBRARY_PATH
@@ -285,7 +286,7 @@ specify a 64-bit version project by:
$ cmake -G "Visual Studio 14 Win64" -DUSE_CUDA=OFF -DUSE_PYTHON=OFF ..
If you get error outputs like "Could NOT find xxxxx" indicating a dependent
-library missing, please configure your library file and include path for cmake or the system.
+library missing, configure your library file and include path for cmake or the system.
For example, you get an error "Could NOT find CBLAS" and suppose you installed
openblas header files at "d:\include" and openblas library at "d:\lib". You should run the
following command to specify your cblas parameters in cmake:
@@ -297,13 +298,13 @@ can run the following command:
$ cmake -LAH
-If you use cmake GUI tool in windows, please make sure you configure the right
+If you use cmake GUI tool in windows, make sure you configure the right
parameters for the singa solution by select "Advanced" box. After generating the VS project,
-please open the "singa.sln" project file under
+open the "singa.sln" project file under
the "build" folder and compile it as a normal VS solution. You will find the
unit tests file named "test_singa" in the project binary folder.
If you get errors when running test_singa.exe due to libglog.dll/libopenblas.dll missing,
-please just copy the dll files into the same folder as test_singa.exe
+just copy the dll files into the same folder as test_singa.exe
## Build the Debian packages
@@ -322,7 +323,7 @@ please just copy the dll files into the same folder as test_singa.exe
$ echo "export LD_LIBRARY_PATH=/home/<yourname>/local/cudnn/lib64:$LD_LIBRARY_PATH" >> ~/.bashrc
- 2. If it is the problem related to protobuf, then please download the newest whl files which have [compiled protobuf and openblas into the whl](https://issues.apache.org/jira/browse/SINGA-255) file of PySINGA.
+ 2. If it is the problem related to protobuf, then download the newest whl files which have [compiled protobuf and openblas into the whl](https://issues.apache.org/jira/browse/SINGA-255) file of PySINGA.
Or you can install protobuf from source into a local folder, say ~/local/;
Decompress the tar file, and then
@@ -331,7 +332,7 @@ please just copy the dll files into the same folder as test_singa.exe
$ echo "export LD_LIBRARY_PATH=/home/<yourname>/local/lib:$LD_LIBRARY_PATH" >> ~/.bashrc
$ source ~/.bashrc
- 3. If it cannot find other libs including python, then please create virtual env using pip or conda;
+ 3. If it cannot find other libs including python, then create virtual env using pip or conda;
and then install SINGA via
$ pip install --upgrade <url of singa wheel>
@@ -339,9 +340,9 @@ please just copy the dll files into the same folder as test_singa.exe
* Q: Error from running `cmake ..`, which cannot find the dependent libraries.
- A: If you haven't installed the libraries, please install them. If you installed
+ A: If you haven't installed the libraries, install them. If you installed
the libraries in a folder that is outside of the system folder, e.g. /usr/local,
- please export the following variables
+ you need to export the following variables
$ export CMAKE_INCLUDE_PATH=<path to your header file folder>
$ export CMAKE_LIBRARY_PATH=<path to your lib file folder>
@@ -430,3 +431,15 @@ please just copy the dll files into the same folder as test_singa.exe
A: Please install numpy and export the path of numpy header files as
$ export CPLUS_INCLUDE_PATH=`python -c "import numpy; print numpy.get_include()"`:$CPLUS_INCLUDE_PATH
+
+* Q: When I run PySINGA in Mac OS X, I got the error "Fatal Python error: PyThreadState_Get: no current thread Abort trap: 6"
+
+ A: This error happens typically when you have multiple version of Python on your system,
+ e.g, the one comes with the OS and the one installed by Homebrew. The Python linked by PySINGA must be the same as the Python interpreter.
+ You can check your interpreter by `which python` and check the Python linked by PySINGA via `otool -L <path to _singa_wrap.so>`.
+ To fix this error, compile SINGA with the correct version of Python.
+ In particular, if you build PySINGA from source, you need to specify the paths when invoking [cmake](http://stackoverflow.com/questions/15291500/i-have-2-versions-of-python-installed-but-cmake-is-using-older-version-how-do)
+
+ $ cmake -DPYTHON_LIBRARY=`python-config --prefix`/lib/libpython2.7.dylib -DPYTHON_INCLUDE_DIR=`python-config --prefix`/include/python2.7/ ..
+
+ If installed PySINGA from binary packages, e.g. debian or wheel, then you need to change the python interpreter, e.g., reset the $PATH to put the correct path of Python at the front position.
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/8621f58b/tool/jenkins/README.md
----------------------------------------------------------------------
diff --git a/tool/jenkins/README.md b/tool/jenkins/README.md
index e631d39..ffb4595 100644
--- a/tool/jenkins/README.md
+++ b/tool/jenkins/README.md
@@ -3,7 +3,7 @@
## Introduction
This documentation is to guide SINGA developers to setup Jenkins service.
-We use jenkins to support continuous integration.
+We use Jenkins to support continuous integration.
After each commit, we want to automatically compile and test SINGA
under different OS and settings.
Those built binaries need to be archived for users to download.
@@ -12,7 +12,7 @@ Those built binaries need to be archived for users to download.
[Jenkins Official Wiki](https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins)
The slave nodes for running different building environments are configured under 'Manage Jenkins'->'Manage nodes'.
-## Configure Jenkins Multi-configuration Project for Unit Testing and Package Generation
+## Configure Jenkins for Unit Testing and Binary Package Generation
Create a multi-configuration project and configure project as follows:
### Description
@@ -34,7 +34,7 @@ This job automatically pulls latest commits from Apache incubator-singa github r
### Configuration Matrix
* User-defined Axis - name ``lang`` values ``CPP CUDA``
- * Slave - name ``env`` Node/label ``tick all nodes``
+ * Slave - name ``env`` Node/label: tick available nodes
### Build
* Execute shell - command - ``bash -ex tool/jenkins/jenkins_wheel.sh $lang``
@@ -48,15 +48,15 @@ This job automatically pulls latest commits from Apache incubator-singa github r
* In `Manage Jenkins`-`Configure System`, configure the SSH for connecting to the remote public server and set the target folder location
* Source files - `build/python/dist/*.tar.gz`
* Remove prefix - `build/python/dist`
- * Remote directory - `wheel`
+ * Remote directory - `wheel/linux`
* Exec a command on the remote server to decompress the package and add a symlink to the latest build. E.g., on a Solaris server the command is
- cd <target_folder>/wheel && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
+ cd <target_folder>/wheel/linux && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
* The file links on the remote public server would be like
- wheel/32/84d56b7/ubuntu16.04-cpp/singa-1.0.1-py2-none-any.whl
- wheel/32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1-py2-none-any.whl
+ wheel/Linux/32/84d56b7/ubuntu16.04-cpp/singa-1.0.1-py2-none-any.whl
+ wheel/Linux/32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1-py2-none-any.whl
* Send build artifacts (Debian package) over SSH for wheel
* ../debian/build.sh packages the .deb file into $BUILD_ID.tar.gz. Inside the tar file,
@@ -74,6 +74,30 @@ This job automatically pulls latest commits from Apache incubator-singa github r
debian/32/84d56b7/ubuntu16.04-cpp/singa-1.0.1-py2-none-any.whl
debian/32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1-py2-none-any.whl
+### Notes for Mac OS X
+The Job configuration is the same as stated above except,
+* Machines (or docker containers) with Mac OS X should be configured as jenkin nodes; the following environmental variables should be exported in the .bashrc or in the node configuration
+
+ # 10.11 or 10.12
+ export OS_VERSION=macosx10.11
+ # if using Homebrew
+ export PATH=/usr/local/bin:$PATH
+ # for numpy.i
+ export CPLUS_INCLUDE_PATH=`python -c "import numpy; print numpy.get_include()"`:$CPLUS_INCLUDE_PATH
+
+* Set User-defined Axis to None and tick only Mac OS X nodes
+
+* Send build artifacts (wheel) over SSH
+ * The remote directly -- `wheel/macosx`
+ * The shell command is
+
+ cd <target_folder>/wheel/macosx && gunzip $BUILD_ID.tar.gz && tar xf $BUILD_ID.tar && chmod -R 755 $BUILD_ID && /bin/rm -f $BUILD_ID.tar && /bin/rm -f latest && ln -s $BUILD_ID/* latest
+### Docker Images
+We provide in `docker` a number of singa docker images for Jenkins to use as slaves.
+To run the docker images,
+
+ nvidia-docker run --name <jenkins-slaveXX> -d <Image ID>
+
## Configure Jenkins for SINGA Website Updates
### Description and Configuration
@@ -110,11 +134,5 @@ To start the slave node
# change any file (add spaces) inside trunk/ to commit a message
$ svn commit -m "test" --username <committer id> --password <passwd>
-## Docker Images
-We provide in `docker` a number of singa docker images for Jenkins to use as slaves.
-To run the docker images,
-
- nvidia-docker run --name <jenkins-slaveXX> -d <Image ID>
-
## Access Control
Use `Role Strategy Plugin` to give read access for anonymous users.
http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/8621f58b/tool/jenkins/jenkins_wheel.sh
----------------------------------------------------------------------
diff --git a/tool/jenkins/jenkins_wheel.sh b/tool/jenkins/jenkins_wheel.sh
index 1c8af27..3f3f0b5 100644
--- a/tool/jenkins/jenkins_wheel.sh
+++ b/tool/jenkins/jenkins_wheel.sh
@@ -45,9 +45,13 @@ echo wheel file folder: build/python/dist/whl/$FOLDER
rm -rf build
mkdir build
+if [ `uname` = "Darwin" ]; then
+ EXTRA_ARGS="-DPYTHON_LIBRARY=`python-config --prefix`/lib/libpython2.7.dylib -DPYTHON_INCLUDE_DIR=`python-config --prefix`/include/python2.7/"
+fi
+
# compile singa c++
cd build
-cmake -DUSE_CUDNN=$CUDNN -DUSE_CUDA=$CUDA -DUSE_MODULES=ON -DUSE_MODULES=ON ../
+cmake -DUSE_CUDNN=$CUDNN -DUSE_CUDA=$CUDA -DUSE_MODULES=ON $EXTRA_ARGS ../
make
# unit test cpp code
./bin/test_singa --gtest_output=xml:./gtest.xml