You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by we...@apache.org on 2017/04/19 19:49:07 UTC
arrow git commit: ARROW-848: [Python] Another pass on conda dev guide
Repository: arrow
Updated Branches:
refs/heads/master 41a8ff9ad -> 391242a17
ARROW-848: [Python] Another pass on conda dev guide
Per feedback in https://github.com/apache/arrow/commit/bb8514cc9d7068c8b62d346577370751d68221d8
Author: Wes McKinney <we...@twosigma.com>
Closes #562 from wesm/conda-quickstart-iterate and squashes the following commits:
881a44d [Wes McKinney] Add system requirements notes about gcc 4.9, use boost shared libs
8c95705 [Wes McKinney] Install cmake in conda env
8c2885e [Wes McKinney] Another pass on conda dev guide, do not require LD_LIBRARY_PATH. Install everything in a single conda environment
Project: http://git-wip-us.apache.org/repos/asf/arrow/repo
Commit: http://git-wip-us.apache.org/repos/asf/arrow/commit/391242a1
Tree: http://git-wip-us.apache.org/repos/asf/arrow/tree/391242a1
Diff: http://git-wip-us.apache.org/repos/asf/arrow/diff/391242a1
Branch: refs/heads/master
Commit: 391242a17d5bdb041b7b1b036b48e69e82ec29b8
Parents: 41a8ff9
Author: Wes McKinney <we...@twosigma.com>
Authored: Wed Apr 19 15:49:00 2017 -0400
Committer: Wes McKinney <we...@twosigma.com>
Committed: Wed Apr 19 15:49:00 2017 -0400
----------------------------------------------------------------------
python/DEVELOPMENT.md | 73 ++++++++++++++++++++++++++++++----------------
python/setup.py | 2 +-
2 files changed, 49 insertions(+), 26 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/arrow/blob/391242a1/python/DEVELOPMENT.md
----------------------------------------------------------------------
diff --git a/python/DEVELOPMENT.md b/python/DEVELOPMENT.md
index ca74462..7f08169 100644
--- a/python/DEVELOPMENT.md
+++ b/python/DEVELOPMENT.md
@@ -16,36 +16,41 @@
### Linux and macOS
-First, set up your thirdparty C++ toolchain using libraries from conda-forge:
+#### System Requirements
+
+On macOS, any modern XCode (6.4 or higher; the current version is 8.3.1) is
+sufficient.
+
+On Linux, for this guide, we recommend using gcc 4.8 or 4.9, or clang 3.7 or
+higher. You can check your version by running
```shell
-conda config --add channels conda-forge
+$ gcc --version
+```
-export ARROW_BUILD_TYPE=Release
+On Ubuntu 16.04 and higher, you can obtain gcc 4.9 with:
-export CPP_TOOLCHAIN=$HOME/cpp-toolchain
-export LD_LIBRARY_PATH=$CPP_TOOLCHAIN/lib:$LD_LIBRARY_PATH
+```shell
+$ sudo apt-get install g++-4.9
+```
-export BOOST_ROOT=$CPP_TOOLCHAIN
-export FLATBUFFERS_HOME=$CPP_TOOLCHAIN
-export RAPIDJSON_HOME=$CPP_TOOLCHAIN
-export THRIFT_HOME=$CPP_TOOLCHAIN
-export ZLIB_HOME=$CPP_TOOLCHAIN
-export SNAPPY_HOME=$CPP_TOOLCHAIN
-export BROTLI_HOME=$CPP_TOOLCHAIN
-export JEMALLOC_HOME=$CPP_TOOLCHAIN
-export ARROW_HOME=$CPP_TOOLCHAIN
-export PARQUET_HOME=$CPP_TOOLCHAIN
+Finally, set gcc 4.9 as the active compiler using:
-conda create -y -q -p $CPP_TOOLCHAIN \
- flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib brotli jemalloc
+```shell
+export CC=gcc-4.9
+export CXX=g++-4.9
```
-Now, activate a conda environment containing your target Python version and
-NumPy installed:
+#### Environment Setup and Build
+
+First, let's create a conda environment with all the C++ build and Python
+dependencies from conda-forge:
```shell
-conda create -y -q -n pyarrow-dev python=3.6 numpy
+conda create -y -q -n pyarrow-dev \
+ python=3.6 numpy six setuptools cython pandas pytest \
+ cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
+ brotli jemalloc -c conda-forge
source activate pyarrow-dev
```
@@ -67,6 +72,26 @@ drwxrwxr-x 12 wesm wesm 4096 Apr 15 19:19 arrow/
drwxrwxr-x 12 wesm wesm 4096 Apr 15 19:19 parquet-cpp/
```
+We need to set a number of environment variables to let Arrow's build system
+know about our build toolchain:
+
+```
+export ARROW_BUILD_TYPE=release
+
+export BOOST_ROOT=$CONDA_PREFIX
+export BOOST_LIBRARYDIR=$CONDA_PREFIX/lib
+
+export FLATBUFFERS_HOME=$CONDA_PREFIX
+export RAPIDJSON_HOME=$CONDA_PREFIX
+export THRIFT_HOME=$CONDA_PREFIX
+export ZLIB_HOME=$CONDA_PREFIX
+export SNAPPY_HOME=$CONDA_PREFIX
+export BROTLI_HOME=$CONDA_PREFIX
+export JEMALLOC_HOME=$CONDA_PREFIX
+export ARROW_HOME=$CONDA_PREFIX
+export PARQUET_HOME=$CONDA_PREFIX
+```
+
Now build and install the Arrow C++ libraries:
```shell
@@ -74,7 +99,7 @@ mkdir arrow/cpp/build
pushd arrow/cpp/build
cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
- -DCMAKE_INSTALL_PREFIX=$CPP_TOOLCHAIN \
+ -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
-DARROW_PYTHON=on \
-DARROW_BUILD_TESTS=OFF \
..
@@ -90,7 +115,7 @@ mkdir parquet-cpp/build
pushd parquet-cpp/build
cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
- -DCMAKE_INSTALL_PREFIX=$CPP_TOOLCHAIN \
+ -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX \
-DPARQUET_BUILD_BENCHMARKS=off \
-DPARQUET_BUILD_EXECUTABLES=off \
-DPARQUET_ZLIB_VENDORED=off \
@@ -102,11 +127,9 @@ make install
popd
```
-Now, install requisite build requirements for pyarrow, then build:
+Now, build pyarrow:
```shell
-conda install -y -q six setuptools cython pandas pytest
-
cd arrow/python
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet --inplace
```
http://git-wip-us.apache.org/repos/asf/arrow/blob/391242a1/python/setup.py
----------------------------------------------------------------------
diff --git a/python/setup.py b/python/setup.py
index ab71e78..1c46617 100644
--- a/python/setup.py
+++ b/python/setup.py
@@ -155,7 +155,7 @@ class build_ext(_build_ext):
cmake_options.append('-DPYARROW_BUNDLE_ARROW_CPP=ON')
cmake_options.append('-DCMAKE_BUILD_TYPE={0}'
- .format(self.build_type))
+ .format(self.build_type.lower()))
if sys.platform != 'win32':
cmake_command = (['cmake', self.extra_cmake_args] +