You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Andrew Palumbo <ap...@outlook.com> on 2019/02/15 00:48:20 UTC

building py-arrow with CUDA

Hello,
I've been trying to get py-arrow built with CUDA support, I've had help from Wes and Perau on user@, and it seems that the docs for building with CUDA are out of date; Wes suggested That I try here.

I have a sript:

conda create -n pyarrow-dev
conda activate pyarrow-dev
conda install python numpy six setuptools cython pandas pytest \
      cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
      gflags brotli jemalloc lz4-c zstd \
      double-conversion glog autoconf hypothesis numba \
      clangdev=6 flake8 gtest gmock \
      -c conda-forge

git clone https://github.com/arrow/arrow.git

conda activate pyarrow-dev
cd arrow
export ARROW_BUILD_TYPE=release
export ARROW_BUILD_TOOLCHAIN=$CONDA_PREFIX
export ARROW_HOME=$CONDA_PREFIX
export PARQUET_HOME=$CONDA_PREFIX
export NUMBAPRO_LIBDEVICE=/usr/local/cuda-9.0/nvvm/libdevice
export NUMBAPRO_NVVM=/usr/local/cuda-9.0/nvvm/lib64/libnvvm.so

cd cpp

cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
      -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
      -DARROW_PARQUET=off  -DARROW_PYTHON=on  \
      -DARROW_PLASMA=off -DARROW_BUILD_TESTS=OFF \
      -DARROW_CUDA=on \
      -DCLANG_FORMAT_BIN=`which clang-format` \
      .
make -j3
make install
cd ../python
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-cuda develop
py.test -sv pyarrow/


which is a slightly modified script given to me by Perau (I'd earlier been trying to build strictly with cmake)

I'm working on the Amazon Deep Learning AMI:
conda 4.6.4
Clang 6.0.1 (need to upgrade this)
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11)
cmake version 3.13.2
GNU Make 4.1
Python 3.6.7
CUDA 9.0


I can build without py-arrow, but my needs involve CUDA support.

Any help would be appreciated.

Thanks in advance,

Andy

Re: building py-arrow with CUDA

Posted by Wes McKinney <we...@gmail.com>.
hi Andrew,

At the top of your script:

./run_conda.sh: 1: ./run_conda.sh: conda: not found
./run_conda.sh: 2: ./run_conda.sh: conda: not found
./run_conda.sh: 3: ./run_conda.sh: conda: not found
fatal: destination path 'arrow' already exists and is not an empty directory.
./run_conda.sh: 12: ./run_conda.sh: conda: not found

these issues are fatal.  I suggest adding `set -ex` so the Pearu's
build script that you are using will error and exit on the first
failure of a command.

It seems that "conda" is not available in the script, do you have the
appropriate shebang (e.g. "#!/usr/bin/env bash") in the script so it
picks up your bash environment including conda initialization?

- Wes


On Thu, Feb 14, 2019 at 6:59 PM Andrew Palumbo <ap...@outlook.com> wrote:
>
> sorry forgot to include the gist:
>
> https://gist.github.com/andrewpalumbo/d85d57063e58ae81134426ca640aded9
> [https://avatars2.githubusercontent.com/u/7681565?s=400&v=4]<https://gist.github.com/andrewpalumbo/d85d57063e58ae81134426ca640aded9>
>
> conda build of py-arrow with CUDA support<https://gist.github.com/andrewpalumbo/d85d57063e58ae81134426ca640aded9>
> conda build of py-arrow with CUDA support. GitHub Gist: instantly share code, notes, and snippets.
> gist.github.com
>
>
>
> Thanks very much,
>
> Andy
> ________________________________
> From: Andrew Palumbo <ap...@outlook.com>
> Sent: Thursday, February 14, 2019 7:48 PM
> To: dev@arrow.apache.org
> Subject: building py-arrow with CUDA
>
> Hello,
> I've been trying to get py-arrow built with CUDA support, I've had help from Wes and Perau on user@, and it seems that the docs for building with CUDA are out of date; Wes suggested That I try here.
>
> I have a sript:
>
> conda create -n pyarrow-dev
> conda activate pyarrow-dev
> conda install python numpy six setuptools cython pandas pytest \
>       cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
>       gflags brotli jemalloc lz4-c zstd \
>       double-conversion glog autoconf hypothesis numba \
>       clangdev=6 flake8 gtest gmock \
>       -c conda-forge
>
> git clone https://github.com/arrow/arrow.git
>
> conda activate pyarrow-dev
> cd arrow
> export ARROW_BUILD_TYPE=release
> export ARROW_BUILD_TOOLCHAIN=$CONDA_PREFIX
> export ARROW_HOME=$CONDA_PREFIX
> export PARQUET_HOME=$CONDA_PREFIX
> export NUMBAPRO_LIBDEVICE=/usr/local/cuda-9.0/nvvm/libdevice
> export NUMBAPRO_NVVM=/usr/local/cuda-9.0/nvvm/lib64/libnvvm.so
>
> cd cpp
>
> cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
>       -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
>       -DARROW_PARQUET=off  -DARROW_PYTHON=on  \
>       -DARROW_PLASMA=off -DARROW_BUILD_TESTS=OFF \
>       -DARROW_CUDA=on \
>       -DCLANG_FORMAT_BIN=`which clang-format` \
>       .
> make -j3
> make install
> cd ../python
> python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-cuda develop
> py.test -sv pyarrow/
>
>
> which is a slightly modified script given to me by Perau (I'd earlier been trying to build strictly with cmake)
>
> I'm working on the Amazon Deep Learning AMI:
> conda 4.6.4
> Clang 6.0.1 (need to upgrade this)
> gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11)
> cmake version 3.13.2
> GNU Make 4.1
> Python 3.6.7
> CUDA 9.0
>
>
> I can build without py-arrow, but my needs involve CUDA support.
>
> Any help would be appreciated.
>
> Thanks in advance,
>
> Andy
>
>

Re: building py-arrow with CUDA

Posted by Andrew Palumbo <ap...@outlook.com>.
sorry forgot to include the gist:

https://gist.github.com/andrewpalumbo/d85d57063e58ae81134426ca640aded9
[https://avatars2.githubusercontent.com/u/7681565?s=400&v=4]<https://gist.github.com/andrewpalumbo/d85d57063e58ae81134426ca640aded9>

conda build of py-arrow with CUDA support<https://gist.github.com/andrewpalumbo/d85d57063e58ae81134426ca640aded9>
conda build of py-arrow with CUDA support. GitHub Gist: instantly share code, notes, and snippets.
gist.github.com



Thanks very much,

Andy
________________________________
From: Andrew Palumbo <ap...@outlook.com>
Sent: Thursday, February 14, 2019 7:48 PM
To: dev@arrow.apache.org
Subject: building py-arrow with CUDA

Hello,
I've been trying to get py-arrow built with CUDA support, I've had help from Wes and Perau on user@, and it seems that the docs for building with CUDA are out of date; Wes suggested That I try here.

I have a sript:

conda create -n pyarrow-dev
conda activate pyarrow-dev
conda install python numpy six setuptools cython pandas pytest \
      cmake flatbuffers rapidjson boost-cpp thrift-cpp snappy zlib \
      gflags brotli jemalloc lz4-c zstd \
      double-conversion glog autoconf hypothesis numba \
      clangdev=6 flake8 gtest gmock \
      -c conda-forge

git clone https://github.com/arrow/arrow.git

conda activate pyarrow-dev
cd arrow
export ARROW_BUILD_TYPE=release
export ARROW_BUILD_TOOLCHAIN=$CONDA_PREFIX
export ARROW_HOME=$CONDA_PREFIX
export PARQUET_HOME=$CONDA_PREFIX
export NUMBAPRO_LIBDEVICE=/usr/local/cuda-9.0/nvvm/libdevice
export NUMBAPRO_NVVM=/usr/local/cuda-9.0/nvvm/lib64/libnvvm.so

cd cpp

cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
      -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
      -DARROW_PARQUET=off  -DARROW_PYTHON=on  \
      -DARROW_PLASMA=off -DARROW_BUILD_TESTS=OFF \
      -DARROW_CUDA=on \
      -DCLANG_FORMAT_BIN=`which clang-format` \
      .
make -j3
make install
cd ../python
python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-cuda develop
py.test -sv pyarrow/


which is a slightly modified script given to me by Perau (I'd earlier been trying to build strictly with cmake)

I'm working on the Amazon Deep Learning AMI:
conda 4.6.4
Clang 6.0.1 (need to upgrade this)
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.11)
cmake version 3.13.2
GNU Make 4.1
Python 3.6.7
CUDA 9.0


I can build without py-arrow, but my needs involve CUDA support.

Any help would be appreciated.

Thanks in advance,

Andy