You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/03 07:55:54 UTC

[GitHub] [arrow] Atharex opened a new issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Atharex opened a new issue #10222:
URL: https://github.com/apache/arrow/issues/10222


   Hi!
   
   I have been trying to compile apache arrow 1.0.1 with CUDA support to use with RAPIDS and I am getting stuck it seems because of some boost dependencies.
   
   These are the commands I run in a docker container that has CUDA libraries preinstalled:
   
   ```
   ## Apache Arrow repo  (instructions from https://randyzwitch.com/pyarrow-cuda-support/)
   export ARROW_HOME=/root/home/arrow
   git clone https://github.com/apache/arrow.git $ARROW_HOME
   cd $ARROW_HOME
   git submodule update --init
   git checkout apache-arrow-1.0.1
   mkdir -p $ARROW_HOME/cpp/build && cd $ARROW_HOME/cpp/build
   
   ## Apache Arrow build dependencies
   bash $ARROW_HOME/cpp/thirdparty/download_dependencies.sh
   
   ## Apache Arrow C++ build
   cmake -DCMAKE_INSTALL_PREFIX=/usr/local/lib/arrow \
   -DCMAKE_INSTALL_LIBDIR=lib \
   -DARROW_FLIGHT=ON \
   -DARROW_ORC=ON \
   -DARROW_WITH_BZ2=ON \
   -DARROW_WITH_ZLIB=ON \
   -DARROW_WITH_ZSTD=ON \
   -DARROW_WITH_LZ4=ON \
   -DARROW_WITH_SNAPPY=ON \
   -DARROW_WITH_BROTLI=ON \
   -DARROW_PARQUET=ON \
   -DARROW_PYTHON=ON \
   -DARROW_PLASMA=ON \
   -DARROW_CUDA=ON \
   ..
   make -j 24
   
   ```
   
   The make command fails quickly with this output:
   
   ```
   CMake Error at /root/home/arrow/cpp/build/boost_ep-prefix/src/boost_ep-stamp/boost_ep-configure-RELEASE.cmake:37 (message):
     Command failed: 1
   
      './bootstrap.sh' '--prefix=/root/home/arrow/cpp/build/boost_ep-prefix/src/boost_ep' '--with-libraries=filesystem,regex,system'
   
     See also
   
       /root/home/arrow/cpp/build/boost_ep-prefix/src/boost_ep-stamp/boost_ep-configure-*.log
   
   
   -- stdout output is:
   Building Boost.Build engine with toolset ... 
   Failed to build Boost.Build build engine
   Consult 'bootstrap.log' for more details
   
   -- stderr output is:
   ./bootstrap.sh: line 196: ./tools/build/src/engine/build.sh: No such file or directory
   
   CMake Error at /root/home/arrow/cpp/build/boost_ep-prefix/src/boost_ep-stamp/boost_ep-configure-RELEASE.cmake:47 (message):
     Stopping after outputting logs.
   ```
   
   Am I missing an installation step here?
   
   
   When I run the download_dependencies.sh script, it seems it fails. Could that be the reason for this missing build.sh script?
   ```
   
   bash $ARROW_HOME/cpp/thirdparty/download_dependencies.sh
   # Environment variables for offline Arrow build
   export ARROW_ABSL_URL=/root/home/arrow/cpp/build/absl-2eba343b51e0923cd3fb919a6abd6120590fc059.tar.gz
   export ARROW_AWSSDK_URL=/root/home/arrow/cpp/build/aws-sdk-cpp-1.7.160.tar.gz
   Failed downloading https://dl.bintray.com/ursalabs/arrow-boost/boost_1_71_0.tar.gz
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] jorisvandenbossche commented on issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on issue #10222:
URL: https://github.com/apache/arrow/issues/10222#issuecomment-831161148


   The bintray url has been retired, and thus the `download_dependencies.sh` script as included in the older Arrow 1.0.1 release no longer works out of the box. 
   
   You can update the URL to download from Sourceforge or https://github.com/ursa-labs/thirdparty instead, see the changes in https://github.com/apache/arrow/pull/9483


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Atharex commented on issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
Atharex commented on issue #10222:
URL: https://github.com/apache/arrow/issues/10222#issuecomment-832492786


   Also the wheel that is built in the end seems to have some troubles during import. 
   
   ```
   bash-4.2# pip3 install $ARROW_HOME/python/dist/pyarrow*.whl
   Processing ./dist/pyarrow-1.0.1.dev0+g886d87bde.d20210505-cp38-cp38-linux_x86_64.whl
   Requirement already satisfied: numpy>=1.14 in /usr/local/lib/python3.8/site-packages (from pyarrow==1.0.1.dev0+g886d87bde.d20210505) (1.19.5)
   Installing collected packages: pyarrow
   Successfully installed pyarrow-1.0.1.dev0+g886d87bde.d20210505
   WARNING: You are using pip version 20.2.4; however, version 21.1.1 is available.
   You should consider upgrading via the '/usr/local/bin/python3.8 -m pip install --upgrade pip' command.
   bash-4.2# python3
   Python 3.8.9 (default, May  3 2021, 14:20:19) 
   [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
   Type "help", "copyright", "credits" or "license" for more information.
   >>> import pyarrow
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "/root/home/arrow/python/pyarrow/__init__.py", line 62, in <module>
       import pyarrow.lib as _lib
   ModuleNotFoundError: No module named 'pyarrow.lib'
   ```
   
   Though the package itself seems correctly installed, as the `lib.pyx` file is present
   ```
   bash-4.2# ls /root/home/arrow/python/pyarrow/
   __init__.pxd  _csv.pxd      _flight.pyx            _json.pyx     _plasma.pyx    builder.pxi  config.pxi  feather.pxi    gandiva.pyx  ipc.pxi  lib.pyx           parquet.py         serialization.py  types.pxi
   __init__.py   _csv.pyx      _fs.pxd                _orc.pxd      _s3fs.pyx      cffi.py      csv.py      feather.py     hdfs.py      ipc.py   memory.pxi        plasma.py          table.pxi         types.py
   __pycache__   _cuda.pxd     _fs.pyx                _orc.pyx      array.pxi      compat.pxi   cuda.py     filesystem.py  includes     json.py  orc.py            public-api.pxi     tensor.pxi        util.py
   _compute.pxd  _cuda.pyx     _generated_version.py  _parquet.pxd  benchmark.pxi  compat.py    dataset.py  flight.py      io-hdfs.pxi  jvm.py   pandas-shim.pxi   scalar.pxi         tensorflow
   _compute.pyx  _dataset.pyx  _hdfs.pyx              _parquet.pyx  benchmark.py   compute.py   error.pxi   fs.py          io.pxi       lib.pxd  pandas_compat.py  serialization.pxi  tests
   ```
   
   Am I missing something in the build steps for the python wheel here?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] edponce commented on issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
edponce commented on issue #10222:
URL: https://github.com/apache/arrow/issues/10222#issuecomment-853375339


   @Atharex The error usually occurs if you run the Python interpreter from `ARROW_ROOT/python` directory because the `import pyarrow` statement will try to use the local directory of `ARROW_ROOT/python/pyarrow` path instead of the installed `pyarrow` wheel.
   
   Also, I noticed from the previous threads that ARROW_HOME is used somewhat incorrectly. ARROW_HOME should point to the installation directory of Arrow C++. Use ARROW_ROOT for the top directory of the repo. Putting all of this together, here are steps you can follow to successfully build/install Arrow 1.0.1 system-wide:
   
   ```
   export ARROW_HOME=/usr/local/lib/arrow
   export ARROW_ROOT=/root/home/arrow
   
   # Download Apache Arrow repo
   git clone https://github.com/apache/arrow.git "$ARROW_ROOT"
   cd "$ARROW_ROOT"
   git checkout apache-arrow-1.0.1
   
   # (Optional) Update submodules for unit tests
   git submodule update --init
   export PARQUET_TEST_DATA="$ARROW_ROOT/cpp/submodules/parquet-testing/data"
   export ARROW_TEST_DATA="$ARROW_ROOT/testing/data"
   
   # Download Arrow C++ third party build dependencies
   cd "$ARROW_ROOT/cpp/thirdparty"
   
   # Fix Boost URL issue
   sed -i 's@dl.bintray.com/ursalabs/arrow-boost@github.com/ursa-labs/thirdparty/releases/download/latest@' versions.txt
   
   bash download_dependencies.sh > environ.sh
   source environ.sh
   
   # Apache Arrow C++ build
   export ARROW_CPP_BUILD="$ARROW_ROOT/cpp/build"
   mkdir -p "$ARROW_CPP_BUILD"
   cd "$ARROW_CPP_BUILD"
   cmake \
       -DCMAKE_BUILD_TYPE=release \
       -DCMAKE_INSTALL_PREFIX="$ARROW_HOME" \
       -DCMAKE_INSTALL_LIBDIR=lib \
       -DARROW_FLIGHT=ON \
       -DARROW_ORC=ON \
       -DARROW_WITH_BZ2=ON \
       -DARROW_WITH_ZLIB=ON \
       -DARROW_WITH_ZSTD=ON \
       -DARROW_WITH_LZ4=ON \
       -DARROW_WITH_SNAPPY=ON \
       -DARROW_WITH_BROTLI=ON \
       -DARROW_PARQUET=ON \
       -DARROW_PYTHON=ON \
       -DARROW_PLASMA=ON \
       -DARROW_CUDA=ON \
       -DARROW_DEPENDENCY_SOURCE=BUNDLED \
       ..
   make -j 24
   make install
   
   # Set up Python requirements and environment
   cd "$ARROW_ROOT/python"
   export PYARROW_BUILD_TYPE=release
   export PYARROW_WITH_FLIGHT=1
   export PYARROW_WITH_ORC=1
   export PYARROW_WITH_PARQUET=1
   export PYARROW_WITH_PLASMA=1
   export PYARROW_WITH_CUDA=1
   pip3 install -r requirements-wheel-build.txt
   python3 setup.py build_ext -j 24 --bundle-arrow-cpp bdist_wheel
   pip3 install "$ARROW_ROOT"/python/dist/pyarrow*.whl
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Atharex closed issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
Atharex closed issue #10222:
URL: https://github.com/apache/arrow/issues/10222


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] edponce commented on issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
edponce commented on issue #10222:
URL: https://github.com/apache/arrow/issues/10222#issuecomment-853375339


   @Atharex The error usually occurs if you run the Python interpreter from `ARROW_ROOT/python` directory because the `import pyarrow` statement will try to use the local directory of `ARROW_ROOT/python/pyarrow` path instead of the installed `pyarrow` wheel.
   
   Also, I noticed from the previous threads that ARROW_HOME is used somewhat incorrectly. ARROW_HOME should point to the installation directory of Arrow C++. Use ARROW_ROOT for the top directory of the repo. Putting all of this together, here are steps you can follow to successfully build/install Arrow 1.0.1 system-wide:
   
   ```
   export ARROW_HOME=/usr/local/lib/arrow
   export ARROW_ROOT=/root/home/arrow
   
   # Download Apache Arrow repo
   git clone https://github.com/apache/arrow.git "$ARROW_ROOT"
   cd "$ARROW_ROOT"
   git checkout apache-arrow-1.0.1
   
   # (Optional) Update submodules for unit tests
   git submodule update --init
   export PARQUET_TEST_DATA="$ARROW_ROOT/cpp/submodules/parquet-testing/data"
   export ARROW_TEST_DATA="$ARROW_ROOT/testing/data"
   
   # Download Arrow C++ third party build dependencies
   cd "$ARROW_ROOT/cpp/thirdparty"
   
   # Fix Boost URL issue
   sed -i 's@dl.bintray.com/ursalabs/arrow-boost@github.com/ursa-labs/thirdparty/releases/download/latest@' versions.txt
   
   bash download_dependencies.sh > environ.sh
   source environ.sh
   
   # Apache Arrow C++ build
   export ARROW_CPP_BUILD="$ARROW_ROOT/cpp/build"
   mkdir -p "$ARROW_CPP_BUILD"
   cd "$ARROW_CPP_BUILD"
   cmake \
       -DCMAKE_BUILD_TYPE=release \
       -DCMAKE_INSTALL_PREFIX="$ARROW_HOME" \
       -DCMAKE_INSTALL_LIBDIR=lib \
       -DARROW_FLIGHT=ON \
       -DARROW_ORC=ON \
       -DARROW_WITH_BZ2=ON \
       -DARROW_WITH_ZLIB=ON \
       -DARROW_WITH_ZSTD=ON \
       -DARROW_WITH_LZ4=ON \
       -DARROW_WITH_SNAPPY=ON \
       -DARROW_WITH_BROTLI=ON \
       -DARROW_PARQUET=ON \
       -DARROW_PYTHON=ON \
       -DARROW_PLASMA=ON \
       -DARROW_CUDA=ON \
       -DARROW_DEPENDENCY_SOURCE=BUNDLED \
       ..
   make -j 24
   make install
   
   # Set up Python requirements and environment
   cd "$ARROW_ROOT/python"
   export PYARROW_BUILD_TYPE=release
   export PYARROW_WITH_FLIGHT=1
   export PYARROW_WITH_ORC=1
   export PYARROW_WITH_PARQUET=1
   export PYARROW_WITH_PLASMA=1
   export PYARROW_WITH_CUDA=1
   pip3 install -r requirements-wheel-build.txt
   python3 setup.py build_ext -j 24 --bundle-arrow-cpp bdist_wheel
   pip3 install "$ARROW_ROOT"/python/dist/pyarrow*.whl
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Atharex commented on issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
Atharex commented on issue #10222:
URL: https://github.com/apache/arrow/issues/10222#issuecomment-831242667


   Thanks @jorisvandenbossche I've manually set the download links to the ursa-labs/thirdparty repo and then the downloads completed successfully. However I still encounter the same error as before...
   
   It seems the build script of boost is the one failing, as I get the same error if I extract the boost from the new thirdparty repo and try to build it separately by hand.
   
   ```
   bash-4.2# bash bootstrap.sh
   bootstrap.sh: line 196: ./tools/build/src/engine/build.sh: No such file or directory
   Building Boost.Build engine with toolset ... 
   Failed to build Boost.Build build engine
   Consult 'bootstrap.log' for more details
   ```
   
   Are there any additional steps I need to do besides the ones I wrote in my first post?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Atharex commented on issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
Atharex commented on issue #10222:
URL: https://github.com/apache/arrow/issues/10222#issuecomment-832486934


   Thanks @edponce this solved the c++ code building issues! 
   My mistake was also running download_dependencies.sh through the bash interpreter, so the exports were never saved in the current session.
   
   I did notice also that if I run cmake with `-DCMAKE_INSTALL_PREFIX=/usr/local/lib/arrow`
   Then I cannot later build the python wheel, as it complains it cannot find ARROW_INCLUDE_DIR and ARROW_LIB_DIR
   
   ```
   cd $ARROW_HOME/python
   export PYARROW_WITH_PARQUET=1
   export PYARROW_WITH_CUDA=1
   python3 setup.py build_ext -j 24 --build-type=release  --bundle-arrow-cpp bdist_wheel
   
   ```
   
   ```
   Call Stack (most recent call first):
     /opt/cmake-3.20.1-linux-x86_64/share/cmake-3.20/Modules/FindPkgConfig.cmake:70 (find_package_handle_standard_args)
     cmake_modules/FindArrow.cmake:39 (include)
     cmake_modules/FindArrowPython.cmake:46 (find_package)
     CMakeLists.txt:210 (find_package)
   This warning is for project developers.  Use -Wno-dev to suppress it.
   
   -- Found PkgConfig: /usr/bin/pkg-config (found version "0.27.1") 
   CMake Error at /opt/cmake-3.20.1-linux-x86_64/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
     Could NOT find Arrow (missing: ARROW_INCLUDE_DIR ARROW_LIB_DIR
     ARROW_FULL_SO_VERSION ARROW_SO_VERSION)
   Call Stack (most recent call first):
     /opt/cmake-3.20.1-linux-x86_64/share/cmake-3.20/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
     cmake_modules/FindArrow.cmake:412 (find_package_handle_standard_args)
     cmake_modules/FindArrowPython.cmake:46 (find_package)
     CMakeLists.txt:210 (find_package)
   ```
   
   
   
   When I use `-DCMAKE_INSTALL_PREFIX=$ARROW_HOME` then the python wheel build finds all those variables and works normally. Any ideas why that would be the case?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] edponce commented on issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
edponce commented on issue #10222:
URL: https://github.com/apache/arrow/issues/10222#issuecomment-831458147


   The [Ursa Labs thirdparty project](https://github.com/ursa-labs/thirdparty) has been updated so you can use `https://github.com/ursa-labs/thirdparty/releases/download/latest` for the `ARROW_BOOST_URL` in `cpp/thirdparty/versions.txt`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] edponce commented on issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
edponce commented on issue #10222:
URL: https://github.com/apache/arrow/issues/10222#issuecomment-831363193


   @Atharex Besides downloading the dependency packages (e.g., Boost) you can also force cmake to use the downloaded (aka bundled) packages and ignore system-wide versions.
   
   I am able to do an offline build of Arrow 1.0.1 via the following steps:
   1. Given that bintray has been retired, in `cpp/thirdparty/versions.txt` change the Boost URL from `https://dl.bintray.com/ursalabs/arrow-boost` to `https://github.com/ursa-labs/thirdparty/releases/download/27apr2021`.
   2. In cmake command, specify the package dependency resolution to use the bundled version, refer to https://arrow.apache.org/docs/developers/cpp/building.html#individual-dependency-resolution. If you want to use all the downloaded packages, add `-DARROW_DEPENDENCY_SOURCE=BUNDLED`, and if you want only the Boost library package, use only `-DBOOST_SOURCE=BUNDLED`.
   3. From your initial build, use the following commands
   
   ```
   ## Apache Arrow build dependencies and source corresponding environment variables
   bash $ARROW_HOME/cpp/thirdparty/download_dependencies.sh > environ.sh
   source environ.sh
   
   ## Apache Arrow C++ build
   cmake -DCMAKE_INSTALL_PREFIX=/usr/local/lib/arrow \
   -DCMAKE_INSTALL_LIBDIR=lib \
   -DARROW_FLIGHT=ON \
   -DARROW_ORC=ON \
   -DARROW_WITH_BZ2=ON \
   -DARROW_WITH_ZLIB=ON \
   -DARROW_WITH_ZSTD=ON \
   -DARROW_WITH_LZ4=ON \
   -DARROW_WITH_SNAPPY=ON \
   -DARROW_WITH_BROTLI=ON \
   -DARROW_PARQUET=ON \
   -DARROW_PYTHON=ON \
   -DARROW_PLASMA=ON \
   -DARROW_CUDA=ON \
   -DARROW_DEPENDENCY_SOURCE=BUNDLED \
   ..
   make -j 24
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [arrow] Atharex commented on issue #10222: Arrow CUDA build v1.0.1 from source missing build.sh

Posted by GitBox <gi...@apache.org>.
Atharex commented on issue #10222:
URL: https://github.com/apache/arrow/issues/10222#issuecomment-855686678


   Ah true, the import was failing because I was testing it from the ARROW_ROOT/python directory. Works fine now!
   
   Also, thanks for the updated steps @edponce 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org