You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Vibhatha Abeykoon <vi...@gmail.com> on 2020/05/01 02:30:48 UTC
Pyarrow building from source along with CPP Libraries to link to
another Cython API
Hi,
I am trying to integrate Arrow with an application that I am developing.
Here I build Arrow from the source (CPP) and use the API to develop some
custom functions to do a scientific calculation after data loaded with
Arrow table API. On top of this, I develop a Cython API to design a python
API.
In the current stage, I have a new necessity where I need to consume Arrow
Cython API for my code.
Here It was hard to link the build libarrow.so.16 with the
libarrow_python.so.16 from the installed pyarrow (separately from pip).
What I realised was everything has to be built from the same source, so
that I can install pyarrow from the source in my virtual environment.
Before going through deeper things, I started by just building from source
(CPP) and then moving towards installing pyarrow from the source.
I tried to follow the guideline form here,
https://arrow.apache.org/docs/developers/python.html,
But when I found issues in the python build, I followed this source,
(but still, I used the clone from the master, not a released version)
https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521
My environmental variables are as follows,
python3 setup.py build_ext --inplace
running build_ext
-- Running cmake for pyarrow
cmake -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3
-DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off
-DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off
-DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on
-DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off
-DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off
-DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off
-DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on
-DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release
/home/vibhatha/sandbox/arrow/repos/arrow/python
-- System processor: x86_64
-- Arrow build warning level: PRODUCTION
Using ld linker
Configured for RELEASE build (set with cmake
-DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: RELEASE
-- Build output directory:
/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release
-- Arrow version: 0.18.0 (HOME:
/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs)
-- Arrow SO and ABI version: 18
-- Arrow full SO version: 18.0.0
-- Found the Arrow core shared library:
/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
-- Found the Arrow core import library:
/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
-- Found the Arrow core static library:
/home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a
CMake Error at /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146
(message):
Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) (found
version "0.18.0")
Call Stack (most recent call first):
/usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393
(_FPHSA_FAILURE_MESSAGE)
cmake_modules/FindArrowPython.cmake:76 (find_package_handle_standard_args)
CMakeLists.txt:210 (find_package)
-- Configuring incomplete, errors occurred!
See also "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
error: command 'cmake' failed with exit status 1
Re: Pyarrow building from source along with CPP Libraries to link to
another Cython API
Posted by Vibhatha Abeykoon <vi...@gmail.com>.
Sorry for the confusion, this must go to the user mailing list.
With Regards,
Vibhatha Abeykoon,
Research Assistant,
Intelligent Systems Engineering,
Indiana University Bloomington,
Cell : +1-812-955-1394
Web: https://www.vibhatha.org
<https://www.linkedin.com/in/vibhathaabeykoon/>
On Thu, Apr 30, 2020 at 10:30 PM Vibhatha Abeykoon <vi...@gmail.com>
wrote:
> Hi,
>
> I am trying to integrate Arrow with an application that I am developing.
> Here I build Arrow from the source (CPP) and use the API to develop some
> custom functions to do a scientific calculation after data loaded with
> Arrow table API. On top of this, I develop a Cython API to design a python
> API.
>
> In the current stage, I have a new necessity where I need to consume Arrow
> Cython API for my code.
>
> Here It was hard to link the build libarrow.so.16 with the
> libarrow_python.so.16 from the installed pyarrow (separately from pip).
> What I realised was everything has to be built from the same source, so
> that I can install pyarrow from the source in my virtual environment.
>
> Before going through deeper things, I started by just building from source
> (CPP) and then moving towards installing pyarrow from the source.
>
> I tried to follow the guideline form here,
>
> https://arrow.apache.org/docs/developers/python.html,
>
> But when I found issues in the python build, I followed this source,
> (but still, I used the clone from the master, not a released version)
>
> https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521
>
> My environmental variables are as follows,
>
> python3 setup.py build_ext --inplace
> running build_ext
> -- Running cmake for pyarrow
> cmake -DPYTHON_EXECUTABLE=/home/vibhatha/sandbox/arrow/repos/arrow/ENVARROW/bin/python3 -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=on -DPYARROW_BUILD_PLASMA=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_USE_TENSORFLOW=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_BOOST=off -DPYARROW_GENERATE_COVERAGE=off -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_PARQUET_USE_SHARED=on -DCMAKE_BUILD_TYPE=release /home/vibhatha/sandbox/arrow/repos/arrow/python
> -- System processor: x86_64
> -- Arrow build warning level: PRODUCTION
> Using ld linker
> Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
> -- Build Type: RELEASE
> -- Build output directory: /home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/release
> -- Arrow version: 0.18.0 (HOME: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs)
> -- Arrow SO and ABI version: 18
> -- Arrow full SO version: 18.0.0
> -- Found the Arrow core shared library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
> -- Found the Arrow core import library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.so
> -- Found the Arrow core static library: /home/vibhatha/sandbox/arrow/repos/arrow/cpp/arrowmylibs/libarrow.a
> CMake Error at /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
> Could NOT find ArrowPython (missing: ARROW_PYTHON_INCLUDE_DIR) (found
> version "0.18.0")
> Call Stack (most recent call first):
> /usr/local/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
> cmake_modules/FindArrowPython.cmake:76 (find_package_handle_standard_args)
> CMakeLists.txt:210 (find_package)
>
>
> -- Configuring incomplete, errors occurred!
> See also "/home/vibhatha/sandbox/arrow/repos/arrow/python/build/temp.linux-x86_64-3.8/CMakeFiles/CMakeOutput.log".
> error: command 'cmake' failed with exit status 1
>
>
>