You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ian Joiner <ia...@gmail.com> on 2021/11/26 18:21:06 UTC
[Python] How to make a new C++ header file available in Python
Hi,
I have added a new C++ header file, arrow/adapters/orc/adapter_options.h which is not recognized in Cython in our Github Action builds. I wonder what kind of changes (in CMake? I haven’t found any relevant piece in the one CMakeLists.txt in the python directory though) are necessary to make the file usable.
Here is the error message I got here (https://github.com/apache/arrow/runs/4336588692?check_suite_focus=true) :
[44/60] Building CXX object CMakeFiles/_orc.dir/_orc.cpp.o
FAILED: CMakeFiles/_orc.dir/_orc.cpp.o
ccache /opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-c++ -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512 -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2 -D_orc_EXPORTS -isystem /opt/conda/envs/arrow/lib/python3.8/site-packages/numpy/core/include -isystem /opt/conda/envs/arrow/include/python3.8 -isystem /arrow/python/src -Wno-noexcept-type -Wall -fno-semantic-interposition -msse4.2 -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /opt/conda/envs/arrow/include -fdiagnostics-color=always -g -O0 -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -g -fPIC -std=c++11 -Wno-unused-function -MD -MT CMakeFiles/_orc.dir/_orc.cpp.o -MF CMakeFiles/_orc.dir/_orc.cpp.o.d -o CMakeFiles/_orc.dir/_orc.cpp.o -c /build/python/temp.linux-x86_64-3.8/_orc.cpp
/build/python/temp.linux-x86_64-3.8/_orc.cpp:736:10: fatal error: arrow/adapters/orc/adapter_options.h: No such file or directory
736 | #include "arrow/adapters/orc/adapter_options.h"
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Thanks Sutou for pointing out how to fix CMakeLists.txt in cpp/src/arrow back then which is why we no longer have any comparable problem in C++.
Thanks,
Ian
Re: [Python] How to make a new C++ header file available in Python
Posted by David Li <li...@apache.org>.
Looking at the CMakeLists: https://github.com/apache/arrow/blob/abfd2c968b765ffa0607cebd55cb9828e1b61a23/cpp/src/arrow/adapters/orc/CMakeLists.txt#L23
It seems the orc/ subdirectory explicitly lists out the headers, while most other subdirectories use arrow_install_all_headers and automatically pick up new headers. Seems you could add the header to the list in that CMakeLists.txt. (Sorry, I haven't tried this though.)
While building Python, the only header files that get picked up are from where Arrow was installed. (You can see this as the last step of the C++ build in the CI logs, right before Python starts building: https://github.com/apache/arrow/runs/4345764853?check_suite_focus=true#step:8:1971.) But during building, they come from the source tree.
-David
On Sun, Nov 28, 2021, at 08:34, Ian Joiner wrote:
> Hi,
>
> What determines which C++ header files go into the include directory in
> Python builds? From CMakeLists.txt all I can find is that if we set
> PYARROW_BUILD_ORC to ON then _orc gets built. What really matters is what
> goes inside _orc and in particular why doesn’t
> arrow/adapters/orc/adapter_options.h get recognized even though it clearly
> exists, is used in _orc.pxd and the C++ builds do recognize its existence.
>
> Here is the PR: https://github.com/apache/arrow/pull/9702
>
> Best,
> Ian
>
> On Friday, November 26, 2021, Ian Joiner <ia...@gmail.com> wrote:
>
> > Hi,
> >
> > I have added a new C++ header file, arrow/adapters/orc/adapter_options.h
> > which is not recognized in Cython in our Github Action builds. I wonder
> > what kind of changes (in CMake? I haven’t found any relevant piece in the
> > one CMakeLists.txt in the python directory though) are necessary to make
> > the file usable.
> >
> > Here is the error message I got here (https://github.com/apache/
> > arrow/runs/4336588692?check_suite_focus=true) :
> >
> > [44/60] Building CXX object CMakeFiles/_orc.dir/_orc.cpp.o
> > FAILED: CMakeFiles/_orc.dir/_orc.cpp.o
> > ccache /opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-c++
> > -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512
> > -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2
> > -D_orc_EXPORTS -isystem /opt/conda/envs/arrow/lib/
> > python3.8/site-packages/numpy/core/include -isystem
> > /opt/conda/envs/arrow/include/python3.8 -isystem /arrow/python/src
> > -Wno-noexcept-type -Wall -fno-semantic-interposition -msse4.2
> > -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona
> > -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> > -ffunction-sections -pipe -isystem /opt/conda/envs/arrow/include
> > -fdiagnostics-color=always -g -O0 -fno-omit-frame-pointer
> > -Wno-unused-variable -Wno-maybe-uninitialized -g -fPIC -std=c++11
> > -Wno-unused-function -MD -MT CMakeFiles/_orc.dir/_orc.cpp.o -MF
> > CMakeFiles/_orc.dir/_orc.cpp.o.d -o CMakeFiles/_orc.dir/_orc.cpp.o -c
> > /build/python/temp.linux-x86_64-3.8/_orc.cpp
> > /build/python/temp.linux-x86_64-3.8/_orc.cpp:736:10: fatal error:
> > arrow/adapters/orc/adapter_options.h: No such file or directory
> > 736 | #include "arrow/adapters/orc/adapter_options.h"
> > | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > compilation terminated.
> >
> > Thanks Sutou for pointing out how to fix CMakeLists.txt in cpp/src/arrow
> > back then which is why we no longer have any comparable problem in C++.
> >
> > Thanks,
> > Ian
>
Re: [Python] How to make a new C++ header file available in Python
Posted by Ian Joiner <ia...@gmail.com>.
Hi,
What determines which C++ header files go into the include directory in
Python builds? From CMakeLists.txt all I can find is that if we set
PYARROW_BUILD_ORC to ON then _orc gets built. What really matters is what
goes inside _orc and in particular why doesn’t
arrow/adapters/orc/adapter_options.h get recognized even though it clearly
exists, is used in _orc.pxd and the C++ builds do recognize its existence.
Here is the PR: https://github.com/apache/arrow/pull/9702
Best,
Ian
On Friday, November 26, 2021, Ian Joiner <ia...@gmail.com> wrote:
> Hi,
>
> I have added a new C++ header file, arrow/adapters/orc/adapter_options.h
> which is not recognized in Cython in our Github Action builds. I wonder
> what kind of changes (in CMake? I haven’t found any relevant piece in the
> one CMakeLists.txt in the python directory though) are necessary to make
> the file usable.
>
> Here is the error message I got here (https://github.com/apache/
> arrow/runs/4336588692?check_suite_focus=true) :
>
> [44/60] Building CXX object CMakeFiles/_orc.dir/_orc.cpp.o
> FAILED: CMakeFiles/_orc.dir/_orc.cpp.o
> ccache /opt/conda/envs/arrow/bin/x86_64-conda-linux-gnu-c++
> -DARROW_HAVE_RUNTIME_AVX2 -DARROW_HAVE_RUNTIME_AVX512
> -DARROW_HAVE_RUNTIME_BMI2 -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_HAVE_SSE4_2
> -D_orc_EXPORTS -isystem /opt/conda/envs/arrow/lib/
> python3.8/site-packages/numpy/core/include -isystem
> /opt/conda/envs/arrow/include/python3.8 -isystem /arrow/python/src
> -Wno-noexcept-type -Wall -fno-semantic-interposition -msse4.2
> -fvisibility-inlines-hidden -std=c++17 -fmessage-length=0 -march=nocona
> -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2
> -ffunction-sections -pipe -isystem /opt/conda/envs/arrow/include
> -fdiagnostics-color=always -g -O0 -fno-omit-frame-pointer
> -Wno-unused-variable -Wno-maybe-uninitialized -g -fPIC -std=c++11
> -Wno-unused-function -MD -MT CMakeFiles/_orc.dir/_orc.cpp.o -MF
> CMakeFiles/_orc.dir/_orc.cpp.o.d -o CMakeFiles/_orc.dir/_orc.cpp.o -c
> /build/python/temp.linux-x86_64-3.8/_orc.cpp
> /build/python/temp.linux-x86_64-3.8/_orc.cpp:736:10: fatal error:
> arrow/adapters/orc/adapter_options.h: No such file or directory
> 736 | #include "arrow/adapters/orc/adapter_options.h"
> | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> compilation terminated.
>
> Thanks Sutou for pointing out how to fix CMakeLists.txt in cpp/src/arrow
> back then which is why we no longer have any comparable problem in C++.
>
> Thanks,
> Ian