[jira] [Created] (ARROW-2616) Cross-compiling Pyarrow

["Gerlando Falauto (JIRA)" <ji...@apache.org>]

Gerlando Falauto created ARROW-2616:
---------------------------------------

             Summary: Cross-compiling Pyarrow
                 Key: ARROW-2616
                 URL: https://issues.apache.org/jira/browse/ARROW-2616
             Project: Apache Arrow
          Issue Type: Bug
          Components: Packaging, Python
    Affects Versions: 0.9.0
            Reporter: Gerlando Falauto


Hi there,

not quite sure this could be filed in as a bug.

I'm trying to cross-compile Pyarrow on a regular linux x86_64 host for an arm32 target, using LEDE/OpenWRT as the build/packaging system.

The process miserably fails as soon as cmake gets involved:

{noformat}
-- Runnning cmake for pyarrow
cmake -DPYTHON_EXECUTABLE=/workdir/rpi/LEDE/staging_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/host/bin/python3.6 -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_BUNDLE_BOOST=OFF -DCMAKE_BUILD_TYPE=debug /workdir/rpi/LEDE/build_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/apache-arrow-0.9.0/python
INFOCompiler command: /workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/arm-openwrt-linux-muslgnueabi-g++
INFOCompiler version: Reading specs from /workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/../lib/gcc/arm-openwrt-linux-muslgnueabi/5.4.0/specs
COLLECT_GCC=/workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/arm-openwrt-linux-muslgnueabi-g++
COLLECT_LTO_WRAPPER=/workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/../libexec/gcc/arm-openwrt-linux-muslgnueabi/5.4.0/lto-wrapper
Target: arm-openwrt-linux-muslgnueabi
Configured with: /workdir/LEDE/build_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/gcc-5.4.0/configure --with-bugurl=http://www.lede-project.org/bugs/ --with-pkgversion='LEDE GCC 5.4.0 r3533-d0bf257c46' --prefix=/workdir/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-openwrt-linux-muslgnueabi --with-gnu-ld --enable-target-optspace --disable-libgomp --disable-libmudflap --disable-multilib --disable-libmpx --disable-nls --without-isl --without-cloog --with-host-libstdcxx=-lstdc++ --with-gmp=/workdir/LEDE/staging_dir/host --with-mpfr=/workdir/LEDE/staging_dir/host --with-mpc=/workdir/LEDE/staging_dir/host --disable-decimal-float --with-diagnostics-color=auto-if-env --disable-libssp --enable-__cxa_atexit --with-float=hard --with-headers=/workdir/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/include --disable-libsanitizer --enable-languages=c,c++ --enable-shared --enable-threads --with-slibdir=/workdir/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/lib --enable-lto --with-libelf=/workdir/LEDE/staging_dir/host
Thread model: posix
gcc version 5.4.0 (LEDE GCC 5.4.0 r3533-d0bf257c46) 

INFOCompiler id: GNU
Selected compiler gcc 5.4.0
Configured for DEBUG build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
-- Build Type: DEBUG
-- Build output directory: /workdir/rpi/LEDE/build_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/apache-arrow-0.9.0/python/build/temp.linux2-3.6/debug/
CMake Error at cmake_modules/FindPythonLibsNew.cmake:124 (message):
 Python config failure: Python is 64-bit, chosen compiler is 32-bit
Call Stack (most recent call first):
 CMakeLists.txt:183 (find_package)


-- Configuring incomplete, errors occurred!
See also "/workdir/rpi/LEDE/build_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/apache-arrow-0.9.0/python/build/temp.linux2-3.6/CMakeFiles/CMakeOutput.log".
error: command 'cmake' failed with exit status 1
{noformat}

 

I had a look at the affected area (FindPythonLibsNew.cmake) and I'm under the impression the whole CMake-based build system for pyarrow *was never designed for cross-compilation in the first place*. It just calls the configured interpreter ({{PYTHON_EXECUTABLE}}) and tries to locate libraries by running sample code.

That's just great for native builds (where you have e.g. multiple installations on the same filesystem), but it's never gonna work on a cross-build environment.

I believe I could patch the cmake files so to be able to pass those paths (e.g. PYTHON_INCLUDE_DIRS) from the command line (i.e. the way it's done by {{CMAKE_BUILD_TYPE}}), and skip the auto-detect portions. Before I go down that path though, I'd like to know whether I'm bound to eventual failure due to inherent design decisions.

[~pitrou] could perhaps back me up here?

Thank you!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)