[jira] [Resolved] (ARROW-2616) [Python] Cross-compiling Pyarrow

["Uwe Korn (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uwe Korn resolved ARROW-2616.
-----------------------------
    Fix Version/s: 3.0.0
         Assignee: Uwe Korn
       Resolution: Fixed

Closing this as we successfully cross-compile {{pyarrow}} on conda-forge for Apple Silicon. We have some patches in the feedstock that were incorporated into master for 3.0.

> [Python] Cross-compiling Pyarrow
> --------------------------------
>
>                 Key: ARROW-2616
>                 URL: https://issues.apache.org/jira/browse/ARROW-2616
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Packaging, Python
>    Affects Versions: 0.9.0
>            Reporter: Gerlando Falauto
>            Assignee: Uwe Korn
>            Priority: Major
>             Fix For: 3.0.0
>
>
> Hi there,
> not quite sure this could be filed in as a bug.
> I'm trying to cross-compile Pyarrow on a regular linux x86_64 host for an arm32 target, using LEDE/OpenWRT as the build/packaging system.
> The process miserably fails as soon as cmake gets involved:
> {noformat}
> -- Runnning cmake for pyarrow
> cmake -DPYTHON_EXECUTABLE=/workdir/rpi/LEDE/staging_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/host/bin/python3.6 -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_BUNDLE_BOOST=OFF -DCMAKE_BUILD_TYPE=debug /workdir/rpi/LEDE/build_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/apache-arrow-0.9.0/python
> INFOCompiler command: /workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/arm-openwrt-linux-muslgnueabi-g++
> INFOCompiler version: Reading specs from /workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/../lib/gcc/arm-openwrt-linux-muslgnueabi/5.4.0/specs
> COLLECT_GCC=/workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/arm-openwrt-linux-muslgnueabi-g++
> COLLECT_LTO_WRAPPER=/workdir/rpi/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/bin/../libexec/gcc/arm-openwrt-linux-muslgnueabi/5.4.0/lto-wrapper
> Target: arm-openwrt-linux-muslgnueabi
> Configured with: /workdir/LEDE/build_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/gcc-5.4.0/configure --with-bugurl=http://www.lede-project.org/bugs/ --with-pkgversion='LEDE GCC 5.4.0 r3533-d0bf257c46' --prefix=/workdir/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=arm-openwrt-linux-muslgnueabi --with-gnu-ld --enable-target-optspace --disable-libgomp --disable-libmudflap --disable-multilib --disable-libmpx --disable-nls --without-isl --without-cloog --with-host-libstdcxx=-lstdc++ --with-gmp=/workdir/LEDE/staging_dir/host --with-mpfr=/workdir/LEDE/staging_dir/host --with-mpc=/workdir/LEDE/staging_dir/host --disable-decimal-float --with-diagnostics-color=auto-if-env --disable-libssp --enable-__cxa_atexit --with-float=hard --with-headers=/workdir/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/include --disable-libsanitizer --enable-languages=c,c++ --enable-shared --enable-threads --with-slibdir=/workdir/LEDE/staging_dir/toolchain-arm_cortex-a53+neon-vfpv4_gcc-5.4.0_musl-1.1.16_eabi/lib --enable-lto --with-libelf=/workdir/LEDE/staging_dir/host
> Thread model: posix
> gcc version 5.4.0 (LEDE GCC 5.4.0 r3533-d0bf257c46) 
> INFOCompiler id: GNU
> Selected compiler gcc 5.4.0
> Configured for DEBUG build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
> -- Build Type: DEBUG
> -- Build output directory: /workdir/rpi/LEDE/build_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/apache-arrow-0.9.0/python/build/temp.linux2-3.6/debug/
> CMake Error at cmake_modules/FindPythonLibsNew.cmake:124 (message):
>  Python config failure: Python is 64-bit, chosen compiler is 32-bit
> Call Stack (most recent call first):
>  CMakeLists.txt:183 (find_package)
> -- Configuring incomplete, errors occurred!
> See also "/workdir/rpi/LEDE/build_dir/target-arm_cortex-a53+neon-vfpv4_musl-1.1.16_eabi/apache-arrow-0.9.0/python/build/temp.linux2-3.6/CMakeFiles/CMakeOutput.log".
> error: command 'cmake' failed with exit status 1
> {noformat}
>  
> I had a look at the affected area (FindPythonLibsNew.cmake) and I'm under the impression the whole CMake-based build system for pyarrow *was never designed for cross-compilation in the first place*. It just calls the configured interpreter ({{PYTHON_EXECUTABLE}}) and tries to locate libraries by running sample code.
> That's just great for native builds (where you have e.g. multiple installations on the same filesystem), but it's never gonna work on a cross-build environment.
> I believe I could patch the cmake files so to be able to pass those paths (e.g. PYTHON_INCLUDE_DIRS) from the command line (i.e. the way it's done by {{CMAKE_BUILD_TYPE}}), and skip the auto-detect portions. Before I go down that path though, I'd like to know whether I'm bound to eventual failure due to inherent design decisions.
> [~pitrou] could perhaps back me up here?
> Thank you!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)