You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "h-vetinari (via GitHub)" <gi...@apache.org> on 2024/10/16 22:23:31 UTC
[I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
h-vetinari opened a new issue, #44448:
URL: https://github.com/apache/arrow/issues/44448
### Describe the bug, including details regarding any error messages, version, and platform.
I'm [testing](https://github.com/conda-forge/arrow-cpp-feedstock/pull/1432) 18.0.0rc0 in conda-forge, and getting a compilation error on osx-arm64 (which for us is cross-compiled from osx-64). It seems something is going wrong in the following code
https://github.com/apache/arrow/blob/5d7987b94a4389e098461ae013522a64eb2fe6fe/cpp/src/arrow/util/simd.h#L34-L39
in the sense that this tries to include `<nmmintrin.h>` in cross-compilation (probably the `HAVE_RUNTIME_X` is based on the architecture of the agent?), despite the target architecture having no support for that.
```
[124/491] Building CXX object src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o
FAILED: src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o
$BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang++ -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_DYN_LINK -DGFLAGS_IS_A_DLL=0 -DGLOG_USE_GFLAGS -DGLOG_USE_GLOG_EXPORT -DURI_STATIC_BUILD -I$SRC_DIR/cpp/build/src -I$SRC_DIR/cpp/src -I$SRC_DIR/cpp/src/generated -fno-aligned-new -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/apache-arrow-18.0.0 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -D_LIBCPP_DISABLE_AVAILABILITY -Qunused-arguments -fcolor-diagnostics -Wall -Wno-unknown-warning-option -Wno-pass-failed -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/apache-arrow-18.0.0 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -D_LIBCPP_DISABLE_AVAILABILITY -O3 -DND
EBUG -O2 -std=c++17 -isysroot /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -mmacosx-version-min=11.0 -fPIC -MD -MT src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o -MF src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o.d -o src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o -c $SRC_DIR/cpp/src/arrow/util/utf8.cc
In file included from $SRC_DIR/cpp/src/arrow/util/utf8.cc:28:
In file included from $SRC_DIR/cpp/src/arrow/util/utf8_internal.h:33:
In file included from $SRC_DIR/cpp/src/arrow/util/simd.h:38:
$BUILD_PREFIX/lib/clang/18/include/nmmintrin.h:14:2: error: "This header is only meant to be used on x86 and x64 architecture"
14 | #error "This header is only meant to be used on x86 and x64 architecture"
| ^
```
Last relevant-looking change in that area is https://github.com/apache/arrow/commit/87d647742b9ae46e86050ac7361d6c3e75613a47 CC @zanmato1984 @pitrou
### Component(s)
C++, Packaging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430574272
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1057709&view=logs&j=d6b58996-039f-5e48-56bf-c3a016e5cd7f&t=dd6a23d9-2356-5264-64bf-579875a5d8d6&l=2571 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #44448: [C++] cross-compilation issues with 18.0.0rc0: inclusion of `<nmmintrin.h>` resp. mis-detection of `grpc_cpp_plugin`
URL: https://github.com/apache/arrow/issues/44448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430593394
https://github.com/grpc/grpc/commit/831d2a6855da36998e24f46e6b2a5b6f6042ad1a
It seems that gRPC changed CMake package since 1.52.0 that was released on 2023-02-04.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430599593
Actually, it seems ubuntu stopped updating libgrc. Even in oracular it's still on 1.51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430563653
So while we solved the CMake bits for not piccking up `nmmintrin.h`, we still have broken cross-compilation currently. This should IMO be considered a blocker, as conda-forge won't be able to release arrow 18 in this state. If you change the milestone on an issue I raise for an rc build without a comment, I don't even get a notification...
I [noted](https://github.com/conda-forge/arrow-cpp-feedstock/pull/1432#issuecomment-2423182583) on the conda-forge side:
> One thing that looks weird to me compared to other such CMake errors is:
>
> ```
> + cmake --build . --config Release
> ninja: error: 'src/arrow/flight/gRPC::grpc_cpp_plugin', needed by 'src/arrow/flight/Flight.pb.cc', missing and no known rule to make it
> ^^^^^^^^^^^^^^^^^
> ?????????????????
> ```
>
> AFAIU this should be detected/used as `gRPC::grpc_cpp_plugin`. Perhaps something is going wrong in https://github.com/apache/arrow/blob/main/cpp/cmake_modules/FindgRPCAlt.cmake ...?
We have correct CMake metadata for grpc (we're [testing](https://github.com/conda-forge/grpc-cpp-feedstock/tree/main/recipe/cmake_test) this as part of the feedstock), and so I suspect there's something going wrong with the custom processing in FindgRPCAlt. I've tried to read that file (also `cpp/cmake_modules/ThirdpartyToolchain.cmake`), and I've tried disabling its use, but to no avail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430597310
> Ubuntu 24.04 ships gRPC 1.51.1.
Wow, that was already 14 months old when 24.04 got released 😱
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430912913
Issue resolved by pull request 44507
https://github.com/apache/arrow/pull/44507
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430572852
Could you share the build log URL that the error message include?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430589265
> Does https://github.com/conda-forge/grpc-cpp-feedstock provide `gRPC::grpc_cpp_plugin` target?
The plugin definitely gets [installed](https://conda-metadata-app.streamlit.app/?q=conda-forge%2Flinux-64%2Flibgrpc-1.65.5-hf5c653b_0.conda); we haven't modified the CMake installation targets, so it's going to be whatever grpc puts in there by default (so I assume `gRPC::grpc_cpp_plugin` to be there, but haven't checked that).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430589532
It seems that https://anaconda.org/conda-forge/libgrpc/1.67.0/download/osx-64/libgrpc-1.67.0-h0fbbd33_0.conda provides `gRPC` CMake package and `gRPCPlugin` CMake package. And `gRPCPlugin` CMake package provides `gRPC::grpc_cpp_plugin`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2436356099
People like Uwe and Keith would equally know how to do it, and I would mentor anyone who invests time on the feedstock. 😉
The offer to maintain it myself also still stands, but I cannot do that out of my free time. Perhaps the situation may change/improve now that QuantStack has jumped into the fray (I'm already collaborating with them). 🙃
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430916958
I don't think that this is a blocker of 18.0.0 RC0 because:
* `<nmmintrin.h>` is cross-compiling problem by conda side
* Cross-compiling didn't work with old releases (It worked accidentally.)
Please patch #44507 for 18.0.0 if we use RC0 for 18.0.0. If we need RC1, we may add #44507 to 18.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2418501681
Hi @h-vetinari , thanks for letting us know. I wonder if this is a regression? (That is, the same build used to pass before?) Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2418553345
I think here is a possible temporary workaround: adding an extra CMake option `-DARROW_RUNTIME_SIMD_LEVEL=NONE`, if you want to proceed with anything you originally intended.
But I can't promise this would be the final solution, which I'll have to talk to Arrow community about and come up with. We'll keep you posted of course.
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2418491374
Sure. This is the CMake call as extracted (and formatted for legibility) from the [logs](https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1056872&view=results):
```
cmake -GNinja \
-DARROW_ACERO=ON
-DARROW_AZURE=ON
-DARROW_BOOST_USE_SHARED=ON
-DARROW_BUILD_BENCHMARKS=OFF
-DARROW_BUILD_STATIC=OFF
-DARROW_BUILD_TESTS=OFF
-DARROW_BUILD_UTILITIES=OFF
-DARROW_COMPUTE=ON
-DARROW_CSV=ON
-DARROW_CXXFLAGS='-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/apache-arrow-18.0.0 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix'
-D_LIBCPP_DISABLE_AVAILABILITY
-DARROW_DATASET=ON
-DARROW_DEPENDENCY_SOURCE=SYSTEM
-DARROW_FILESYSTEM=ON
-DARROW_FLIGHT=ON
-DARROW_FLIGHT_REQUIRE_TLSCREDENTIALSOPTIONS=ON
-DARROW_FLIGHT_SQL=ON
-DARROW_GANDIVA=ON
-DARROW_GANDIVA_PC_CXX_FLAGS=-D_LIBCPP_DISABLE_AVAILABILITY
-DARROW_GCS=ON
-DARROW_GDB_INSTALL_DIR=replace_this_section_with_absolute_slashed_path_to_CONDA_PREFIX/lib
-DARROW_HDFS=ON
-DARROW_JEMALLOC=ON
-DARROW_JSON=ON
-DARROW_MIMALLOC=ON
-DARROW_ORC=ON
-DARROW_PACKAGE_PREFIX=$PREFIX
-DARROW_PARQUET=ON
-DARROW_S3=ON
-DARROW_SIMD_LEVEL=NONE
-DARROW_SUBSTRAIT=ON
-DARROW_USE_GLOG=ON
-DARROW_USE_LD_GOLD=ON
-DARROW_WITH_BROTLI=ON
-DARROW_WITH_BZ2=ON
-DARROW_WITH_LZ4=ON
-DARROW_WITH_NLOHMANN_JSON=ON
-DARROW_WITH_OPENTELEMETRY=OFF
-DARROW_WITH_SNAPPY=ON
-DARROW_WITH_UCX=OFF
-DARROW_WITH_ZLIB=ON
-DARROW_WITH_ZSTD=ON
-DBUILD_SHARED_LIBS=ON
-DCMAKE_BUILD_TYPE=release
-DCMAKE_CXX_STANDARD=17
-DCMAKE_INSTALL_LIBDIR=lib
-DCMAKE_INSTALL_PREFIX=$PREFIX
-DLLVM_TOOLS_BINARY_DIR=$PREFIX/bin
-DMAKE=$BUILD_PREFIX/bin/make
-DPARQUET_REQUIRE_ENCRYPTION=ON
-DPython3_EXECUTABLE=
-DARROW_CUDA=OFF
-DProtobuf_PROTOC_EXECUTABLE=$BUILD_PREFIX/bin/protoc
-DCLANG_EXECUTABLE=$BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang
-DLLVM_LINK_EXECUTABLE=$BUILD_PREFIX/bin/llvm-link
-DARROW_JEMALLOC_LG_PAGE=16 \
..
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430607448
> It seems that `gRPC` CMake package includes `gRPCPluginTargets.cmake` automatically EXCEPT `CROSS_COMPILING`:
Yeah, that makes sense... It's not possible to use the plugins for the target architecture. But in our case we actually supply _both_, i.e. there's a libgrpc in the build environment (where we can execute the plugins), and then one in the host environment (where we don't need the plugins but need to match the target architecture).
> Does conda really need to use cross-compiling?
Yes, and this is impossible to change. We don't have native runners for `linux-aarch64`, `linux-ppc64le` or `osx-arm64`, but they are important platforms we support, so the only way to do it is to cross-compile.
This is where things like `Protobuf_PROTOC_EXECUTABLE` come from; i.e. it can be explicitly overridden. We could easily provide some CMake variable for example `-DGRPC_PLUGIN=$BUILD_PREFIX/bin/grpc_cpp_plugin`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430594551
I did a quick check on our package content, and also found that this was introduced in 1.52... :)
We've been using grpc newer than that in conda-forge for a long time already, so it seems to have worked by accident somehow (or perhaps there were some compatibility in the old CMake module that eventually got removed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2437166171
> Perhaps the situation may change/improve now that QuantStack has jumped into the fray (I'm already collaborating with them).
Yes, hopefully. We might get to work together :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2435531299
If there would be no patch for this, I would consider the issue a release blocker. The release is blocked until we have a patch. Mainly because breaking conda-forge being such a big distribution channel is a release blocker for me.
In my opinion on how to proceed now that we have a patch and due to the current effort on creating a new Release Candidate with the current limited maintenance resources we have, I feel it's reasonable to port the patch on conda and release RC0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2418649378
OK. By looking at the log more carefully I seem to get to the root cause.
From the log, there is a preset `CMAKE_ARGS`:
```
2024-10-17T00:53:32.7864890Z +CMAKE_ARGS= ... -DCMAKE_SYSTEM_PROCESSOR=arm64 ...
```
This is well expected for a cross-compiling config.
Then from the output of Arrow CMake:
```
2024-10-17T00:53:45.7948650Z -- System processor: x86_64
```
This is printed by:
https://github.com/apache/arrow/blob/b17546322a57f67b2adf1fcdb212685ac89210f6/cpp/cmake_modules/SetupCxxFlags.cmake#L24
And it's not surprising that the subsequent SIMD options are set for x86 platform and eventually lead to the erroring header.
With a little googling there are [discussions](https://stackoverflow.com/a/76632595/4485168) seemed to be related:
> I had a similar issue. The answer is that when CMAKE_SYSTEM_NAME is not set, CMake overwrites CMAKE_SYSTEM_PROCESSOR ([source code](https://github.com/Kitware/CMake/blob/fbea5d9f9945363f35aa16c3edcf0924e3dc4686/Modules/CMakeDetermineSystem.cmake#L178)). This is arguably a bug but regardless is how CMake behaves.
I can locally reproduce the questioning CMake behavior on my M1 Mac CMake version 3.25.2:
```
cmake -DCMAKE_SYSTEM_PROCESSOR=x86_64 ..
-- Building using CMake version: 3.25.2
...
-- System processor: arm64
```
```
cmake -DCMAKE_SYSTEM_NAME=Darwin -DCMAKE_SYSTEM_PROCESSOR=x86_64 ..
-- Building using CMake version: 3.25.2
...
-- System processor: x86_64
```
Worth a try and hope this solves your issue, @h-vetinari . Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430599550
Ah, wait.
It seems that `gRPC` CMake package includes `gRPCPluginTargets.cmake` automatically EXCEPT `CROSS_COMPILING`:
```console
$ cat libgrpc-1.67.0-h0fbbd33_0/lib/cmake/grpc/gRPCConfig.cmake
# Module path
list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR}/modules)
# Depend packages
if(NOT ZLIB_FOUND)
find_package(ZLIB)
endif()
include(CMakeFindDependencyMacro)
find_dependency(Protobuf CONFIG)
if(NOT OPENSSL_FOUND)
find_package(OpenSSL)
endif()
if(NOT c-ares_FOUND)
find_package(c-ares)
endif()
if(NOT TARGET absl::strings)
find_package(absl CONFIG)
endif()
if(NOT re2_FOUND)
find_package(re2)
endif()
# Targets
include(${CMAKE_CURRENT_LIST_DIR}/gRPCTargets.cmake)
if(NOT CMAKE_CROSSCOMPILING)
include(${CMAKE_CURRENT_LIST_DIR}/gRPCPluginTargets.cmake)
endif()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2431106942
It seems that we need to define support level or something like other projects.
For example, problems in the support level A environments must be blockers, problems in the support level B environments may be blockers, the support level C is best effort and so on. The support level A environments' CI must be prepared.
We may not be able to support all environments because we have limited maintenance resources.
How about discuss this on the mailing list? https://arrow.apache.org/community/#mailing-lists
Could you open a discussion thread?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430616875
Hmm. Why can we cross-compile 17.0.0...? We don't have any changes around it...
Anyway, could you try GH-44448?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430605864
Does conda really need to use cross-compiling?
In general, executables (`grpc_cpp_plugin` in this case) on host isn't usable in cross-compiling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compiling osx-arm64 fails due to inclusion of `` with 18.0.0rc0 [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430639246
> Anyway, could you try [GH-44448](https://github.com/apache/arrow/issues/44448)?
Thanks a lot! Started a new [job](https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1061282&view=results)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2435603500
> it may be nice to have at least one CI build to exercise cross-compiling
I've opened an issue to track that https://github.com/apache/arrow/issues/44531
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430595962
https://packages.ubuntu.com/search?keywords=libgrpc
Ubuntu 24.04 ships gRPC 1.51.1. So our CI doesn't use system gRPC 1.52.0 or later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430590162
Apache Arrow C++ uses only `gRPC` CMake package. So we don't have `gRPC::grpc_cpp_plugin`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2435548242
As a side note, it may be nice to have at least one CI build to exercise cross-compiling (perhaps as a Crossbow build?). That needs someone to set up, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2436107541
Yes, it's unmaintained because it comes with the entire cognitive overhead of conda-forge that none of us is familiar with (except you ;-)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2418466494
Could you share the actual CMake options used for the cross-compilation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2418509283
Yes, definitely a regression. Conda-forge has been cross-compiling osx-arm64 from osx-64 since arrow v0.17, and never ran into this (to my knowledge; some of it was before my time).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compiling osx-arm64 fails due to inclusion of `` with 18.0.0rc0 [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430621938
> Hmm. Why can we cross-compile 17.0.0...? We don't have any changes around it...
I think our CMake got confused and didn't actually enter into the `NOT CMAKE_CROSSCOMPILING`, so the target got found. Now that we fixed the CMake confusion (which became necessary due to the failure at the top of the ticket), the target suddenly was gone...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430601816
> It seems that https://anaconda.org/conda-forge/libgrpc/1.67.0/download/osx-64/libgrpc-1.67.0-h0fbbd33_0.conda provides `gRPC` CMake package and `gRPCPlugin` CMake package. And `gRPCPlugin` CMake package provides `gRPC::grpc_cpp_plugin`.
I was wrong. There is only `gRPC` CMake package. It just has `gRPCTargets.cmake` and `gRPCPluginTargets.cmake`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430591842
Yeah, that makes sense. Though again, we're using the default CMake install of upstream grpc, that split does not come from conda-forge. So in that sense, it would be good for arrow to be(come) compatible with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430587443
> Could you share the build log URL that includes the error message?
https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1060564&view=results
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of `` [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2430584548
Does https://github.com/conda-forge/grpc-cpp-feedstock provide `gRPC::grpc_cpp_plugin` target?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2436097001
> As a side note, it may be nice to have at least one CI build to exercise cross-compiling (perhaps as a Crossbow build?).
The in-repo conda(-forge) CI does cross compilation already. Not sure what's the state of that though, presumably unmaintained?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] cross-compilation issues with 18.0.0rc0: inclusion of `` resp. mis-detection of `grpc_cpp_plugin` [arrow]
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #44448:
URL: https://github.com/apache/arrow/issues/44448#issuecomment-2431000046
> I don't think that this is a blocker of 18.0.0 RC0 because:
> * [...] Cross-compiling didn't work with old releases (It worked accidentally.)
Accident or not, it worked for years. When a major distribution channel ends up being broken, that should be the definition of a release blocker, even though we'll manage to backport the patch (again, thanks for that). It's not about the effort to backport, it's about what role conda-forge has. With several million downloads, the arrow feedstock should IMO be important enough to be a release consideration (that's originally why the conda-forge CI was integrated into this repo...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org