You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "h-vetinari (via GitHub)" <gi...@apache.org> on 2023/01/26 13:11:29 UTC
[GitHub] [arrow] h-vetinari opened a new issue, #33882: [CMake] gRPCAlt takes 15 minutes on linux
h-vetinari opened a new issue, #33882:
URL: https://github.com/apache/arrow/issues/33882
### Describe the bug, including details regarding any error messages, version, and platform.
In conda-forge, running the code related to (or around) `gRPCAlt` is near-instantaneous on osx & windows,
```
2023-01-26T06:03:55.9218860Z -- Providing CMake module for gRPCAlt as part of Arrow CMake package
2023-01-26T06:03:55.9220610Z -- pkg-config package for grpc++ for static link isn't found
```
while on linux it takes 15 minutes
```
2023-01-26T06:02:28.6762155Z -- Providing CMake module for gRPCAlt as part of Arrow CMake package
2023-01-26T06:17:03.7331385Z -- Using pkg-config package for grpc++ for static link
```
This flew under the radar a bit because the linux builds are generally the fastest of the bunch, but it still looks like a massive waste of time (almost a third of the build time).
It happens on all versions 7-10; I didn't test earlier versions, and haven't tested 11 yet.
### Component(s)
Continuous Integration
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou closed issue #33882: [C++][CMake] gRPCAlt takes 15 minutes on linux
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou closed issue #33882: [C++][CMake] gRPCAlt takes 15 minutes on linux
URL: https://github.com/apache/arrow/issues/33882
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #33882: [CMake] gRPCAlt takes 15 minutes on linux
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #33882:
URL: https://github.com/apache/arrow/issues/33882#issuecomment-1407312758
> Could you try this?
Done in https://github.com/conda-forge/arrow-cpp-feedstock/pull/941; I'll get back with timing infos as soon as the CI completes.
> We can add an option to disable pkg-config support. But if we disable pkg-config support, conda users can't find Apache Arrow C++ by pkg-config. I'm not familiar with conda but I think that there are users who use pkg-config to find external packages.
AFAIU there's a difference between _looking_ for pkgconfig metadata of other packages (not necessary for us) and _producing_ it for arrow's consumers (that would still be possible). It might be all bundled up in the `pkg_check_modules` function, but conceptually those things are very different (and need quite different CMake code).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #33882: [CMake] gRPCAlt takes 15 minutes on linux
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #33882:
URL: https://github.com/apache/arrow/issues/33882#issuecomment-1407332445
Here are the timings including the debug statements from the faster agent:
```
2023-01-28T06:58:13.4947939Z -- Found utf8proc: /home/conda/feedstock_root/build_artifacts/apache-arrow_1674888786737/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/libutf8proc.so (found suitable version "2.8.0", minimum required is "2.2.0")
2023-01-28T06:58:13.4949077Z -- Providing CMake module for utf8proc as part of Arrow CMake package
2023-01-28T06:58:13.4949545Z -- Before find_package(PkgConfig)
2023-01-28T06:58:13.4995903Z -- After find_package(PkgConfig)
2023-01-28T06:58:13.5629462Z -- After pkg_check_modules(libutf8proc)
2023-01-28T06:58:13.5630196Z -- Using pkg-config package for libutf8proc for static link
2023-01-28T06:58:13.5684245Z -- Found c-ares: /home/conda/feedstock_root/build_artifacts/apache-arrow_1674888786737/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/cmake/c-ares/c-ares-config.cmake (found version "1.18.1")
2023-01-28T06:58:13.5837826Z -- Providing CMake module for gRPCAlt as part of Arrow CMake package
2023-01-28T06:58:13.5838361Z -- Before find_package(PkgConfig)
2023-01-28T06:58:13.5885242Z -- After find_package(PkgConfig)
2023-01-28T07:13:49.8029312Z -- After pkg_check_modules(grpc++)
2023-01-28T07:13:49.8030662Z -- Using pkg-config package for grpc++ for static link
2023-01-28T07:13:49.8059077Z -- Found nlohmann_json: /home/conda/feedstock_root/build_artifacts/apache-arrow_1674888786737
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #33882: [CMake] gRPCAlt takes 15 minutes on linux
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #33882:
URL: https://github.com/apache/arrow/issues/33882#issuecomment-1407308357
Could you try this?
```diff
diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index 4310d6018b..5a88900fe9 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -286,14 +286,17 @@ macro(resolve_dependency DEPENDENCY_NAME)
endif()
if(${DEPENDENCY_NAME}_SOURCE STREQUAL "SYSTEM" AND ARG_IS_RUNTIME_DEPENDENCY)
provide_find_module(${PACKAGE_NAME} "Arrow")
+ message(STATUS "Before find_package(PkgConfig)")
list(APPEND ARROW_SYSTEM_DEPENDENCIES ${PACKAGE_NAME})
find_package(PkgConfig QUIET)
+ message(STATUS "After find_package(PkgConfig)")
foreach(ARG_PC_PACKAGE_NAME ${ARG_PC_PACKAGE_NAMES})
pkg_check_modules(${ARG_PC_PACKAGE_NAME}_PC
${ARG_PC_PACKAGE_NAME}
NO_CMAKE_PATH
NO_CMAKE_ENVIRONMENT_PATH
QUIET)
+ message(STATUS "After pkg_check_modules(${ARG_PC_PACKAGE_NAME})")
if(${${ARG_PC_PACKAGE_NAME}_PC_FOUND})
message(STATUS "Using pkg-config package for ${ARG_PC_PACKAGE_NAME} for static link"
)
```
> Can we just disable the whole pkg-config search completely? Our builds have CMake metadata, and `find_package(${PACKAGE_NAME} REQUIRED)` should work in the vast majority of cases (even moreso with `CONFIG` mode), and especially for gprc.
We can add an option to disable pkg-config support. But if we disable pkg-config support, conda users can't find Apache Arrow C++ by pkg-config. I'm not familiar with conda but I think that there are users who use pkg-config to find external packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #33882: [CMake] gRPCAlt takes 15 minutes on linux
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #33882:
URL: https://github.com/apache/arrow/issues/33882#issuecomment-1414967493
> OK. We can disable pkg-config search only with `-DARROW_BUILD_STATIC=OFF`. Does conda package specify it?
Yes, it [does](https://github.com/conda-forge/arrow-cpp-feedstock/blob/5945aacd49f0941b6e4c545cced68374fa7bc0d0/recipe/build-arrow.sh#L80) (and has for a long time, so all the recent runs I looked at would have had it).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #33882: [C++][CMake] gRPCAlt takes 15 minutes on linux
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #33882:
URL: https://github.com/apache/arrow/issues/33882#issuecomment-1416747510
Thanks. I've merged it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #33882: [C++][CMake] gRPCAlt takes 15 minutes on linux
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #33882:
URL: https://github.com/apache/arrow/issues/33882#issuecomment-1416663599
So the results from #34019 look great, now takes <1 sec rather than 15min - thanks a lot! :)
```
2023-02-04T03:42:59.9003207Z -- Providing CMake module for utf8proc as part of Arrow CMake package
2023-02-04T03:42:59.9065300Z -- Found c-ares: [...]/lib/cmake/c-ares/c-ares-config.cmake (found version "1.18.1")
2023-02-04T03:42:59.9213892Z -- Providing CMake module for gRPCAlt as part of Arrow CMake package
2023-02-04T03:42:59.9237717Z -- Found nlohmann_json: [...]/share/cmake/nlohmann_json/nlohmann_jsonConfig.cmake (found version "3.11.2")
```
PS: The hunk for the changes around `BZip2` did not apply, so I dropped it while backporting the patch to v11.0.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] h-vetinari commented on issue #33882: [CMake] gRPCAlt takes 15 minutes on linux
Posted by "h-vetinari (via GitHub)" <gi...@apache.org>.
h-vetinari commented on issue #33882:
URL: https://github.com/apache/arrow/issues/33882#issuecomment-1405966845
> Hmm. It's strange. It's happen in https://github.com/apache/arrow/blob/master/cpp/cmake_modules/ThirdpartyToolchain.cmake#L288-L305 .
>
> Can we add debug logs (`message(...)`) in the block or enable something general CMake's debug log to debug this?
That code is really hard to understand. I'm happy to try with a patch, but not sure what/where.
Just now I got worried because on linux, pkg-config claims to find a static grpc++ somehwere, though the build coming in through conda-forge [definitely](https://github.com/regro/libcfgraph/blob/master/artifacts/libgrpc/conda-forge/linux-64/libgrpc-1.51.1-h30feacc_0.json#L513-L548) does not contain any static libs. However, it looks like that's just the log-message being unaware of which flavour of dynamic/static is actually being found by pkg-config.
Can we just disable the whole pkg-config search completely? Our builds have CMake metadata, and `find_package(${PACKAGE_NAME} REQUIRED)` should work in the vast majority of cases (even moreso with `CONFIG` mode), and especially for gprc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #33882: [CMake] gRPCAlt takes 15 minutes on linux
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #33882:
URL: https://github.com/apache/arrow/issues/33882#issuecomment-1405948244
Hmm. It's strange. It's happen in https://github.com/apache/arrow/blob/master/cpp/cmake_modules/ThirdpartyToolchain.cmake#L288-L305 .
Can we add debug logs (`message(...)`) in the block or enable something general CMake's debug log to debug this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on issue #33882: [CMake] gRPCAlt takes 15 minutes on linux
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #33882:
URL: https://github.com/apache/arrow/issues/33882#issuecomment-1414952841
Thanks. I noticed that `pkg-config grpc++` is slow because it has many `absl_*` dependencies.
OK. We can disable pkg-config search only with `-DARROW_BUILD_STATIC=OFF`. Does conda package specify it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org