You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "WillAyd (via GitHub)" <gi...@apache.org> on 2025/09/08 21:19:21 UTC
[PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
WillAyd opened a new pull request, #47535:
URL: https://github.com/apache/arrow/pull/47535
### Rationale for this change
This detects conda-installed packages in Meson CI, so that dependencies which are installed via conda do not need to be built from scratch
### What changes are included in this PR?
An update to the meson setup invocation
### Are these changes tested?
Yes
### Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #47535:
URL: https://github.com/apache/arrow/pull/47535#issuecomment-3269132127
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 2b203ff616b7a16fb1eb160c93dc447096eda255.
There weren't enough matching historic benchmark results to make a call on whether there were regressions.
The [full Conbench report](https://github.com/apache/arrow/runs/49916039929) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on PR #47535:
URL: https://github.com/apache/arrow/pull/47535#issuecomment-3268556104
> Can we use `dependency('snappy', 'Snappy')` for Snappy?
Ah cool! I wasn't aware of this syntax but it seems to work locally. Let's try it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on PR #47535:
URL: https://github.com/apache/arrow/pull/47535#issuecomment-3268540850
> bzip2 does not provide any information about its dependency (neither pkg-config nor CMake files), so I think that would need an upstream patch
I agree with it.
Can we use `find_library()` as a workaround like the following?
```diff
diff --git a/cpp/src/arrow/meson.build b/cpp/src/arrow/meson.build
index fbd9cc8551..fd80539c80 100644
--- a/cpp/src/arrow/meson.build
+++ b/cpp/src/arrow/meson.build
@@ -223,7 +223,14 @@ endif
if needs_bz2
arrow_util_srcs += ['util/compression_bz2.cc']
- arrow_util_deps += dependency('bzip2')
+ bzip2 = cpp_compiler.find_library('bz2',
+ has_headers: ['bzlib.h'],
+ required: false)
+ if bzip2.found()
+ arrow_util_deps += bzip2
+ else
+ arrow_util_deps += dependency('bzip2')
+ endif
endif
if needs_lz4
```
Can we use `dependency('snappy', 'Snappy')` for Snappy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #47535:
URL: https://github.com/apache/arrow/pull/47535#discussion_r2331447344
##########
ci/scripts/cpp_build.sh:
##########
@@ -142,6 +142,7 @@ if [ "${ARROW_USE_MESON:-OFF}" = "ON" ]; then
meson setup \
--prefix=${MESON_PREFIX:-${ARROW_HOME}} \
--buildtype=${ARROW_BUILD_TYPE:-debug} \
+ -Dpkg_config_path="${CONDA_PREFIX}/lib/pkgconfig/" \
Review Comment:
How about using `--pkg-config-path` instead?
```suggestion
--pkg-config-path="${CONDA_PREFIX}/lib/pkgconfig/" \
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on PR #47535:
URL: https://github.com/apache/arrow/pull/47535#issuecomment-3268094017
Looks like this is working now and cuts the Meson build time down by about ~35 minutes! @kou
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou merged PR #47535:
URL: https://github.com/apache/arrow/pull/47535
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #47535:
URL: https://github.com/apache/arrow/pull/47535#issuecomment-3268058706
:warning: GitHub issue #47534 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on code in PR #47535:
URL: https://github.com/apache/arrow/pull/47535#discussion_r2331471322
##########
ci/scripts/cpp_build.sh:
##########
@@ -142,6 +142,7 @@ if [ "${ARROW_USE_MESON:-OFF}" = "ON" ]; then
meson setup \
--prefix=${MESON_PREFIX:-${ARROW_HOME}} \
--buildtype=${ARROW_BUILD_TYPE:-debug} \
+ -Dpkg_config_path="${CONDA_PREFIX}/lib/pkgconfig/" \
Review Comment:
Ah great call - thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-47534: [C++] Detect conda-installed packages in Meson CI [arrow]
Posted by "WillAyd (via GitHub)" <gi...@apache.org>.
WillAyd commented on PR #47535:
URL: https://github.com/apache/arrow/pull/47535#issuecomment-3268431211
Nice observation @kou - there appear to be various inconsistencies with the packages that are not found:
bzip2 does not provide any information about its dependency (neither pkg-config nor CMake files), so I think that would need an upstream patch
snappy does not provide a pkg-config file. It *does* provide a CMake file, but unfortunately only for MODULE search mode and not CONFIG search mode. As search, it provides a file located in "${CONDA_PREFIX}/lib/cmake" called `Snappy/SnappyConfig.cmake`. The case-sensitive nature of the search overlooks that file.
If we cared enough, we could patch our meson configuration to try something like:
```
snappy_dep = dependency('Snappy', required: false)
if not snappy_dep.found()
snappy_dep = dependency('snappy')
endif
```
Although that does special-case snappy. We could alternatively just see if they can distribute pkg-config information, or a CMake file using the MODULE search mode
Regarding `benchmark-main`, it appears that there is an inconsistency in the wrapdb. Google benchmark distributes a file by the name of `benchmark_main` (underscore not hyphen). We can therefore change our code to:
```python
benchmark_main_dep = dependency(
'benchmark_main',
fallback: ['benchmark', 'google_benchmark_main_dep'],
)
```
I've opened an upstream issue about that in https://github.com/mesonbuild/wrapdb/issues/2388, and an issue for a workaround here in https://github.com/apache/arrow/issues/47537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org