You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "abnobdoss (via GitHub)" <gi...@apache.org> on 2026/07/26 19:42:51 UTC
[PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
abnobdoss opened a new pull request, #853:
URL: https://github.com/apache/iceberg-cpp/pull/853
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#discussion_r4013197908
##########
.github/workflows/sanitizer_test.yml:
##########
@@ -64,6 +66,10 @@ jobs:
mkdir build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Debug -DICEBERG_ENABLE_ASAN=ON -DICEBERG_ENABLE_UBSAN=ON \
-DCMAKE_C_COMPILER_LAUNCHER=sccache -DCMAKE_CXX_COMPILER_LAUNCHER=sccache
+ if [[ -d _deps/vendoredarrow-src ]]; then
Review Comment:
can this check be moved into CMake?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#issuecomment-5325726626
Sorry for the late review! Could you help rebase it? We have recently upgraded arrow to use 25.0.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#issuecomment-5325765814
Is it possible to fix the Windows CI to leverage it? We should also leave at least one CI job to use bundled build to make sure it does not break.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "abnobdoss (via GitHub)" <gi...@apache.org>.
abnobdoss commented on PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#issuecomment-6514437435
Apologies for the delay on this. I've come back to it a few times and gone through your comments, and I think conda is the wrong thing to target. CI will keep changing before 1.0.0, and as @wgtmac mentioned in #799 it should be easy to maintain, and @zhjwpku's comment convinced me this isn't. @manuzhang's question about Hive with bundled Thrift and @wgtmac's ask to keep at least one bundled leg get at the other problem: balancing coverage and CI time gets tricky when we still want some legs running the full from-source build.
Going back to the goal in #799, I was wrong that Arrow is most of the cache pressure. It's about 15% of the sccache bytes, debug info is 60 to 80%. Each push to main saves about 10.6 GB against the 10 GB limit, so at least one entry from the previous set gets evicted, and about a quarter of jobs on main since August ran fully cold.
Dropping debug info on the Debug legs (still Debug builds, asserts and -O0 stay) got that to about 4.7 GB on my fork with all tests passing, cold builds got faster too, and build types and coverage stay the same. So my leaning is to close this and open a small PR for that instead. What do you all think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#issuecomment-6517127947
> So my leaning is to close this and open a small PR for that instead. What do you all think?
That sounds perfect to me. Thanks for your continuous investigation on this! @abnobdoss
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "abnobdoss (via GitHub)" <gi...@apache.org>.
abnobdoss commented on PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#issuecomment-5577820619
Thanks for the review, @wgtmac. Apologies for the delay getting back to this. The branch has been rebased onto Arrow 25.0.0, and I've incorporated the Hive and C++ linter conversions from #854.
Builds that meet the intended outcome (build types are unchanged):
| Build (type) | Main | This PR |
|---|---|---|
| AMD64 Ubuntu 26.04 (Debug) | Vendored | Conda |
| AMD64 Ubuntu 26.04 (RelWithDebInfo) | Vendored | Conda |
| AArch64 macOS 26 (Debug) | Vendored | Conda |
| AMD64 Ubuntu 26.04 Hive (Debug) | Vendored | Conda + system Thrift |
| ASAN and UBSAN Tests (Debug) | Vendored | Conda; Iceberg remains instrumented, Arrow is prebuilt without sanitizer instrumentation |
| SQL Catalog (AMD64 Ubuntu 26.04) (Debug) | Vendored | Conda |
| SQL Catalog (AArch64 macOS 26) (Debug) | Vendored | Conda |
| AWS (Ubuntu 26.04, S3 + SigV4, system AWS SDK) (Debug) | Vendored Arrow + vcpkg AWS SDK | Conda Arrow + AWS SDK |
| cpp-linter (Debug) | Vendored | Conda + system Thrift |
| Meson - AMD64 Ubuntu 26.04 (Debug) | No Arrow | No Arrow |
| Meson - AArch64 macOS 26 (Debug) | No Arrow | No Arrow |
| Meson - AMD64 Windows 2025 (Debug) | No Arrow | No Arrow |
| Verify (macos-26) (Release) | Vendored | Vendored |
| Verify (ubuntu-26.04) (Release) | Vendored | Vendored |
Open exceptions:
| Build (type) | Main | This PR | Question |
|---|---|---|---|
| AMD64 Windows 2025 (Debug) | Vendored | Vendored | See question 1. |
| SQL Catalog (AMD64 Windows 2025) (Debug) | Vendored | Vendored | See question 1. |
| AWS (Ubuntu 26.04, S3 + SigV4, bundled AWS SDK) (Debug) | Vendored | Vendored | See question 2. |
| AWS (macOS 26 ARM64, S3, bundled AWS SDK) (Debug) | Vendored | Vendored | See question 2. |
1. If I understand correctly the only way to use conda's prebuilt Arrow on Windows, we would have to change our Windows CMake builds from Debug to [RelWithDebInfo or Release](https://arrow.apache.org/docs/developers/cpp/windows.html#using-conda-forge-for-build-dependencies:~:text=If%20you%20intend%20to%20use%20%2DDCMAKE_BUILD_TYPE%3Ddebug%20then%20you%20must%20build%20the%20packages%20from%20source). Is that something we would want to pursue?
2. Do we want to retain both bundled AWS jobs in PR CI, or keep one and convert the other to conda with a system AWS SDK?
3. Both Hive-enabled jobs now use system Thrift. Given your point about bundled coverage, should we add bundled Hive/Thrift coverage to release verification, or revert one of those jobs to bundled dependencies?
More crucially, although this speeds up builds and reduces reliance on warm caches, the extra exceptions above feel like they make CI harder to maintain. I'm leaning toward dropping this change for now because I don't see a simple way to preserve coverage without adding additional complexity to the CI which is already slightly complex. What do you think: should we continue pursuing this change or drop it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#issuecomment-5654060082
I agree with your judgment. Mixing conda-forge with vcpkg is not a good idea. We should just make it as maintainable as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on code in PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#discussion_r4013179303
##########
.github/workflows/test.yml:
##########
@@ -94,7 +96,9 @@ jobs:
env:
SCCACHE_DIR: ${{ github.workspace }}/.sccache
SCCACHE_CACHE_SIZE: "2G"
- ICEBERG_EXTRA_CMAKE_ARGS: "-DICEBERG_BUILD_HIVE=ON"
+ # Prebuilt Arrow exports no Thrift target, so resolve the Thrift runtime
Review Comment:
Do we lose the test coverage of hive with bundled thrift?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "zhjwpku (via GitHub)" <gi...@apache.org>.
zhjwpku commented on PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#issuecomment-5843556838
I share the concern about the maintenance cost: adopting prebuilt Arrow also changes how we obtain Thrift and the AWS SDK, I'd prefer to preserve the existing build configurations and coverage, particularly Windows Debug and Hive with bundled Thrift. The latter currently loses coverage when Hive-enabled jobs switch to system Thrift.
Could we first evaluate reusing dependency build artifacts with appropriate cache keys? For Linux, a versioned CI image with prebuilt dependencies could also be worth a small experiment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] ci: resolve Arrow and Parquet from conda-forge on Unix legs [iceberg-cpp]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #853:
URL: https://github.com/apache/iceberg-cpp/pull/853#issuecomment-6430869002
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org