You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "kevinjqliu (via GitHub)" <gi...@apache.org> on 2026/03/27 01:18:42 UTC
[PR] Reduce peak memory usage during release builds to fix OOM on manylinux runners [datafusion-python]
kevinjqliu opened a new pull request, #1445:
URL: https://github.com/apache/datafusion-python/pull/1445
## Which issue does this PR close?
Fixes OOM (`Killed process ... (rustc) total-vm:25086084kB, anon-rss:15361808kB`) during manylinux x86_64 release builds, where `rustc` consumed ~15 GB and exhausted the runner's memory.
## Rationale
As the dependency tree has grown (DataFusion + Substrait + Arrow + object_store with aws/gcp/azure/http features), the release build's peak memory during LTO linking has exceeded what the GitHub runner can provide.
## What changes are included in this PR?
**Cargo profile (`Cargo.toml`):**
- Switch from fat LTO (`lto = true`) to thin LTO (`lto = "thin"`) -- this is the biggest win, reducing peak memory by ~50-70% since LLVM no longer needs to merge all bitcode into a single module
- Increase `codegen-units` from 1 to 2 -- splits LLVM's workload, further reducing peak RSS
**CI workflow (`.github/workflows/build.yml`):**
- Add 8 GB swap to the `build-manylinux-x86_64` job as a safety net (matching the existing pattern in the aarch64 job)
- Reduce `build-manylinux-aarch64` swap from 16 GB to 8 GB for consistency
## Tradeoffs
Thin LTO + codegen-units=2 may produce binaries that are ~1-4% slower in micro-benchmarks vs fat LTO + codegen-units=1. In practice, this is unlikely to be measurable for a Python extension where the Python-Rust FFI boundary and PyArrow serialization dominate execution time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Reduce peak memory usage during release builds to fix OOM on manylinux runners [datafusion-python]
Posted by "timsaucer (via GitHub)" <gi...@apache.org>.
timsaucer merged PR #1445:
URL: https://github.com/apache/datafusion-python/pull/1445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Reduce peak memory usage during release builds to fix OOM on manylinux runners [datafusion-python]
Posted by "kevinjqliu (via GitHub)" <gi...@apache.org>.
kevinjqliu commented on code in PR #1445:
URL: https://github.com/apache/datafusion-python/pull/1445#discussion_r2998496727
##########
.github/workflows/build.yml:
##########
@@ -232,7 +245,7 @@ jobs:
set -euxo pipefail
sudo swapoff -a || true
sudo rm -f /swapfile
- sudo fallocate -l 16G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=16384
+ sudo fallocate -l 8G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=8192
Review Comment:
we dont need all 16GB, take less disk space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org