You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/11/03 17:11:08 UTC
[I] `datafusion-cli` requires more than 8GB to build [datafusion]
alamb opened a new issue, #18471:
URL: https://github.com/apache/datafusion/issues/18471
### Describe the bug
@pmcgleenon reports on https://github.com/apache/datafusion/issues/17721#issuecomment-3476400608
> The instance type [c6a.xlarge](https://instances.vantage.sh/aws/ec2/c6a.xlarge) only has 8GB RAM (the clickbench dataset is 15GB) and there are a number of issues with it, including
>
> datafusion compilation fails on this instance. We can workaround this by using brew install
### To Reproduce
You can reproduce this locally on docker using this command:
```shell
cd .devcontainer
# create a docker container
docker build -t datafusion-build .
cd ..
docker run -m 4G -v `pwd`:/datafusion -it datafusion-build /bin/bash
```
Then try to run
```
cargo build --release -p datafusion-cli
```
This will fail with an error like this:
```
Caused by:
process didn't exit successfully: `/usr/local/rustup/toolchains/1.90.0-aarch64-unknown-linux-gnu/bin/rustc --crate-name datafusion_functions_aggregate --edition=2021 datafusion/functions-aggregate/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --diagnostic-width=160 --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C codegen-units=1 '--warn=clippy::used_underscore_binding' --deny=unused_qualifications '--warn=clippy::unnecessary_lazy_evaluations' '--warn=clippy::uninlined_format_args' --warn=unexpected_cfgs '--warn=clippy::or_fun_call' '--warn=clippy::large_futures' '--warn=clippy::inefficient_to_string' --check-cfg 'cfg(datafusion_coop, values("tokio", "tokio_fallback", "per_stream"))' --check-cfg 'cfg(tarpaulin)' --check-cfg 'cfg(tarpaulin_include)' --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values())' -C metadata=727ec34a83a3b23a -C extra-filename=-41481de704778bc6 --out-dir /datafusion/
target/release/deps -C strip=symbols -L dependency=/datafusion/target/release/deps --extern ahash=/datafusion/target/release/deps/libahash-16bfcefaf019ce0a.rmeta --extern arrow=/datafusion/target/release/deps/libarrow-7a0c4e030b71b835.rmeta --extern datafusion_common=/datafusion/target/release/deps/libdatafusion_common-ca5a073002b6fd0e.rmeta --extern datafusion_doc=/datafusion/target/release/deps/libdatafusion_doc-8ab038afb63bc362.rmeta --extern datafusion_execution=/datafusion/target/release/deps/libdatafusion_execution-93c2cc956c650df0.rmeta --extern datafusion_expr=/datafusion/target/release/deps/libdatafusion_expr-87611f1ca5ce8f65.rmeta --extern datafusion_functions_aggregate_common=/datafusion/target/release/deps/libdatafusion_functions_aggregate_common-05b65785638f4212.rmeta --extern datafusion_macros=/datafusion/target/release/deps/libdatafusion_macros-0bed692e75dab347.so --extern datafusion_physical_expr=/datafusion/target/release/deps/libdatafusion_physical_expr-93e2a25460e
fd9b7.rmeta --extern datafusion_physical_expr_common=/datafusion/target/release/deps/libdatafusion_physical_expr_common-79bb18eb023feafb.rmeta --extern half=/datafusion/target/release/deps/libhalf-feba467d2af9eeb5.rmeta --extern log=/datafusion/target/release/deps/liblog-d456af582e58c114.rmeta --extern paste=/datafusion/target/release/deps/libpaste-5dba9edad0280e64.so -L native=/datafusion/target/release/build/zstd-sys-85e0b1f1ab4f39ce/out -L native=/datafusion/target/release/build/lzma-sys-d68250324f9032a4/out -L native=/datafusion/target/release/build/aws-lc-sys-441c9757ee7585c9/out -L native=/datafusion/target/release/build/ring-dcbccc510e078148/out -L native=/datafusion/target/release/build/psm-2382bc49ac467ad4/out` (signal: 9, SIGKILL: kill)
```
Note the `signal: 9, SIGKILL: kill`
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `datafusion-cli` requires more than 8GB to build [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #18471:
URL: https://github.com/apache/datafusion/issues/18471#issuecomment-3481733898
BTW I am not sure this is a bug we need to solve necessairly -- using a pre-built binary might be a better option
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] `datafusion-cli` requires more than 8GB to build [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #18471:
URL: https://github.com/apache/datafusion/issues/18471#issuecomment-3481665585
I tried limiting concurrency to work around the issue but this still fails with a SIGKILL:
```shell
cargo build -j 1 --release -p datafusion-cli
```
I was able to build it using nonlto mode:
```shell
cargo build --profile=release-nonlto -p datafusion-cli
...
Finished `release-nonlto` profile [optimized] target(s) in 2m 26s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org