You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "2010YOUY01 (via GitHub)" <gi...@apache.org> on 2024/11/06 14:19:39 UTC
[I] Make benchmark binaries (like tpch) compile faster [datafusion]
2010YOUY01 opened a new issue, #13277:
URL: https://github.com/apache/datafusion/issues/13277
### Is your feature request related to a problem or challenge?
I found benchmark binaries takes a lot of time to compile, and most time is spent on compiling the binary itself.
I profiled the compiling time using:
```sh
# under benchmark/
cargo build --release --timings --bin tpch
```
On my m1 pro MacBook, the report shows 2 minutes is spent compiling dependencies including `datafusion` crate, and 4 minutes is spent on compiling a single `tpch` binary.
As a reference, `datafusion-cli` binary only took 10 sec to build on my machine.
This seems unusual, and I'm wondering is it possible to make it compile faster.
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make benchmark binaries (like tpch) compile faster [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on issue #13277:
URL: https://github.com/apache/datafusion/issues/13277#issuecomment-2461288805
> For the quick test during dev, maybe can try following build mode, it will be much faster:
>
> ```
> cargo build --profile release-nonlto
> ```
Thank you, it works!
I believe most time is spent on LTO: only set `lto = false` can significantly reduce compile time. I also tried faster linker (ld -> lld) but didn't help much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make benchmark binaries (like tpch) compile faster [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #13277:
URL: https://github.com/apache/datafusion/issues/13277#issuecomment-2461731437
I think we have a PR that avoid to use release-nonlto optimization because we want to benchmark the code as close as to the release version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make benchmark binaries (like tpch) compile faster [datafusion]
Posted by "Rachelint (via GitHub)" <gi...@apache.org>.
Rachelint commented on issue #13277:
URL: https://github.com/apache/datafusion/issues/13277#issuecomment-2460338230
I think it may be due to the release compile options. It will be useful when performing formal benchmark (like what will be done before merging pr about performance)
```
[profile.release]
codegen-units = 1
lto = true
```
For the quick test during dev, maybe can try following build mode, it will be much faster:
```
cargo build --profile release-nonlto
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Make benchmark binaries (like tpch) compile faster [datafusion]
Posted by "eejbyfeldt (via GitHub)" <gi...@apache.org>.
eejbyfeldt commented on issue #13277:
URL: https://github.com/apache/datafusion/issues/13277#issuecomment-2463139903
I think it my be worth running some tests with thin lto (https://doc.rust-lang.org/cargo/reference/profiles.html#lto) if it can deliver the promised
> "thin": Performs [“thin” LTO](http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html). This is similar to “fat”, but takes substantially less time to run while still achieving performance gains similar to “fat”.
the it seems like it would be a good solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org