You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/12/17 15:27:31 UTC
[I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
alamb opened a new issue, #13814:
URL: https://github.com/apache/datafusion/issues/13814
### Is your feature request related to a problem or challenge?
Compiling the datafusion crate currently takes 40 seconds on my machine, far longer than any other crate
This slows down CI builds as well as my own local development workflow
For example, running
```shell
# start from clean checking
rm -rf target
cargo build --timings
```
Generates a chart as follows (attached here): [carg-timings.zip](https://github.com/user-attachments/files/18167599/carg-timings.zip)

### Describe the solution you'd like
I would like to speed up compilation somehow -- likely by decreasing the time required for datafusion-core
### Describe alternatives you've considered
I think the first thing would be to figure out if possible what is taking up so much time when building the core crate
I suspect it has to do with listing table / some of the various file format support, but I don't have data to justify that
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2554458154
FWIW the only reliable mechanism I've found to measure this is to comment out modules and measure the impact on compilation time. Llvm-lines and cargo-bloat can be informative, but not all lines are created equal. This is especially true for code with complex lifetimes and/or async.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-3096076027
> This is super frustrating - I need to wait around 10 seconds on each code change for rust-analyzer to finish, which spends all of the time on `cargo check`, which I believe does some datafusion related builds (although I'm not sure why this happens as I expect it to be built once as I do not change any datafusion code itself, only what is in my project).
Yes, does sound fustrating -- dependent crates (projects that use datafusion) typically don't end up recompiling it for each run
One thing I have seen that is not DataFusion related is the interplay of feature flags can cause this -- so I suggest look at your project and if you have crates that use other crates in `Cargo.toml` with different subsets of feature flags, use one common set of feature flags
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-5281799225
https://github.com/apache/datafusion/pull/24325
https://github.com/apache/datafusion/pull/24326
This addresses part of it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2711668760
might be compiler related?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2774221510
https://github.com/rust-lang/rust/issues/87012#issuecomment-995262877
This thread mentioned a similar issue to what we have.
What we can do is to find a way to avoid async method at all or remove the `Send` bound
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "logan-keede (via GitHub)" <gi...@apache.org>.
logan-keede commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2770505817
> Thanks [@logan-keede](https://github.com/logan-keede) btw is it a release build? can you also check with LTO and non LTO, sometimes this link optimizations takes time as well
>
> UPD: What rustc version used?
It is debug build, I will try to check to with LTO sometime soon.
rustc version:- 1.84.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2713644414
I also tried removing all the doctests from lib.rs and that didn't help (I didn't expect it to but I wanted to try)
```diff
diff --git a/datafusion/core/src/lib.rs b/datafusion/core/src/lib.rs
index b4d5f9740..9e1f9795e 100644
--- a/datafusion/core/src/lib.rs
+++ b/datafusion/core/src/lib.rs
@@ -824,282 +824,3 @@ pub mod test;
mod schema_equivalence;
pub mod test_util;
-
-#[cfg(doctest)]
-doc_comment::doctest!("../../../README.md", readme_example_test);
-
-// Instructions for Documentation Examples
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2767043847
I wonder since catalog is a smaller crate, we could more easily figure out what was taking time to compile...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "logan-keede (via GitHub)" <gi...@apache.org>.
logan-keede commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2779521254
> Thanks [@logan-keede](https://github.com/logan-keede) btw is it a release build? can you also check with LTO and non LTO, sometimes this link optimizations takes time as well
>
> UPD: What rustc version used?
changing to release build lto or non-lto did not make much of a difference from debug build except the extra process that seems to be taking some time.
rustc 1.86.0 + release build

changing to rustc 1.86 did reduce the build time from 18sec -> 14 sec and then to 12 sec when I made `insert_into` sync (only on `catalog` level so not mergeable)
rustc 1.86.0 + debug build

`insert_into` sync + rustc 1.86.0 + debug build

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-5292456101
This is pretty amazing (something that we have struggled with for many years, finally got fixed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2815659788
> Given our bottleneck is on datafusion-core crate which takes 40s. I guess running modules parallelly is not helping much. Find a way to avoid async trait might be a better option.
Definitely worth to try, if that helps it would be an awesome article 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2814289642
Given our bottleneck is on datafusion-core crate which takes 40s. I guess running modules parallelly is not helping much. Find a way to avoid async trait might be a better option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "logan-keede (via GitHub)" <gi...@apache.org>.
logan-keede commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2786971990
>FYI @logan-keede -- maybe we can see how much this helps the build speed
_Originally posted by @alamb in https://github.com/apache/datafusion/issues/15625#issuecomment-2786005044_
It actually does not seem to have much impact on CI.
after some digging, or rather building with different version of rustc I observed that build time for 1.85.1 and 1.86.0 are similar but better than that of 1.84.1(the version I was using during initial profiling).
for reference, datafusion-catalog takes ~50 sec if rustc>1.85 but >60 sec if rustc ==1.84.1 on my laptop
TLDR: Improvements I observed were actually introduced in 1.85 and we were already benefiting from them. Sorry for the false positive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-3097559543
For `cargo check` it might be useful to start with `RUSTFLAGS="-Ztime-passes" cargo +nightly check -Zunstable-options` to see what step most of the time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2549178206
fwiw I tried to gather a bit more info with [llvm-lines](https://github.com/dtolnay/cargo-llvm-lines):
`cargo llvm-lines -p datafusion --lib | head -20`
```
Lines Copies Function name
----- ------ -------------
1220519 36986 (TOTAL)
44221 (3.6%, 3.6%) 423 (1.1%, 1.1%) <alloc::vec::Vec<T> as alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
25426 (2.1%, 5.7%) 265 (0.7%, 1.9%) alloc::vec::Vec<T,A>::extend_desugared
22436 (1.8%, 7.5%) 142 (0.4%, 2.2%) <arrow_array::array::primitive_array::PrimitiveArray<T> as core::iter::traits::collect::FromIterator<Ptr>>::from_iter
21263 (1.7%, 9.3%) 45 (0.1%, 2.4%) <core::iter::adapters::flatten::FlattenCompat<I,U> as core::iter::traits::iterator::Iterator>::size_hint
16910 (1.4%, 10.7%) 151 (0.4%, 2.8%) <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::fold
15949 (1.3%, 12.0%) 172 (0.5%, 3.2%) alloc::vec::Vec<T,A>::extend_trusted
12204 (1.0%, 13.0%) 116 (0.3%, 3.6%) <core::iter::adapters::GenericShunt<I,R> as core::iter::traits::iterator::Iterator>::try_fold::{{closure}}
10585 (0.9%, 13.8%) 88 (0.2%, 3.8%) <alloc::vec::into_iter::IntoIter<T,A> as core::iter::traits::iterator::Iterator>::try_fold
10438 (0.9%, 14.7%) 295 (0.8%, 4.6%) <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::next
9702 (0.8%, 15.5%) 53 (0.1%, 4.7%) alloc::vec::in_place_collect::from_iter_in_place
8645 (0.7%, 16.2%) 13 (0.0%, 4.8%) std::io::default_read_to_end
8256 (0.7%, 16.9%) 116 (0.3%, 5.1%) core::iter::adapters::try_process
7939 (0.7%, 17.5%) 356 (1.0%, 6.0%) alloc::boxed::Box<T>::new
7673 (0.6%, 18.2%) 236 (0.6%, 6.7%) core::iter::adapters::map::map_fold::{{closure}}
7375 (0.6%, 18.8%) 1 (0.0%, 6.7%) datafusion::physical_planner::DefaultPhysicalPlanner::map_logical_node_to_physical::{{closure}}
7352 (0.6%, 19.4%) 241 (0.7%, 7.3%) tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
7317 (0.6%, 20.0%) 102 (0.3%, 7.6%) core::iter::traits::iterator::Iterator::try_fold
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2711642808
BTW I did some profiling on this today and it seems a non trivial amount of time is spent type checking / resolving functions somehow...

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "Omega359 (via GitHub)" <gi...@apache.org>.
Omega359 commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2549003093
install sccache and tell rust to use it
```
export RUSTC_WRAPPER=sccache
```
```
alias cargo="RUSTFLAGS='-Z threads=8' cargo +nightly"
```
With the above two options I see a full build on my machine taking 100.8s (1m 40.8s).
https://corrode.dev/blog/tips-for-faster-rust-compile-times/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "logan-keede (via GitHub)" <gi...@apache.org>.
logan-keede commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2725515168
using following command on release 42.0.0, 45.0.0, and current main.
```bash
cargo build -p datafusion-cli --timings -j 10
```
[42.0.0]


[45.0.0]


[46.0.0]


There is some improvement in build time (201sec -> 190 sec-->183sec), for datafusion crate(65->65->49).(some progress!!)
also, [this article](https://fasterthanli.me/articles/why-is-my-rust-build-so-slow#rustc-self-profiling) and the issue mentioned above by @tustvold(https://github.com/rust-lang/rust/issues/87012) looks promising. The problem cause also seems to be similar between evaluate_obligation[1] and `async_trait`. Though I am not sure if we can apply the same tricks in our usecase. Is it technically possible[2] and worth a try?
[1]: I think evaluate_obligation and process_obligation are same, correct me if I am wrong.
[2]: I am guessing it should be possible but not too sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #13814: Building project takes a *long* time (esp compilation time for `datafusion` core crate)
URL: https://github.com/apache/datafusion/issues/13814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-5292709677
For anyone following along, I think what @Dandandan did was figure out that `rustc` was re-checking the whole `Expr` / `LogicalPlan` tree to prove futures returned from the catalog's `#[async_trait]` methods were `Send`/`Sync`. Apparently `rustc` only caches the result of trait selection sometimes and was not in this case -- see https://rustc-dev-guide.rust-lang.org/traits/caching.html.
The fix in the four PRs (https://github.com/apache/datafusion/pull/24325, https://github.com/apache/datafusion/pull/24326, https://github.com/apache/datafusion/pull/24329, https://github.com/apache/datafusion/pull/24330) is to build the future in a helper function that can be cached, so the Send/Sync proof lands in the global cache and is done once, instead of once per method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2767041075
I ran some profiling recently and it seems like catalog is also getting pretty long to compile

(this is before https://github.com/apache/datafusion/pull/15459)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-3094637555
Hey @anovv thanks for reporting it, what does `cargo build --timings` show?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2549024835
> With the above two options I see a full build on my machine taking 100.8s (1m 40.8s).
This makes sense -- thank you @Omega359
However, I think it will not help CI where the use of such caching doesn't help (and since the intermediate rust files are so large trying to use cargo cache wasn't effective in the past when we tried)
I would love to figure out how to break the datafusion core crate into smaller pieces / crates that can be compiled in parallel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2550624455
> I would love to figure out how to break the datafusion core crate into smaller pieces / crates that can be compiled in parallel
yes! and move around sqlparser dependency when doing so :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-5290642488
In my measurements on Apple M1 cpu, cold compile time performance (after `cargo clean`) for the workspace improved more than 3x 🤯 after the combined changes.
(From around 200s -> 63s).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-5440755560
Possibly related: https://github.com/apache/datafusion/issues/24727 tracks reducing unnecessary monomorphization (generic functions that compile to many near-identical copies). Less generated code means less work for LLVM and the linker, so progress there should also improve compile times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2813309410
Interesting material comes up this week
https://www.feldera.com/blog/cutting-down-rust-compile-times-from-30-to-2-minutes-with-one-thousand-crates
And the nature of this project is similar to DataFusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2711737185
Thanks @tustvold it was nice attempt to solve it 3 years ago, hopefully someone is planning to close it out one day
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2713669607
I also tried cutting out (ablating) the contents of b/datafusion/core/src/physical_planner.rs but that diidn't seem to make any difference either
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2774240019
`ExecutionPlan`'s `execution` is changed to sync in https://github.com/apache/datafusion/issues/2307. It returns `SendableRecordBatchStream` instead.
Is it possible to change `scan` and `insert_into` to sync and returns `Stream`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2726072268
> There is some improvement in build time (201sec -> 190 sec-->183sec), for datafusion crate(65->65->49).(some progress!!)
Pulling stuff out of core helped! Thank you !
I was wondering recently if the the probably could be related to all the re-exports (`pub use ....`) we do in DataFusion. Maybe we could see if reducing them helped 🤔
I think before we spend a lot of time engineering some solution we should try and do something quick and dirty to prove it will help. If it does then we can figure out how to make it work reasonably / get it in chunks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2770840025
It seems like 1.86 won't be released for 3 more days: https://releases.rs/docs/1.86.0/
It would be cool to update and try it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-5616896896
I think we should close this issue and open a new one for additional build time issues so that we can start a clean slate. I think the root cause here was fixed, and we can always improve build times
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2744490926
New Rust compiler has notable performance changes, https://nnethercote.github.io/2025/03/19/how-to-speed-up-the-rust-compiler-in-march-2025.html
looking fwd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2770542695
> rustc version:- 1.84.1
1.86 introduced some compilation improvements https://releases.rs/docs/1.86.0/#internal-changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2770334767
> default
Thanks @logan-keede btw is it a release build? can you also check with LTO and non LTO, sometimes this link optimizations takes time as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2770161361
A break in the case! 🕵
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2774185206
> I did some profiling.
>
> RUSTC_BOOTSTRAP=1 cargo rustc -p datafusion-catalog -- -Z self-profile -Z self-profile-events=default,args
> 
>
> 
>
> culprits are some asynchronous functions(Memtable::{scan, insert_into}, StreamTable::{scan, insert_into} StreamTableFactory::create, StreamWrite::write_all, ViewTable::scan ):- Each of them taking 1-2 sec(5-10% of total time) in analysis phase only.
>
> there are a couple hundred evaluate_obligation call under each type_check of pattern `{...{some_datafusion_object} as std::marker::Sync/Send> ... }` which means that compiler is trying to check if `some_datafusion_object` fulfils the trait bound of being Send/Sync.
>
> This does not happen with other asynchronous method eg. `ListingSchemaProvider::refresh`, so maybe we can try to check the difference between them and see what is going wrong with above mentioned functions.
ListingSchemaProvider::refresh is async method in the struct while `scan` and `insert_into` are async trait method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2713604463
One thing that is strange is that we have moved most of the code out of datafusion-core now.
I am still hoping we can get https://github.com/apache/datafusion/tree/main/datafusion/core/src/datasource/physical_plan out of the core somehow (I know @AdamGS is thinking about that)
However, I don't think that is the problem either as all the formts are feature gated and when I build with no default feaatures
When I build with no default features:
```shell
cargo build --timings -p datafusion --no-default-features
```
I get

And then adding parquet (by far the largest / most complex), it goes up by only 3 seconds
```shell
cargo build --timings -p datafusion --no-default-features --features=parquet
```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-5395015191
> [@crepererum](https://github.com/crepererum) pointed out recently that there is a Rust blog post about the new solver in the Rust compiler (and it mentions that datafusion specifically as something that gets 10x faster with the new solver):
>
> https://blog.rust-lang.org/2026/08/21/enabling-next-solver-on-nightly/
>
> I am not sure if this will be in addition to what [@Dandandan](https://github.com/Dandandan) did in [#13814 (comment)](https://github.com/apache/datafusion/issues/13814#issuecomment-5292709677), but it is pretty cool to see the shoutout
Yes - it is the same issue, so this will probably no longer give any speed up - in fact I measured compilation to be a bit slower in the next gen trait solver (I guess it probably will be optimized a bit further before stabilization).
- 10x (or 4x in the linked article) refers to optimizing the new solver (which was slower than the old one first), also it probably `cargo check` or something on that module only (i.e. not all dependent crates). The 10x happens to correspond with https://github.com/apache/datafusion/pull/24329 as well.
- I traced the issues in the old trait solver: https://github.com/rust-lang/rust/issues/157595#issuecomment-5312290980
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2740776452
> I was wondering recently if the the probably could be related to all the re-exports (`pub use ....`) we do in DataFusion. Maybe we could see if reducing them helped 🤔
it might be good idea for reasons other than compilation time.
For example, my IDE asking me which `ScalarValue` to import (`datafusion::common::ScalarValue` or `datafusion::scalar::ScalarValue`) isn't helpful at all. I would strongly prefer if a public entity had only one public path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2813842960
> Interesting material comes up this week https://www.feldera.com/blog/cutting-down-rust-compile-times-from-30-to-2-minutes-with-one-thousand-crates And the nature of this project is similar to DataFusion
>
> UPD: These guys proposing something we already started, namely split large modules into smaller ones which allows compile parallelization
Interesting! Though that is via codegen (they are code gening rust so can make thousands of modules)
I also found it fascainting they use DataFusion too!
https://github.com/feldera/feldera/blob/7550dc4a024f41020d98233885705d733d14a7e8/Cargo.toml#L73
🐶 😎
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2711684269
IIRC that relates to type checking expressions, and therefore this would suggest the compiler is spending a lot of time resolving generics. At least historically non-boxed async has been a good source of this, as async "desugars" to quite complex generic code, and often has quite complex Send / lifetime constraints.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "logan-keede (via GitHub)" <gi...@apache.org>.
logan-keede commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2770108919
I did some profiling.
```sh
RUSTC_BOOTSTRAP=1 cargo rustc -p datafusion-catalog -- -Z self-profile -Z self-profile-events=default,args
```


culprits are some asynchronous functions(Memtable::{scan, insert_into}, StreamTable::{scan, insert_into} StreamTableFactory::create, StreamWrite::write_all, ViewTable::scan ):- Each of them taking 1-2 sec(5-10% of total time) in analysis phase only.
there are a couple hundred evaluate_obligation call under each type_check of pattern
`{...{some_datafusion_object} as std::marker::Sync/Send> ... }` which means that compiler is trying to check if `some_datafusion_object` fulfils the trait bound of being Send/Sync.
This does not happen with other asynchronous method eg. `ListingSchemaProvider::refresh`, so maybe we can try to check the difference between them and see what is going wrong with above mentioned functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "logan-keede (via GitHub)" <gi...@apache.org>.
logan-keede commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2774239938
> [rust-lang/rust#87012 (comment)](https://github.com/rust-lang/rust/issues/87012#issuecomment-995262877)
>
> This thread mentioned a similar issue to what we have. ~What we can do is to find a way to avoid async method at all or remove the `Send` bound~
>
> [#13814 (comment)](https://github.com/apache/datafusion/issues/13814#issuecomment-2725515168) has the conclusion already 😆
still good to get a heads up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "anovv (via GitHub)" <gi...@apache.org>.
anovv commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-3093213557
Hi all, not sure if it is related, but I have a problem with `cargo check` being super slow after adding datafusion crate to my project.
This is super frustrating - I need to wait around 7 seconds on each code change for rust-analyzer to finish, which spends all of the time on `cargo check`, which I believe does some datafusion related builds (although I'm not sure why this happens as I expect it to be built once as I do not change any datafusion code itself, only what is in my project).
I tried using sscache and it did not help much. Any tips?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "findepi (via GitHub)" <gi...@apache.org>.
findepi commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-3096736712
Sometimes different feature flags are result of different crates depending on different features. In such case, having workspace-level definition of features is a solution, as @alamb pointed out.
Sometimes, however, different flags are result of dependencies needing different features -- of datafusion, or more likely, of one of datafusion dependencies. Workspace-wide `Cargo.toml` is not sufficient to help with this, because the recompilation may be triggered by a 3rd party crate not depended-on explicitly.
I found two things to be helpful
- Hakari "workspace hack" https://crates.io/crates/hakari, which unifies features across transitive dependencies
- Adding this to root `Cargo.toml`
```
[profile.dev.build-override]
# Some libraries (e.g. syn, proc-macro2 and all their dependencies) are compiled
# *for* the build, and also depended-on by tests (because we have proc-macro crates).
# Tests compile with debug symbols enabled, otherwise debug symbols are disabled by default.
# Force debug symbols to be always enabled to increase cache reuse between `cargo test`
# and `cargo build/run --bin` invocations.
debug = true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-5286468958
Amazing -- thank you @Dandandan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-5394208715
@crepererum pointed out recently that there is a Rust blog post about the new solver in the Rust compiler (and it mentions that datafusion specifically as something that gets 10x faster with the new solver):
https://blog.rust-lang.org/2026/08/21/enabling-next-solver-on-nightly/
I am not sure if this will be in addition to what @Dandandan did in https://github.com/apache/datafusion/issues/13814#issuecomment-5292709677, but it is pretty cool to see the shoutout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org