Re: [I] Building project takes a *long* time (esp compilation time for `datafusion` core crate) [datafusion]

["logan-keede (via GitHub)" <gi...@apache.org>]

logan-keede commented on issue #13814:
URL: https://github.com/apache/datafusion/issues/13814#issuecomment-2770108919

   I did some profiling.
   ```sh
   RUSTC_BOOTSTRAP=1 cargo rustc -p datafusion-catalog -- -Z self-profile -Z self-profile-events=default,args
   ```
   ![Image](https://github.com/user-attachments/assets/86ff5fe6-61ed-4ebc-beeb-6f9170bb1ba9)
   
   ![Image](https://github.com/user-attachments/assets/4a749967-b8f2-4ba2-87e2-05b05a5c8343)
   
   culprits are some asynchronous functions(Memtable::{scan,  insert_into}, StreamTable::{scan, insert_into}  StreamTableFactory::create, StreamWrite::write_all, ViewTable::scan ):- Each of them taking 1-2 sec(5-10% of total time) in analysis phase only.
   
   there are a couple hundred evaluate_obligation call under each type_check of pattern 
   `{...{some_datafusion_object} as std::marker::Sync/Send> ... }` which means that compiler is trying to check if `some_datafusion_object` fulfils the trait bound of being Send/Sync.
   
   This does not happen with other asynchronous method eg. `ListingSchemaProvider::refresh`, so maybe we can try to check the difference between them and see what is going wrong with above mentioned functions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org