You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/05/15 17:05:43 UTC
[GitHub] [arrow-datafusion] jorgecarleitao opened a new issue #348: Reduce compile time of DataFusion?
jorgecarleitao opened a new issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348
During development of DataFusion I am getting a compile time that is hurting development. I am not sure if other feel the same, and I would like to gauge this here.
I admit I do not have a great machine, but I wonder if other also feel some pain in this front.
Note that this is unrelated with the dependencies; it is e.g. when changing a single physical node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb edited a comment on issue #348: Reduce compile time of DataFusion?
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348#issuecomment-1029413337
I have a coworker reports that they can't work on datafusion with a mac mini with 8GB of RAM due to `rust_analyzer` swapping too much
It would be great to make this better somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on issue #348: Reduce compile time of DataFusion?
Posted by GitBox <gi...@apache.org>.
andygrove commented on issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348#issuecomment-841817861
FWIW, a full release build after `cargo clean` takes 1 min 35 seconds for me. The non-release build takes 58 seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb edited a comment on issue #348: Reduce compile time of DataFusion?
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348#issuecomment-1030602865
@Jimexist has a proposal to break datafusion into smaller crates here: https://github.com/apache/arrow-datafusion/issues/1750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on issue #348: Reduce compile time of DataFusion?
Posted by GitBox <gi...@apache.org>.
houqp commented on issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348#issuecomment-847291524
Being spoiled by Golang, I have to say it's relatively slow to the extend where i had to switch to do something else to wait for the build whenever I needed to test a change, even if it's just a single line.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #348: Reduce compile time of DataFusion?
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348#issuecomment-846043608
`cargo test -p datafusion` goes much faster for me, btw, than `cargo test` -- the dependency stack for ballista and some datafusion examples is *distinguished*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #348: Reduce compile time of DataFusion?
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348#issuecomment-1029413337
I coworker reports that they can't work on datafusion with a mac mini with 8GB of RAM due to `rust_analyzer` swapping too much
It would be great to make this better somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #348: Reduce compile time of DataFusion?
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348#issuecomment-1029436437
I ran the following to see where the time was going in my normal development loop:
```
cargo +nightly test -p datafusion -Z timings
```
And the output was instructive. Specifically it seems to take 28.6 seconds to compile the actual datafusion crate, but a whopping 65.6s to compile the tests (aka `cfg(test)`). I'll keep plugging away at moving test code out of the `datafusion` crate and into integration tests
```
Completed datafusion v6.0.0 in 28.6s
Completed datafusion v6.0.0 test "merge_fuzz" (test) in 4.6s
Completed datafusion v6.0.0 test "order_spill_fuzz" (test) in 4.7s
Completed datafusion v6.0.0 test "simplification" (test) in 4.8s
Completed datafusion v6.0.0 test "dataframe" (test) in 6.1s
Completed datafusion v6.0.0 test "provider_filter_pushdown" (test) in 6.7s
Completed datafusion v6.0.0 test "custom_sources" (test) in 6.7s
Completed datafusion v6.0.0 test "statistics" (test) in 6.8s
Completed datafusion v6.0.0 test "path_partition" (test) in 6.9s
Completed datafusion v6.0.0 test "user_defined_plan" (test) in 7.0s
Completed datafusion v6.0.0 test "dataframe_functions" (test) in 7.4s
Completed datafusion v6.0.0 test "parquet_pruning" (test) in 7.9s
Completed datafusion v6.0.0 test "sql_integration" (test) in 19.2s
Completed datafusion v6.0.0 lib (test) in 65.6s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #348: Reduce compile time of DataFusion?
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348#issuecomment-1030602865
@Jimexist has a proposal to break datafusion into smaller crates here: https://github.com/apache/arrow-datafusion/issues/348
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] andygrove commented on issue #348: Reduce compile time of DataFusion?
Posted by GitBox <gi...@apache.org>.
andygrove commented on issue #348:
URL: https://github.com/apache/arrow-datafusion/issues/348#issuecomment-841817861
FWIW, a full release build after `cargo clean` takes 1 min 35 seconds for me. The non-release build takes 58 seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org