You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/07/19 15:46:28 UTC
[GitHub] [arrow] jorgecarleitao opened a new pull request #7799: ARROW-7136: [Rust] Added caching to the docker image
jorgecarleitao opened a new pull request #7799:
URL: https://github.com/apache/arrow/pull/7799
This allow us to build the crate without having to re-build nor download all its dependencies at the expense of a larger (build) image.
Here I am extending the ideas of a non-workspace rust project to a workspace rust project. See [here](https://github.com/rust-lang/cargo/issues/2644) for a _long_ discussion on the issue. Essentially, it is not easy to do this atm with cargo.
IMO I think that this is worth it, but I need your help to judge it; it incurs some maintenance (duplication of configurations).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs closed pull request #7799: ARROW-7136: [Rust] Added caching to the docker image
Posted by GitBox <gi...@apache.org>.
kszucs closed pull request #7799:
URL: https://github.com/apache/arrow/pull/7799
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao edited a comment on pull request #7799: ARROW-7136: [Rust] Added caching to the docker image
Posted by GitBox <gi...@apache.org>.
jorgecarleitao edited a comment on pull request #7799:
URL: https://github.com/apache/arrow/pull/7799#issuecomment-660824105
Some benchmarks:
* Current: 19m with image caching; 21m without image caching
* This PR: ~8m with image caching; 13m without image caching
(~8 assumes 1m download time of the image vs the current 40s, due to bigger image size)
2m:20s are used to build the current layers in both cases
In this PR, 2m40s is used to compile rust dependencies (within docker build) that are cached. The remaining difference is explained by the second commit on this PR, which was causing a complete cache miss of all built dependencies by one of the executions.
Small drill-down:
* build project (all workspaces + small dependencies): 3m
* build of `cargo check --all-targets --no-default-features`: 40s
* build of `cargo test --features simd`: 1m:30s
* build of examples: ~40s
Overall, I think that there is not so much more to save; it is mostly about building our own code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on pull request #7799: ARROW-7136: [Rust] Added caching to the docker image
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #7799:
URL: https://github.com/apache/arrow/pull/7799#issuecomment-666590004
@kszucs , thank you very much for all the improvements and integration that you've made here. Really useful and appreciated!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7799: ARROW-7136: [Rust] Added caching to the docker image
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7799:
URL: https://github.com/apache/arrow/pull/7799#issuecomment-660949359
Thanks @jorgecarleitao, it looks like a nice improvement. Currently I'm working on the next release candidate, but I'll take a look at it afterwards.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7799: ARROW-7136: [Rust] Added caching to the docker image [WIP]
Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #7799:
URL: https://github.com/apache/arrow/pull/7799#issuecomment-665086095
Master - 19m 32s (including docker pull)
No Cache - 13m 28s
Download Cache - 13m 43s
Download and Build Cache - 12m 20s
Download and Build Cache on a Volume - 9m 8s
See build logs [here](https://github.com/apache/arrow/runs/918765051).
The environment variables may not work as expected, but still an improvement over the current master (even if we add the 5 minute build time which we won't need to pay after the docker image gets updated).
I'm going to finalize the PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7799: ARROW-7136: [Rust] Added caching to the docker image
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7799:
URL: https://github.com/apache/arrow/pull/7799#issuecomment-660666121
https://issues.apache.org/jira/browse/ARROW-7136
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorgecarleitao commented on pull request #7799: ARROW-7136: [Rust] Added caching to the docker image
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #7799:
URL: https://github.com/apache/arrow/pull/7799#issuecomment-660824105
Some benchmarks:
* Current: 19m with image caching; 21m without image caching
* This PR: ~8m with image caching; 13m without image caching
(~8 assumes 1m download time of the image vs the current 40s, due to bigger image size)
2m:20s are used to build the current layers in both cases
In this PR, 2m40s is used to compile rust dependencies (within docker build). The remaining difference is explained by the second commit on this PR, which was causing a complete cache miss of all built dependencies by one of the executions.
Small drill-down:
* build project (all workspaces + small dependencies): 3m
* build of `cargo check --all-targets --no-default-features`: 40s
* build of `cargo test --features simd`: 1m:30s
* build of examples: ~40s
Overall, I think that there is not so much more to save; it is mostly about building our own code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org