[Rust] Heads up -- change to the Rust CI system

[Andrew Lamb <al...@influxdata.com>]

I just wanted to give everyone a heads up that we have merged an overhaul
of the Rust CI system.

You can see more details and rationale on the PR:
https://github.com/apache/arrow/pull/8821 but I wanted to bring it to the
attention of a wider audience as the PR checks will look different for Rust
PRs.

Thanks to @Jorge Cardoso Leitão <jo...@gmail.com>  for driving
this work

Andrew

Re: [Rust] Heads up -- change to the Rust CI system

[Jorge Cardoso Leitão <jo...@gmail.com>]

Thanks Andrew,

To give a bit more context, It now looks as follows (this is from a PR adds
a new build to it, wasm32, taken from here
<https://github.com/apache/arrow/actions/runs/408710849>):

[image: Screenshot 2020-12-08 at 18.54.36.png]

The main consequences were:

* build time halved in both cache hits and misses
* reduced cache misses
* code coverage is now part of the build (only ran in cron-job)
* test of the c data interface against pyarrow is now part of the build
(only ran in cron-job)
* use only official rust docker images and stopped pulling and pushing
custom images
* easier to extend to variations of compiler flags, archs, compiler
version, etc.

The above was achieved through the following changes:

* jobs that share artifacts share a common parent job (a common "need") and
use github's cache action to share built artifacts
* any independent work is assigned its own job, for increased parallelism
(the reason for more jobs)
* stopped using docker images to store intermediary state via github cache
actions
* stopped using archery and docker-compose to drive the execution. Instead,
we run the official Rust docker images and use officially curated github
actions to prepare the environment.

The main build can be reproduced by a single command: docker run --rm -v
$(pwd)/rust:/rust -it rust /bin/bash -c "cd /rust && cargo build" (this is
notoriously slow on mac due to how docker volumes work).

There are 3 items that can be further improved / optimized:
* The integration test, that takes 20-30m, often re-builds entire artifacts
even when no changes are made. I proposed something on a separate mailing
thread to address this
* Travis build and other github jobs are often triggered by rust-specific
changes to `ci/` that do not affect them.
* The lint step (rustfmt) currently runs together with all other
implementations in a single job. It could be split off and run only to
changes on rust.

Best,
Jorge

On Tue, Dec 8, 2020 at 5:51 PM Andrew Lamb <al...@influxdata.com> wrote:

> I just wanted to give everyone a heads up that we have merged an overhaul
> of the Rust CI system.
>
> You can see more details and rationale on the PR:
> https://github.com/apache/arrow/pull/8821 but I wanted to bring it to the
> attention of a wider audience as the PR checks will look different for Rust
> PRs.
>
> Thanks to @Jorge Cardoso Leitão <jo...@gmail.com>  for driving
> this work
>
> Andrew
>