You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/12/02 20:39:37 UTC

[GitHub] [arrow] jorgecarleitao opened a new pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

jorgecarleitao opened a new pull request #8821:
URL: https://github.com/apache/arrow/pull/8821


   This PR refactors the CI with the purpose of making it faster, easier to extend, and more robust. The main consequences of this PR are:
   
   1. build time of non-integration jobs reduced by half (11m vs 20m)
   2. coverage is now part of all builds
   3. significantly reduced the risk of cache misses, and the size of the caches
   4. macos now have caching in place, like other builds
   5. significantly easier to extend to other architectures, os, etc.
   
   The overall design of this PR is that most steps of the CI process should be independent of all others as much as possible, so that
   
   * they can run in parallel
   * it is clearer where the failure is
   * it is easier to extend the build to other variations (arch, os, flags, etc.)
   
   Intrinsically, the CI pipeline is just a DAG where certain jobs benefit from shared compute and thus depend on common nodes, while others run independently.
   
   This PR re-writes most of our CI as simple github jobs that share caches whenever makes sense, thereby describing the aforementioned DAG. Consequently, many of the sequential steps are now executed in parallel, causing the items describe above.
   
   There are two tasks depending on rust that are not in the `rust.yaml`:
   
   * lint, that happens via archery
   * integration tests, that happens via archery
   
   The bottleneck of a green pipeline are the integration tests, that currently re-compile a significant number of components (as I already flagged on the mailing list), but at least we get faster failures for compilation (2.5m) and clippy (4.5m).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] codecov-io edited a comment on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-737491170


   # [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=h1) Report
   > Merging [#8821](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=desc) (fe124f3) into [master](https://codecov.io/gh/apache/arrow/commit/322cd01fff055c4d4a03839fe5ac435948e75b09?el=desc) (322cd01) will **decrease** coverage by `31.72%`.
   > The diff coverage is `58.22%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow/pull/8821/graphs/tree.svg?width=650&height=150&src=pr&token=LpTCFbqVT1)](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #8821       +/-   ##
   ===========================================
   - Coverage   84.54%   52.82%   -31.73%     
   ===========================================
     Files         185      165       -20     
     Lines       46176    29837    -16339     
   ===========================================
   - Hits        39041    15761    -23280     
   - Misses       7135    14076     +6941     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [rust/arrow/src/array/transform/boolean.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2Jvb2xlYW4ucnM=) | `76.92% <0.00%> (-23.08%)` | :arrow_down: |
   | [rust/arrow/src/array/transform/list.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2xpc3QucnM=) | `36.36% <0.00%> (-8.09%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/comparison.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2NvbXBhcmlzb24ucnM=) | `96.27% <ø> (ø)` | |
   | [rust/arrow/src/compute/kernels/filter.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2ZpbHRlci5ycw==) | `58.72% <ø> (-2.55%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/limit.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2xpbWl0LnJz) | `100.00% <ø> (ø)` | |
   | [rust/arrow/src/record\_batch.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvcmVjb3JkX2JhdGNoLnJz) | `84.04% <ø> (-6.31%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/builder.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vYnVpbGRlci5ycw==) | `0.00% <0.00%> (-85.10%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/plan.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | `0.00% <ø> (-83.69%)` | :arrow_down: |
   | [...tafusion/src/physical\_plan/distinct\_expressions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Rpc3RpbmN0X2V4cHJlc3Npb25zLnJz) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
   | [rust/datafusion/src/physical\_plan/functions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Z1bmN0aW9ucy5ycw==) | `0.00% <ø> (-82.13%)` | :arrow_down: |
   | ... and [156 more](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=footer). Last update [17805f3...8803c13](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kszucs commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r536302049



##########
File path: .github/workflows/rust.yml
##########
@@ -21,145 +21,286 @@ on:
   push:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
   pull_request:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
 
-env:
-  DOCKER_BUILDKIT: 0
-  COMPOSE_DOCKER_CLI_BUILD: 1
-  ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
-  ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
-
 jobs:
 
-  debian:
-    name: AMD64 Debian 10 Rust ${{ matrix.rust }}
+  # build the library, a compilation step used by multiple steps below
+  linux-build-lib:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build libraries

Review comment:
       I see, lowercase is fine then.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r536518433



##########
File path: docker-compose.yml
##########
@@ -903,35 +902,6 @@ services:
       /bin/bash -c "
         /arrow/ci/scripts/r_sanitize.sh /arrow"
 
-  ################################ Rust #######################################
-
-  debian-rust:

Review comment:
       I have added a section on the README on how to compile, which includes both pure `cargo` and `docker` variants.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r536358928



##########
File path: .github/workflows/rust.yml
##########
@@ -21,145 +21,286 @@ on:
   push:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
   pull_request:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
 
-env:
-  DOCKER_BUILDKIT: 0
-  COMPOSE_DOCKER_CLI_BUILD: 1
-  ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
-  ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
-
 jobs:
 
-  debian:
-    name: AMD64 Debian 10 Rust ${{ matrix.rust }}
+  # build the library, a compilation step used by multiple steps below
+  linux-build-lib:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build libraries
     runs-on: ubuntu-latest
-    if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
     strategy:
-      fail-fast: false
       matrix:
+        arch: [amd64]
         rust: [nightly-2020-11-24]
-    env:
-      RUST: ${{ matrix.rust }}
+    container:
+      image: ${{ matrix.arch }}/rust
     steps:
-      - name: Checkout Arrow
-        uses: actions/checkout@v2
+      - uses: actions/checkout@v2
+      - name: Cache Cargo
+        uses: actions/cache@v2
         with:
-          fetch-depth: 0
-      - name: Fetch Submodules and Tags
-        shell: bash
-        run: ci/scripts/util_checkout.sh
-      - name: Free Up Disk Space
-        shell: bash
-        run: ci/scripts/util_cleanup.sh
-      - name: Cache Docker Volumes
-        uses: actions/cache@v1
-        with:
-          path: .docker
-          key: debian-10-rust-${{ hashFiles('rust/**/**.rs', 'rust/**/Cargo.toml') }}
-          restore-keys: debian-10-rust-
-      - name: Setup Python
-        uses: actions/setup-python@v1
-        with:
-          python-version: 3.8
-      - name: Setup Archery
-        run: pip install -e dev/archery[docker]
-      - name: Execute Docker Build
+          # these represent dependencies downloaded by cargo
+          # and thus do not depend on the OS, arch nor rust version.
+          path: /github/home/.cargo
+          key: cargo-cache-
+      - name: Cache Rust dependencies
+        uses: actions/cache@v2
+        with:
+          # these represent compiled steps of both dependencies and arrow
+          # and thus are specific for a particular OS, arch and rust version.
+          path: /github/home/target
+          key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: ${{ matrix.rust }}
+          default: true
+          components: rustfmt
+      - name: Build
         run: |
-          sudo sysctl -w kernel.core_pattern="core.%e.%p"
-          ulimit -c unlimited
-          archery docker run debian-rust
-      - name: Fix Cache Permissions
-        run: sudo chmod -R o+r .docker
-      - name: Docker Push
-        if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
-        continue-on-error: true
-        shell: bash
-        run: archery docker push debian-rust
+          export CARGO_HOME="/github/home/.cargo"
+          export CARGO_TARGET_DIR="/github/home/target"
+          cd rust
+          cargo build
 
-  windows:
-    name: AMD64 Windows 2019 Rust ${{ matrix.rust }}
-    runs-on: windows-latest
-    if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+  # test the crate
+  linux-test:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} test workspace
+    needs: [linux-build-lib]
+    runs-on: ubuntu-latest
     strategy:
-      fail-fast: false
       matrix:
+        arch: [amd64]
         rust: [nightly-2020-11-24]
+    container:
+      image: ${{ matrix.arch }}/rust
+      env:
+        ARROW_TEST_DATA: /__w/arrow/arrow/testing/data

Review comment:
       It is the absolute path to where the container action places the checkout code on. I tried using relative paths but I was unable to make it work.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kszucs commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r536304032



##########
File path: .github/workflows/rust.yml
##########
@@ -21,145 +21,286 @@ on:
   push:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
   pull_request:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
 
-env:
-  DOCKER_BUILDKIT: 0
-  COMPOSE_DOCKER_CLI_BUILD: 1
-  ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
-  ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
-
 jobs:
 
-  debian:
-    name: AMD64 Debian 10 Rust ${{ matrix.rust }}
+  # build the library, a compilation step used by multiple steps below
+  linux-build-lib:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build libraries

Review comment:
       On the other hand we won't be able to execute non-amd64 build on github actions since we would need hosted agents with different arch, so you can also hardcode 'AMD64'.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r536378073



##########
File path: docker-compose.yml
##########
@@ -903,35 +902,6 @@ services:
       /bin/bash -c "
         /arrow/ci/scripts/r_sanitize.sh /arrow"
 
-  ################################ Rust #######################################
-
-  debian-rust:

Review comment:
       The arguments I used to remove it:
   
   * this compose and associated dockerfile are no longer tested under the CI, and their primary purpose was to be used in the CI
   * the dockerfile is not needed, as the default rust image is sufficient
   * the compose contains non-trivial aspects, such as:
      * volume mounting with `delegate`
      * using the no-longer needed `rust_test.sh`
      * non-default `CARGO_*` env variables
   * `docker-compose run debian-rust` is IMO less transparent to a user about what is happening, and the person needs to go to a +1k LOC file to understand.
   
   In opposition, this whole project can now be built using
   
   ```bash
   docker run --rm -v $(pwd)/rust:/rust -it rust /bin/bash -c "cd /rust && cargo build"
   ```
   
   which IMO is really simple to understand and reproduce. Wouldn't this command on the README be sufficient for contributors?
   
   In the end, I evaluated that having these around would be more confusing than just having the user to follow standard rust practices of building a rust project (including using the official docker image), and therefore there was no need for the maintenance burden of having these around.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-737490236


   https://issues.apache.org/jira/browse/ARROW-10792


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-737487996


   @nevi-me , a follow-up of the issue that we were discussing in another PR. IMO this design would enable us to much more easily add new architectures to the CI (we just need to add them to the matrix and would run in parallel).
   
   cc @kszucs 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] codecov-io edited a comment on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-737491170


   # [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=h1) Report
   > Merging [#8821](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=desc) (aa78f32) into [master](https://codecov.io/gh/apache/arrow/commit/322cd01fff055c4d4a03839fe5ac435948e75b09?el=desc) (322cd01) will **decrease** coverage by `31.73%`.
   > The diff coverage is `57.37%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow/pull/8821/graphs/tree.svg?width=650&height=150&src=pr&token=LpTCFbqVT1)](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #8821       +/-   ##
   ===========================================
   - Coverage   84.54%   52.81%   -31.74%     
   ===========================================
     Files         185      165       -20     
     Lines       46176    29844    -16332     
   ===========================================
   - Hits        39041    15761    -23280     
   - Misses       7135    14083     +6948     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [rust/arrow/src/array/transform/boolean.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2Jvb2xlYW4ucnM=) | `76.92% <0.00%> (-23.08%)` | :arrow_down: |
   | [rust/arrow/src/array/transform/list.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2xpc3QucnM=) | `36.36% <0.00%> (-8.09%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/comparison.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2NvbXBhcmlzb24ucnM=) | `96.27% <ø> (ø)` | |
   | [rust/arrow/src/compute/kernels/filter.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2ZpbHRlci5ycw==) | `58.72% <ø> (-2.55%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/limit.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2xpbWl0LnJz) | `100.00% <ø> (ø)` | |
   | [rust/arrow/src/record\_batch.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvcmVjb3JkX2JhdGNoLnJz) | `84.04% <ø> (-6.31%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/builder.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vYnVpbGRlci5ycw==) | `0.00% <0.00%> (-85.10%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/plan.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | `0.00% <ø> (-83.69%)` | :arrow_down: |
   | [...tafusion/src/physical\_plan/distinct\_expressions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Rpc3RpbmN0X2V4cHJlc3Npb25zLnJz) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
   | [rust/datafusion/src/physical\_plan/functions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Z1bmN0aW9ucy5ycw==) | `0.00% <ø> (-82.13%)` | :arrow_down: |
   | ... and [156 more](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=footer). Last update [17805f3...aa78f32](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-739325071


   I rebased it against latest master and placed the c interface as part of the tests, instead of the cronjob, which will allow us to verify that part on every pr, as it should be.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao closed pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao closed pull request #8821:
URL: https://github.com/apache/arrow/pull/8821


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r536518280



##########
File path: .github/workflows/rust.yml
##########
@@ -21,145 +21,286 @@ on:
   push:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
   pull_request:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
 
-env:
-  DOCKER_BUILDKIT: 0
-  COMPOSE_DOCKER_CLI_BUILD: 1
-  ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
-  ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
-
 jobs:
 
-  debian:
-    name: AMD64 Debian 10 Rust ${{ matrix.rust }}
+  # build the library, a compilation step used by multiple steps below
+  linux-build-lib:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build libraries

Review comment:
       That is a good point. Reverted that. :)




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] nevi-me commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
nevi-me commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r534656496



##########
File path: .github/workflows/rust.yml
##########
@@ -22,144 +22,300 @@ on:
     paths:
       - '.github/workflows/rust.yml'
       - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
   pull_request:
     paths:
       - '.github/workflows/rust.yml'
       - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
 
-env:
-  DOCKER_BUILDKIT: 0
-  COMPOSE_DOCKER_CLI_BUILD: 1
-  ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
-  ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
-
 jobs:
 
-  debian:
-    name: AMD64 Debian 10 Rust ${{ matrix.rust }}
+  # build the docker image with rust and necessary dependencies
+  linux-build-image:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build image
     runs-on: ubuntu-latest
-    if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
     strategy:
-      fail-fast: false
       matrix:
-        rust: [nightly-2020-11-24]
+        arch: [amd64]
+        rust: [stable, nightly-2020-11-24]
     env:
-      RUST: ${{ matrix.rust }}
+      REGISTRY: docker.pkg.github.com
+      IMAGE_NAME: docker.pkg.github.com/${{ github.repository }}/${{ matrix.arch }}-linux-rust-${{ matrix.rust }}
     steps:
-      - name: Checkout Arrow
-        uses: actions/checkout@v2
-        with:
-          fetch-depth: 0
-      - name: Fetch Submodules and Tags
+      # based on https://github.com/dtinth/github-actions-docker-layer-caching-poc/blob/master/.github/workflows/dockerimage.yml
+      - uses: actions/checkout@v2
+      - name: Build build image
         shell: bash
-        run: ci/scripts/util_checkout.sh
-      - name: Free Up Disk Space
-        shell: bash
-        run: ci/scripts/util_cleanup.sh
-      - name: Cache Docker Volumes
-        uses: actions/cache@v1
-        with:
-          path: .docker
-          key: debian-10-rust-${{ hashFiles('rust/**/**.rs', 'rust/**/Cargo.toml') }}
-          restore-keys: debian-10-rust-
-      - name: Setup Python
-        uses: actions/setup-python@v1
-        with:
-          python-version: 3.8
-      - name: Setup Archery
-        run: pip install -e dev/archery[docker]
-      - name: Execute Docker Build
         run: |
-          sudo sysctl -w kernel.core_pattern="core.%e.%p"
-          ulimit -c unlimited
-          archery docker run debian-rust
-      - name: Fix Cache Permissions
-        run: sudo chmod -R o+r .docker
-      - name: Docker Push
-        if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
-        continue-on-error: true
-        shell: bash
-        run: archery docker push debian-rust
+          echo ${{ secrets.GITHUB_TOKEN }} | docker login $REGISTRY -u $GITHUB_ACTOR --password-stdin
+          docker pull $IMAGE_NAME || true
+          docker build -f ci/docker/debian-10-rust.dockerfile . -t $IMAGE_NAME --cache-from $IMAGE_NAME
+          docker push $IMAGE_NAME || true
 
-  windows:
-    name: AMD64 Windows 2019 Rust ${{ matrix.rust }}
-    runs-on: windows-latest
-    if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+  # build the library, a compilation step used by multiple steps below
+  linux-build-lib:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build libraries
+    needs: [linux-build-image]
+    runs-on: ubuntu-latest
     strategy:
-      fail-fast: false
       matrix:
+        arch: [amd64]
         rust: [nightly-2020-11-24]
+    container:
+      image: docker.pkg.github.com/${{ github.repository }}/${{ matrix.arch }}-linux-rust-${{ matrix.rust }}
+      credentials:
+        username: ${{ github.actor }}
+        password: ${{ github.token }}
+      env:
+        # so that cache takes it
+        CARGO_HOME: /github/home/.cargo
+        CARGO_TARGET_DIR: /github/home/target/
     steps:
-      - name: Install Rust
-        uses: actions-rs/toolchain@v1
-        with:
-            toolchain: ${{ matrix.rust }}
-            override: true
-            components: rustfmt
-      - name: Install Flatbuffers
-        shell: bash
-        run: choco install flatc
-      - name: Checkout Arrow
-        uses: actions/checkout@v2
+      - uses: actions/checkout@v2
+      - name: Cache Cargo
+        uses: actions/cache@v2
         with:
-          fetch-depth: 0
-      - name: Fetch Submodules and Tags
-        shell: bash
-        run: ci/scripts/util_checkout.sh
-      - name: Cache Build Artifacts
-        uses: actions/cache@v1
+          # these represent dependencies downloaded by cargo
+          # and thus do not depend on the OS, arch nor rust version.
+          path: /github/home/.cargo
+          key: cargo-cache-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: cargo-cache-
+      - name: Cache Rust dependencies
+        uses: actions/cache@v2
         with:
-          path: rust/target
-          key: windows-rust-v1-${{ hashFiles('rust/**/**.rs', 'rust/**/Cargo.toml') }}
-          restore-keys: windows-rust-v1-
+          # these represent compiled steps of both dependencies and arrow
+          # and thus are specific for a particular OS, arch and rust version.
+          path: /github/home/target
+          key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-
       - name: Build
-        shell: bash
-        run: ci/scripts/rust_build.sh $(pwd) $(pwd)/build
-      - name: Test
-        shell: bash
-        run: ci/scripts/rust_test.sh $(pwd) $(pwd)/build
+        run: cd rust && cargo build
 
-  macos:
-    name: AMD64 MacOS 10.15 Rust ${{ matrix.rust }}
-    runs-on: macos-latest
-    if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+  # test the crate
+  linux-test:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} test workspace
+    needs: [linux-build-lib]
+    runs-on: ubuntu-latest
     strategy:
-      fail-fast: false
       matrix:
+        arch: [amd64]
         rust: [nightly-2020-11-24]
+    container:
+      image: docker.pkg.github.com/${{ github.repository }}/${{ matrix.arch }}-linux-rust-${{ matrix.rust }}
+      credentials:
+        username: ${{ github.actor }}
+        password: ${{ github.token }}
+      env:
+        CARGO_HOME: /github/home/.cargo
+        CARGO_TARGET_DIR: /github/home/target
+
+        ARROW_TEST_DATA: /__w/arrow/arrow/testing/data
+        PARQUET_TEST_DATA: /__w/arrow/arrow/cpp/submodules/parquet-testing/data
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+      - name: Cache Cargo
+        uses: actions/cache@v2
+        with:
+          path: /github/home/.cargo
+          # these keys equal the ones on `linux-build-lib` for re-use
+          key: cargo-cache-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: cargo-cache-
+      - name: Cache Rust dependencies
+        uses: actions/cache@v2
+        with:
+          path: /github/home/target
+          # these keys equal the ones on `linux-build-lib` for re-use
+          key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-
+      - name: Run tests
+        run: |
+          cd rust
+          cargo test
+          # test datafusion examples
+          cd datafusion
+          cargo run --example csv_sql
+          cargo run --example parquet_sql
+
+  # test the --features "simd" of the arrow crate
+  linux-test-simd:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} test arrow simd
+    needs: [linux-build-lib]
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        arch: [amd64]
+        rust: [nightly-2020-11-24]
+    container:
+      image: docker.pkg.github.com/${{ github.repository }}/${{ matrix.arch }}-linux-rust-${{ matrix.rust }}
+      credentials:
+        username: ${{ github.actor }}
+        password: ${{ github.token }}
+      env:
+        CARGO_HOME: /github/home/.cargo
+        CARGO_TARGET_DIR: /github/home/target
+
+        ARROW_TEST_DATA: /__w/arrow/arrow/testing/data
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+      - name: Cache Cargo
+        uses: actions/cache@v2
+        with:
+          path: /github/home/.cargo
+          # these keys equal the ones on `linux-build-lib` for re-use
+          key: cargo-cache-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: cargo-cache-
+      - name: Cache Rust dependencies
+        uses: actions/cache@v2
+        with:
+          path: /github/home/target
+          # these keys equal the ones on `linux-build-lib` for re-use
+          key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-
+      - name: Run tests
+        run: |
+          cd rust/arrow
+          cargo test --features "simd"
+
+  # test the arrow crate with stable rust
+  linux-test-stable:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} test arrow
+    needs: [linux-build-image]
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        arch: [amd64]
+        rust: [stable]
+    container:
+      image: docker.pkg.github.com/${{ github.repository }}/${{ matrix.arch }}-linux-rust-${{ matrix.rust }}
+      credentials:
+        username: ${{ github.actor }}
+        password: ${{ github.token }}
+      env:
+        CARGO_HOME: /github/home/.cargo
+        CARGO_TARGET_DIR: /github/home/target
+
+        ARROW_TEST_DATA: /__w/arrow/arrow/testing/data
     steps:
-      - name: Install Rust
-        uses: actions-rs/toolchain@v1
+      - uses: actions/checkout@v2
+        with:
+          submodules: true
+      - name: Cache Cargo
+        uses: actions/cache@v2
+        with:
+          path: /github/home/.cargo
+          # these keys equal the ones on `linux-build-lib` for re-use
+          key: cargo-cache-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: cargo-cache-
+      - name: Cache Rust dependencies
+        uses: actions/cache@v2
+        with:
+          path: /github/home/target
+          # these keys equal the ones on `linux-build-lib` for re-use
+          key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-${{ hashFiles('**/Cargo.lock') }}
+          restore-keys: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-
+      - name: Run tests
+        run: |
+          cd rust/arrow

Review comment:
       @GregBowyer we'd be able to add parquet to stable tests here, so we can leave out adding it to CI in your PR




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kszucs edited a comment on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
kszucs edited a comment on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-738629896


   Thanks Jorge for working on this! 
   
   In general it looks good to me, but this PR makes harder to reproduce the exact same environment locally. Considering that cargo makes the building process straightforward and if it indeed speeds up the build execution by a factor of 2 I can accept that tradeoff. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] alamb commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-740627085


   @jorgecarleitao  -- sounds good (with you writing the note)
   
   The windows C++ test is still failing in a strange way -- I can't seem to find `AMD64 Windows 2019 C++` running on any other (recent) PRs. So it looks to me like it may not run regularly anymore?
   
   ![Screen Shot 2020-12-08 at 8 43 33 AM](https://user-images.githubusercontent.com/490673/101491199-75f36c00-3931-11eb-8d47-2f2f1fcae919.png)
   
   
   https://github.com/apache/arrow/pull/8831/files#r537152984


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-739683711


   cc @alamb and @andygrove , as this will have a (hopefully positive) impact on all our builds.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] alamb commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-740602719


   I retriggered the C++ CI tests on github in hopes it passes on a subsequent run.
   
   @jorgecarleitao I propose we merge this PR in and then send a note to the dev@arrow mailing list with a heads up ("major refactor of rust CI configuration, please let us know if you see issues")  -- I am happy to do so if you would prefer


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kszucs commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r535900056



##########
File path: .github/workflows/rust.yml
##########
@@ -21,145 +21,286 @@ on:
   push:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
   pull_request:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
 
-env:
-  DOCKER_BUILDKIT: 0
-  COMPOSE_DOCKER_CLI_BUILD: 1
-  ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
-  ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
-
 jobs:
 
-  debian:
-    name: AMD64 Debian 10 Rust ${{ matrix.rust }}
+  # build the library, a compilation step used by multiple steps below
+  linux-build-lib:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build libraries

Review comment:
       I'd prefer the previous title to better align with the rest of the builds, e.g.:
   `AMD64 Debian 10 Rust ${{ matrix.rust }} build libraries`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] alamb commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-740176227


   I will try and read this more carefully later this evening  -- it sounds pretty awesome


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] codecov-io edited a comment on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-737491170


   # [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=h1) Report
   > Merging [#8821](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=desc) (c151b60) into [master](https://codecov.io/gh/apache/arrow/commit/322cd01fff055c4d4a03839fe5ac435948e75b09?el=desc) (322cd01) will **decrease** coverage by `7.08%`.
   > The diff coverage is `63.74%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow/pull/8821/graphs/tree.svg?width=650&height=150&src=pr&token=LpTCFbqVT1)](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #8821      +/-   ##
   ==========================================
   - Coverage   84.54%   77.45%   -7.09%     
   ==========================================
     Files         185      166      -19     
     Lines       46176    39218    -6958     
   ==========================================
   - Hits        39041    30378    -8663     
   - Misses       7135     8840    +1705     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [rust/arrow/src/array/transform/boolean.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2Jvb2xlYW4ucnM=) | `76.92% <0.00%> (-23.08%)` | :arrow_down: |
   | [rust/arrow/src/array/transform/list.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2xpc3QucnM=) | `36.36% <0.00%> (-8.09%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/comparison.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2NvbXBhcmlzb24ucnM=) | `96.27% <ø> (ø)` | |
   | [rust/arrow/src/compute/kernels/filter.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2ZpbHRlci5ycw==) | `58.72% <ø> (-2.55%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/limit.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2xpbWl0LnJz) | `100.00% <ø> (ø)` | |
   | [rust/arrow/src/record\_batch.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvcmVjb3JkX2JhdGNoLnJz) | `88.29% <ø> (-2.06%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/builder.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vYnVpbGRlci5ycw==) | `0.00% <0.00%> (-85.10%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/plan.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | `0.00% <ø> (-83.69%)` | :arrow_down: |
   | [...tafusion/src/physical\_plan/distinct\_expressions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Rpc3RpbmN0X2V4cHJlc3Npb25zLnJz) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
   | [rust/datafusion/src/physical\_plan/functions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Z1bmN0aW9ucy5ycw==) | `0.00% <ø> (-82.13%)` | :arrow_down: |
   | ... and [112 more](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=footer). Last update [17805f3...c151b60](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-740634083


   @alamb , these are unrelated and I (unfortunately) see them often whenever I need to change dockerfiles and other CI files. I have been ignoring them, as I expect / hope that people from the respective implementations to be handling them already.
   
   I have been also trying to reduce the number of builds that we trigger when we change rust-specific code, e.g. https://github.com/apache/arrow/pull/8824


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] alamb closed pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
alamb closed pull request #8821:
URL: https://github.com/apache/arrow/pull/8821


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kszucs commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r535900733



##########
File path: .github/workflows/rust.yml
##########
@@ -21,145 +21,286 @@ on:
   push:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
   pull_request:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
 
-env:
-  DOCKER_BUILDKIT: 0
-  COMPOSE_DOCKER_CLI_BUILD: 1
-  ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
-  ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
-
 jobs:
 
-  debian:
-    name: AMD64 Debian 10 Rust ${{ matrix.rust }}
+  # build the library, a compilation step used by multiple steps below
+  linux-build-lib:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build libraries
     runs-on: ubuntu-latest
-    if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
     strategy:
-      fail-fast: false
       matrix:
+        arch: [amd64]
         rust: [nightly-2020-11-24]
-    env:
-      RUST: ${{ matrix.rust }}
+    container:
+      image: ${{ matrix.arch }}/rust
     steps:
-      - name: Checkout Arrow
-        uses: actions/checkout@v2
+      - uses: actions/checkout@v2
+      - name: Cache Cargo
+        uses: actions/cache@v2
         with:
-          fetch-depth: 0
-      - name: Fetch Submodules and Tags
-        shell: bash
-        run: ci/scripts/util_checkout.sh
-      - name: Free Up Disk Space
-        shell: bash
-        run: ci/scripts/util_cleanup.sh
-      - name: Cache Docker Volumes
-        uses: actions/cache@v1
-        with:
-          path: .docker
-          key: debian-10-rust-${{ hashFiles('rust/**/**.rs', 'rust/**/Cargo.toml') }}
-          restore-keys: debian-10-rust-
-      - name: Setup Python
-        uses: actions/setup-python@v1
-        with:
-          python-version: 3.8
-      - name: Setup Archery
-        run: pip install -e dev/archery[docker]
-      - name: Execute Docker Build
+          # these represent dependencies downloaded by cargo
+          # and thus do not depend on the OS, arch nor rust version.
+          path: /github/home/.cargo
+          key: cargo-cache-
+      - name: Cache Rust dependencies
+        uses: actions/cache@v2
+        with:
+          # these represent compiled steps of both dependencies and arrow
+          # and thus are specific for a particular OS, arch and rust version.
+          path: /github/home/target
+          key: ${{ runner.os }}-${{ matrix.arch }}-target-cache-${{ matrix.rust }}-
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: ${{ matrix.rust }}
+          default: true
+          components: rustfmt
+      - name: Build
         run: |
-          sudo sysctl -w kernel.core_pattern="core.%e.%p"
-          ulimit -c unlimited
-          archery docker run debian-rust
-      - name: Fix Cache Permissions
-        run: sudo chmod -R o+r .docker
-      - name: Docker Push
-        if: success() && github.event_name == 'push' && github.repository == 'apache/arrow'
-        continue-on-error: true
-        shell: bash
-        run: archery docker push debian-rust
+          export CARGO_HOME="/github/home/.cargo"
+          export CARGO_TARGET_DIR="/github/home/target"
+          cd rust
+          cargo build
 
-  windows:
-    name: AMD64 Windows 2019 Rust ${{ matrix.rust }}
-    runs-on: windows-latest
-    if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
+  # test the crate
+  linux-test:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} test workspace
+    needs: [linux-build-lib]
+    runs-on: ubuntu-latest
     strategy:
-      fail-fast: false
       matrix:
+        arch: [amd64]
         rust: [nightly-2020-11-24]
+    container:
+      image: ${{ matrix.arch }}/rust
+      env:
+        ARROW_TEST_DATA: /__w/arrow/arrow/testing/data

Review comment:
       What is the `__w` prefix?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kszucs commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r536430639



##########
File path: docker-compose.yml
##########
@@ -903,35 +902,6 @@ services:
       /bin/bash -c "
         /arrow/ci/scripts/r_sanitize.sh /arrow"
 
-  ################################ Rust #######################################
-
-  debian-rust:

Review comment:
       Okay, sounds good to me - lucky us that we have cargo around. 
   
   Please document how to reproduce the builds locally with docker.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kszucs commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r535902088



##########
File path: docker-compose.yml
##########
@@ -903,35 +902,6 @@ services:
       /bin/bash -c "
         /arrow/ci/scripts/r_sanitize.sh /arrow"
 
-  ################################ Rust #######################################
-
-  debian-rust:

Review comment:
       Even if we plan to use plain GHA jobs without docker-compose it'd be nice to keep it around to let contributors execute rust builds without installing the toolchain locally.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kszucs commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
kszucs commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-738629896


   Thanks Jorge for working on this! 
   
   In general it looks good to me, but this PR makes local reproducibility harder. Considering that cargo makes the building process straightforward and if it indeed speeds up the build execution by a factor of 2 I can accept that tradeoff. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] alamb commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-740635592


   Ok, 🚀  it is then!
   
   cc @nevi-me @andygrove @kszucs 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-740619423


   Let's do that. I can write the note, as I can also describe the changes in more depth, as IMO they are relevant to other parts of the project also.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorgecarleitao commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r536296786



##########
File path: .github/workflows/rust.yml
##########
@@ -21,145 +21,286 @@ on:
   push:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
   pull_request:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
 
-env:
-  DOCKER_BUILDKIT: 0
-  COMPOSE_DOCKER_CLI_BUILD: 1
-  ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
-  ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
-
 jobs:
 
-  debian:
-    name: AMD64 Debian 10 Rust ${{ matrix.rust }}
+  # build the library, a compilation step used by multiple steps below
+  linux-build-lib:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build libraries

Review comment:
       Agreed. The issue with hard-coding AMD64 is that it does not depend on `${{ matrix.arch }}`. Would it be ok to be amd64, since that is the name of the image?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] kszucs commented on a change in pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
kszucs commented on a change in pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#discussion_r536304032



##########
File path: .github/workflows/rust.yml
##########
@@ -21,145 +21,286 @@ on:
   push:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
   pull_request:
     paths:
       - '.github/workflows/rust.yml'
-      - 'ci/docker/*rust*'
-      - 'ci/scripts/rust_*.sh'
-      - 'ci/scripts/util_*.sh'
       - 'rust/**'
       - 'format/Flight.proto'
 
-env:
-  DOCKER_BUILDKIT: 0
-  COMPOSE_DOCKER_CLI_BUILD: 1
-  ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
-  ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
-
 jobs:
 
-  debian:
-    name: AMD64 Debian 10 Rust ${{ matrix.rust }}
+  # build the library, a compilation step used by multiple steps below
+  linux-build-lib:
+    name: Linux ${{ matrix.arch }} Rust ${{ matrix.rust }} build libraries

Review comment:
       On the other hand we won't be able to execute non-amd64 build on github actions since we would need hosted agents with different arch (but there aren't any), so you can also hardcode 'AMD64'.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] codecov-io commented on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-737491170


   # [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=h1) Report
   > Merging [#8821](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=desc) (590b58a) into [master](https://codecov.io/gh/apache/arrow/commit/322cd01fff055c4d4a03839fe5ac435948e75b09?el=desc) (322cd01) will **decrease** coverage by `31.77%`.
   > The diff coverage is `59.74%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow/pull/8821/graphs/tree.svg?width=650&height=150&src=pr&token=LpTCFbqVT1)](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #8821       +/-   ##
   ===========================================
   - Coverage   84.54%   52.77%   -31.78%     
   ===========================================
     Files         185      165       -20     
     Lines       46176    29813    -16363     
   ===========================================
   - Hits        39041    15734    -23307     
   - Misses       7135    14079     +6944     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [rust/arrow/src/array/transform/boolean.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2Jvb2xlYW4ucnM=) | `76.92% <0.00%> (-23.08%)` | :arrow_down: |
   | [rust/arrow/src/array/transform/list.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2xpc3QucnM=) | `36.36% <0.00%> (-8.09%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/comparison.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2NvbXBhcmlzb24ucnM=) | `96.27% <ø> (ø)` | |
   | [rust/arrow/src/compute/kernels/filter.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2ZpbHRlci5ycw==) | `58.72% <ø> (-2.55%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/limit.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2xpbWl0LnJz) | `100.00% <ø> (ø)` | |
   | [rust/arrow/src/record\_batch.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvcmVjb3JkX2JhdGNoLnJz) | `84.04% <ø> (-6.31%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/builder.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vYnVpbGRlci5ycw==) | `0.00% <0.00%> (-85.10%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/plan.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | `0.00% <ø> (-83.69%)` | :arrow_down: |
   | [...tafusion/src/physical\_plan/distinct\_expressions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Rpc3RpbmN0X2V4cHJlc3Npb25zLnJz) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
   | [rust/datafusion/src/physical\_plan/functions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Z1bmN0aW9ucy5ycw==) | `0.00% <ø> (-82.13%)` | :arrow_down: |
   | ... and [152 more](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=footer). Last update [17805f3...590b58a](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] codecov-io edited a comment on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-737491170


   # [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=h1) Report
   > Merging [#8821](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=desc) (8b5cb6f) into [master](https://codecov.io/gh/apache/arrow/commit/322cd01fff055c4d4a03839fe5ac435948e75b09?el=desc) (322cd01) will **decrease** coverage by `31.75%`.
   > The diff coverage is `59.60%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow/pull/8821/graphs/tree.svg?width=650&height=150&src=pr&token=LpTCFbqVT1)](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree)
   
   ```diff
   @@             Coverage Diff             @@
   ##           master    #8821       +/-   ##
   ===========================================
   - Coverage   84.54%   52.79%   -31.76%     
   ===========================================
     Files         185      165       -20     
     Lines       46176    29829    -16347     
   ===========================================
   - Hits        39041    15748    -23293     
   - Misses       7135    14081     +6946     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [rust/arrow/src/array/transform/boolean.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2Jvb2xlYW4ucnM=) | `76.92% <0.00%> (-23.08%)` | :arrow_down: |
   | [rust/arrow/src/array/transform/list.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvYXJyYXkvdHJhbnNmb3JtL2xpc3QucnM=) | `36.36% <0.00%> (-8.09%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/comparison.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2NvbXBhcmlzb24ucnM=) | `96.27% <ø> (ø)` | |
   | [rust/arrow/src/compute/kernels/filter.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2ZpbHRlci5ycw==) | `58.72% <ø> (-2.55%)` | :arrow_down: |
   | [rust/arrow/src/compute/kernels/limit.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvY29tcHV0ZS9rZXJuZWxzL2xpbWl0LnJz) | `100.00% <ø> (ø)` | |
   | [rust/arrow/src/record\_batch.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9hcnJvdy9zcmMvcmVjb3JkX2JhdGNoLnJz) | `84.04% <ø> (-6.31%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/builder.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vYnVpbGRlci5ycw==) | `0.00% <0.00%> (-85.10%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/plan.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | `0.00% <ø> (-83.69%)` | :arrow_down: |
   | [...tafusion/src/physical\_plan/distinct\_expressions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Rpc3RpbmN0X2V4cHJlc3Npb25zLnJz) | `0.00% <0.00%> (-90.91%)` | :arrow_down: |
   | [rust/datafusion/src/physical\_plan/functions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Z1bmN0aW9ucy5ycw==) | `0.00% <ø> (-82.13%)` | :arrow_down: |
   | ... and [152 more](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=footer). Last update [17805f3...8b5cb6f](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] codecov-io edited a comment on pull request #8821: ARROW-10792: [Rust] [CI] Modularize builds for faster build and smaller caches

Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #8821:
URL: https://github.com/apache/arrow/pull/8821#issuecomment-737491170


   # [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=h1) Report
   > Merging [#8821](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=desc) (774e704) into [master](https://codecov.io/gh/apache/arrow/commit/1d2b4a55770fa4dbe24959b3b40c745964c3184e?el=desc) (1d2b4a5) will **decrease** coverage by `7.08%`.
   > The diff coverage is `n/a`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/arrow/pull/8821/graphs/tree.svg?width=650&height=150&src=pr&token=LpTCFbqVT1)](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree)
   
   ```diff
   @@            Coverage Diff             @@
   ##           master    #8821      +/-   ##
   ==========================================
   - Coverage   84.48%   77.39%   -7.09%     
   ==========================================
     Files         190      170      -20     
     Lines       46854    39528    -7326     
   ==========================================
   - Hits        39584    30593    -8991     
   - Misses       7270     8935    +1665     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=tree) | Coverage Δ | |
   |---|---|---|
   | [rust/datafusion/src/test/variable.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy90ZXN0L3ZhcmlhYmxlLnJz) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [rust/datafusion/src/optimizer/filter\_push\_down.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9vcHRpbWl6ZXIvZmlsdGVyX3B1c2hfZG93bi5ycw==) | `0.00% <0.00%> (-99.33%)` | :arrow_down: |
   | [rust/datafusion/src/datasource/parquet.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9kYXRhc291cmNlL3BhcnF1ZXQucnM=) | `0.00% <0.00%> (-97.00%)` | :arrow_down: |
   | [rust/datafusion/src/physical\_plan/hash\_join.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2hhc2hfam9pbi5ycw==) | `0.00% <0.00%> (-96.12%)` | :arrow_down: |
   | [...tafusion/src/physical\_plan/datetime\_expressions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2RhdGV0aW1lX2V4cHJlc3Npb25zLnJz) | `0.00% <0.00%> (-95.09%)` | :arrow_down: |
   | [rust/datafusion/src/logical\_plan/display.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9sb2dpY2FsX3BsYW4vZGlzcGxheS5ycw==) | `0.00% <0.00%> (-94.81%)` | :arrow_down: |
   | [...tafusion/src/physical\_plan/distinct\_expressions.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2Rpc3RpbmN0X2V4cHJlc3Npb25zLnJz) | `0.00% <0.00%> (-91.31%)` | :arrow_down: |
   | [rust/datafusion/src/execution/dataframe\_impl.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9leGVjdXRpb24vZGF0YWZyYW1lX2ltcGwucnM=) | `0.00% <0.00%> (-91.25%)` | :arrow_down: |
   | [rust/datafusion/src/physical\_plan/aggregates.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL2FnZ3JlZ2F0ZXMucnM=) | `0.00% <0.00%> (-91.14%)` | :arrow_down: |
   | [rust/datafusion/src/physical\_plan/sort.rs](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree#diff-cnVzdC9kYXRhZnVzaW9uL3NyYy9waHlzaWNhbF9wbGFuL3NvcnQucnM=) | `0.00% <0.00%> (-91.13%)` | :arrow_down: |
   | ... and [78 more](https://codecov.io/gh/apache/arrow/pull/8821/diff?src=pr&el=tree-more) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=continue).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=footer). Last update [1d2b4a5...774e704](https://codecov.io/gh/apache/arrow/pull/8821?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org