You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "sarutak (via GitHub)" <gi...@apache.org> on 2023/09/27 18:40:51 UTC

[GitHub] [arrow-datafusion] sarutak opened a new pull request, #7678: Improve cache usage in CI

sarutak opened a new pull request, #7678:
URL: https://github.com/apache/arrow-datafusion/pull/7678

   ## Which issue does this PR close?
   
   Closes #7677
   
   ## Rationale for this change
   Making the CI faster accelerate development.
   
   ## What changes are included in this PR?
   This PR includes improvement of cache usage in CI.
   In the current workflow, the critical path seems `cargo check` to `verify benchmark results (amd64)`.So `actions/cache` is used only for the jobs.
   Cached contents are also revised.
   
   ## Are these changes tested?
   Done by CI.
   With my experiment, the total duration of workflow defined by `rust.yml` was about 30 mins.
   https://github.com/sarutak/arrow-datafusion/actions/runs/6329626199
   
   ## Are there any user-facing changes?
   No.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] sarutak commented on a diff in pull request #7678: Improve cache usage in CI

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #7678:
URL: https://github.com/apache/arrow-datafusion/pull/7678#discussion_r1341481055


##########
.github/workflows/rust.yml:
##########
@@ -47,18 +47,24 @@ jobs:
       image: amd64/rust
     steps:
       - uses: actions/checkout@v4
-      - name: Cache Cargo
-        uses: actions/cache@v3
-        with:
-          # these represent dependencies downloaded by cargo
-          # and thus do not depend on the OS, arch nor rust version.
-          path: /github/home/.cargo
-          key: cargo-cache-
       - name: Setup Rust toolchain
         uses: ./.github/actions/setup-builder
         with:
           rust-version: stable
 
+      - name: Cache Cargo
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cargo/bin/
+            ~/.cargo/registry/index/
+            ~/.cargo/registry/cache/
+            ~/.cargo/git/db/
+            ./target/
+            ./datafusion-cli/target/
+          # this key equals the ones on `linux-build-lib` for re-use
+          key: cargo-cache-benchmark-${{ hashFiles('datafusion/**/Cargo.toml', 'benchmarks/Cargo.toml', 'datafusion-cli/Cargo.toml') }}

Review Comment:
   Sorry for the late reply.
   > This cache key doesn't change unless we update the dependencies, right?
   
   Yes.
   
   > What about the local code changes?
   
   Local code change (except for Cargo.toml specified in `hashFiles`) doesn't affect the cache.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] sarutak commented on a diff in pull request #7678: Improve cache usage in CI

Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on code in PR #7678:
URL: https://github.com/apache/arrow-datafusion/pull/7678#discussion_r1341492294


##########
.github/workflows/rust.yml:
##########
@@ -241,9 +229,14 @@ jobs:
       - name: Cache Cargo
         uses: actions/cache@v3
         with:
-          path: /github/home/.cargo
+          path: |
+            ~/.cargo/bin/
+            ~/.cargo/registry/index/
+            ~/.cargo/registry/cache/
+            ~/.cargo/git/db/
+            ./target/
           # this key equals the ones on `linux-build-lib` for re-use
-          key: cargo-cache-
+          key: cargo-cache-benchmark-${{ hashFiles('datafusion/**/Cargo.toml', 'benchmarks/Cargo.toml') }}

Review Comment:
   Yes, it's intended.
   This job (verify benchmark results) seems not to touch `datafusion-cli`, so this cache should not be stale on changing `datafusion-cli/Cargo.toml`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb merged pull request #7678: Improve cache usage in CI

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #7678:
URL: https://github.com/apache/arrow-datafusion/pull/7678


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #7678: Improve cache usage in CI

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7678:
URL: https://github.com/apache/arrow-datafusion/pull/7678#discussion_r1340516995


##########
.github/workflows/rust.yml:
##########
@@ -47,18 +47,24 @@ jobs:
       image: amd64/rust
     steps:
       - uses: actions/checkout@v4
-      - name: Cache Cargo
-        uses: actions/cache@v3
-        with:
-          # these represent dependencies downloaded by cargo
-          # and thus do not depend on the OS, arch nor rust version.
-          path: /github/home/.cargo
-          key: cargo-cache-
       - name: Setup Rust toolchain
         uses: ./.github/actions/setup-builder
         with:
           rust-version: stable
 
+      - name: Cache Cargo
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cargo/bin/
+            ~/.cargo/registry/index/
+            ~/.cargo/registry/cache/
+            ~/.cargo/git/db/
+            ./target/
+            ./datafusion-cli/target/
+          # this key equals the ones on `linux-build-lib` for re-use
+          key: cargo-cache-benchmark-${{ hashFiles('datafusion/**/Cargo.toml', 'benchmarks/Cargo.toml', 'datafusion-cli/Cargo.toml') }}

Review Comment:
   This cache key doesn't change unless we update the dependencies, right? What about the local code changes?



##########
.github/workflows/rust.yml:
##########
@@ -241,9 +229,14 @@ jobs:
       - name: Cache Cargo
         uses: actions/cache@v3
         with:
-          path: /github/home/.cargo
+          path: |
+            ~/.cargo/bin/
+            ~/.cargo/registry/index/
+            ~/.cargo/registry/cache/
+            ~/.cargo/git/db/
+            ./target/
           # this key equals the ones on `linux-build-lib` for re-use
-          key: cargo-cache-
+          key: cargo-cache-benchmark-${{ hashFiles('datafusion/**/Cargo.toml', 'benchmarks/Cargo.toml') }}

Review Comment:
   The key above also includes `'datafusion-cli/Cargo.toml'` -- is in intentional that these keys do not match?
   
   ```
             key: cargo-cache-benchmark-${{ hashFiles('datafusion/**/Cargo.toml', 'benchmarks/Cargo.toml', 'datafusion-cli/Cargo.toml') }}
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org