You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@iceberg.apache.org by ga...@apache.org on 2026/06/29 03:42:52 UTC
(iceberg-cpp) branch main updated: ci: key Windows sccache cache on MSVC version (#787)
This is an automated email from the ASF dual-hosted git repository.
wgtmac pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/iceberg-cpp.git
The following commit(s) were added to refs/heads/main by this push:
new 10c08b27 ci: key Windows sccache cache on MSVC version (#787)
10c08b27 is described below
commit 10c08b27bd58ce9f6956a9c8f87e07a7a8360fa6
Author: Abanoub Doss <ab...@gmail.com>
AuthorDate: Sun Jun 28 22:42:47 2026 -0500
ci: key Windows sccache cache on MSVC version (#787)
## What
Add the `cl.exe` version to the Windows sccache cache key (and its
restore-keys prefix) in `test` and `sql_catalog_test`, so each MSVC
build keeps its own cache. On the `sql_catalog_test` matrix the suffix
stays empty for the non-Windows legs, so their keys don't change.
## Why
sccache decides whether a cached object is still valid from the compiler
binary. GitHub patches the Windows runner image regularly, roughly once
a week, and rolls each update out across the hosted fleet over a few
days rather than all at once. During that window back-to-back runs can
land on different `cl.exe` builds. When that happens the two builds
share one cache key and keep evicting each other, and a run on a newer
compiler restores the old cache, misses everything, and rebuilds the
whole stack (around 35 to 50 minutes). Putting the `cl.exe` version in
the key gives each compiler its own cache, so a run stays warm instead
of recompiling.
## Validation
On the `windows-2025` runner the resolve step read `cl.exe` as
19.51.36248, and the Windows `test` and `sql_catalog_test` builds passed
with the version in the key. The cache is only saved on `main`, so the
warm reuse shows up there rather than on a branch run.
Co-authored-by: Abanoub Doss <ab...@gmail.com>
---
.github/workflows/sql_catalog_test.yml | 26 +++++++++++++++++++++++---
.github/workflows/test.yml | 24 +++++++++++++++++++++---
2 files changed, 44 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/sql_catalog_test.yml b/.github/workflows/sql_catalog_test.yml
index 3227e93e..905b4a06 100644
--- a/.github/workflows/sql_catalog_test.yml
+++ b/.github/workflows/sql_catalog_test.yml
@@ -74,6 +74,26 @@ jobs:
uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756 # v1.13.0
with:
arch: x64
+ # GitHub patches the Windows image roughly weekly via gradual rollouts, so
+ # back-to-back runs can hit different cl.exe builds. sccache keys on the compiler,
+ # so a shared key would miss after each bump; the version keeps caches separate.
+ # Non-Windows legs get an empty suffix, leaving their keys unchanged.
+ - name: Resolve MSVC version for sccache key
+ if: ${{ startsWith(matrix.runs-on, 'windows') }}
+ shell: pwsh
+ run: |
+ $PSNativeCommandUseErrorActionPreference = $false
+ $banner = (cl.exe 2>&1 | Out-String)
+ if ($banner -match 'Version ([\d.]+)') {
+ $suffix = "-$($Matches[1])"
+ } else {
+ # The key is only a cache hint, so degrade to a shared bucket rather than
+ # fail the build; the warning flags that the parse needs fixing.
+ $suffix = '-unknown'
+ Write-Host "::warning::could not parse cl.exe version for sccache key, using '$suffix'; banner was: $banner"
+ }
+ Add-Content -Path $env:GITHUB_ENV -Value "SCCACHE_KEY_SUFFIX=$suffix"
+ Write-Host "SCCACHE_KEY_SUFFIX=$suffix"
- name: Install dependencies on Ubuntu
if: ${{ startsWith(matrix.runs-on, 'ubuntu') }}
shell: bash
@@ -99,9 +119,9 @@ jobs:
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
- key: sccache-sqlcatalog-${{ matrix.runs-on }}-${{ github.run_id }}
+ key: sccache-sqlcatalog-${{ matrix.runs-on }}${{ env.SCCACHE_KEY_SUFFIX }}-${{ github.run_id }}
restore-keys: |
- sccache-sqlcatalog-${{ matrix.runs-on }}-
+ sccache-sqlcatalog-${{ matrix.runs-on }}${{ env.SCCACHE_KEY_SUFFIX }}-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Configure Iceberg
@@ -129,7 +149,7 @@ jobs:
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
- key: sccache-sqlcatalog-${{ matrix.runs-on }}-${{ github.run_id }}
+ key: sccache-sqlcatalog-${{ matrix.runs-on }}${{ env.SCCACHE_KEY_SUFFIX }}-${{ github.run_id }}
- name: Run SQL catalog tests
shell: bash
run: ctest --test-dir build -R '^sql_catalog_test$' --output-on-failure -C ${{ matrix.cmake_build_type }}
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 4c68ebd0..23fcc2d1 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -144,6 +144,24 @@ jobs:
uses: ilammy/msvc-dev-cmd@0b201ec74fa43914dc39ae48a89fd1d8cb592756 # v1.13.0
with:
arch: x64
+ # GitHub patches the Windows image roughly weekly via gradual rollouts, so
+ # back-to-back runs can hit different cl.exe builds. sccache keys on the compiler,
+ # so a shared key would miss after each bump; the version keeps caches separate.
+ - name: Resolve MSVC version for sccache key
+ shell: pwsh
+ run: |
+ $PSNativeCommandUseErrorActionPreference = $false
+ $banner = (cl.exe 2>&1 | Out-String)
+ if ($banner -match 'Version ([\d.]+)') {
+ $msvc = $Matches[1]
+ } else {
+ # The key is only a cache hint, so degrade to a shared bucket rather than
+ # fail the build; the warning flags that the parse needs fixing.
+ $msvc = 'unknown'
+ Write-Host "::warning::could not parse cl.exe version for sccache key, using '$msvc'; banner was: $banner"
+ }
+ Add-Content -Path $env:GITHUB_ENV -Value "MSVC_VER=$msvc"
+ Write-Host "Resolved MSVC_VER=$msvc"
- name: Cache vcpkg packages
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
id: vcpkg-cache
@@ -159,9 +177,9 @@ jobs:
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
- key: sccache-test-windows-${{ github.run_id }}
+ key: sccache-test-windows-${{ env.MSVC_VER }}-${{ github.run_id }}
restore-keys: |
- sccache-test-windows-
+ sccache-test-windows-${{ env.MSVC_VER }}-
- name: Setup sccache
uses: mozilla-actions/sccache-action@9e7fa8a12102821edf02ca5dbea1acd0f89a2696 # v0.0.10
- name: Build Iceberg
@@ -176,7 +194,7 @@ jobs:
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ github.workspace }}/.sccache
- key: sccache-test-windows-${{ github.run_id }}
+ key: sccache-test-windows-${{ env.MSVC_VER }}-${{ github.run_id }}
- name: Build Example
shell: pwsh
run: |