(spark-connect-gateway) branch main updated: [SPARK-59161] Run the Redis-backed integration tests in CI

[vi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-gateway.git


The following commit(s) were added to refs/heads/main by this push:
     new 28a6b19  [SPARK-59161] Run the Redis-backed integration tests in CI
28a6b19 is described below

commit 28a6b19c9c9faf9330919ed7c9450c9701aab571
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Wed Sep 2 13:12:33 2026 -0700

    [SPARK-59161] Run the Redis-backed integration tests in CI
    
    ### What changes were proposed in this pull request?
    
    The Redis-backed integration tests in `scg-store-redis` and `scg-ratelimit` are
    marked `#[ignore = "requires Docker"]`, so `cargo test --workspace` skips them and
    they have never run in CI. That leaves 9 integration tests uncovered:
    
    - **`scg-store-redis`** — the Redis `AffinityStore` round-trips:
      bind/resolve/forget, session TTL expiry, atomic bind-if-absent, the op reverse
      index, and two stores sharing one Redis.
    - **`scg-ratelimit`** — the distributed token bucket: burst admit-then-reject,
      refill after wait, per-tenant isolation, and two replicas sharing a bucket.
    
    These tests bring up their own Redis container through `testcontainers`, and the
    GitHub-hosted runners already provide a Docker daemon, so no `services:` block or
    extra setup is needed. This adds one CI step that runs them explicitly:
    
    ```
    cargo test -p scg-store-redis -p scg-ratelimit -- --ignored
    ```
    
    The `#[ignore]` markers stay, so a local `cargo test` still needs no Docker — CI
    opts in instead.
    
    This lands as a **separate `integration` job** rather than another step in the
    existing one, so that:
    
    - a Docker-related failure reads as its own red check instead of being reported as
      an "fmt / clippy / build / test" failure,
    - it can be re-run on its own (useful for anything container-backed), and
    - later integration suites have a natural home.
    
    It runs in parallel with the existing job.
    
    ### Why are the changes needed?
    
    The Redis backends are what make multi-replica HA work (cross-replica session
    affinity and a shared rate-limit bucket), and none of that behaviour was verified
    by CI. A regression in either would have gone unnoticed.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Ran the tests locally with a Docker daemon available:
    `cargo test -p scg-store-redis -p scg-ratelimit -- --ignored` — 9 passed
    (store-redis 5, ratelimit 4) in about 10 seconds including container startup, so
    the added CI time is small. The rest of the quartet is unchanged. This PR's own
    CI run exercises the new step on a GitHub-hosted runner.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Yes, co-authored with Claude Code.
    
    Closes #17 from viirya/ci-01-redis-integration.
    
    Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
    Signed-off-by: Liang-Chi Hsieh <li...@databricks.com>
---
 .github/workflows/ci.yml | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 101e76c..fffb276 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -67,3 +67,31 @@ jobs:
 
       - name: Test
         run: cargo test --workspace
+
+  integration:
+    name: integration tests (Docker)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+        with:
+          persist-credentials: false
+
+      - name: Setup Rust toolchain
+        uses: ./.github/actions/setup-builder
+        with:
+          rust-version: stable
+
+      - name: Install protoc
+        uses: arduino/setup-protoc@c65c819552d16ad3c9b72d9dfd5ba5237b9c906b # v3.0.0
+        with:
+          version: "29.x"
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - uses: swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
+
+      # The Redis-backed integration tests are `#[ignore]` by default so that a
+      # local `cargo test` needs no Docker. The GitHub-hosted runners do provide
+      # a Docker daemon, and these tests bring up their own Redis container via
+      # testcontainers, so run them explicitly here.
+      - name: Redis integration tests
+        run: cargo test -p scg-store-redis -p scg-ratelimit -- --ignored


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org