You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/09/03 06:23:40 UTC
[PR] [SPARK-59196] Run the graceful-drain and multi-replica HA smoke examples in CI [spark-connect-gateway]
viirya opened a new pull request, #19:
URL: https://github.com/apache/spark-connect-gateway/pull/19
### What changes were proposed in this pull request?
Two binaries under `crates/proxy/examples/` assert end-to-end invariants, but were
only ever *compiled* by `cargo clippy --all-targets` and never executed — so the
behaviour they cover had no CI verification at all:
- **`drain_smoke`** — exercises the two-step drain the gateway performs on SIGTERM
(in-process, triggered through the same channel `shutdown_signal()` uses).
Asserts that `/readyz` flips to not-ready when the drain starts, that in-flight
streams keep receiving messages through the drain, and that the gateway shuts
down cleanly afterwards. **No external dependencies.**
- **`ha_smoke`** — spawns two real `SparkConnectProxy` gateways sharing one Redis
affinity store and one static pool of two fake backends, then verifies three
multi-replica HA properties: shared affinity state across replicas, that
failover preserves an existing session binding, and that the op-id reverse index
resolves across replicas. Needs a Redis (`REDIS_URL`, default
`redis://127.0.0.1:6399`).
Both signal failure by panicking on an assert, so a plain `cargo run` is a
sufficient CI check. This adds them as two steps in the `integration` job, plus a
Redis **service container** on `:6399` for `ha_smoke`.
The Redis *tests* in that job are unaffected — they start their own container
through `testcontainers` and do not use the service container.
The kind/Helm end-to-end walkthroughs under `deploy/examples/` remain out of
scope; they need a much heavier harness and are better suited to a separate
scheduled workflow.
### Why are the changes needed?
Graceful drain and multi-replica HA are both production behaviours — the first is
what makes a rolling restart not drop in-flight queries, the second is what makes
running more than one replica correct. Neither had any automated coverage, even
though the code to verify them was already in the tree.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Locally, with a Redis on `:6399` standing in for the service container, running
all four steps of the job in sequence: the testcontainers Redis tests still pass
(4 + 5), the tracing test passes, and both smoke examples exit 0 — "all drain
invariants passed" and "all HA invariants passed". This PR's own CI run exercises
the new steps and the service container.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59196] Run the graceful-drain and multi-replica HA smoke examples in CI [spark-connect-gateway]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #19: [SPARK-59196] Run the graceful-drain and multi-replica HA smoke examples in CI
URL: https://github.com/apache/spark-connect-gateway/pull/19
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59196] Run the graceful-drain and multi-replica HA smoke examples in CI [spark-connect-gateway]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #19:
URL: https://github.com/apache/spark-connect-gateway/pull/19#issuecomment-5529202236
Thanks @dongjoon-hyun !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org