You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/08/28 06:11:51 UTC
[PR] [SPARK-59074] Add scg-pool-k8s crate [spark-connect-gateway]
viirya opened a new pull request, #13:
URL: https://github.com/apache/spark-connect-gateway/pull/13
### What changes were proposed in this pull request?
Part of importing the Spark Connect Gateway reference implementation into
`apache/spark-connect-gateway` as a series of small, bottom-up PRs (tracked in
the do-not-merge reference PR #1).
This adds the `scg-pool-k8s` crate — a Kubernetes service-watch backend `Pool`.
It watches a `Service`'s `Endpoints` object in a given namespace and exposes the
ready endpoints as the routing backend set, so the gateway's backend membership
tracks the Spark Connect pods dynamically. This is the dynamic counterpart to
the fixed-membership `scg-pool-static`.
`scg-pool-k8s` depends on the `scg-routing` crate whose `Pool` trait it
implements, and pulls in the kube / k8s-openapi client stack. It is imported as
its own PR (rather than grouped) because of that larger dependency surface. It
compiles and tests on its own (unit tests plus a router integration test) and is
appended to the workspace `members` list. It is consumed later by the `gateway`
binary.
### Why are the changes needed?
Implements the reference gateway from the approved SPIP (SPARK-58455), imported
one reviewable piece at a time.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
`cargo fmt --all --check`, `cargo clippy --workspace --all-targets -- -D warnings`,
`cargo build --workspace`, and `cargo test --workspace` all pass locally.
### Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with Claude Code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59074] Add scg-pool-k8s crate [spark-connect-gateway]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #13:
URL: https://github.com/apache/spark-connect-gateway/pull/13#issuecomment-5461125386
Thanks @dongjoon-hyun @HyukjinKwon !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59074] Add scg-pool-k8s crate [spark-connect-gateway]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #13: [SPARK-59074] Add scg-pool-k8s crate
URL: https://github.com/apache/spark-connect-gateway/pull/13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59074] Add scg-pool-k8s crate [spark-connect-gateway]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #13:
URL: https://github.com/apache/spark-connect-gateway/pull/13#issuecomment-5461129020
> Could you file a SPARK sub-task under [SPARK-58455](https://issues.apache.org/jira/browse/SPARK-58455) for user-aware placement, or tell me which key to use? I have `OwnerLabelStrategy` written against `scg-routing` and cannot create the issue myself.
@alexandrefimov Could you wait until I finish all code import? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59074] Add scg-pool-k8s crate [spark-connect-gateway]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #13:
URL: https://github.com/apache/spark-connect-gateway/pull/13#issuecomment-5453118623
All code reading, not a run against a cluster.
**A session bound to a pod that leaves `Endpoints`.** `resolve_session_detailed` returns the stored binding before it consults members (`crates/routing/src/lib.rs:479`), and `stickiness_holds_across_membership_changes` pins that: the session keeps resolving to `a:1` after membership becomes `x:1, y:1, z:1`. With the static pool that needs a config change; here membership changes when a pod dies, so the binding outlives the backend. For Spark Connect that may be right — the session state died with the pod, and re-routing would hand the client a different session under the same id — so the question is whether it is deliberate: forget a binding whose address is no longer a member, or let the RPC fail?
**`Delete` empties the pool.** A Service deleted for good deserves that; a recreate or an operator resync empties it while the pods are healthy. `mark_unhealthy` refuses to evict optimistically so it does not race the watcher — does the same argument make "keep the last membership and log" the better default here?
**`warn!("k8s pool: watcher stream ended")` cannot fire.** `watcher()` is a `stream::unfold` whose closure always returns `Some`, so the loop never exits; errors arrive as `Err` items instead. For a permanent failure — RBAC revoked, namespace deleted — that means an endless recover-and-warn loop with the pool frozen at its last membership and `scg_backend_pool_size` still reporting it. Worth surfacing as something other than a log line?
**Doc note.** "The watcher fully replaces the Vec on every event" holds because the field selector pins one object. It stops holding for the `EndpointSlice` source mentioned above: a Service's endpoints are spread over several slices, so each `Apply` would clobber the others.
Could you file a SPARK sub-task under SPARK-58455 for user-aware placement, or tell me which key to use? I have `OwnerLabelStrategy` written against `scg-routing` and cannot create the issue myself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59074] Add scg-pool-k8s crate [spark-connect-gateway]
Posted by "alexandrefimov (via GitHub)" <gi...@apache.org>.
alexandrefimov commented on PR #13:
URL: https://github.com/apache/spark-connect-gateway/pull/13#issuecomment-5461169265
> > Could you file a SPARK sub-task under [SPARK-58455](https://issues.apache.org/jira/browse/SPARK-58455) for user-aware placement, or tell me which key to use? I have `OwnerLabelStrategy` written against `scg-routing` and cannot create the issue myself.
>
> @alexandrefimov Could you wait until I finish all code import? Thanks!
Sure, my bad for the early ping
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org