[PR] [SPARK-59201] Add an end-to-end workflow running the gateway on kind with a real PySpark client [spark-connect-gateway]

["viirya (via GitHub)" <gi...@apache.org>]

viirya opened a new pull request, #20:
URL: https://github.com/apache/spark-connect-gateway/pull/20

   ### What changes were proposed in this pull request?
   
   The `deploy/examples/e2e-smoke` walkthrough exercises the whole deployment path —
   gateway image, Helm chart, Kubernetes Endpoints discovery, session affinity, audit
   and metrics — but only by hand. Nothing in CI covered it, so a break in the chart,
   the Dockerfile or the K8s pool would only surface when someone next ran the
   walkthrough manually.
   
   This adds an `E2E` workflow that automates it:
   
   1. build the gateway image
   2. create a kind cluster, load the image into it
   3. deploy two `apache/spark:4.0.0` Spark Connect servers
   4. install the gateway with the Helm chart + the example values
   5. assert the gateway discovered **both** backends (K8s Endpoints watch)
   6. run the repo's own `test/integration/client_smoke.py` through a port-forward
   7. assert `scg_backend_pool_size`, the `ExecutePlan` counters, and that the audit
      log recorded `ExecutePlan`
   
   `kind` and `helm` are installed with plain `curl` at pinned versions rather than
   third-party actions — both to stay clear of the ASF GitHub Actions allowlist and to
   keep the versions explicit. The only action used is `actions/checkout`. On failure
   the job dumps pod state and gateway/Spark logs; the kind cluster is always deleted.
   
   **Note on where this runs:** it is wired to `pull_request` here so we can measure
   what it actually costs on a hosted runner. Locally the walkthrough takes ~7.5 min
   (image build 5m23s, Spark image pull + pod readiness 54s, the rest seconds); on a
   2-core runner it will be slower. If the measured time turns out too high to gate
   PRs on, the trigger can move to a nightly `schedule` + `workflow_dispatch` instead —
   that is a two-line change.
   
   ### Why are the changes needed?
   
   This is the only coverage of the real deployment path: the Helm chart, the
   Kubernetes Endpoints-watch pool, and session affinity as observed by an actual
   Spark client. The in-process integration tests cover the proxy logic, but nothing
   else verifies that the chart deploys a working gateway.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Ran the entire walkthrough locally before writing the workflow. The PySpark client
   returned correct results including a TempView query — the meaningful check that
   session affinity held, since a TempView lives in one driver's memory and a
   misrouted follow-up RPC would fail it. `scg_backend_pool_size 2`, the RPC counters
   and the audit records were all as expected. This PR's own run exercises the
   workflow on a hosted runner.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Yes, co-authored with Claude Code.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-59224] Add an end-to-end workflow running the gateway on kind with a real PySpark client [spark-connect-gateway]

["viirya (via GitHub)" <gi...@apache.org>]

viirya closed pull request #20: [SPARK-59224] Add an end-to-end workflow running the gateway on kind with a real PySpark client
URL: https://github.com/apache/spark-connect-gateway/pull/20


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-59224] Add an end-to-end workflow running the gateway on kind with a real PySpark client [spark-connect-gateway]

["viirya (via GitHub)" <gi...@apache.org>]

viirya commented on PR #20:
URL: https://github.com/apache/spark-connect-gateway/pull/20#issuecomment-5537263327

   Thanks @dongjoon-hyun !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org