You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by "otterc (via GitHub)" <gi...@apache.org> on 2026/07/20 20:05:52 UTC

[PR] [SPARK-58160] Add Gateway API HTTPRoute/GRPCRoute support for driver endpoints [spark-kubernetes-operator]

otterc opened a new pull request, #763:
URL: https://github.com/apache/spark-kubernetes-operator/pull/763

### What changes were proposed in this pull request?
Adds opt-in native support for Kubernetes Gateway API v1 routes on SparkApplication, parallel to the existing driverServiceIngressList. Two new optional lists on ApplicationSpec — driverHttpRouteList and driverGrpcRouteList — each pair a Service (fronting the driver pod) with an HTTPRoute or GRPCRoute. When the user omits the route rules, a default rule targeting the Service's first port is emitted.

No behavior change when unset. Not tied to a specific Gateway controller — Istio, Envoy Gateway, and Cilium are all GA on HTTPRoute/GRPCRoute v1.

Includes:
- DriverHttpRouteSpec / DriverGrpcRouteSpec CRD types
- DriverGatewayRouteUtils in spark-submission-worker
- Wiring through SparkAppSubmissionWorker / SparkAppResourceSpec
- Helm RBAC for gateway.networking.k8s.io httproutes/grpcroutes
- Regenerated v1 CRD
- Unit tests for default and user-provided rule shapes

### Why are the changes needed?
Kubernetes Gateway API v1 is the successor to Ingress and is GA across major implementations (Istio, Envoy Gateway, Cilium, etc.). Today SparkApplication only supports Ingress via driverServiceIngressList; users on Gateway-based clusters have to create HTTPRoute / GRPCRoute objects out-of-band, which don't share the driver pod's lifecycle and can leak on cleanup. Native support lets the operator emit and GC these routes alongside the driver, on par with the existing Ingress path, and enables exposing Spark Connect (gRPC) endpoints — something Ingress can't cleanly do.

### Does this PR introduce _any_ user-facing change?
Yes. Adds two optional fields on SparkApplication.spec:

- driverHttpRouteList — emits a Service + Gateway API v1 HTTPRoute per entry for the driver.
- driverGrpcRouteList — emits a Service + Gateway API v1 GRPCRoute per entry.

Both are opt-in and parallel to the existing driverServiceIngressList. The emitted resources are owned by the driver pod, so they GC automatically. If a route list is set, httpRouteSpec.parentRefs / grpcRouteSpec.parentRefs and the service/route metadata names are required — otherwise the app moves to Failed with a message naming the offending field. No behavior change when both fields are unset.

### How was this patch tested?
Unit tests and on an internal dev cluster

### Was this patch authored or co-authored using generative AI tooling?
Yes.
Generated-by: Claude Code (Claude Opus 4.7)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58160] Add Gateway API HTTPRoute/GRPCRoute support for driver endpoints [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #763:
URL: https://github.com/apache/spark-kubernetes-operator/pull/763#issuecomment-5055043908

   Congratulations for your first contribution to Apache Spark K8s Operator repository, @otterc .
   
   I assigned SPARK-58160 to you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58160] Add Gateway API HTTPRoute/GRPCRoute support for driver endpoints [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #763:
URL: https://github.com/apache/spark-kubernetes-operator/pull/763#issuecomment-5055068288

   BTW, @otterc , your PR has a different author in your commit. Who is @ceelion ?
   
   <img width="358" height="350" alt="Screenshot 2026-07-22 at 23 10 05" src="https://github.com/user-attachments/assets/ff8cdfa6-de2e-4af3-a4ef-836907c4ddd3" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58160] Add Gateway API HTTPRoute/GRPCRoute support for driver endpoints [spark-kubernetes-operator]

Posted by "otterc (via GitHub)" <gi...@apache.org>.

otterc commented on PR #763:
URL: https://github.com/apache/spark-kubernetes-operator/pull/763#issuecomment-5059081473

   > BTW, @otterc , your PR has a different GitHub author in your commit. Who is @ceelion ?
   > 
   > <img alt="Screenshot 2026-07-22 at 23 10 05" width="358" height="350" src="https://private-user-images.githubusercontent.com/9700541/625501562-ff8cdfa6-de2e-4af3-a4ef-836907c4ddd3.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3ODQ4MTQwNjksIm5iZiI6MTc4NDgxMzc2OSwicGF0aCI6Ii85NzAwNTQxLzYyNTUwMTU2Mi1mZjhjZGZhNi1kZTJlLTRhZjMtYTRlZi04MzY5MDdjNGRkZDMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI2MDcyMyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNjA3MjNUMTMzNjA5WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9ZDkyODQ0OGNkMjgyYTFjYjExNzRlYzU0ZTE1NTA5MjhhY2EyY2IyNTVmMjlmZTc2MTU1MWQ3MDE3NDcyYjUyMCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmcmVzcG9uc2UtY29udGVudC10eXBlPWltYWdlJTJGcG5nIn0.VTH1fEkkgt8Za9C9Q590uR8SrMfaAZawKtfIU02g5-8">
   > For the commit message, you are the author of course
   > 
   > ```
   > $ git log -n1 | head -n3
   > commit 746602722928ce7963d4de37997ea82852682d6b
   > Author: Chandni Singh <cs...@cloudera.com>
   > Date:   Wed Jul 22 23:05:15 2026 -0700
   > 
   > $ git log -n1 | tail -n2
   >     Authored-by: Chandni Singh <cs...@cloudera.com>
   >     Signed-off-by: Dongjoon Hyun <do...@apache.org>
   > ```
   
   @ceelion is my username on Cloudera’s GitHub instance. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58160] Add Gateway API HTTPRoute/GRPCRoute support for driver endpoints [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #763:
URL: https://github.com/apache/spark-kubernetes-operator/pull/763#issuecomment-5063647049

   Got it, @otterc . 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-58160] Add Gateway API HTTPRoute/GRPCRoute support for driver endpoints [spark-kubernetes-operator]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun closed pull request #763: [SPARK-58160] Add Gateway API HTTPRoute/GRPCRoute support for driver endpoints
URL: https://github.com/apache/spark-kubernetes-operator/pull/763


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org