[jira] [Created] (SPARK-57793) [SparkConnect] Support url path in SparkConnect connection string

["BoYang (Jira)" <ji...@apache.org>]

BoYang created SPARK-57793:
------------------------------

             Summary: [SparkConnect] Support url path in SparkConnect connection string
                 Key: SPARK-57793
                 URL: https://issues.apache.org/jira/browse/SPARK-57793
             Project: Spark
          Issue Type: New Feature
          Components: Connect
    Affects Versions: 4.1.0
            Reporter: BoYang


See [https://github.com/apache/spark/issues/56816]

Current Spark Connect uses connection string like {{{}sc://host1:15002{}}}, which does not support a path in the url, e.g. not support value like {{{}sc://host1/path1:15002{}}}.

In Kubernetes, people can set up an Ingress to expose Spark Connect driver endpoint by a matching url, e.g. {{http://host1/path1}} can be routed to the driver endpoint. Need to update Spark Connect client to support this scenario.

This could be a Spark Client side change, e.g. in Python Spark Connect client, connect to server based on the host name, and intercept GPRC call to avoid path interfering GRPC protocol.

I am working on a prototype, also open to hear feedback / suggestions.

 

Also from [~viirya] in the Github issue discussion:

For an Ingress-routing-hint prototype, a gRPC metadata header (e.g. an x- header the client sets from the URL path) is probably less invasive than changing :path, and Ingress controllers can route on headers.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org