[jira] [Updated] (SPARK-58113) Support publishNotReadyAddresses in driver service

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-58113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-58113:
----------------------------------
    Summary: Support publishNotReadyAddresses in driver service  (was: [K8S] Support publishNotReadyAddresses in driver service)

> Support publishNotReadyAddresses in driver service
> --------------------------------------------------
>
>                 Key: SPARK-58113
>                 URL: https://issues.apache.org/jira/browse/SPARK-58113
>             Project: Spark
>          Issue Type: Improvement
>          Components: Kubernetes
>    Affects Versions: 4.3.0
>            Reporter: Jiwon Park
>            Assignee: Jiwon Park
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> {{DriverServiceFeatureStep}} creates the driver service as headless ({{clusterIP: None}}). Kubernetes publishes DNS records for a headless service only for *Ready* pods — the EndpointSlice controller computes endpoint readiness as {{service.Spec.PublishNotReadyAddresses || (serving && !terminating)}}, and the legacy {{tolerate-unready-endpoints}} annotation is not honored by it.
> Users commonly attach a readiness probe to the driver pod through {{spark.kubernetes.driver.podTemplateFile}} — a typical setup for long-running Spark Connect servers, where pod readiness gates traffic routing. Probes are not Spark-managed fields, so they pass through to the driver pod. When such a probe gates on a port that only binds late in startup — e.g. the Spark Connect gRPC port, which binds after {{SparkContext}} initialization completes — a circular dependency arises:
> executors need the driver service DNS to register -> DNS requires the driver pod to be Ready -> Ready requires the probe to pass -> the probe requires initialization to complete -> initialization keeps replacing executors that die with {{UnknownHostException}}.
> SPARK-32975 already acknowledges this coupling: {{ExecutorPodsAllocator.start()}} blocks on {{waitUntilReady}} with the comment _"as the headless service won't be resolvable by DNS until the driver pod is ready"_. However, that wait runs *inside* {{SparkContext}} initialization, while a probe on a post-initialization port can only pass *after* initialization completes — so for this class of deployment the wait can never succeed, and raising {{spark.kubernetes.allocation.driver.readinessTimeout}} (the guidance in SPARK-49079) only burns the full timeout before the race resumes unchanged.
> The failure is fatal with static allocation: {{maxNumExecutorFailures = max(3, 2 * spark.executor.instances)}} is 3 for a single executor, and executor churn during the NotReady window (one failure every ~5s) exceeds it before the 30s {{maxRegisteredResourcesWaitingTime}} escape. The driver exits with code 11 ({{EXCEED_MAX_EXECUTOR_FAILURES}}) on every attempt, so the application can never start. Observed in production; trivially reproducible with a driver readiness probe on the Connect port plus {{spark.executor.instances=1}}.
> *Proposal*: add {{spark.kubernetes.driver.service.publishNotReadyAddresses}} (boolean, default {{false}}, preserving current behavior). When enabled, {{DriverServiceFeatureStep}} sets the corresponding field on the driver service spec, so DNS records exist from pod creation and executors register over the already-bound driver RPC port. This also makes the SPARK-32975 wait unnecessary for such deployments.
> The driver service is a single-pod discovery endpoint (executor->driver RPC, block manager, UI, Spark Connect since SPARK-52193), not a load-balancing service; publishing not-ready addresses is the standard Kubernetes pattern for headless discovery services (cf. StatefulSet peer discovery). This follows the existing precedent of exposing driver service spec fields as configurations (SPARK-39490: {{ipFamilyPolicy}}/{{ipFamilies}}; {{deleteOnTermination}}).
> There is no first-class knob for this today; workarounds require excluding {{DriverServiceFeatureStep}} via {{spark.kubernetes.driver.pod.excludedFeatureSteps}} and re-implementing the service (including the {{spark.driver.host}} wiring) in a custom feature step, or cluster-level admission webhooks.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org