(datafusion-ballista) branch main updated: fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 (#2289)

[av...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

avantgardnerio pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-ballista.git


The following commit(s) were added to refs/heads/main by this push:
     new b6b6a21b7 fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 (#2289)
b6b6a21b7 is described below

commit b6b6a21b78c7deed9a65b2ed7e535ff6aeea55df
Author: Brent Gardner <bg...@squarelabs.net>
AuthorDate: Wed Aug 12 14:35:23 2026 -0600

    fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 (#2289)
    
    * fix(docs): point k8s executors at the scheduler registration Service
    
    The example cluster in kubernetes.md gave executors
    `--scheduler-host=ballista-scheduler`, the client-facing Service, which
    respects /readyz. The scheduler's /readyz returns 503 until at least
    `--min-ready-executors` (default 1) executors have registered, so on a cold
    start that Service has no ready Endpoints, executors cannot reach the
    scheduler to register, and the scheduler never becomes ready. The cluster
    does not converge.
    
    `ballista-scheduler-registration` already exists in the same manifest for
    exactly this purpose, with `publishNotReadyAddresses: true`, and the file's
    own comment describes the chicken-and-egg it is there to prevent. Nothing
    dialed it.
    
    #2088 introduced the /readyz gate and readinessProbe; #2245 added the
    registration Service to solve the resulting bootstrap problem but did not
    repoint the executor at it.
    
    Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
    
    * fix(docs): give the scheduler Deployment maxUnavailable: 1
    
    A new scheduler pod is not Ready until an executor registers with it, but
    running executors keep their connections to the pod it is replacing. With
    `maxUnavailable: 0` Kubernetes cannot retire that outgoing pod until the
    incoming one is Ready, so the rollout stalls and the cluster keeps serving the
    old image with nothing reported as failed. The default policy has the same
    effect, since 25% of 1 replica rounds down to 0.
    
    `maxUnavailable: 1` lets the outgoing pod go first; its executors then
    re-register through `ballista-scheduler-registration` and the incoming pod
    reaches Ready. This is not equivalent to `type: Recreate` -- with replicas > 1
    Kubernetes still rolls one scheduler at a time.
    
    The executor Deployment keeps `maxSurge: 25% / maxUnavailable: 0`, which is
    correct there: it preserves capacity through the roll. Both strategies are now
    in the example manifest rather than only in prose.
    
    Co-Authored-By: Claude Opus 5 (1M context) <no...@anthropic.com>
    
    ---------
    
    Co-authored-by: Claude Opus 5 (1M context) <no...@anthropic.com>
---
 docs/source/user-guide/deployment/kubernetes.md | 52 ++++++++++++++++++++-----
 1 file changed, 43 insertions(+), 9 deletions(-)

diff --git a/docs/source/user-guide/deployment/kubernetes.md b/docs/source/user-guide/deployment/kubernetes.md
index 721f7705c..a1c445e3e 100644
--- a/docs/source/user-guide/deployment/kubernetes.md
+++ b/docs/source/user-guide/deployment/kubernetes.md
@@ -152,6 +152,14 @@ metadata:
   name: ballista-scheduler
 spec:
   replicas: 1
+  # The outgoing pod must be allowed to go first: a new scheduler is not Ready
+  # until an executor registers with it, and the executors stay attached to the
+  # pod it is replacing. See Rolling both Deployments together.
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 1
+      maxUnavailable: 1
   selector:
     matchLabels:
       app: ballista-scheduler
@@ -194,6 +202,13 @@ metadata:
   name: ballista-executor
 spec:
   replicas: 2
+  # Surge new executors before retiring old ones, so capacity stays at roughly
+  # 100% through the roll.
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 25%
+      maxUnavailable: 0
   selector:
     matchLabels:
       app: ballista-executor
@@ -208,7 +223,11 @@ spec:
           image: <your-repo>/datafusion-ballista-executor:latest
           args:
             - "--bind-port=50051"
-            - "--scheduler-host=ballista-scheduler"
+            # Registration Service, not the client-facing one: the client-facing
+            # `ballista-scheduler` respects /readyz, which stays 503 until an
+            # executor registers, so its Endpoints are empty and executors can
+            # never reach it to register in the first place.
+            - "--scheduler-host=ballista-scheduler-registration"
             - "--scheduler-port=50050"
           ports:
             - containerPort: 50051
@@ -319,11 +338,10 @@ The rejection happens only when a _new_ scheduler sees an _old_ executor,
 which is the case the version check is designed to protect against —
 mismatched pairs never exchange work, so they cannot corrupt state.
 
-Give each Deployment a `maxSurge` so Kubernetes brings new pods up before
-tearing old ones down; this keeps executor capacity at roughly 100% during
-the roll:
+The two Deployments want different rollout strategies.
 
 ```yaml
+# executor Deployment: keeps capacity at roughly 100% through the roll
 spec:
   strategy:
     type: RollingUpdate
@@ -332,11 +350,27 @@ spec:
       maxUnavailable: 0
 ```
 
-The scheduler's `/readyz` gate (`--min-ready-executors`, default `1`) means
-a fresh scheduler pod does not receive `Service` traffic until at least one
-matching-version executor has registered with it, so clients keep hitting
-the old scheduler until the new one has real capacity behind it. Set
-`--min-ready-executors=0` if you want the scheduler to advertise readiness
+```yaml
+# scheduler Deployment
+spec:
+  strategy:
+    type: RollingUpdate
+    rollingUpdate:
+      maxSurge: 1
+      maxUnavailable: 1
+```
+
+`maxUnavailable: 1` lets the outgoing scheduler go first, so its executors lose
+their connection and re-register through `ballista-scheduler-registration`,
+which is what makes the incoming pod Ready. With `maxUnavailable: 0` the
+outgoing pod cannot be retired until the incoming one is Ready, and it cannot
+become Ready while the executors are still attached to the pod it is replacing.
+The cost is a short window with no scheduler, so queries in flight fail.
+
+This is not `type: Recreate`. With `replicas: 3` Kubernetes still rolls one
+scheduler at a time and keeps the other two serving.
+
+Set `--min-ready-executors=0` if you want the scheduler to advertise readiness
 immediately (e.g. single-node development clusters).
 
 ## Port Forwarding


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org