You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "avantgardnerio (via GitHub)" <gi...@apache.org> on 2026/08/12 18:38:44 UTC
[PR] fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 [datafusion-ballista]
avantgardnerio opened a new pull request, #2289:
URL: https://github.com/apache/datafusion-ballista/pull/2289
Two fixes to the Kubernetes deployment guide. Docs only, no code changes.
## 1. Executors dialed the /readyz-gated Service, so the example cluster could not bootstrap
The executor Deployment was configured with `--scheduler-host=ballista-scheduler`,
the client-facing Service, which respects `/readyz`. The scheduler's `/readyz`
returns 503 until at least `--min-ready-executors` (default 1) executors have
registered, so on a cold start that Service has no ready Endpoints, executors
cannot reach the scheduler to register, and the scheduler never becomes ready.
`ballista-scheduler-registration` is already defined in the same manifest for
exactly this purpose, with `publishNotReadyAddresses: true`, and the file's own
comment describes the chicken-and-egg it exists to prevent. Nothing dialed it.
#2088 introduced the `/readyz` gate and readinessProbe; #2245 added the
registration Service to solve the resulting bootstrap problem but did not
repoint the executor at it. So the remedy already shipped, it just was not
connected.
## 2. The scheduler Deployment needs `maxUnavailable: 1`
The guide recommended `maxSurge: 25% / maxUnavailable: 0` for both Deployments.
That is right for the executors, where it preserves capacity through the roll,
and it deadlocks the scheduler.
A new scheduler pod is not Ready until an executor registers with it, but
running executors hold established connections to the pod being replaced and
have no reason to re-register. With `maxUnavailable: 0` Kubernetes cannot retire
that outgoing pod until the incoming one is Ready, so the rollout stalls and the
cluster keeps serving the old image with nothing reported as failed. The default
policy behaves the same way, since 25% of 1 replica rounds down to 0.
`maxUnavailable: 1` lets the outgoing pod go first. Its executors lose their
connection, re-register through `ballista-scheduler-registration`, and the
incoming pod reaches Ready. The cost is a short window with no scheduler, so
queries in flight fail.
This is not equivalent to `type: Recreate`: with `replicas: 3` Kubernetes still
rolls one scheduler at a time and keeps the other two serving, so concurrent
schedulers remain possible.
Both strategies are now in the example manifest rather than only in prose.
## Validation
Fix 2 is confirmed on a 32-executor EKS cluster (Kubernetes 1.34, TPC-H SF1000).
Rolling the scheduler to a different image with the default policy stalled on
"1 old replicas are pending termination" until timeout while the old pod kept
serving. With `maxUnavailable: 1` the same change rolled cleanly twice,
reporting `0 of 1 updated replicas are available` and then success, with all 32
executors re-registering against the new pod.
Fix 1 is reasoned rather than reproduced: a Service with no ready Endpoints
cannot be dialed. It also matches the wiring our working cluster uses, which is
how the discrepancy surfaced.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 [datafusion-ballista]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on PR #2289:
URL: https://github.com/apache/datafusion-ballista/pull/2289#issuecomment-5272428573
@avantgardnerio makes sense 👍 My only comment would be the reference to multi-scheduler deployments mentioned in the description, which isn't really supported on upstream right now, but manifest changes look good.
Slightly off topic, but how would the community feel about publishing an official Helm chart for deploying Ballista clusters? It would make it much easier to provide a reference template that users could track and easily upgrade to pull in new fixes/features/defaults. Now they have to poll the docs + hand roll their own manifests, which can be painful. I've just recently configured a fully automated Artifact Hub pipeline for the Apache Superset Kubernetes Operator chart, complete with automatic `helm-docs` based README rendering and `helm-unittest` based tests to ensure the chart renders correctly (see Artifact Hub page [here](https://artifacthub.io/packages/helm/superset-kubernetes-operator/superset-operator)). I'd be happy to build the same here if there's interest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 [datafusion-ballista]
Posted by "villebro (via GitHub)" <gi...@apache.org>.
villebro commented on PR #2289:
URL: https://github.com/apache/datafusion-ballista/pull/2289#issuecomment-5272772745
@milenkovicm I'm ok either way, I just feel we should centralize it somewhere. Right now we're in practice maintaining in-docs reference manifests, which run the same risk of going stale, but are less convenient to use.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 [datafusion-ballista]
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio commented on PR #2289:
URL: https://github.com/apache/datafusion-ballista/pull/2289#issuecomment-5271251484
@phillipleblanc FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 [datafusion-ballista]
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio merged PR #2289:
URL: https://github.com/apache/datafusion-ballista/pull/2289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 [datafusion-ballista]
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio commented on PR #2289:
URL: https://github.com/apache/datafusion-ballista/pull/2289#issuecomment-5271247171
@villebro I am not a devops expert - I'd be interested in your thoughts on this approach. @andygrove do we have an issue for the return of HA backing stores?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 [datafusion-ballista]
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio commented on PR #2289:
URL: https://github.com/apache/datafusion-ballista/pull/2289#issuecomment-5272497973
> feel about publishing an official Helm chart
I would feel about 500% better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(docs): k8s executors dial the registration Service; scheduler needs maxUnavailable: 1 [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #2289:
URL: https://github.com/apache/datafusion-ballista/pull/2289#issuecomment-5272594043
I would not support helm chart here as I removed last one as nobody maintained them, we do not have to provide everything, it's just few of us maintaining code.
I would suggest if we want to provide helms to provide them as part of datafusion contrib project we create
Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org