You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "avantgardnerio (via GitHub)" <gi...@apache.org> on 2026/07/17 22:28:14 UTC
[PR] feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes [datafusion-ballista]
avantgardnerio opened a new pull request, #2088:
URL: https://github.com/apache/datafusion-ballista/pull/2088
## Summary
Closes #2071.
Adds a compile-time `BALLISTA_PROTOCOL_VERSION: u32 = 1` in
`ballista_core` that gates executor↔scheduler traffic, plus
Kubernetes-style `/healthz` and `/readyz` probes on both binaries so
live upgrades have something to observe.
**Protocol version**
- New `uint32 ballista_protocol_version = 7` on `ExecutorRegistration`.
- Executor populates it on registration, heartbeat, and `poll_work`.
- Scheduler rejects mismatches (or missing metadata on heartbeat) with
`Status::failed_precondition`, an `info!` log, and increments
`ballista_scheduler_rejected_by_protocol_version_total` (prometheus
feature).
- Executor `/readyz` flips off on RPC failure, and after 5 consecutive
heartbeat failures the executor self-terminates so a mismatched pod
crash-loops until its image is bumped rather than sitting idle.
**Health probes**
- Scheduler: mounted on the existing axum router (same port as gRPC/REST).
`/healthz` is always 200. `/readyz` returns 200 iff at least
`--min-ready-executors` executors are registered (default 1; set to 0
for dev/single-node).
- Executor: new HTTP server on `--bind-health-port` (default `50053`),
since Arrow Flight isn't HTTP. `/healthz` is always 200 — deliberately
independent of scheduler connectivity to prevent restart cascades.
`/readyz` reflects the last heartbeat.
**Rolling upgrade semantics** (documented in `kubernetes.md`)
| scheduler ↔ executor | outcome |
|---|---|
| old ↔ old | works |
| old ↔ new | works (old scheduler ignores the unknown field) |
| new ↔ new | works |
| new ↔ old | rejected — the case the check exists for |
Bump both Deployments' image tags together in the same release; each
rolls independently under its own `maxSurge`, and the scheduler's
`/readyz` gate keeps `Service` traffic on the old scheduler until the
new one has matching-version executors registered.
**Bump policy**: bump `BALLISTA_PROTOCOL_VERSION` only on releases that
change the executor↔scheduler wire format. Most releases won't.
## Test plan
- [x] `cargo test -p ballista-scheduler --lib` (236 pass, incl. 8 new)
- [x] `cargo test -p ballista-executor --lib` (45 pass)
- [x] `cargo clippy --all-targets -p ballista-core -p ballista-executor -p ballista-scheduler` (clean)
- [x] `cargo fmt --all`
- [x] End-to-end smoke against a 2-executor local cluster: all four
`/healthz` and `/readyz` endpoints return 200; scheduler `/readyz`
shows `ready: 2 executors registered`.
New unit tests cover:
- Register rejected on mismatched protocol version
- Heartbeat rejected on mismatched protocol version
- Heartbeat rejected when metadata is missing entirely
- `poll_work` rejected on mismatched protocol version
- `/healthz` always OK
- `/readyz` fails with no executors
- `/readyz` OK when `min_ready_executors = 0`
- `/readyz` OK after an active heartbeat is recorded
## Notes for reviewers
- The zero default of the new proto field naturally means "old
executor" and is rejected once we start at
`BALLISTA_PROTOCOL_VERSION = 1`. This is intentional per the issue's
"no dual-implementation window" design.
- The old scheduler is permissive because it doesn't know the field
exists — so during a rolling upgrade, old-scheduler ↔ new-executor
keeps working. Only new-scheduler ↔ old-executor is rejected.
- Health probes are unconditional: they mount regardless of the
`rest-api` feature or `--disable-rest-api`, since they're
operational, not part of the REST API surface.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes [datafusion-ballista]
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio commented on code in PR #2088:
URL: https://github.com/apache/datafusion-ballista/pull/2088#discussion_r3609461366
##########
ballista/executor/src/executor_process.rs:
##########
@@ -126,6 +128,8 @@ pub struct ExecutorProcessConfig {
pub port: u16,
/// Port for the executor's gRPC service.
pub grpc_port: u16,
+ /// Port for the executor's HTTP health server (`/healthz` and `/readyz`).
+ pub health_port: u16,
Review Comment:
@phillipleblanc , I wasn't aware you (or anyone) was using it that way. Given your use case (should we add this to a persona @andygrove ?) I think you are absolutely correct about architecture, and I was negligent in the design. PTAL at the current state. I have:
1. removed axum from the executor library
2. exposed a boolean is_healthy() method in both scheduler and executor
3. moved axum & the health check server into the executor binary
4. not changed the scheduler (this is the one I'm not sure about)
My thought is that if schedulers are already starting a REST API, it is natural to put the health check endpoint there so as to not require another port binding. However, I can see how this might be a separate concern from the REST API.
Happy to entertain either option...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes [datafusion-ballista]
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio commented on PR #2088:
URL: https://github.com/apache/datafusion-ballista/pull/2088#issuecomment-5008090260
@phillipleblanc FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes [datafusion-ballista]
Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc commented on code in PR #2088:
URL: https://github.com/apache/datafusion-ballista/pull/2088#discussion_r3606748576
##########
ballista/executor/src/executor_process.rs:
##########
@@ -126,6 +128,8 @@ pub struct ExecutorProcessConfig {
pub port: u16,
/// Port for the executor's gRPC service.
pub grpc_port: u16,
+ /// Port for the executor's HTTP health server (`/healthz` and `/readyz`).
+ pub health_port: u16,
Review Comment:
In our project we embed Ballista in our process that already defines health/readiness probes that are specific to our implementation. This would add a second health surface that we can't turn off without removing this code in our fork.
Perhaps similar to the `rest-api` feature flag, we could add a `health-probes` flag that controls this? (or bundle this with the `rest-api` feature flag?) Alternatively a separate option to control whether ballista starts the health service at all that we could disable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes [datafusion-ballista]
Posted by "phillipleblanc (via GitHub)" <gi...@apache.org>.
phillipleblanc commented on code in PR #2088:
URL: https://github.com/apache/datafusion-ballista/pull/2088#discussion_r3609654628
##########
ballista/executor/src/executor_process.rs:
##########
@@ -126,6 +128,8 @@ pub struct ExecutorProcessConfig {
pub port: u16,
/// Port for the executor's gRPC service.
pub grpc_port: u16,
+ /// Port for the executor's HTTP health server (`/healthz` and `/readyz`).
+ pub health_port: u16,
Review Comment:
@avantgardnerio no problem! Originally Ballista was started as a standalone binary process, and it was relatively recently that @milenkovicm started to focus it to be more of a library (which is one of the reasons we decided to go with Ballista rather than something like [`datafusion-distributed`](https://github.com/datafusion-contrib/datafusion-distributed) which is also a library, but has different design goals)
I agree that putting the health APIs on the scheduler rest API make sense. Everything else looks good now, thanks for reworking this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on PR #2088:
URL: https://github.com/apache/datafusion-ballista/pull/2088#issuecomment-5014901143
https://datafusion.apache.org/ballista/contributors-guide/user-personas.html
Persona 3 should cover @phillipleblanc case if im not mistaken
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: add BALLISTA_PROTOCOL_VERSION + k8s health probes [datafusion-ballista]
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio merged PR #2088:
URL: https://github.com/apache/datafusion-ballista/pull/2088
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org