[I] Support rolling upgrades without an outage window [datafusion-ballista]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #2525:
URL: https://github.com/apache/datafusion-ballista/issues/2525

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   Operators want to upgrade a running Ballista deployment without an outage. Today we only partly support that.
   
   What works on `main`:
   
   - Executors can roll without losing queries. On SIGTERM an executor stops taking new tasks, tells the scheduler it's stopping, and finishes its running tasks. The scheduler then re-runs any work whose shuffle output went away with it, and if the last executor disappears it waits a grace period for new ones before failing jobs (#2029).
   - `BALLISTA_PROTOCOL_VERSION` (#2071, #2088) keeps a new scheduler from handing work to old executors, and in the documented Kubernetes setup the client-facing Service only routes to a scheduler that passes `/readyz`. The steps are in the Kubernetes guide under "Rolling both Deployments together".
   
   What doesn't:
   
   - **Replacing the scheduler fails every query in flight.** Scheduler state is in memory only (`ClusterStorage::Memory` is the only backend), so the new scheduler starts empty. The Kubernetes guide says so: "a short window with no scheduler, so queries in flight fail". Running several scheduler replicas keeps new queries flowing, but queries on the replaced scheduler still fail, because replicas don't share state.
   - **Clients aren't part of the upgrade.** Within a major version, clients and schedulers work together in practice. Across a major version nothing guarantees that, because DataFusion's plan encoding and Ballista's client-facing protocol both change between majors (#2367, #2370, #2376). Today a mismatched client is accepted and can get wrong results. #2514 (draft) would make that an error, which means clients and the cluster have to move to a new major together, and any client on the other side gets errors until it catches up.
   
   This is getting more pressing. Users run clusters around the clock (#2071), and now that the Python client is released separately from the Rust crates (#2512), Python users will often be a major version behind the cluster.
   
   **Describe the solution you'd like**
   
   This needs a design, and I don't think we know the answer yet. Some questions it should cover:
   
   - **Scheduler handover.** Can an outgoing scheduler drain, finishing the jobs it has while a new scheduler takes new ones? Or do we need shared job state so another scheduler can pick up a running job? #2349 is adding the public APIs for scheduler failover, which could be a building block. But it expects an incompatible execution graph version to fail with a clear error, so handing over between versions isn't in scope there.
   - **Clients across a major version.** Could a scheduler accept clients from the previous major for a while? That only works if it can decode their plans with their original meaning, which DataFusion doesn't guarantee. The alternatives are running old and new clusters side by side and moving clients over, or some form of version negotiation.
   - **What we promise.** Which surfaces stay compatible across which versions, and for how long. #2261 covers the executor protocol, the REST API, and the event log. Clients of the scheduler's gRPC API would be a new surface.
   
   **Describe alternatives you've considered**
   
   - Upgrade clients and the cluster together and accept an outage window. This is what happens today, and #2514 would make it explicit for major upgrades.
   - Blue/green at the deployment level: stand up a second cluster on the new version, move clients over, then retire the old one. This needs no Ballista changes, but it doubles the cluster for the length of the upgrade and leaves the client cutover to each user.
   
   **Additional context**
   
   The compatibility table under "Rolling both Deployments together" describes the first release with the protocol check ("old scheduler pre-dates the check"). From the next `BALLISTA_PROTOCOL_VERSION` bump on, an old scheduler rejects new executors too, so that table needs updating after 55.0.0.
   
   Related: #2029, #2071, #2088, #2261, #2349, #2512, #2514.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Support rolling upgrades without an outage window [datafusion-ballista]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove commented on issue #2525:
URL: https://github.com/apache/datafusion-ballista/issues/2525#issuecomment-5951023133

   cc @phillipleblanc 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org