[I] Store job event logs in an object store so history survives the scheduler [datafusion-ballista]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #2267:
URL: https://github.com/apache/datafusion-ballista/issues/2267

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   The event log added in #2264 is hard-wired to the local filesystem. `EventLogWriter` (`ballista/history/src/writer.rs`) opens a `tokio::fs::File` per job and appends JSONL to `<event_log_dir>/<job_id>.eventlog`, the reader (`ballista/history/src/reader.rs`) takes a `&Path` and opens it with `std::fs::File::open`, and `HistoryStore::load` (`ballista/scheduler/src/history/mod.rs`) discovers jobs with `std::fs::read_dir`.
   
   That is fine for a single scheduler on a box with a persistent disk, but it is the wrong shape for how Ballista is usually deployed:
   
   - In Kubernetes the scheduler pod's filesystem is ephemeral, so job history dies with the pod unless someone attaches and manages a PVC.
   - The history server has to run somewhere that can see the same directory the scheduler wrote to, which in practice means a shared volume rather than a separately deployed read-only service.
   - With multiple schedulers there is no single directory holding the whole cluster's history.
   
   Writing logs to an object store (S3, GCS, Azure, or anything else `object_store` supports) removes all three constraints: the scheduler ships logs to a durable remote location, and the history server reads them from anywhere.
   
   **Describe the solution you'd like**
   
   Put the `object_store` crate behind the event log's read and write paths, so `--event-log-dir` accepts a URL (`s3://bucket/prefix`, `file:///var/log/ballista`, plain paths staying local for compatibility):
   
   - `EventLogWriter::new` takes an `Arc<dyn ObjectStore>` plus a base `Path` instead of a `PathBuf`.
   - The reader takes a store and a path rather than `&Path`, and `HistoryStore::load` uses `ObjectStore::list` in place of `read_dir`.
   - The history server binary resolves its own store from the same URL, so it can be deployed on its own with no shared volume.
   
   **Describe alternatives you've considered**
   
   - *Keep it local and let operators sync the directory out of band* (sidecar, `aws s3 sync`, PVC + backup). Works, but pushes the durability problem onto every deployment and leaves the history server needing volume access.
   - *Trait-abstract the storage layer ourselves* rather than depending on `object_store`. More code for less coverage; `object_store` is already a workspace dependency, already used by `ballista-scheduler`, and already how the rest of the stack talks to remote storage.
   
   **Additional context**
   
   The main design question is append semantics. Object stores have no append: the current writer holds one open handle per job for the life of the process and appends a line per event. Options, roughly in increasing order of effort:
   
   1. **Buffer per job, write once on `JobEnd`.** Simplest, and it fits how the log is consumed today — the reader only looks at the terminal `JobEnd` record and the timeline records are unread. The cost is that a scheduler crash loses the whole job's log, and memory is held for the duration of long-running jobs.
   2. **Multipart upload per job**, flushing parts as events accumulate. Preserves incremental durability, but needs careful handling of aborted uploads for jobs whose scheduler dies mid-flight.
   3. **Write locally, upload on completion.** Keeps crash-durability on local disk and gets the object into remote storage once the job finishes; needs a cleanup/retry story for uploads that fail.
   
   Worth deciding this before a future UI starts consuming the incremental timeline records, since option 1 rules out live progress from a remote store.
   
   Follow-on from https://github.com/apache/datafusion-ballista/pull/2264#issuecomment-5233620333.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Store job event logs in an object store so history survives the scheduler [datafusion-ballista]

["milenkovicm (via GitHub)" <gi...@apache.org>]

milenkovicm commented on issue #2267:
URL: https://github.com/apache/datafusion-ballista/issues/2267#issuecomment-5480442751

   As its would be very hard to provide one size fit all solution i would suggest slight protocol change which would simplify implementations.
   
   I would suggest small changes to file names for running jobs :
   
   - running jobs to be named: `<event_log_dir>/<job_id>.eventlog.running`  (or some prefix like that)
   - completed jobs, to be renamed from `<event_log_dir>/<job_id>.eventlog.running` to `<event_log_dir>/<job_id>.eventlog`
   
   this would give us ability to listen for file changes in `event_log_dir` and know when job has finished and event log completed. 
   
   user could either listen for a directory changes or scheduler job notification to implement synchronisation any way it suits them
   
   wdyt @andygrove 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org