You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "danielhumanmod (via GitHub)" <gi...@apache.org> on 2025/11/20 06:17:52 UTC
[I] Support `EXPLAIN ANALYZE` in Ballista [datafusion-ballista]
danielhumanmod opened a new issue, #1344:
URL: https://github.com/apache/datafusion-ballista/issues/1344
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The initial EXPLAIN support was merged in #1293 , but the EXPLAIN ANALYZE portion is not implemented yet. This issue tracks the remaining work needed to support it.
**Describe the solution you'd like**
The exact design is still TBD, but the general direction is to create a new exec to gather per-executor metrics from scheduler so we can generate a full EXPLAIN ANALYZE output
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `EXPLAIN ANALYZE` in Ballista [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1344:
URL: https://github.com/apache/datafusion-ballista/issues/1344#issuecomment-3641482816
thank you @danielhumanmod for tackling this issue, I think it would bring few new possibilities if implemented .
I'm not sure if we should utilise shuffles as base building block. Let's say we want to implement distributed cache functionality or some kind of job preparation, they may be reading from source which is not a shuffle file.
Would it be possible to keep job submission pending until another job finishes? If we can attach a callback on a "leading" job which would change dependant job status from "pending" to "ready to run" or to "cancel".
I'm just throwing ideas, will have to take a deeper look at this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `EXPLAIN ANALYZE` in Ballista [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1344:
URL: https://github.com/apache/datafusion-ballista/issues/1344#issuecomment-3827939026
Thanks @danielhumanmod will have a look asap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `EXPLAIN ANALYZE` in Ballista [datafusion-ballista]
Posted by "danielhumanmod (via GitHub)" <gi...@apache.org>.
danielhumanmod commented on issue #1344:
URL: https://github.com/apache/datafusion-ballista/issues/1344#issuecomment-3693741643
Thanks for the insight, @milenkovicm — job-level dependency is a really good idea and definitely aligns well with where the scheduler could evolve. It also fits nicely on top of the current stage/shuffle-level dependency logic.
I took a quick look at the implementation, and the main changes I’m thinking about are:
1. If needed, split jobs and build the dependency context in the scheduler (execute_query()).
2. Add job-level dependency tracking in TaskManager for pending jobs.
3. Trigger downstream tasks on JobFinished events.
I’ll start with a small PoC around this, but happy to adjust the direction based on your feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `EXPLAIN ANALYZE` in Ballista [datafusion-ballista]
Posted by "danielhumanmod (via GitHub)" <gi...@apache.org>.
danielhumanmod commented on issue #1344:
URL: https://github.com/apache/datafusion-ballista/issues/1344#issuecomment-3827697831
Hey @milenkovicm , I created a PR to implements the job-level dependency in scheduler, as the phase 1, appreciate any feedback you have!
Once phase 1 completed, I will apply this pattern to `EXPLAIN ANALYZE` in a follow up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `EXPLAIN ANALYZE` in Ballista [datafusion-ballista]
Posted by "danielhumanmod (via GitHub)" <gi...@apache.org>.
danielhumanmod commented on issue #1344:
URL: https://github.com/apache/datafusion-ballista/issues/1344#issuecomment-3640412709
> An alternative would be to add ability to register job completion callback which triggers change of job B once the job A finishes. Trigger can change UnresolvedShuffle to actual Exec (we could use same logic like we use for simple EXPLAIN).
I believe job finish callback would be useful for other cases, I'm not sure at this point how complicated would it be to implement, may have a look this weekend. (https://github.com/apache/datafusion-ballista/pull/1309#issuecomment-3414552929)
Hey @milenkovicm I did some investigation on this approach. The current processing logic for `QueryStageSchedulerEvent::TaskUpdating` even is already behaves like a built-in callback chain:
```
TaskUpdating event
→ update_task_statuses
→ update_task_status
→ update_stage_output_links
→ processing_stages_update
→ resolve_stage
→ remove_unresolved_shuffles
```
This whole chain is tightly coupled, extracting it into a standalone job-completion callback looks non-trivial
But I find `remove_unresolved_shuffles()` is actually the point where `UnresolvedShuffleExec` -> `ShuffleReaderExec`, so another reasonable approach might be adding a lightweight abstraction for controlling how shuffles get resolved. Something like a `ShuffleResolver` that lets us plug in different UnresolvedShuffleExec replace strategies:
```
trait ShuffleResolver {
fn should_resolve(...): bool;
fn resolve(...): Result<Arc<dyn ExecutionPlan>>;
}
```
The flow will looks like:
```
remove_unresolved_shuffles()
↓
resolver_manager.resolve()
↓
select_resolver()
├─ AnalyzeShuffleResolver.should_resolve()?
│ ├─ true → Use this resolver and stop
│ └─ false → continue
│
├─ DefaultShuffleResolver.should_resolve()?
│ └─ true → use this and stop
│
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `EXPLAIN ANALYZE` in Ballista [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1344:
URL: https://github.com/apache/datafusion-ballista/issues/1344#issuecomment-3598676795
Would you be interested to investigate this @danielhumanmod? I got stacked with some other functionality I'd like to add to ballista ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support `EXPLAIN ANALYZE` in Ballista [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm closed issue #1344: Support `EXPLAIN ANALYZE` in Ballista
URL: https://github.com/apache/datafusion-ballista/issues/1344
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org