[GitHub] [arrow-ballista] liurenjie1024 commented on issue #30: [Discuss] Ballista Future Direction

[GitBox <gi...@apache.org>]

liurenjie1024 commented on issue #30:
URL: https://github.com/apache/arrow-ballista/issues/30#issuecomment-1274413864

   cc @mingmwang @thinkharderdev 
   Sorry for late reply from a long vacation.
   
   > Some questions are still not clear to me as of now, particularly if we plan to bring the pipeline based scheduling(
   (Morsel-Driven Parallelism) into Ballista, then what's the relationship between Bubble, Stage and Pipeline,
   and how to divide the DAG into Bubbles, how to split the Stage into multiple Pipelines etc.
   
   ##  Bubble and stage
   
   My motivation of dividing dag into bubble comes from two cases:
   
   1. Subplan reuse in complex olap queries, e.g. the spool operator in [this paper](https://dl.acm.org/doi/abs/10.1145/1247480.1247540) which materialize results into disk.
   2. The dataflow api which can persist data explicitly.
   
   From the bubble execution paper you mentioned, it takes into account the resource limitation of query exection, and add vertical cut when performing a bubble.
   
   The relationship between stage and bubble is clearly described by this pic from alibaba's article:
   ![pic](https://miro.medium.com/max/1400/0*KPg8EOFdnihiQgy1.png)
   
   ## Stage and pipeline
   
   In fact in my proposal I didn't mention bringing morsal driven into Ballista. The key problem with morsel driven is that it uses shared state so that each pipeline is paralleled, and it's not practical in distributed execution. Presto uses morsel driven approach to execute each stage, e.g. breaking stage plan fragments into pipelines and executes them in parallel. This way it can implement two level scheduling, e.g. each stage has a parallelism and each pipeline has another parallelism. They claim that this is more flexible, e.g. a query can have more resources when cluster is idle, and less resources when it's busy. I'm not a big fan of this approach and not sure whether it's state of art.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org