[jira] [Created] (ARROW-12255) [Rust] [Ballista] Integrate scheduler with DataFusion

["Andy Grove (Jira)" <ji...@apache.org>]

Andy Grove created ARROW-12255:
----------------------------------

             Summary: [Rust] [Ballista] Integrate scheduler with DataFusion
                 Key: ARROW-12255
                 URL: https://issues.apache.org/jira/browse/ARROW-12255
             Project: Apache Arrow
          Issue Type: New Feature
          Components: Rust - Ballista, Rust - DataFusion
            Reporter: Andy Grove
            Assignee: Andy Grove
             Fix For: 5.0.0


The Ballista scheduler breaks a query down into stages based on changes in partitioning int he plan, where each stage is broken down into tasks that can be executed concurrently.

Rather than trying to run all the partitions at once, Ballista executors process n concurrent tasks at a time and then request new tasks from the scheduler.

This approach would help DataFusion scale better and it would be ideal to use the same scheduler to scale across cores in DataFusion and across nodes in Ballista.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)