[jira] [Created] (ARROW-11606) [Rust] [DataFusion] Need guidance on HashAggregateExec reconstruction

["Andy Grove (Jira)" <ji...@apache.org>]

Andy Grove created ARROW-11606:
----------------------------------

             Summary: [Rust] [DataFusion] Need guidance on HashAggregateExec reconstruction
                 Key: ARROW-11606
                 URL: https://issues.apache.org/jira/browse/ARROW-11606
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Rust - DataFusion
            Reporter: Andy Grove


We have run into an issue in the Ballista project where we are reconstructing the Final and Partial HashAggregateExec operators [1] for distributed execution and we need some guidance.

The Partial HashAggregateExec gets created OK and executes correctly.

However, when we create the Final HashAggregateExec, it is not finding the expected schema in the input operator. The partial exec outputs field names ending with "[sum]" and "[count]" and so on but the final aggregate doesn't seem to be looking for those names.

It is also worth noting that the Final and Partial executors are not connected directly in this usage.

The Partial exec is executed and output streamed to disk.

The Final exec then runs against the output from the Partial exec.

We may need to make changes in DataFusion to allow other crates to support this kind of use case?

 [1] https://github.com/ballista-compute/ballista/pull/491

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)