You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by Yijie Shen <he...@gmail.com> on 2021/11/12 09:49:21 UTC

[DISCUSS][Rust][DataFusion] Introduce memory manager for query execution

Hi,

I propose adding memory management capabilities to DataFusion, expanding
the scale of data it can handle, and increasing its applicable scenarios.

Here's the design doc [1] available for comments, and an experimental PR
[2] illustrates the usage of the proposed API through an ExternalSorter
example.

I want to raise it in this forum for a broader audience and more valuable
suggestions.

[1]
https://docs.google.com/document/d/1BT5HH-2sKq-Jxo51PNE6l9NNd_F-FyyYcyC3SKTnkIA/edit
[2] https://github.com/yjshen/arrow-datafusion/pull/3


Best,
Yijie

Re: [DISCUSS][Rust][DataFusion] Introduce memory manager for query execution

Posted by Andrew Lamb <al...@influxdata.com>.

Thank you for doing this Yijie,

I encourage everyone else to have a look too -- there are some great ideas
in both the document and the PR

Andrew



On Fri, Nov 12, 2021 at 4:49 AM Yijie Shen <he...@gmail.com>
wrote:

> Hi,
>
> I propose adding memory management capabilities to DataFusion, expanding
> the scale of data it can handle, and increasing its applicable scenarios.
>
> Here's the design doc [1] available for comments, and an experimental PR
> [2] illustrates the usage of the proposed API through an ExternalSorter
> example.
>
> I want to raise it in this forum for a broader audience and more valuable
> suggestions.
>
> [1]
>
> https://docs.google.com/document/d/1BT5HH-2sKq-Jxo51PNE6l9NNd_F-FyyYcyC3SKTnkIA/edit
> [2] https://github.com/yjshen/arrow-datafusion/pull/3
>
>
> Best,
> Yijie
>