Re: [Rust] Async record batch reader?

[Andrew Lamb <al...@influxdata.com>]

A (very belated) follow up to this thread --  there is now a POC PR from
Raphael Taylor-Davies with an async Arrow reader for parquet files [1] if
anyone is interested.

Andrew

[1] https://github.com/apache/arrow-rs/pull/1154#

On Wed, Aug 26, 2020 at 12:52 PM Andy Grove <an...@gmail.com> wrote:

> Hi Max,
>
> I have been experimenting with an async record batch reader and was able to
> get a working version, but I had to use channels to communicate with the
> parquet reader, which ran on its own thread.
>
> I have taken a step back now that I have some experience of this and look
> forward to working with others on a design for async. I know that there are
> some other contributors who are interested in this as well.
>
> I can take an action to start a Google doc where we can discuss this. I
> think this will be useful while we all align on what problems we are trying
> to solve and then we can take the findings back to JIRA. I'll create a doc
> and reply to this thread in the next few days.
>
> Here's the JIRA issue that Mahmut was referring to:
>
> https://issues.apache.org/jira/browse/ARROW-9275
>
> There are some related issues as well, but without much detail:
>
> https://issues.apache.org/jira/browse/ARROW-9674
> https://issues.apache.org/jira/browse/ARROW-9754
>
> Thanks,
>
> Andy.
>
>
> On Wed, Aug 26, 2020 at 10:31 AM Vertexclique <ve...@gmail.com>
> wrote:
>
> > Hi Max;
> >
> > There is an open issue in the tracker which needs to gather feedback to
> > finalize how we will do overall async interface which spans to arrow
> > crates. Please check that issue, it is mentioning sans IO and several
> > design considerations. Imo we can carry async discussion under it.
> >
> > Best,
> > Mahmut
> >
> > On Aug 26, 2020, 19:18, at 19:18, Max Burke <ma...@urbanlogiq.com> wrote:
> > >Out of curiosity, is anyone working on a record batch reader that's
> > >async
> > >friendly?
> > >
> > >Wanting to know if it's something I could wait on/help out with, or if
> > >it's
> > >something we could start working on too.
> > >
> > >--
> > >-Max
> >
>