[Rust] Async record batch reader?

[Max Burke <ma...@urbanlogiq.com>]

Out of curiosity, is anyone working on a record batch reader that's async
friendly?

Wanting to know if it's something I could wait on/help out with, or if it's
something we could start working on too.

-- 
-Max

Re: [Rust] Async record batch reader?

[Andrew Lamb <al...@influxdata.com>]

A (very belated) follow up to this thread --  there is now a POC PR from
Raphael Taylor-Davies with an async Arrow reader for parquet files [1] if
anyone is interested.

Andrew

[1] https://github.com/apache/arrow-rs/pull/1154#

On Wed, Aug 26, 2020 at 12:52 PM Andy Grove <an...@gmail.com> wrote:

> Hi Max,
>
> I have been experimenting with an async record batch reader and was able to
> get a working version, but I had to use channels to communicate with the
> parquet reader, which ran on its own thread.
>
> I have taken a step back now that I have some experience of this and look
> forward to working with others on a design for async. I know that there are
> some other contributors who are interested in this as well.
>
> I can take an action to start a Google doc where we can discuss this. I
> think this will be useful while we all align on what problems we are trying
> to solve and then we can take the findings back to JIRA. I'll create a doc
> and reply to this thread in the next few days.
>
> Here's the JIRA issue that Mahmut was referring to:
>
> https://issues.apache.org/jira/browse/ARROW-9275
>
> There are some related issues as well, but without much detail:
>
> https://issues.apache.org/jira/browse/ARROW-9674
> https://issues.apache.org/jira/browse/ARROW-9754
>
> Thanks,
>
> Andy.
>
>
> On Wed, Aug 26, 2020 at 10:31 AM Vertexclique <ve...@gmail.com>
> wrote:
>
> > Hi Max;
> >
> > There is an open issue in the tracker which needs to gather feedback to
> > finalize how we will do overall async interface which spans to arrow
> > crates. Please check that issue, it is mentioning sans IO and several
> > design considerations. Imo we can carry async discussion under it.
> >
> > Best,
> > Mahmut
> >
> > On Aug 26, 2020, 19:18, at 19:18, Max Burke <ma...@urbanlogiq.com> wrote:
> > >Out of curiosity, is anyone working on a record batch reader that's
> > >async
> > >friendly?
> > >
> > >Wanting to know if it's something I could wait on/help out with, or if
> > >it's
> > >something we could start working on too.
> > >
> > >--
> > >-Max
> >
>

Re: [Rust] Async record batch reader?

[Andy Grove <an...@gmail.com>]

Hi Max,

I have been experimenting with an async record batch reader and was able to
get a working version, but I had to use channels to communicate with the
parquet reader, which ran on its own thread.

I have taken a step back now that I have some experience of this and look
forward to working with others on a design for async. I know that there are
some other contributors who are interested in this as well.

I can take an action to start a Google doc where we can discuss this. I
think this will be useful while we all align on what problems we are trying
to solve and then we can take the findings back to JIRA. I'll create a doc
and reply to this thread in the next few days.

Here's the JIRA issue that Mahmut was referring to:

https://issues.apache.org/jira/browse/ARROW-9275

There are some related issues as well, but without much detail:

https://issues.apache.org/jira/browse/ARROW-9674
https://issues.apache.org/jira/browse/ARROW-9754

Thanks,

Andy.


On Wed, Aug 26, 2020 at 10:31 AM Vertexclique <ve...@gmail.com>
wrote:

> Hi Max;
>
> There is an open issue in the tracker which needs to gather feedback to
> finalize how we will do overall async interface which spans to arrow
> crates. Please check that issue, it is mentioning sans IO and several
> design considerations. Imo we can carry async discussion under it.
>
> Best,
> Mahmut
>
> On Aug 26, 2020, 19:18, at 19:18, Max Burke <ma...@urbanlogiq.com> wrote:
> >Out of curiosity, is anyone working on a record batch reader that's
> >async
> >friendly?
> >
> >Wanting to know if it's something I could wait on/help out with, or if
> >it's
> >something we could start working on too.
> >
> >--
> >-Max
>

Re: [Rust] Async record batch reader?

[Vertexclique <ve...@gmail.com>]

Hi Max;

There is an open issue in the tracker which needs to gather feedback to finalize how we will do overall async interface which spans to arrow crates. Please check that issue, it is mentioning sans IO and several design considerations. Imo we can carry async discussion under it.

Best,
Mahmut

On Aug 26, 2020, 19:18, at 19:18, Max Burke <ma...@urbanlogiq.com> wrote:
>Out of curiosity, is anyone working on a record batch reader that's
>async
>friendly?
>
>Wanting to know if it's something I could wait on/help out with, or if
>it's
>something we could start working on too.
>
>-- 
>-Max