You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Yaron Gvili <rt...@hotmail.com> on 2023/02/16 13:33:35 UTC
testing of back-pressure
Hi,
What testing of back-pressure exist in Acero? I'm mostly interested in testing of back-pressure that applies to any ExecNode, but could also learn from more specific testing. If this is not well covered, I'd look into implementing such testing.
Cheers,
Yaron.
Re: testing of back-pressure
Posted by Li Jin <ic...@gmail.com>.
Thanks Weston for the information.
On Thu, Feb 16, 2023 at 1:32 PM Weston Pace <we...@gmail.com> wrote:
> There is a little bit at the end-to-end level. One goal is to be able to
> repartition a very large dataset. This means we read from something bigger
> than memory and then write to it. This workflow is tested in
> `test_write_dataset_with_backpresure` in test_dataset.py in pyarrow.
>
> Then there is a one unit test in plan_test.cc (ExecPlanExecution,
> SinkNodeBackpressure). And of course, there is some testing in the asof
> join test.
>
> The dataset writer and scanner have their own concepts of backpressure and
> these are independently unit tested. However, this is more or less
> external to Acero.
>
> So I think there is certainly room for improvement here.
>
> On Thu, Feb 16, 2023 at 5:34 AM Yaron Gvili <rt...@hotmail.com> wrote:
>
> > Hi,
> >
> > What testing of back-pressure exist in Acero? I'm mostly interested in
> > testing of back-pressure that applies to any ExecNode, but could also
> learn
> > from more specific testing. If this is not well covered, I'd look into
> > implementing such testing.
> >
> >
> > Cheers,
> > Yaron.
> >
>
Re: testing of back-pressure
Posted by Weston Pace <we...@gmail.com>.
There is a little bit at the end-to-end level. One goal is to be able to
repartition a very large dataset. This means we read from something bigger
than memory and then write to it. This workflow is tested in
`test_write_dataset_with_backpresure` in test_dataset.py in pyarrow.
Then there is a one unit test in plan_test.cc (ExecPlanExecution,
SinkNodeBackpressure). And of course, there is some testing in the asof
join test.
The dataset writer and scanner have their own concepts of backpressure and
these are independently unit tested. However, this is more or less
external to Acero.
So I think there is certainly room for improvement here.
On Thu, Feb 16, 2023 at 5:34 AM Yaron Gvili <rt...@hotmail.com> wrote:
> Hi,
>
> What testing of back-pressure exist in Acero? I'm mostly interested in
> testing of back-pressure that applies to any ExecNode, but could also learn
> from more specific testing. If this is not well covered, I'd look into
> implementing such testing.
>
>
> Cheers,
> Yaron.
>