You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Weston Pace (Jira)" <ji...@apache.org> on 2021/10/01 11:24:00 UTC

[jira] [Created] (ARROW-14191) [C++][Dataset] Dataset writes should respect backpressure

Weston Pace created ARROW-14191:
-----------------------------------

             Summary: [C++][Dataset] Dataset writes should respect backpressure
                 Key: ARROW-14191
                 URL: https://issues.apache.org/jira/browse/ARROW-14191
             Project: Apache Arrow
          Issue Type: Improvement
            Reporter: Weston Pace
            Assignee: Weston Pace


If the write destination is slow then the dataset writer should back off and apply backpressure to pause the reader.  This will allow simple dataset API scans to operate on large out of core datasets.

This is dependent on ARROW-13611 which adds a backpressure feature for regular scanning (but not writing data) and on ARROW-13542 which moves the dataset write to a node in the exec plan.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)