You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@arrow.apache.org by "Neal Richardson (Jira)" <ji...@apache.org> on 2022/02/01 17:21:00 UTC

[jira] [Created] (ARROW-15517) [R] Use WriteNode in write_dataset()

Neal Richardson created ARROW-15517:
---------------------------------------

             Summary: [R] Use WriteNode in write_dataset()
                 Key: ARROW-15517
                 URL: https://issues.apache.org/jira/browse/ARROW-15517
             Project: Apache Arrow
          Issue Type: Improvement
          Components: R
            Reporter: Neal Richardson
            Assignee: Neal Richardson
             Fix For: 8.0.0


Currently, write_dataset uses the Scanner interface, which can't handle everything that the ExecPlan does. So if your arrow_dplyr_query contains things like aggregations or (more importantly) joins, you have to materialize the Table in memory before you can write to disk. The WriteNode added in ARROW-13542 is a special sink node that can be put at the end of an ExecPlan, so data should be able to stream to disk in more cases, and will benefit from future improvements to ExecPlan memory usage and spillover.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)