[jira] [Commented] (ARROW-15297) [C++] The write node options shouldn't require a schema

["Vibhatha Lakmal Abeykoon (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-15297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17486211#comment-17486211 ] 

Vibhatha Lakmal Abeykoon commented on ARROW-15297:
--------------------------------------------------

[~westonpace] The modification looks very clear and important. But there are a few contraints assoicated with this modification. 

I am going to think out loud and please correct me if I am wrong. 

One thing is if there is metadata or an state that needs to be consistent within an execution plan, we may face some difficulties in acessing that condition because if we don't handle it per each node in the pipeline such information could be lost. 

So should there be a mechanism like an option associated with each node option to preserve that, or the best should be to decide such and handle it in each node. 

For instance, in the current implementation (before the modification), the schema is directly passed to the write node via the scanner's information. We get the scanner's projected_schema as the output_schema to the write node. But when we use a projection as the Write function is basically scan->filter->project->write, we lose that information and cannot get the schema information from the project accurately. Should we explicitly keep an option in WriteNode to preserve that or should we handle it within the Project node. 

 

> [C++] The write node options shouldn't require a schema
> -------------------------------------------------------
>
>                 Key: ARROW-15297
>                 URL: https://issues.apache.org/jira/browse/ARROW-15297
>             Project: Apache Arrow
>          Issue Type: Sub-task
>          Components: C++
>            Reporter: Weston Pace
>            Assignee: Vibhatha Lakmal Abeykoon
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> The schema should be the output schema of the input node.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)