You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Vibhatha Lakmal Abeykoon (Jira)" <ji...@apache.org> on 2022/09/14 16:07:00 UTC
[jira] [Commented] (ARROW-17648) [C++] Add ScanOptions to support projection and filter in ToProto Read
[ https://issues.apache.org/jira/browse/ARROW-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17604816#comment-17604816 ]
Vibhatha Lakmal Abeykoon commented on ARROW-17648:
--------------------------------------------------
[~westonpace]
In the projection part, Substrait uses `MaskExpression` which doesn't support what Acero wants to do with the projection field which is equivalent to `substrait::Expression`. I have a doubt if we can support this feature. Would appreciate your thoughts on this?
> [C++] Add ScanOptions to support projection and filter in ToProto Read
> ----------------------------------------------------------------------
>
> Key: ARROW-17648
> URL: https://issues.apache.org/jira/browse/ARROW-17648
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Vibhatha Lakmal Abeykoon
> Assignee: Vibhatha Lakmal Abeykoon
> Priority: Major
> Labels: query-engine, substrait, substrait-roundtrip
>
> When converting from an Acero plan to a Substrait plan not all scan options are fully supported. The `projection` (partial column selection) and `filter` (pushdown filtering) support needs to be added.
> Ref:
>
> {code:java}
> Result<std::unique_ptr<substrait::ReadRel>> ScanRelationConverter(const std::shared_ptr<Schema>& schema, const compute::Declaration& declaration, ExtensionSet* ext_set, const ConversionOptions& conversion_options) {
> auto read_rel = make_unique<substrait::ReadRel>();
> const auto& scan_node_options = checked_cast<const dataset::ScanNodeOptions&>(*declaration.options);
> auto dataset = dynamic_cast<dataset::FileSystemDataset*>(scan_node_options.dataset.get());
> // TODO handle here{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)