[jira] [Created] (ARROW-13340) [C++][Dataset] Simplify ScanOptions after complexity has moved to ScanNode

["Ben Kietzman (Jira)" <ji...@apache.org>]

Ben Kietzman created ARROW-13340:
------------------------------------

             Summary: [C++][Dataset] Simplify ScanOptions after complexity has moved to ScanNode
                 Key: ARROW-13340
                 URL: https://issues.apache.org/jira/browse/ARROW-13340
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++
            Reporter: Ben Kietzman
             Fix For: 6.0.0


ScanOptions currently has a number of constraints between members, which violates the contract of a public struct:

- {{filter}} must be bound to {{dataset_schema}}
- {{projection}} must be bound to {{dataset_schema}}
- {{projected_schema}} must be {{schema<...fields>}}, where the type of projection is {{struct<...fields>}}

These are currently required to support {{FilterAndProjectScanTask}}, but after ARROW-13328 this complexity can be removed and ScanOptions can be a pure struct argument to {{MakeScanNode}}. Specifically, it should be possible to:

- remove the {{projected_schema}} field (ScanNode doesn't need to know the schemas of any subsequent nodes)
- remove the {{projection}} field (ScanNode doesn't need to know how or if scanned batches will be projected)
- provide a simple vector of {{FieldRef}} to indicate which fields should be materialized (MakeScanNode can validate that this includes every field referenced by {{filter}})
- allow {{filter}} to be unbound (MakeScanNode can bind it to the dataset schema)

{{dataset_schema}} seems slightly redundant too since MakeScanNode also takes a Dataset as an argument but it is currently used by CsvFileFormat to derive column types



--
This message was sent by Atlassian Jira
(v8.3.4#803005)