[jira] [Created] (ARROW-17959) [C++][Dataset]

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

Joris Van den Bossche created ARROW-17959:
---------------------------------------------

             Summary: [C++][Dataset] 
                 Key: ARROW-17959
                 URL: https://issues.apache.org/jira/browse/ARROW-17959
             Project: Apache Arrow
          Issue Type: Improvement
          Components: C++
            Reporter: Joris Van den Bossche


Currently, when reading a subfield of a nested column of a Parquet file using the Dataset API, we read the full parent column instead of only the requested field. This should be optimized to only read the field itself.

This was left as a TODO in ARROW-14658 (https://github.com/apache/arrow/pull/11704) which added the initial support for nested field refs in dataset scanning (https://github.com/apache/arrow/blob/c29ca51f44eaf41c3a2f6f72e3e23a7b428211c2/cpp/src/arrow/dataset/file_parquet.cc#L240-L246):

{code}
  if (field) {
    // TODO(ARROW-1888): support fine-grained column projection. We should be
    // able to materialize only the child fields requested, and not the entire
    // top-level field.
    // Right now, if enabled, projection/filtering will fail when they cast the
    // physical schema to the dataset schema.
    AddColumnIndices(*toplevel, columns_selection);
{code}

Some relevant comments at https://github.com/apache/arrow/pull/11704#discussion_r749733765. ARROW-1888 was mentioned as a blocker back then, but this is resolved in the meantime.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)