[jira] [Commented] (ORC-97) Support column name selection in ReaderOptions

["Chunyang Wen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ORC-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15444533#comment-15444533 ] 

Chunyang Wen commented on ORC-97:
---------------------------------

In Parquet, there is a class called ColumnPath which represented nested column as dot separated string.

I plan to first build a map from column path( dot separated strings) to its id so that users can specify dot separated columns like (a.b.c).

When receiving include name request from ReaderOptions, we can just turn it into its type id and then call includeTypes from ReaderOptions (omalley has commited includeTypes).

The cost is an in-memory data structure of a map, but it simplify the implementation of include_name for nested column names. By the way, we do not need to add any public API and it is compatible.

For struct type: it is easy to understand (we just add a dot separated field  to the column path). For other non-primitive type like map, union, list, we have to make it clear that how to specify them.

struct <m:map<string, primitive_type>>
struct <m:map<string, non_primitive_type>>


> Support column name selection in ReaderOptions
> ----------------------------------------------
>
>                 Key: ORC-97
>                 URL: https://issues.apache.org/jira/browse/ORC-97
>             Project: Orc
>          Issue Type: New Feature
>          Components: C++
>    Affects Versions: 1.2.0
>            Reporter: Chunyang Wen
>            Assignee: Chunyang Wen
>
> After orc-92 patch, column id selection is supported. But actually select sub-type by name is more useful.
> In my project, we use period(.) to separate nested field names.
> <s1:struct<s2:struct<int1: int>>>
> we choose int1 by s1.s2.int1 which will be passed include(std::list<std:string>).
> In my implementation: first I build a map for name and column id, and then   direct the function call to includeTypes. If this is a candidate solution, I will provide a patch for review soon.
> When a sub-type is selected, all his child types should be selected also, as O'Malley pointed out in orc-92.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)