Re: complex types and ORC

[Owen O'Malley <om...@apache.org>]

On Mon, Apr 29, 2013 at 4:26 PM, Sean McNamara
<Se...@webtrends.com>wrote:

>  If I create a table that has a map<string, string> field, will ORC files
> columnarize by the keys in the map?  Or will all the pairs in the map be
> grouped together?
>

It will break the map keys into one sub-column and the map values into a
separate sub-column. The map keys will end up in a dictionary and the
values will be in a different dictionary. It would be interesting to have a
map encoding that columnarized low cardinality maps, but I haven't heard
anyone thinking about it.

-- Owen


>  My goal is to columnarize the storage of a variable number of fields
> (where the names and number of fields are unknown at schema creation).  It
> does not appear to be possible with RCFILE, and I'm curious if ORC just
> adds better type support, or if they also columnarize the fields within
> complex types.
>
>  Thanks,
>
>  Sean
>