complex types and ORC

[Sean McNamara <Se...@Webtrends.com>]

If I create a table that has a map<string, string> field, will ORC files columnarize by the keys in the map?  Or will all the pairs in the map be grouped together?

My goal is to columnarize the storage of a variable number of fields (where the names and number of fields are unknown at schema creation).  It does not appear to be possible with RCFILE, and I'm curious if ORC just adds better type support, or if they also columnarize the fields within complex types.

Thanks,

Sean

Re: complex types and ORC

[Owen O'Malley <om...@apache.org>]

On Mon, Apr 29, 2013 at 4:26 PM, Sean McNamara
<Se...@webtrends.com>wrote:

>  If I create a table that has a map<string, string> field, will ORC files
> columnarize by the keys in the map?  Or will all the pairs in the map be
> grouped together?
>

It will break the map keys into one sub-column and the map values into a
separate sub-column. The map keys will end up in a dictionary and the
values will be in a different dictionary. It would be interesting to have a
map encoding that columnarized low cardinality maps, but I haven't heard
anyone thinking about it.

-- Owen


>  My goal is to columnarize the storage of a variable number of fields
> (where the names and number of fields are unknown at schema creation).  It
> does not appear to be possible with RCFILE, and I'm curious if ORC just
> adds better type support, or if they also columnarize the fields within
> complex types.
>
>  Thanks,
>
>  Sean
>