You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by ameet chaubal <am...@yahoo.com> on 2011/12/20 20:08:53 UTC

millions of columns in Hive

Hi all,

is there a limitation in Hive to create a large number of columns, say a few million or so? If one is using MySQL for metastore, where would this limitation come from, if so? 

thanks,
 
Sincerely,


Ameet

Re: millions of columns in Hive

Posted by ameet chaubal <am...@yahoo.com>.

thanks Kirk,
Actually, it saves the columns in a row format in a table "COLUMNS"
So all the columns are stored as records in this table. i.e. for a million row table, there will be a million rows which MySQL should have no problem with, right?

I did see a post regarding the 4000 bytes,but that was for serde definition to map an HBase table with a large number of columns, which required that the serde string would contain a large number of characters.

However, in this case, since the columns would be stored as rows, it ought to create a million rows for this table... I hope ... 

Sincerely,

Ameet

________________________________
 From: Kirk True <ki...@mustardgrain.com>
To: user@hive.apache.org 
Cc: ameet chaubal <am...@yahoo.com> 
Sent: Thursday, December 22, 2011 1:49 PM
Subject: Re: millions of columns in Hive

IIRC, the names and types of the columns are kept in a *_PARAMS table in the metastore. At one point (depending on your DB) this was only ~4000 characters and thus posed a limitation on the number of columns one could use.

I believe there was work to turn some of the *_PARAMS table columns
    into CLOBs (or equivalent).

Kirk

On 12/20/11 11:08 AM, ameet chaubal wrote: 
Hi all,
>
>
>is there a limitation in Hive to create a large number of columns, say a few million or so? If one is using MySQL for metastore, where would this limitation come from, if so? 
>
>thanks,
>  
>Sincerely,
>
>
>Ameet

Re: millions of columns in Hive

Posted by Kirk True <ki...@mustardgrain.com>.

IIRC, the names and types of the columns are kept in a *_PARAMS table in 
the metastore. At one point (depending on your DB) this was only ~4000 
characters and thus posed a limitation on the number of columns one 
could use.

I believe there was work to turn some of the *_PARAMS table columns into 
CLOBs (or equivalent).

Kirk

On 12/20/11 11:08 AM, ameet chaubal wrote:
> Hi all,
>
> is there a limitation in Hive to create a large number of columns, say 
> a few million or so? If one is using MySQL for metastore, where would 
> this limitation come from, if so?
>
> thanks,
> Sincerely,
>
> Ameet