You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by "Connell, Chuck" <Ch...@nuance.com> on 2012/10/04 17:48:09 UTC

Limit to columns or nesting of Hive table?

I am trying to create a large Hive table, with many columns and deeply nested structs. It is failing with java.lang.ArrayIndexOutOfBoundsException: 10.

Before I spend a lot of time debugging my table declaration, is there some limit here I should know about? Max number of columns? Max depth of struct nesting?

Thanks,
Chuck

RE: Limit to columns or nesting of Hive table?

Posted by "Connell, Chuck" <Ch...@nuance.com>.

The issue apparently is not just the number of levels of nesting. I just created a Hive table with 20 levels of structs within each other. It created fine. This is more levels than the table that was failing for me. The failing table had many more fields throughout the levels.

Chuck

-----Original Message-----
From: Connell, Chuck [mailto:Chuck.Connell@nuance.com] 
Sent: Thursday, October 04, 2012 12:09 PM
To: user@hive.apache.org
Subject: RE: Limit to columns or nesting of Hive table?

Thanks. So is the nesting limit 10 now? Does your 2nd paragraph mean that this limit cannot easily be raised?

Chuck

-----Original Message-----
From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
Sent: Thursday, October 04, 2012 11:57 AM
To: user@hive.apache.org
Subject: Re: Limit to columns or nesting of Hive table?

There is an open jira ticket on this. There is a hard coded limit but it could be raised with some mostly minor code changes.

One of the bigger problems is that hive stores the definition of a column in JDBC "column" and for some databases larger nested structs can case issues.

Edward

On Thu, Oct 4, 2012 at 11:48 AM, Connell, Chuck <Ch...@nuance.com> wrote:
> I am trying to create a large Hive table, with many columns and deeply 
> nested structs. It is failing with java.lang.ArrayIndexOutOfBoundsException:
> 10.
>
>
>
> Before I spend a lot of time debugging my table declaration, is there 
> some limit here I should know about? Max number of columns? Max depth 
> of struct nesting?
>
>
>
> Thanks,
>
> Chuck
>
>

RE: Limit to columns or nesting of Hive table?

Posted by "Connell, Chuck" <Ch...@nuance.com>.

Thanks. So is the nesting limit 10 now? Does your 2nd paragraph mean that this limit cannot easily be raised?

Chuck

-----Original Message-----
From: Edward Capriolo [mailto:edlinuxguru@gmail.com] 
Sent: Thursday, October 04, 2012 11:57 AM
To: user@hive.apache.org
Subject: Re: Limit to columns or nesting of Hive table?

There is an open jira ticket on this. There is a hard coded limit but it could be raised with some mostly minor code changes.

One of the bigger problems is that hive stores the definition of a column in JDBC "column" and for some databases larger nested structs can case issues.

Edward

On Thu, Oct 4, 2012 at 11:48 AM, Connell, Chuck <Ch...@nuance.com> wrote:
> I am trying to create a large Hive table, with many columns and deeply 
> nested structs. It is failing with java.lang.ArrayIndexOutOfBoundsException:
> 10.
>
>
>
> Before I spend a lot of time debugging my table declaration, is there 
> some limit here I should know about? Max number of columns? Max depth 
> of struct nesting?
>
>
>
> Thanks,
>
> Chuck
>
>

Re: Limit to columns or nesting of Hive table?

Posted by Edward Capriolo <ed...@gmail.com>.

There is an open jira ticket on this. There is a hard coded limit but
it could be raised with some mostly minor code changes.

One of the bigger problems is that hive stores the definition of a
column in JDBC "column" and for some databases larger nested structs
can case issues.

Edward

On Thu, Oct 4, 2012 at 11:48 AM, Connell, Chuck
<Ch...@nuance.com> wrote:
> I am trying to create a large Hive table, with many columns and deeply
> nested structs. It is failing with java.lang.ArrayIndexOutOfBoundsException:
> 10.
>
>
>
> Before I spend a lot of time debugging my table declaration, is there some
> limit here I should know about? Max number of columns? Max depth of struct
> nesting?
>
>
>
> Thanks,
>
> Chuck
>
>