You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hive.apache.org by "kulkarni.swarnim@gmail.com" <ku...@gmail.com> on 2012/07/11 23:10:05 UTC

Separators in struct

Hello,

I am not sure I understand the significance of separators very well in case
of structs. For instance, for deeply nested structs I usually hit this
exception:

java.lang.ArrayIndexOutOfBoundsException: 9
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
 at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at
org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)

Digging deeper into the code I found that the size of separators have been
hard-coded to be an array of size "8" with a comment on it.

// Read the separators: We use 8 levels of separators by default, but we should
change this when we allow users to specify more than 10 levels of
separators through DDL.

serdeParams.separators = new byte[8];


If someone can explain this to me, I would really appreciate that. Also is
there a way to change the number of separators so that this exception is
not thrown?

Thanks,

-- 
Swarnim

Re: Separators in struct

Posted by "kulkarni.swarnim@gmail.com" <ku...@gmail.com>.

Issue logged. https://issues.apache.org/jira/browse/HIVE-3253

On Wed, Jul 11, 2012 at 4:13 PM, Edward Capriolo <ed...@gmail.com>wrote:

> We surely can make it bigger. However there is a sublte problem. I
> have ran into maximum column length limitations in the metastore with
> heavily nested columns. MySQL varchar maxes etc. You should open a
> jirra issues on issues.apache.org/jira/hive
>
> Edward
>
> On Wed, Jul 11, 2012 at 5:10 PM, kulkarni.swarnim@gmail.com
> <ku...@gmail.com> wrote:
> > Hello,
> >
> > I am not sure I understand the significance of separators very well in
> case
> > of structs. For instance, for deeply nested structs I usually hit this
> > exception:
> >
> > java.lang.ArrayIndexOutOfBoundsException: 9
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> > at
> >
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> >
> > Digging deeper into the code I found that the size of separators have
> been
> > hard-coded to be an array of size "8" with a comment on it.
> >
> > // Read the separators: We use 8 levels of separators by default, but we
> > should change this when we allow users to specify more than 10 levels of
> > separators through DDL.
> >
> > serdeParams.separators = new byte[8];
> >
> >
> > If someone can explain this to me, I would really appreciate that. Also
> is
> > there a way to change the number of separators so that this exception is
> not
> > thrown?
> >
> > Thanks,
> >
> >
> > --
> > Swarnim
>



-- 
Swarnim

Re: Separators in struct

Posted by Edward Capriolo <ed...@gmail.com>.

We surely can make it bigger. However there is a sublte problem. I
have ran into maximum column length limitations in the metastore with
heavily nested columns. MySQL varchar maxes etc. You should open a
jirra issues on issues.apache.org/jira/hive

Edward

On Wed, Jul 11, 2012 at 5:10 PM, kulkarni.swarnim@gmail.com
<ku...@gmail.com> wrote:
> Hello,
>
> I am not sure I understand the significance of separators very well in case
> of structs. For instance, for deeply nested structs I usually hit this
> exception:
>
> java.lang.ArrayIndexOutOfBoundsException: 9
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at
> org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
>
> Digging deeper into the code I found that the size of separators have been
> hard-coded to be an array of size "8" with a comment on it.
>
> // Read the separators: We use 8 levels of separators by default, but we
> should change this when we allow users to specify more than 10 levels of
> separators through DDL.
>
> serdeParams.separators = new byte[8];
>
>
> If someone can explain this to me, I would really appreciate that. Also is
> there a way to change the number of separators so that this exception is not
> thrown?
>
> Thanks,
>
>
> --
> Swarnim