You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Swarnim Kulkarni (JIRA)" <ji...@apache.org> on 2012/07/12 15:26:34 UTC
[jira] [Created] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Swarnim Kulkarni created HIVE-3253:
--------------------------------------
Summary: ArrayIndexOutOfBounds exception for deeply nested structs
Key: HIVE-3253
URL: https://issues.apache.org/jira/browse/HIVE-3253
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.9.0
Reporter: Swarnim Kulkarni
It was observed that while creating table with deeply nested structs might throw this exception:
{code}
java.lang.ArrayIndexOutOfBoundsException: 9
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
{code}
// Read the separators: We use 8 levels of separators by default, but we
// should change this when we allow users to specify more than 10 levels
// of separators through DDL.
serdeParams.separators = new byte[8];
{code}
We possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Travis Crawford (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424998#comment-13424998 ]
Travis Crawford commented on HIVE-3253:
---------------------------------------
I ran into this issue too. I'm running tests with that value doubled to 16.
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Swarnim Kulkarni
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Swarnim Kulkarni (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489837#comment-13489837 ]
Swarnim Kulkarni commented on HIVE-3253:
----------------------------------------
@Ashutosh - What I fail to understand is why do we have to hardcode this value? Instead why can't we go for a better solution where array can be dynamically resized based on requirement?
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0, 0.10.0
> Reporter: Swarnim Kulkarni
> Assignee: Travis Crawford
> Attachments: HIVE-3253_moar_nesting.1.patch, jsonout.hive
>
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Travis Crawford (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Travis Crawford updated HIVE-3253:
----------------------------------
Affects Version/s: 0.10.0
Status: Patch Available (was: Open)
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0, 0.10.0
> Reporter: Swarnim Kulkarni
> Assignee: Travis Crawford
> Attachments: HIVE-3253_moar_nesting.1.patch
>
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Mark Grover (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490281#comment-13490281 ]
Mark Grover commented on HIVE-3253:
-----------------------------------
+1 on Swarnim's comment.
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0, 0.10.0
> Reporter: Swarnim Kulkarni
> Assignee: Travis Crawford
> Attachments: HIVE-3253_moar_nesting.1.patch, jsonout.hive
>
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Travis Crawford (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Travis Crawford updated HIVE-3253:
----------------------------------
Attachment: HIVE-3253_moar_nesting.1.patch
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0, 0.10.0
> Reporter: Swarnim Kulkarni
> Assignee: Travis Crawford
> Attachments: HIVE-3253_moar_nesting.1.patch
>
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Chuck Connell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chuck Connell updated HIVE-3253:
--------------------------------
Attachment: jsonout.hive
Test case of a large, deeply nested table definition. This is our real case that is failing.
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0, 0.10.0
> Reporter: Swarnim Kulkarni
> Assignee: Travis Crawford
> Attachments: HIVE-3253_moar_nesting.1.patch, jsonout.hive
>
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Swarnim Kulkarni (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swarnim Kulkarni updated HIVE-3253:
-----------------------------------
Description:
It was observed that while creating table with deeply nested structs might throw this exception:
{code}
java.lang.ArrayIndexOutOfBoundsException: 9
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
{code}
The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
{code}
// Read the separators: We use 8 levels of separators by default, but we
// should change this when we allow users to specify more than 10 levels
// of separators through DDL.
serdeParams.separators = new byte[8];
{code}
If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
was:
It was observed that while creating table with deeply nested structs might throw this exception:
{code}
java.lang.ArrayIndexOutOfBoundsException: 9
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
{code}
The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
{code}
// Read the separators: We use 8 levels of separators by default, but we
// should change this when we allow users to specify more than 10 levels
// of separators through DDL.
serdeParams.separators = new byte[8];
{code}
We possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Swarnim Kulkarni
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490443#comment-13490443 ]
Namit Jain commented on HIVE-3253:
----------------------------------
[~swarnim], I agree with you that this needs a better, dynamic solution. But, dont you think we should get it in
to get a work-around for the current scenario, and then proceed for the general solution ?
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0, 0.10.0
> Reporter: Swarnim Kulkarni
> Assignee: Travis Crawford
> Attachments: HIVE-3253_moar_nesting.1.patch, jsonout.hive
>
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Travis Crawford (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Travis Crawford reassigned HIVE-3253:
-------------------------------------
Assignee: Travis Crawford
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Swarnim Kulkarni
> Assignee: Travis Crawford
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Swarnim Kulkarni (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swarnim Kulkarni updated HIVE-3253:
-----------------------------------
Description:
It was observed that while creating table with deeply nested structs might throw this exception:
{code}
java.lang.ArrayIndexOutOfBoundsException: 9
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
{code}
The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
// Read the separators: We use 8 levels of separators by default, but we
// should change this when we allow users to specify more than 10 levels
// of separators through DDL.
serdeParams.separators = new byte[8];
{code}
We possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
was:
It was observed that while creating table with deeply nested structs might throw this exception:
{code}
java.lang.ArrayIndexOutOfBoundsException: 9
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
{code}
// Read the separators: We use 8 levels of separators by default, but we
// should change this when we allow users to specify more than 10 levels
// of separators through DDL.
serdeParams.separators = new byte[8];
{code}
We possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Swarnim Kulkarni
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> We possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472487#comment-13472487 ]
Ashutosh Chauhan commented on HIVE-3253:
----------------------------------------
This needs a testcase. I will suggest to add one testcase for each of text, avro and thrift. Also, 16 sounds lower. Assuming its safe, we should bump it higher. 64?
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0, 0.10.0
> Reporter: Swarnim Kulkarni
> Assignee: Travis Crawford
> Attachments: HIVE-3253_moar_nesting.1.patch
>
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Travis Crawford (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13425468#comment-13425468 ]
Travis Crawford commented on HIVE-3253:
---------------------------------------
I ran the full test suite ([test report|https://travis.ci.cloudbees.com/job/HIVE-3318_moar_nesting/1/testReport/]) increasing the number of separators to 16 and there were two failures:
* org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nulls
* org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
Neither look related to this change, and they still failed after reverting the change & rerunning just those tests. AFAICT increasing this constant is a safe thing to do.
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Swarnim Kulkarni
> Assignee: Travis Crawford
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> If possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3253) ArrayIndexOutOfBounds exception for
deeply nested structs
Posted by "Swarnim Kulkarni (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Swarnim Kulkarni updated HIVE-3253:
-----------------------------------
Description:
It was observed that while creating table with deeply nested structs might throw this exception:
{code}
java.lang.ArrayIndexOutOfBoundsException: 9
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
{code}
The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
{code}
// Read the separators: We use 8 levels of separators by default, but we
// should change this when we allow users to specify more than 10 levels
// of separators through DDL.
serdeParams.separators = new byte[8];
{code}
We possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
was:
It was observed that while creating table with deeply nested structs might throw this exception:
{code}
java.lang.ArrayIndexOutOfBoundsException: 9
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
{code}
The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
// Read the separators: We use 8 levels of separators by default, but we
// should change this when we allow users to specify more than 10 levels
// of separators through DDL.
serdeParams.separators = new byte[8];
{code}
We possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
> ArrayIndexOutOfBounds exception for deeply nested structs
> ---------------------------------------------------------
>
> Key: HIVE-3253
> URL: https://issues.apache.org/jira/browse/HIVE-3253
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.9.0
> Reporter: Swarnim Kulkarni
>
> It was observed that while creating table with deeply nested structs might throw this exception:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 9
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:281)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:263)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyObjectInspector(LazyFactory.java:276)
> at org.apache.hadoop.hive.serde2.lazy.LazyFactory.createLazyStructInspector(LazyFactory.java:354)
> {code}
> The reason being that currently the separators array has been hardcoded to be of size 8 in the LazySimpleSerde.
> {code}
> // Read the separators: We use 8 levels of separators by default, but we
> // should change this when we allow users to specify more than 10 levels
> // of separators through DDL.
> serdeParams.separators = new byte[8];
> {code}
> We possible, we should increase this size or at least make it configurable to properly handle deeply nested structs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira