[jira] [Comment Edited] (TEZ-3402) SplitGrouper: Integer overflow

["Gopal V (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15409041#comment-15409041 ] 

Gopal V edited comment on TEZ-3402 at 8/5/16 7:15 AM:
------------------------------------------------------

[~ozawa]: there's no real need to solve it to handle > INT_MAX now  - the real number of splits was 121k in this case. The error was that trying to pre-allocate such a large array was entirely wasteful.


was (Author: gopalv):
[~ozawa]: there's no real need to solve it right now  - the real number of splits was 121k in this case. The error was that trying to pre-allocate such a large array was entirely wasteful.

> SplitGrouper: Integer overflow
> ------------------------------
>
>                 Key: TEZ-3402
>                 URL: https://issues.apache.org/jira/browse/TEZ-3402
>             Project: Apache Tez
>          Issue Type: Bug
>    Affects Versions: 0.8.4
>            Reporter: Gopal V
>            Assignee: Tsuyoshi Ozawa
>         Attachments: TEZ-3402.001.patch
>
>
> Bad configs triggers integer overflow. This is a 5Tb query which tries to group by max-size of 4096.
> {code}        
>         // splits too small to work. Need to override with size.
>         int newDesiredNumSplits = (int)(totalLength/minLengthPerGroup) + 1;
> {code}
> {code}
> diagnostics=[Vertex vertex_1470081722620_0072_3_00 [Map 2] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: srvc_fee initializer failed, vertex=vertex_1470081722620_0072_3_00 [Map 2], java.lang.IllegalArgumentException: Illegal Capacity: -1401168103
>         at java.util.ArrayList.<init>(ArrayList.java:156)
>         at org.apache.hadoop.mapred.split.TezMapredSplitsGrouper.getGroupedSplits(TezMapredSplitsGrouper.java:230)
>         at org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.group(SplitGrouper.java:89)
>         at org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:168)
>         at org.apache.hadoop.hive.ql.exec.tez.SplitGrouper.generateGroupedSplits(SplitGrouper.java:138)
>         at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:159)
>         at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:273)
>         at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:266)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)