You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tez.apache.org by "Rajesh Balamohan (JIRA)" <ji...@apache.org> on 2017/06/12 23:07:00 UTC
[jira] [Comment Edited] (TEZ-3757) Integer overflow in
PipelinedSorter
[ https://issues.apache.org/jira/browse/TEZ-3757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16047190#comment-16047190 ]
Rajesh Balamohan edited comment on TEZ-3757 at 6/12/17 11:06 PM:
-----------------------------------------------------------------
Thanks for reporting the bug [~tmwoodruff]. Due to the overflow, it would end up using {{16 *1024 * 1024}} as the metasize instead of reduced metasize for larger items. Changing the datatype in sortSpan constructor for dataSize would fix the issue.
was (Author: rajesh.balamohan):
Thanks for reporting the bug [~tmwoodruff]. Due to the overflow, it would end up using 16 *1024 *1024 as the metasize instead of reduced metasize for larger items. Changing the datatype in sortSpan constructor for dataSize would fix the issue.
> Integer overflow in PipelinedSorter
> -----------------------------------
>
> Key: TEZ-3757
> URL: https://issues.apache.org/jira/browse/TEZ-3757
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.8.4, 0.8.5
> Reporter: Travis Woodruff
>
> This code in {{PipelinedSorter.sort()}} passes {{(1024*1024)}} as maxItems to the {{SortSpan}} constructor:
> {code}
> //TODO: fix per item being passed.
> span = new SortSpan((ByteBuffer)buffers.get(bufferIndex).clear(), (1024*1024),
> perItem, ConfigUtils.getIntermediateOutputKeyComparator(this.conf));
> {code}
> {{SortSpan}}'s constructor then calculates {{dataSize}} as follows:
> {code}
> int dataSize = maxItems * perItem;
> {code}
> This means that if {{perItem}} is >= 2048, {{dataSize}} overflows, which (usually?) ends up causing the capacity check to not work correctly, which causes subsequent buffer operations to fail.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)