[jira] [Commented] (MAPREDUCE-6724) Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()

["Haibo Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345523#comment-15345523 ] 

Haibo Chen commented on MAPREDUCE-6724:
---------------------------------------

The implementation of InMemoryMapOutput uses a byte array, which only support a maximum size of Integer.MAX_VALUE. The conversion of requested size from long to int seems unavoidable if we want to stick with the byte array implementation. 
We could cap the value of maxSingleShuffleLimit at Integer.MAX_VALUE, so shuffle is done on disk if the requested size is bigger than Integer.MAX_VALUE.

> Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
> -----------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6724
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>
> When shuffle is done in memory, MergeManagerImpl converts the requested size to an int to allocate an instance of InMemoryMapOutput. This results in an overflow if the requested size is bigger than Integer.MAX_VALUE and eventually causes the reducer to fail.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org