You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Haibo Chen (JIRA)" <ji...@apache.org> on 2016/07/08 20:02:10 UTC
[jira] [Commented] (MAPREDUCE-6724) Unsafe conversion from long to
int in MergeManagerImpl.unconditionalReserve()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368329#comment-15368329 ]
Haibo Chen commented on MAPREDUCE-6724:
---------------------------------------
Thanks a lot for your reviews, [~jira.shegalov]! And my apologies for being late to update the patch. I have updated the patch to accommodate your comments except for one thing. Instead of creating a method that just takes targetSingleShuffleLimit as its parameter, I have four parameters for the new private method, since the setup of testLargeMemoryLimits involves REDUCE_MEMORY_TOTAL_BYTES, SHUFFLE_INPUT_BUFFER_PERCENT and REDUCE_INPUT_BUFFER_PERCENT whereas the new test method involves REDUCE_MEMORY_TOTAL_BYTES, SHUFFLE_INPUT_BUFFER_PERCENT and SHUFFLE_MEMORY_LIMIT_PERCENT (testLargeMemoryLimits does not care about the value of targetSingleShuffleLimit). Thanks again for your review and please let me know any more comments you have.
> Unsafe conversion from long to int in MergeManagerImpl.unconditionalReserve()
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-6724
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6724
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Attachments: mapreduce6724.001.patch, mapreduce6724.002.patch, mapreduce6724.003.patch
>
>
> When shuffle is done in memory, MergeManagerImpl converts the requested size to an int to allocate an instance of InMemoryMapOutput. This results in an overflow if the requested size is bigger than Integer.MAX_VALUE and eventually causes the reducer to fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org