You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Benjamin Teke (Jira)" <ji...@apache.org> on 2023/08/01 13:50:00 UTC
[jira] [Updated] (MAPREDUCE-7446) NegativeArraySizeException when running MR jobs with large data size
[ https://issues.apache.org/jira/browse/MAPREDUCE-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Benjamin Teke updated MAPREDUCE-7446:
-------------------------------------
Hadoop Flags: Reviewed
Resolution: Fixed
Status: Resolved (was: Patch Available)
> NegativeArraySizeException when running MR jobs with large data size
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-7446
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7446
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Peter Szucs
> Assignee: Peter Szucs
> Priority: Major
> Labels: pull-request-available
>
> We are using bit shifting to double the byte array in IFile's [nextRawValue|https://github.infra.cloudera.com/CDH/hadoop/blob/bef14a39c7616e3b9f437a6fb24fc7a55a676b57/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/IFile.java#L437] method to store the byte values in it. With large dataset it can easily happen that we shift the leftmost bit when we are calculating the size of the array, which can lead to a negative number as the array size, causing the NegativeArraySizeException.
> It would be safer to expand the backing array with a 1.5x factor, and have a check not to extend Integer's max value during that.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org