[jira] [Commented] (TEZ-3212) IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB

["Jonathan Eagles (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TEZ-3212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15237992#comment-15237992 ] 

Jonathan Eagles commented on TEZ-3212:
--------------------------------------

This patch is really just to start a discussion on what to do with really large values. It is arguable that IFile should be able to read what was written. MAX_ARRAY_LENGTH is due to the limitation where byte array of Integer.MAX_VALUE cannot be allocated due to jvm limits

https://plumbr.eu/outofmemoryerror/requested-array-size-exceeds-vm-limit



> IFile throws NegativeArraySizeException for value sizes between 1GB and 2GB
> ---------------------------------------------------------------------------
>
>                 Key: TEZ-3212
>                 URL: https://issues.apache.org/jira/browse/TEZ-3212
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>         Attachments: TEZ-3212.1.patch
>
>
> This is not a regression with respect to MR, just an issue that was encountered with a job whose IFile record values (which can be of max size 2GB) which can be successfully written but not successfully read.
> Failure while running task:java.lang.NegativeArraySizeException
> 	at org.apache.tez.runtime.library.common.sort.impl.IFile$Reader.nextRawValue(IFile.java:765)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)