[jira] [Updated] (PIG-3251) Bzip2TextInputFormat requires double the memory of maximum record size

["Koji Noguchi (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Koji Noguchi updated PIG-3251:
------------------------------

    Attachment: pig-3251-trunk-v01.patch

In Bzip2TextInputFormat, it says

{code}
        /**
         * Provide a bridge to get the bytes from the ByteArrayOutputStream without
         * creating a new byte array.
         */
        private static class TextStuffer extends OutputStream {

{code}
However, in reality, Text just creates a new bytearray and copy the content.

Attaching a patch that is similar to the approach taken by org.apache.hadoop.util.LineReader but with less changes since HADOOP-4012(added in 0.21) was a huge patch.

This patch basically reads into the fixed-length-buffer and appends to Text whenever it gets full.

Touching BZip2LineRecordReader makes me nervous so I wanted the changes to be small.

I need to do more testings to see if this approach works or not.
                
> Bzip2TextInputFormat requires double the memory of maximum record size
> ----------------------------------------------------------------------
>
>                 Key: PIG-3251
>                 URL: https://issues.apache.org/jira/browse/PIG-3251
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>            Priority: Minor
>         Attachments: pig-3251-trunk-v01.patch
>
>
> While looking at user's OOM heap dump, noticed that pig's Bzip2TextInputFormat consumes memory at both
> Bzip2TextInputFormat.buffer (ByteArrayOutputStream) 
> and actual Text that is returned as line.
> For example, when having one record with 160MBytes, buffer was 268MBytes and Text was 160MBytes.  
> We can probably eliminate one of them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira