[jira] [Updated] (FLUME-2128) HDFS Sink rollSize is calculated based off of uncompressed size of cumulative events.

["Ted Malaska (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Malaska updated FLUME-2128:
-------------------------------

    Attachment: FLUME-2128-2.patch

First implementation of Mike's idea.

The code can take a starting expectedCompressionRatio then with every rolling of the file it will up date the expectedCompressionRatio.  The new expectedCompressionRatio will equal (((processSize / resulting file size) * 2) + lastExceptedCompressionRatio) / 3



> HDFS Sink rollSize is calculated based off of uncompressed size of cumulative events.
> -------------------------------------------------------------------------------------
>
>                 Key: FLUME-2128
>                 URL: https://issues.apache.org/jira/browse/FLUME-2128
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.4.0, v1.3.1
>            Reporter: Jeff Lord
>            Assignee: Ted Malaska
>              Labels: features
>         Attachments: FLUME-2128-0.patch, FLUME-2128-1.patch, FLUME-2128-2.patch
>
>
> The hdfs sink rollSize parameter is compared against uncompressed event sizes.
> The net of this is that if you are using compression and expect the size of your files on HDFS to be rolled/sized based on the value set for rollSize than your files will be much smaller due to compression.
> We should take into account when compression is set and roll based on the compressed size on hdfs.



--
This message was sent by Atlassian JIRA
(v6.1#6144)