[jira] [Updated] (FLUME-2233) MemoryChannel lock contention on every put due to bytesRemaining Semaphore

["Ted Malaska (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-2233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ted Malaska updated FLUME-2233:
-------------------------------

    Attachment: FLUME-2128-2.patch

First implementation of Mike's idea

The code takes a given estimate on the resulting compression and then updates that estimate with every roll over.  

It will compare the processSize to the real size of the final file.  Then it will adjust the expectedCompressionRatio to 66% the last roll over size and 33% the previous expectedCompressionRatio value. 

> MemoryChannel lock contention on every put due to bytesRemaining Semaphore
> --------------------------------------------------------------------------
>
>                 Key: FLUME-2233
>                 URL: https://issues.apache.org/jira/browse/FLUME-2233
>             Project: Flume
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>            Assignee: Hari Shreedharan
>
> This semaphore is checked every time there is a put (unlike the queueRemaining semaphore which is checked only on transaction commits), causing the channel to slow down, even when the user does not care about memory usage. 
> We must add a new parameter to make sure that we look at bytesRemaining only when required which the user can disable if memory is not something they worry about because they know the channel size will be sufficiently small. By default, we will need to still check bytesRemaining to avoid breaking existing configurations.



--
This message was sent by Atlassian JIRA
(v6.1#6144)