[jira] [Updated] (FLUME-1864) Allow hdfs idle callback to clean up closed bucket writers

["Juhani Connolly (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Juhani Connolly updated FLUME-1864:
-----------------------------------

    Attachment: FLUME-1864
    
> Allow hdfs idle callback to clean up closed bucket writers
> ----------------------------------------------------------
>
>                 Key: FLUME-1864
>                 URL: https://issues.apache.org/jira/browse/FLUME-1864
>             Project: Flume
>          Issue Type: Improvement
>            Reporter: Juhani Connolly
>         Attachments: FLUME-1864
>
>
> In the original implementation of the idle file closing behaviour the callback was cancelled on close(). This makes sense assuming everything else behaves in a desirable manner.
> On the other hand, rollInterval will close a file and leave the bucketWriter in the writer map. This allows for incrementally named files to be created as the same path is reopened. However in some situations(primarily with time bucketed data), this leaves a lot of abandoned bucket writers that may be closed by rollInterval and thus never removed by idle.
> In FLUME-1850 a couple of approaches were suggested, and I originally intended to fix this by using the callback from rollInterval to remove the writer from the map. However this would break incremental naming.
> Until(if we ever do) we change the rolling logic it leaves the more viable option to be allowing the idle timer to persist after a close. In this way it can be used to release resources that have become unnecessary but already closed by rollInterval, also reducing hard to understand interactions between configuration variables by one.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira