[jira] [Updated] (FLUME-2357) HDFS sink should retry closing files that previously had close errors

["Hari Shreedharan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hari Shreedharan updated FLUME-2357:
------------------------------------

    Attachment: FLUME-2357.patch

Much of the size of the patch is due to a couple of file renames. Otherwise the patch itself is pretty simple. In the Bucketwriter, if a close fails, we simply reschedule the close to happen sometime later until it finally succeeds or till we hit a maximum count. Added a test case too. This depends on the presence of the isFileClosed method in the HDFS client API. If the method is absent, reattempts are not done.

> HDFS sink should retry closing files that previously had close errors
> ---------------------------------------------------------------------
>
>                 Key: FLUME-2357
>                 URL: https://issues.apache.org/jira/browse/FLUME-2357
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.4.0
>            Reporter: Patrick Dvorak
>         Attachments: FLUME-2357.patch
>
>
> When the AbstractHDFSWriter fails to close a file (due to exceeding the callTimeout or other hdfs issues), it will leave the the file open and never try again.  The only way to close the open files is to restart the flume agent.  There should be a  configurable option to allow the sink to retry to close files that had previously failed to close.



--
This message was sent by Atlassian JIRA
(v6.2#6252)