[jira] [Resolved] (FLUME-3085) HDFS Sink can skip flushing some BucketWriters, might lead to data loss

["Mike Percy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mike Percy resolved FLUME-3085.
-------------------------------
    Resolution: Fixed

Pushed to trunk.

> HDFS Sink can skip flushing some BucketWriters, might lead to data loss
> -----------------------------------------------------------------------
>
>                 Key: FLUME-3085
>                 URL: https://issues.apache.org/jira/browse/FLUME-3085
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: 1.7.0
>            Reporter: Denes Arvay
>            Assignee: Denes Arvay
>            Priority: Critical
>             Fix For: 1.8.0
>
>
> The {{HDFSEventSink.process()}} is already prepared for a rare race condition, namely when the BucketWriter acquired in [line 389|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L389] gets closed by an other thread (e.g. because the {{idleTimeout}} or the {{rollInterval}}) before the {{append()}} is called in [line 406|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L406].
> If this is the case the {{BucketWriter.append()}} call throws a {{BucketClosedException}} and the sink creates a new {{BucketWriter}} instance and appends to it.
> But this newly created instance won't be added to the {{writers}} list, which means that it won't be flushed after the processing loop finished: https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java#L429
> This has multiple consequences:
> - unflushed data might get lost
> - the {{BucketWriter}}'s {{idleAction}} won't be scheduled (https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/BucketWriter.java#L450), which means that it won't be closed nor renamed if the idle timeout is the only trigger for closing the file.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)