[jira] [Updated] (FLUME-2320) Deadlock in DatasetSink

["Ryan Blue (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLUME-2320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ryan Blue updated FLUME-2320:
-----------------------------

    Attachment: FLUME-2320-1.patch

This patch implements the proposed solution: it removes the roll thread and writer lock, and moves the rolling logic into the process method. It also adds a note in the documentation that file rolling may be slightly delayed, in some cases up to 5 seconds.

> Deadlock in DatasetSink
> -----------------------
>
>                 Key: FLUME-2320
>                 URL: https://issues.apache.org/jira/browse/FLUME-2320
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.5.0
>            Reporter: Ryan Blue
>         Attachments: FLUME-2320-1.patch
>
>
> Lines [251-252|https://github.com/apache/flume/blob/trunk/flume-ng-sinks/flume-dataset-sink/src/main/java/org/apache/flume/sink/kite/DatasetSink.java#L251] in the DatasetSink are a potential deadlock: if the transaction throws an exception, then the writer lock is not released, but the same thread tries to lock in the error handling.
> While the simplest solution is to move those two lines inside the try/finally statement, I think we can actually remove the lock completely by reverting to the original version that rolled the files in the process() method. The original concern about that design was that there needs to be some guarantee that all files will be rolled. Because the SinkRunner has a max backoff, there is a guaranteed maximum amount of time between calls to process.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)