[jira] [Commented] (FLUME-2887) Source fan-out and event duplication

["Ambud Sharma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15209832#comment-15209832 ] 

Ambud Sharma commented on FLUME-2887:
-------------------------------------

This is not a bug but expected behavior in my opinion, Flume's exactly once semantics are supported using Transactions concept. If a transaction fails it will be replayed so will all events in that transaction.

Option 1:
Use is Failover Sink Processor and create pairs of replicated sinks if you are looking for an HA configuration.

Option 2:
Reduce Transaction Size to a manageable number so failures are detected quickly rather than building them up in channel. Reducing transaction size could have performance implications.

Hope this helps.

Note:
Usually sinks need to be idempotent (e.g. Hbase or Elasticsearch) to prevent duplication of data there. 

> Source fan-out and event duplication
> ------------------------------------
>
>                 Key: FLUME-2887
>                 URL: https://issues.apache.org/jira/browse/FLUME-2887
>             Project: Flume
>          Issue Type: Bug
>          Components: Channel, Sinks+Sources
>    Affects Versions: v1.5.0
>         Environment: VMs using RedHat
>            Reporter: Adam Alloul
>
> Hi there, 
> in my current work I have come to discover a behaviour of Flume that doesn't seem to be expected. Explanation:
> 1- Architecture (flume 1.5, not tested on 1.6)
>   a- flume server = virtual machine running one (and one only) flume server
>   b- First Flume server (receives 1 flow of data and transfers it using AVRO to another Flume Server.
>   c- the 2nd Flume replicates the source 4 times to 4 different sinks (the final sinks)
>   d- All flume servers use the file channel
> 2- Problem description:
>   a- one of the final sinks fails
>   b- the channel feeding it buffers and fills up
>   c- the first flume then starts buffering until channel is full even though the 3 other final sinks are still receiving and working correctly
>   d- the channel is full on flume 1 and starts complaining.
>   e- when the faulty sink is back again, flume 1 flushes its events
>   f- this causes the events to be written multiple times into the 3 other non faulty final sinks in addition to the faulty one => very very high number of duplication (up to 24)
> I believe that this is a bug and I did not see it reported by someone else. Please let me know if I am mistaken.
> thank you,
> Adam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)