[jira] [Commented] (FLUME-2580) Sink side interceptors

["Nicolas Laduguie (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883682#comment-16883682 ] 

Nicolas Laduguie commented on FLUME-2580:
-----------------------------------------

Hi, what about this feature request ?

We would appreciate it because we have the use case where we need to consume events from Kafka and directly write them to HDFS, without passing through a channel which would either not prevent data loss (file channel, memory channel), or duplicate data to guarantee data integrity (jdbc channel, kafka channel).

Our last need is to compute HDFS path according to Kafka messages specific fields like date or anything else, but we can't do it because we need an interceptor either on the channel side or on the sink side.

Thank you for your support.

Regards

> Sink side interceptors
> ----------------------
>
>                 Key: FLUME-2580
>                 URL: https://issues.apache.org/jira/browse/FLUME-2580
>             Project: Flume
>          Issue Type: New Feature
>            Reporter: Hari Shreedharan
>            Assignee: Hari Shreedharan
>            Priority: Major
>         Attachments: SinkSideInterceptors.pdf
>
>
> Currently, we only have source-side interceptors that help routing. But if we use something like Kafka Channel, having a sink-side interceptor can help us modify events as they come in. We could also do validation on event schemas and drop them before they hit the sink, rather than have an infinite loop due to such events. 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org