[jira] [Commented] (FLUME-2848) Kafka Source could not work with Kafka Sink

["Li Ye (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221124#comment-15221124 ] 

Li Ye commented on FLUME-2848:
------------------------------

The optinal properties topicHeader, keyHeader, timestampHeader are also helpful for performance of Kafka Sink.

> Kafka Source could not work with Kafka Sink
> -------------------------------------------
>
>                 Key: FLUME-2848
>                 URL: https://issues.apache.org/jira/browse/FLUME-2848
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.6.0
>         Environment: IMHO any
>            Reporter: Arkadiusz Firus
>              Labels: easyfix
>         Attachments: FLUME-2789-1.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> KafkaSource stores information from which topic a record was read. This information is stored in an event header with the key "topic" (see org.apache.flume.source.kafka.KafkaSource line 108).
> KafkaSink can be instructed to store record in a different topic than the one in configuration (dynamic destination). To do that "topic" header have to be set in an event (see org.apache.flume.sink.kafka.KafkaSink line 107).
> When this two features are combined we have infinite loop - KafkaSink writes records to the topic which is read by KafkaSource.
> Two solve the problem fields:
> org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
> org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
> need to have different values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)