You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flume.apache.org by "Arkadiusz Firus (JIRA)" <ji...@apache.org> on 2015/12/01 10:45:10 UTC

[jira] [Created] (FLUME-2848) Kafka Source could not work with Kafka Sink

Arkadiusz Firus created FLUME-2848:
--------------------------------------

             Summary: Kafka Source could not work with Kafka Sink
                 Key: FLUME-2848
                 URL: https://issues.apache.org/jira/browse/FLUME-2848
             Project: Flume
          Issue Type: Bug
          Components: Sinks+Sources
    Affects Versions: v1.6.0
         Environment: IMHO any
            Reporter: Arkadiusz Firus


KafkaSource stores information from which topic record was read. This information is stored in event header with key "topic" (see org.apache.flume.source.kafka.KafkaSource line 108).
KafkaSink can be instructed to store record in a different topic than the one in configuration (dynamic destination). To do that "topic" header have to be set in event (see org.apache.flume.sink.kafka.KafkaSink line 107).

When this two features are combined we have infinite loop - KafkaSink writes record to topic which is read by KafkaSource.

Two solve the problem fields:
org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
need to have different values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)