You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Arkadiusz Firus (JIRA)" <ji...@apache.org> on 2015/12/01 10:47:10 UTC
[jira] [Updated] (FLUME-2848) Kafka Source could not work with
Kafka Sink
[ https://issues.apache.org/jira/browse/FLUME-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arkadiusz Firus updated FLUME-2848:
-----------------------------------
Description:
KafkaSource stores information from which topic a record was read. This information is stored in an event header with the key "topic" (see org.apache.flume.source.kafka.KafkaSource line 108).
KafkaSink can be instructed to store record in a different topic than the one in configuration (dynamic destination). To do that "topic" header have to be set in an event (see org.apache.flume.sink.kafka.KafkaSink line 107).
When this two features are combined we have infinite loop - KafkaSink writes records to the topic which is read by KafkaSource.
Two solve the problem fields:
org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
need to have different values.
was:
KafkaSource stores information from which topic record was read. This information is stored in event header with key "topic" (see org.apache.flume.source.kafka.KafkaSource line 108).
KafkaSink can be instructed to store record in a different topic than the one in configuration (dynamic destination). To do that "topic" header have to be set in event (see org.apache.flume.sink.kafka.KafkaSink line 107).
When this two features are combined we have infinite loop - KafkaSink writes record to topic which is read by KafkaSource.
Two solve the problem fields:
org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
need to have different values.
> Kafka Source could not work with Kafka Sink
> -------------------------------------------
>
> Key: FLUME-2848
> URL: https://issues.apache.org/jira/browse/FLUME-2848
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.6.0
> Environment: IMHO any
> Reporter: Arkadiusz Firus
> Labels: easyfix
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> KafkaSource stores information from which topic a record was read. This information is stored in an event header with the key "topic" (see org.apache.flume.source.kafka.KafkaSource line 108).
> KafkaSink can be instructed to store record in a different topic than the one in configuration (dynamic destination). To do that "topic" header have to be set in an event (see org.apache.flume.sink.kafka.KafkaSink line 107).
> When this two features are combined we have infinite loop - KafkaSink writes records to the topic which is read by KafkaSource.
> Two solve the problem fields:
> org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
> org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
> need to have different values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)