You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Darshan Mehta (JIRA)" <ji...@apache.org> on 2018/11/14 17:36:00 UTC
[jira] [Updated] (KAFKA-7629) Mirror maker goes into infinite loop
[ https://issues.apache.org/jira/browse/KAFKA-7629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Darshan Mehta updated KAFKA-7629:
---------------------------------
Description:
*Setup:*
I have 2 kafka images running (Spoify Kafka image : [https://hub.docker.com/r/spotify/kafka/)]
Config:
Image 1:
* host: kafka1
* zk port : 2181
* broker port : 9092
Image 2:
* host: kafka2
* zk port : 1181
* broker port : 8092
Producer Properties for Mirror maker:
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker:
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
*Steps to replicate :*
# Start mirror maker with following command :
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config <producer_config_location> --consumer.config <consumer_config_location> --num.streams 1 --whitelist topic-1
{code}
# Start local kafka console consumer to listen to topic-1 for kafka2:8092
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server kafka2:8092 --topic topic-1
{code}
# Produce an event to kafka1:9092 - topic-1 -> It will be printed on the console in Step 2
# Stop mirror maker with ctrl+C (started in step 1)
# Restart mirror maker with same command
# Produce an event onto the same topic (i.e. repeat step 3)
# Both source and destination will be flooded with the same messages until mirror maker is stopped
Surprisingly, source kafka also gets flooded with the same message. I believe when restarted, the mirror maker is unable to read the state?
was:
*Setup:*
I have 2 kafka images running (Spoify Kafka image : [https://hub.docker.com/r/spotify/kafka/)]
Config:
Image 1:
* host: kafka1
* zk port : 2181
* broker port : 9092
Image 2:
* host: kafka2
* zk port : 1181
* broker port : 8092
Producer Properties for Mirror maker:
{code:java}
bootstrap.servers=kafka2:8092
{code}
Consumer Properties for Mirror maker:
{code:java}
bootstrap.servers=kafka1:9092
group.id=test-consumer-group
exclude.internal.topics=true
{code}
*Steps to replicate :*
# Start mirror maker with following command :
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config <producer_config_location> --consumer.config <consumer_config_location> --num.streams 1 --whitelist topic-1
{code}
# Start local kafka console consumer to listen to topic-1 for kafka2:8092
{code:java}
$KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server kafka2:8092 --topic topic-1
{code}
# Produce an event to kafka1:9092 - topic-1 -> It will be printed on the console in Step 2
# Stop mirror maker with ctrl+C (started in step 1)
# Restart mirror maker with same command
# Produce an event onto the same topic (i.e. repeat step 3)
# Both source and destination will be flooded with the same messages until mirror maker is stopped
Surprisingly, source kafka also gets flooded with the same message. I believe when restarted, the mirror maker is unable to read the state?
> Mirror maker goes into infinite loop
> ------------------------------------
>
> Key: KAFKA-7629
> URL: https://issues.apache.org/jira/browse/KAFKA-7629
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.0.0
> Environment: local
> Reporter: Darshan Mehta
> Priority: Major
>
> *Setup:*
> I have 2 kafka images running (Spoify Kafka image : [https://hub.docker.com/r/spotify/kafka/)]
> Config:
> Image 1:
> * host: kafka1
> * zk port : 2181
> * broker port : 9092
> Image 2:
> * host: kafka2
> * zk port : 1181
> * broker port : 8092
> Producer Properties for Mirror maker:
> {code:java}
> bootstrap.servers=kafka2:8092
> {code}
> Consumer Properties for Mirror maker:
> {code:java}
> bootstrap.servers=kafka1:9092
> group.id=test-consumer-group
> exclude.internal.topics=true
> {code}
>
> *Steps to replicate :*
> # Start mirror maker with following command :
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-mirror-maker.sh --producer.config <producer_config_location> --consumer.config <consumer_config_location> --num.streams 1 --whitelist topic-1
> {code}
> # Start local kafka console consumer to listen to topic-1 for kafka2:8092
> {code:java}
> $KAFKA_INSTALLATION_DIR/bin/kafka-console-consumer.sh --bootstrap-server kafka2:8092 --topic topic-1
> {code}
> # Produce an event to kafka1:9092 - topic-1 -> It will be printed on the console in Step 2
> # Stop mirror maker with ctrl+C (started in step 1)
> # Restart mirror maker with same command
> # Produce an event onto the same topic (i.e. repeat step 3)
> # Both source and destination will be flooded with the same messages until mirror maker is stopped
> Surprisingly, source kafka also gets flooded with the same message. I believe when restarted, the mirror maker is unable to read the state?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)