You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Kay (Jira)" <ji...@apache.org> on 2020/05/19 04:31:00 UTC
[jira] [Updated] (KAFKA-10019) MirrorMaker 2 did not function
properly after restart (message lost, messages arriving slowly)
[ https://issues.apache.org/jira/browse/KAFKA-10019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kay updated KAFKA-10019:
------------------------
Description:
MM2 did not function properly after stopping a running MM2 process then starting it again. Consumer did not receive all messages (even messages being sent after MM2 restarted). The messages arriving to the consumer were no longer at the rate as specified in "--message" --and "--timeout".
To reproduce the issue
# Environment:
## Region 1: one Kafka cluster, two MM2 instances, 1 producer instance
## Region 2: one Kafka cluster, two MM2 instances, 1 consumer instance
# Producer (in region 1) started sending 1000 messages.
## ./bin/kafka-producer-perf-test.sh --producer.config config/producer.properties --topic topic1 --record-size 4800000 --num-records 1000 --throughput 17
# Consumer (in region 2) started receiving messages.
## while true; do ./bin/kafka-consumer-perf-test.sh --threads 60 *--timeout 5000* --consumer.config config/consumer.properties --topic region1.topic1 *--messages 250* --group region2-consume-region1topic1 --broker-list $KAFKA_BROKERS; done > consumer.log &
# Consumer received the first 500 messages (250, 250), as "--message" specified.
# Killed the MM2 process on one of two instances in both regions.
# Consumer started receiving the remaining messages at a much slower "rate" (160, 29, 19, 11, 9, 6, 5, 5, 0,.. 3, 0,... 2, 0,... 1).
# Restarted the MM2 processes killed at (4).
# Producer sent another 1000 messages.
# Still, messages no longer arrived at the "--message" rate (250 * N), but e.g. 37, 30, 23, 13, 9, 0, 1, 3...
# And consumer did not receive all new 1000 messages sent after MM2 restarted.
Please see the producer and consumer log files attached. In the consumer log file, you can see that after the first 2 consecutive "250" messages arrived, the message arrived differently.
*Issue Summary*
# MM2 does not recover from restarting its process.
# After killing a MM2 process in the MM2 EC2 instance, a Consumer no longer received the messages at the rate of "--message" and "--timeout".
# Consumer did not receive all messages even those messages were published after the mm2 process restarted.
# Consumer no longer received messages at the rate of "--message" and "-timeout" even after the mm2 process restarted.
was:
MM2 did not function properly after stopping a running MM2 process then starting it again. Consumer did not receive all messages (even messages being sent after MM2 restarted). The messages arriving to the consumer were no longer at the rate as specified in "--message" and "--timeout".
To reproduce the issue
# Environment:
## Region 1: one Kafka cluster, two MM2 instances, 1 producer instance
## Region 2: one Kafka cluster, two MM2 instances, 1 consumer instance
# **Producer (in region 1) started sending 1000 messages.
## ./bin/kafka-producer-perf-test.sh --producer.config config/producer.properties --topic topic1 --record-size 4800000 --num-records 1000 --throughput 17
# Consumer (in region 2) started receiving messages.
## while true; do ./bin/kafka-consumer-perf-test.sh --threads 60 *--timeout 5000* --consumer.config config/consumer.properties --topic region1.topic1 *--messages 250* --group region2-consume-region1topic1 --broker-list $KAFKA_BROKERS; done > consumer.log &
# Consumer received the first 500 messages (250, 250), as "--message" specified.
# Killed the MM2 process on one of two instances in both regions.
# Consumer started receiving the remaining messages at a much slower "rate" (160, 29, 19, 11, 9, 6, 5, 5, 0,.. 3, 0,... 2, 0,... 1).
# Restarted the MM2 processes killed at (4).
# Producer sent another 1000 messages.
# Still, messages no longer arrived at the "--message" rate (250 * N), but e.g. 37, 30, 23, 13, 9, 0, 1, 3...
# And consumer did not receive all new 1000 messages sent after MM2 restarted.
Please see the producer and consumer log files attached.
In the consumer log file, you can see that after the first 2 consecutive "250" messages arrived, the message arrived differently.
*Issue Summary*
# MM2 does not recover from restarting its process.
# After killing a MM2 process in the MM2 EC2 instance, a Consumer no longer received the messages at the rate of "--message" and "--timeout".
# Consumer did not receive all messages even those messages were published after the mm2 process restarted.
# Consumer no longer received messages at the rate of "--message" and "-timeout" even after the mm2 process restarted.
> MirrorMaker 2 did not function properly after restart (message lost, messages arriving slowly)
> ----------------------------------------------------------------------------------------------
>
> Key: KAFKA-10019
> URL: https://issues.apache.org/jira/browse/KAFKA-10019
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 2.4.1
> Environment: Amazon Linux 2
> MSK clusters: kafka.m5.large, 3 AZ, 3 brokers
> MM2 instances: c5.2xlarge
> Producer/Consumer instances: c5.2xlarge
> Reporter: Kay
> Priority: Major
> Attachments: 2a-consumer.log, 2a-producer.log
>
>
> MM2 did not function properly after stopping a running MM2 process then starting it again. Consumer did not receive all messages (even messages being sent after MM2 restarted). The messages arriving to the consumer were no longer at the rate as specified in "--message" --and "--timeout".
> To reproduce the issue
> # Environment:
> ## Region 1: one Kafka cluster, two MM2 instances, 1 producer instance
> ## Region 2: one Kafka cluster, two MM2 instances, 1 consumer instance
> # Producer (in region 1) started sending 1000 messages.
> ## ./bin/kafka-producer-perf-test.sh --producer.config config/producer.properties --topic topic1 --record-size 4800000 --num-records 1000 --throughput 17
> # Consumer (in region 2) started receiving messages.
> ## while true; do ./bin/kafka-consumer-perf-test.sh --threads 60 *--timeout 5000* --consumer.config config/consumer.properties --topic region1.topic1 *--messages 250* --group region2-consume-region1topic1 --broker-list $KAFKA_BROKERS; done > consumer.log &
> # Consumer received the first 500 messages (250, 250), as "--message" specified.
> # Killed the MM2 process on one of two instances in both regions.
> # Consumer started receiving the remaining messages at a much slower "rate" (160, 29, 19, 11, 9, 6, 5, 5, 0,.. 3, 0,... 2, 0,... 1).
> # Restarted the MM2 processes killed at (4).
> # Producer sent another 1000 messages.
> # Still, messages no longer arrived at the "--message" rate (250 * N), but e.g. 37, 30, 23, 13, 9, 0, 1, 3...
> # And consumer did not receive all new 1000 messages sent after MM2 restarted.
> Please see the producer and consumer log files attached. In the consumer log file, you can see that after the first 2 consecutive "250" messages arrived, the message arrived differently.
> *Issue Summary*
> # MM2 does not recover from restarting its process.
> # After killing a MM2 process in the MM2 EC2 instance, a Consumer no longer received the messages at the rate of "--message" and "--timeout".
> # Consumer did not receive all messages even those messages were published after the mm2 process restarted.
> # Consumer no longer received messages at the rate of "--message" and "-timeout" even after the mm2 process restarted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)