You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Alex (JIRA)" <ji...@apache.org> on 2016/11/02 11:11:59 UTC
[jira] [Updated] (KAFKA-4367) MirrorMaker shuts down gracefully
without being stopped
[ https://issues.apache.org/jira/browse/KAFKA-4367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alex updated KAFKA-4367:
------------------------
Description:
Start:
bin/kafka-mirror-maker.sh --new.consumer --consumer.config config/ssl_mirroring_consumer.properties --producer.config config/ssl_mirroring_producer.properties --whitelist "TOPIC1|TOPIC2|TOPIC3|TOPIC4" --num.streams 20 &> /dev/null &
MirrorMaker stops working without being stopped. No clue why this problem occurs.
--------------------------------------------------------------------------------
kafka-mirror-maker.log
--------------------------------------------------------------------------------
[2016-11-01 19:23:32,010] INFO Shutting down consumer threads. (kafka.tools.MirrorMaker$)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-0] mirrormaker-thread-0 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-1] mirrormaker-thread-1 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-2] mirrormaker-thread-2 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-3] mirrormaker-thread-3 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-4] mirrormaker-thread-4 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-5] mirrormaker-thread-5 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-6] mirrormaker-thread-6 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-7] mirrormaker-thread-7 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-8] mirrormaker-thread-8 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-9] mirrormaker-thread-9 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-10] mirrormaker-thread-10 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-11] mirrormaker-thread-11 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-12] mirrormaker-thread-12 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-13] mirrormaker-thread-13 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-14] mirrormaker-thread-14 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-15] mirrormaker-thread-15 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-16] mirrormaker-thread-16 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-17] mirrormaker-thread-17 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-18] mirrormaker-thread-18 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-9] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-11] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-12] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-6] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-1] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] INFO [mirrormaker-thread-19] mirrormaker-thread-19 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-13] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] INFO [mirrormaker-thread-13] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] TRACE Flushing accumulated records in producer. (org.apache.kafka.clients.producer.KafkaProducer)
[2016-11-01 19:23:32,012] TRACE [mirrormaker-thread-4] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-19] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] INFO [mirrormaker-thread-19] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] TRACE Flushing accumulated records in producer. (org.apache.kafka.clients.producer.KafkaProducer)
[2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-2] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] INFO [mirrormaker-thread-2] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
--------------------------------------------------------------------------------
server.log
--------------------------------------------------------------------------------
[2016-11-01 19:23:32,031] INFO [GroupCoordinator 0]: Preparing to restabilize group MirrorringGroup with old generation 2 (kafka.coordinator.GroupCoordinator)
[2016-11-01 19:24:01,880] INFO [GroupCoordinator 0]: Group MirrorringGroup generation 2 is dead and removed (kafka.coordinator.GroupCoordinator)
[2016-11-01 19:30:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-11-01 19:40:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-11-01 19:50:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
was:
MirrorMaker stops working without being stopped. No clue why this problem occurs.
--------------------------------------------------------------------------------
kafka-mirror-maker.log
--------------------------------------------------------------------------------
[2016-11-01 19:23:32,010] INFO Shutting down consumer threads. (kafka.tools.MirrorMaker$)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-0] mirrormaker-thread-0 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-1] mirrormaker-thread-1 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-2] mirrormaker-thread-2 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-3] mirrormaker-thread-3 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-4] mirrormaker-thread-4 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,011] INFO [mirrormaker-thread-5] mirrormaker-thread-5 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-6] mirrormaker-thread-6 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-7] mirrormaker-thread-7 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-8] mirrormaker-thread-8 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-9] mirrormaker-thread-9 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-10] mirrormaker-thread-10 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-11] mirrormaker-thread-11 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-12] mirrormaker-thread-12 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-13] mirrormaker-thread-13 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-14] mirrormaker-thread-14 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-15] mirrormaker-thread-15 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-16] mirrormaker-thread-16 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-17] mirrormaker-thread-17 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,012] INFO [mirrormaker-thread-18] mirrormaker-thread-18 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-9] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-11] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-12] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-6] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-1] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] INFO [mirrormaker-thread-19] mirrormaker-thread-19 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-13] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] INFO [mirrormaker-thread-13] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] TRACE Flushing accumulated records in producer. (org.apache.kafka.clients.producer.KafkaProducer)
[2016-11-01 19:23:32,012] TRACE [mirrormaker-thread-4] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-19] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] INFO [mirrormaker-thread-19] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] TRACE Flushing accumulated records in producer. (org.apache.kafka.clients.producer.KafkaProducer)
[2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-2] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
[2016-11-01 19:23:32,014] INFO [mirrormaker-thread-2] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
--------------------------------------------------------------------------------
server.log
--------------------------------------------------------------------------------
[2016-11-01 19:23:32,031] INFO [GroupCoordinator 0]: Preparing to restabilize group MirrorringGroup with old generation 2 (kafka.coordinator.GroupCoordinator)
[2016-11-01 19:24:01,880] INFO [GroupCoordinator 0]: Group MirrorringGroup generation 2 is dead and removed (kafka.coordinator.GroupCoordinator)
[2016-11-01 19:30:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-11-01 19:40:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2016-11-01 19:50:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
> MirrorMaker shuts down gracefully without being stopped
> -------------------------------------------------------
>
> Key: KAFKA-4367
> URL: https://issues.apache.org/jira/browse/KAFKA-4367
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 0.9.0.1
> Environment: RHEL 7
> Reporter: Alex
>
> Start:
> bin/kafka-mirror-maker.sh --new.consumer --consumer.config config/ssl_mirroring_consumer.properties --producer.config config/ssl_mirroring_producer.properties --whitelist "TOPIC1|TOPIC2|TOPIC3|TOPIC4" --num.streams 20 &> /dev/null &
> MirrorMaker stops working without being stopped. No clue why this problem occurs.
> --------------------------------------------------------------------------------
> kafka-mirror-maker.log
> --------------------------------------------------------------------------------
> [2016-11-01 19:23:32,010] INFO Shutting down consumer threads. (kafka.tools.MirrorMaker$)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-0] mirrormaker-thread-0 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-1] mirrormaker-thread-1 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-2] mirrormaker-thread-2 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-3] mirrormaker-thread-3 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-4] mirrormaker-thread-4 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,011] INFO [mirrormaker-thread-5] mirrormaker-thread-5 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-6] mirrormaker-thread-6 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-7] mirrormaker-thread-7 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-8] mirrormaker-thread-8 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-9] mirrormaker-thread-9 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-10] mirrormaker-thread-10 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-11] mirrormaker-thread-11 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-12] mirrormaker-thread-12 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-13] mirrormaker-thread-13 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-14] mirrormaker-thread-14 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-15] mirrormaker-thread-15 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-16] mirrormaker-thread-16 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-17] mirrormaker-thread-17 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,012] INFO [mirrormaker-thread-18] mirrormaker-thread-18 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-9] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-11] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-12] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-6] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-1] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] INFO [mirrormaker-thread-19] mirrormaker-thread-19 shutting down (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,013] TRACE [mirrormaker-thread-13] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] INFO [mirrormaker-thread-13] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] TRACE Flushing accumulated records in producer. (org.apache.kafka.clients.producer.KafkaProducer)
> [2016-11-01 19:23:32,012] TRACE [mirrormaker-thread-4] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-19] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] INFO [mirrormaker-thread-19] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] TRACE Flushing accumulated records in producer. (org.apache.kafka.clients.producer.KafkaProducer)
> [2016-11-01 19:23:32,014] TRACE [mirrormaker-thread-2] Caught ConsumerWakeupException, continue iteration. (kafka.tools.MirrorMaker$MirrorMakerThread)
> [2016-11-01 19:23:32,014] INFO [mirrormaker-thread-2] Flushing producer. (kafka.tools.MirrorMaker$MirrorMakerThread)
> --------------------------------------------------------------------------------
> server.log
> --------------------------------------------------------------------------------
> [2016-11-01 19:23:32,031] INFO [GroupCoordinator 0]: Preparing to restabilize group MirrorringGroup with old generation 2 (kafka.coordinator.GroupCoordinator)
> [2016-11-01 19:24:01,880] INFO [GroupCoordinator 0]: Group MirrorringGroup generation 2 is dead and removed (kafka.coordinator.GroupCoordinator)
> [2016-11-01 19:30:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
> [2016-11-01 19:40:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
> [2016-11-01 19:50:00,946] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 1 milliseconds. (kafka.coordinator.GroupMetadataManager)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)