You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Ackermann, Manfred (RINDUS) [EXTERNAL]" <M....@ext.douglas.eu> on 2021/06/01 09:23:05 UTC
MirrorMaker only mirrors a subset of configured topics
Dear all,
I’m stuck in getting a setup to work where MirrorMaker consumes from a Confluent Kafka ensemble and produces to Azure Event Hub.
The MirrorMaker is started as a service like
/etc/systemd/system/kafka-azure-event-hub-mm.service
[Unit]
Description=Kafka Azure-Event-Hub MM Service
After=syslog.target network.target
[Service]
ExecStart=/opt/kafka_2.13-2.8.0/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config /etc/kafka/consumer-azure-event-hub.properties --producer.config /etc/kafka/producer-azure-event-hub.properties --whitelist "OrderPlacedEvent,OrderReturnedEvent,OrderCancelledEvent,LogisticOrderReturn,PointsChangedEvent,PointsChangedEventIT,PointsChangedEventNL,POSReceiptCreatedEvent_20210304"
EnvironmentFile=/etc/sysconfig/kafka-azure-event-hub-mm
SuccessExitStatus=143
LimitNOFILE=500000
LimitNPROC=500000
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
supposed to mirror the whitelisted topics, but as of KafkaManager only
OrderCancelledEvent
OrderPlacedEvent
LogisticOrderReturn
PointsChangedEventIT
POSReceiptCreatedEvent_20210304
show up reviewing the consumer. Checking with
kafka-consumer-groups --bootstrap-server some-kafka-node:9092 --describe --group prd-mm-azure-event-hub-zey
it shows that
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
OrderPlacedEvent 0 836583 836583 0 prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d prd-mm-azure-event-hub-zey-0
PointsChangedEvent 0 - 1508747 - prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d prd-mm-azure-event-hub-zey-0
OrderCancelledEvent 0 12410 12410 0 prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d prd-mm-azure-event-hub-zey-0
LogisticOrderReturn 0 35537 35537 0 prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d prd-mm-azure-event-hub-zey-0
OrderReturnedEvent 0 - 0 - prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d prd-mm-azure-event-hub-zey-0
PointsChangedEventIT 0 214602 214602 0 prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d prd-mm-azure-event-hub-zey-0
POSReceiptCreatedEvent_20210304 0 66729 66729 0 prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d prd-mm-azure-event-hub-zey-0
PointsChangedEventNL 0 - 263448 - prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d prd-mm-azure-event-hub-zey-0
...
e.g. the PointsChangedEventNL topic has no current offset but OrderPlacedEvent has and I don’t have a clue why.
Any hint or advice would be highly appreciated (please be gentle, it’s my first time here)
Thank you all so much, Manfred