You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Ackermann, Manfred (RINDUS) [EXTERNAL]" <M....@ext.douglas.eu> on 2021/06/01 09:23:05 UTC

MirrorMaker only mirrors a subset of configured topics

Dear all,

I’m stuck in getting a setup to work where MirrorMaker consumes from a Confluent Kafka ensemble and produces to Azure Event Hub.

The MirrorMaker is started as a service like

/etc/systemd/system/kafka-azure-event-hub-mm.service
[Unit]
Description=Kafka Azure-Event-Hub MM Service
After=syslog.target network.target
[Service]
ExecStart=/opt/kafka_2.13-2.8.0/bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config /etc/kafka/consumer-azure-event-hub.properties --producer.config /etc/kafka/producer-azure-event-hub.properties --whitelist "OrderPlacedEvent,OrderReturnedEvent,OrderCancelledEvent,LogisticOrderReturn,PointsChangedEvent,PointsChangedEventIT,PointsChangedEventNL,POSReceiptCreatedEvent_20210304"
EnvironmentFile=/etc/sysconfig/kafka-azure-event-hub-mm
SuccessExitStatus=143
LimitNOFILE=500000
LimitNPROC=500000
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target

supposed to mirror the whitelisted topics, but as of KafkaManager only 

OrderCancelledEvent
OrderPlacedEvent 
LogisticOrderReturn
PointsChangedEventIT
POSReceiptCreatedEvent_20210304

show up reviewing the consumer. Checking with 

kafka-consumer-groups --bootstrap-server some-kafka-node:9092 --describe --group prd-mm-azure-event-hub-zey

it shows that 

TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG        CONSUMER-ID                                       HOST                           CLIENT-ID
OrderPlacedEvent               0          836583          836583          0          prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d                 prd-mm-azure-event-hub-zey-0
PointsChangedEvent             0          -               1508747         -          prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d                 prd-mm-azure-event-hub-zey-0
OrderCancelledEvent            0          12410           12410           0          prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d                 prd-mm-azure-event-hub-zey-0
LogisticOrderReturn            0          35537           35537           0          prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d                 prd-mm-azure-event-hub-zey-0
OrderReturnedEvent             0          -               0               -          prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d                 prd-mm-azure-event-hub-zey-0
PointsChangedEventIT           0          214602          214602          0          prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d                 prd-mm-azure-event-hub-zey-0
POSReceiptCreatedEvent_20210304 0          66729           66729           0          prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d                 prd-mm-azure-event-hub-zey-0
PointsChangedEventNL           0          -               263448          -          prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d                 prd-mm-azure-event-hub-zey-0
...

e.g. the PointsChangedEventNL topic has no current offset but OrderPlacedEvent has and I don’t have a clue why.

Any hint or advice would be highly appreciated (please be gentle, it’s my first time here)

Thank you all so much, Manfred