You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Rens Groothuijsen (Jira)" <ji...@apache.org> on 2022/07/24 17:32:00 UTC
[jira] [Commented] (KAFKA-13796) MM2 - Topics Exclude/Blacklist not working
[ https://issues.apache.org/jira/browse/KAFKA-13796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17570518#comment-17570518 ]
Rens Groothuijsen commented on KAFKA-13796:
-------------------------------------------
From what I can see, adding quote marks around the topic name makes a difference for topics.exclude. 'topics.exclude=topicname' produced the expected result, whereas 'topics.exclude="topicname"' did not.
> MM2 - Topics Exclude/Blacklist not working
> ------------------------------------------
>
> Key: KAFKA-13796
> URL: https://issues.apache.org/jira/browse/KAFKA-13796
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.0.0
> Reporter: David Bros
> Priority: Blocker
> Fix For: 3.0.0
>
>
> Hi all,
> We are testing Kafka 3.0.0 in a development environment with Java 17 in a Centos 7 machine.
> When setting up MM2 replication between two clusters we exclude some topics we don't want MM2 to replicate via topics.exclude (we have also tried topics.blacklist)
> Even though the configuration is parsed correctly, we still get those topics replicated among clusters.
> Here is our config for the us_central cluster:
> ```
> # Clusters and bootstrap servers
> clusters=eu_west, us_central
> eu_west.bootstrap.servers=XXnet:9092,XX:9092
> us_central.bootstrap.servers=XX:9092
> us_central.consumer.auto.offset.reset=latest
> us_central.consumer.fetch.max.bytes=31457280
> us_central.consumer.fetch.max.wait.ms=10000
> us_central.consumer.max.poll.records=3000
> us_central.consumer.request.timeout.ms=60000
> eu_west.consumer.auto.offset.reset=latest
> eu_west.consumer.fetch.max.bytes=52428800
> eu_west.consumer.fetch.max.wait.ms=10000
> eu_west.consumer.max.poll.records=3000
> eu_west.consumer.request.timeout.ms=60000
> # Custom producer settings
> us_central.producer.max.request.size=27262976
> us_central.producer.batch.size=22000
> us_central.producer.compression.type=none
> us_central.producer.send.buffer.bytes=26214400
> us_central.producer.receive.buffer.bytes=26214400
> eu_west.producer.max.request.size=27262976
> eu_west.producer.batch.size=22000
> eu_west.producer.compression.type=none
> eu_west.producer.send.buffer.bytes=26214400
> eu_west.producer.receive.buffer.bytes=26214400
> # Topics configuration
> topics=.*
> topics.blacklist="mm2_test_0"
> # Tasks (threads)
> tasks.max=10
> # Groups
> groups=phx_netflow
> # Replication factors, these are 1 for lab
> replication.factor=1
> config.storage.replication.factor=1
> offset.storage.replication.factor=1
> status.storage.replication.factor=1
> checkpoints.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> sync.group.offsets.replication.factor=1
> ## Config
> config.properties.exclude=local.retention.ms
> # Refresh rates
> us_central.refresh.topics.interval.seconds=15
> us_central.refresh.groups.interval.seconds=15
> eu_west.refresh.topics.interval.seconds=15
> eu_west.refresh.groups.interval.seconds=15
> ## Sync options
> sync.topic.acls.enabled=true
> offset-syncs.topic.replication.enabled=true
> sync.group.offsets.enabled=true
> # EU pulls us, syd, nyc
> us_central->eu_west.enabled=true
> us_central->eu_west.sync.group.offsets.enabled=false
> # US pulls eu, syd, nyc
> eu_west->us_central.enabled=true
> eu_west->us_central.sync.group.offsets.enabled=true
> ```
> Here is the configuration for eu_west cluster:
> ```
> # Clusters and bootstrap servers
> clusters=eu_west, us_central
> eu_west.bootstrap.servers=XX,XX:9092
> us_central.bootstrap.servers=XX:9092
> us_central.consumer.auto.offset.reset=latest
> us_central.consumer.fetch.max.bytes=31457280
> us_central.consumer.fetch.max.wait.ms=10000
> us_central.consumer.max.poll.records=3000
> us_central.consumer.request.timeout.ms=60000
> eu_west.consumer.auto.offset.reset=latest
> eu_west.consumer.fetch.max.bytes=52428800
> eu_west.consumer.fetch.max.wait.ms=10000
> eu_west.consumer.max.poll.records=3000
> eu_west.consumer.request.timeout.ms=60000
> # Custom producer settings
> us_central.producer.max.request.size=27262976
> us_central.producer.batch.size=22000
> us_central.producer.compression.type=none
> us_central.producer.send.buffer.bytes=26214400
> us_central.producer.receive.buffer.bytes=26214400
> eu_west.producer.max.request.size=27262976
> eu_west.producer.batch.size=22000
> eu_west.producer.compression.type=none
> eu_west.producer.send.buffer.bytes=26214400
> eu_west.producer.receive.buffer.bytes=26214400
> # Topics configuration
> topics=.*
> topics.blacklist="mm2_test_0"
> # Tasks (threads)
> tasks.max=10
> # Groups
> groups=ams_netflow
> # Replication factors, these are 1 for lab
> replication.factor=1
> config.storage.replication.factor=1
> offset.storage.replication.factor=1
> status.storage.replication.factor=1
> checkpoints.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> sync.group.offsets.replication.factor=1
> ## Config
> config.properties.exclude=local.retention.ms
> # Refresh rates
> us_central.refresh.topics.interval.seconds=15
> us_central.refresh.groups.interval.seconds=15
> eu_west.refresh.topics.interval.seconds=15
> eu_west.refresh.groups.interval.seconds=15
> ## Sync options
> sync.topic.acls.enabled=true
> offset-syncs.topic.replication.enabled=true
> sync.group.offsets.enabled=true
> # EU pulls us, syd, nyc
> us_central->eu_west.enabled=true
> us_central->eu_west.sync.group.offsets.enabled=true
> # US pulls eu, syd, nyc
> eu_west->us_central.enabled=true
> eu_west->us_central.sync.group.offsets.enabled=false
> ```
> When starting MM2 freshly installed (deleted all mm2 related topics), we still get the excluded topics replicated between clusters.
> It's also worth mentioning that the connector seems to be reading the configuration correctly however it is (probably) at a later stage where the topics get filtered incorrectly.
> I attach a log from our connect.log where the exclude field is correctly parsed from our connect-mirrormaker.properties.
> ```
> groups.exclude = [console-consumer-.*, connect-.*, __.*]
> topics.exclude = [mm2_test_0]
> exclude.internal.topics = true
> ```
> After much testing and over 20 combinations of regex we have not found a way to exclude the topics via the configuration properties specified in the documentation for MM2.
> The *whitelist* does work though, so we are a bit confused in why the blacklist does not.
> We request help to further test this, we are not sure if we are doing something wrong.
>
> Thanks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)