You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Jamie <ja...@aol.co.uk.INVALID> on 2021/09/20 07:43:15 UTC
Re: MM2 - Overriding MirrorSourceConnector Consumer & Producer
values
Hi All,
Has anyone been able to override the producer and consumer values of the MM2 MirrorSourceConnector as described below?
Many Thanks,
Jamie
-----Original Message-----
From: Jamie <ja...@aol.co.uk>
To: users@kafka.apache.org <us...@kafka.apache.org>
Sent: Thu, 16 Sep 2021 15:52
Subject: MM2 - Overriding MirrorSourceConnector Consumer & Producer values
Hi All,
I've trying to override the properties of the consumer and producer in MM2 to tune them for high throughput. For example, I'm trying to set the consumers fetch.min.bytes to 100000.
I'm running MM2 in a dedicated mirror maker cluster (https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-RunningadedicatedMirrorMakercluster) and using version 2.7.1 of Kafka.
I have the equivalent of the following in my mirror maker properties file:
clusters = A, B
A.bootstrap.servers = A_host1:9092, A_host2:9092, A_host3:9092 B.bootstrap.servers = B_host1:9092, B_host2:9092, B_host3:9092
# enable and configure individual replication flows A->B.enabled = true
# regex which defines which topics gets replicated. For eg "foo-.*" A->B.topics = .test-topic
I'm trying to override the properties of the consumer which fetches records from cluster "A" and the producer that sends records to cluster "B".
I've tried the following in the config file:
A.consumer.fetch.min.bytes = 100000
A->B.consumer.fetch.min.bytes = 100000
A.fetch.min.bytes = 100000
B.consumer.fetch.min.bytes = 100000
B.fetch.min.bytes = 100000
None of which seem to work, when I start MM2 and go into the logs and look at the value using the the MirrorSourceConnector tasks consumer and producer config I still see the default value for fetch.min.bytes (1) being used.
Am I trying to override the values of the consumer incorrectly or do I need to set these in a different place?
Many Thanks,
Jamie
Re: MM2 - Overriding MirrorSourceConnector Consumer & Producer values
Posted by Men Lim <zu...@gmail.com>.
did you try setting these values in the distributor.properties file?
On Mon, Sep 20, 2021 at 12:43 AM Jamie <ja...@aol.co.uk.invalid> wrote:
> Hi All,
> Has anyone been able to override the producer and consumer values of the
> MM2 MirrorSourceConnector as described below?
> Many Thanks,
> Jamie
>
>
> -----Original Message-----
> From: Jamie <ja...@aol.co.uk>
> To: users@kafka.apache.org <us...@kafka.apache.org>
> Sent: Thu, 16 Sep 2021 15:52
> Subject: MM2 - Overriding MirrorSourceConnector Consumer & Producer values
>
> Hi All,
> I've trying to override the properties of the consumer and producer in MM2
> to tune them for high throughput. For example, I'm trying to set the
> consumers fetch.min.bytes to 100000.
> I'm running MM2 in a dedicated mirror maker cluster (
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP382:MirrorMaker2.0-RunningadedicatedMirrorMakercluster)
> and using version 2.7.1 of Kafka.
> I have the equivalent of the following in my mirror maker properties file:
> clusters = A, B
> A.bootstrap.servers = A_host1:9092, A_host2:9092, A_host3:9092
> B.bootstrap.servers = B_host1:9092, B_host2:9092, B_host3:9092
> # enable and configure individual replication flows A->B.enabled =
> true
> # regex which defines which topics gets replicated. For eg "foo-.*"
> A->B.topics = .test-topic
>
> I'm trying to override the properties of the consumer which fetches
> records from cluster "A" and the producer that sends records to cluster
> "B".
> I've tried the following in the config file:
> A.consumer.fetch.min.bytes = 100000
> A->B.consumer.fetch.min.bytes = 100000
> A.fetch.min.bytes = 100000
> B.consumer.fetch.min.bytes = 100000
> B.fetch.min.bytes = 100000
> None of which seem to work, when I start MM2 and go into the logs and look
> at the value using the the MirrorSourceConnector tasks consumer and
> producer config I still see the default value for fetch.min.bytes (1) being
> used.
> Am I trying to override the values of the consumer incorrectly or do I
> need to set these in a different place?
> Many Thanks,
> Jamie
>
>
>
>