You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vishal Santoshi <vi...@gmail.com> on 2019/11/03 15:40:26 UTC
MirrorMaker 2.0 ( working off origin/2.4 )
Hello folks,
Was doing stress tests and realized that the replication
to the target cluster and thus the configuration of the KafkaProducer has a
default acks of -1 ( all ) and that was prohibitively expensive. It should
have been a simple a->b.producer.acks = 1 ( or b.producer.acks = 1 or acks
= 1, have tried all ) override in the mm2.properties, but it does not take.
I verified by turning on TRACE and a little instrumenting the code to
print out the acks .
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L806
.
Am I missing something?
Regards
Re: MirrorMaker 2.0 ( working off origin/2.4 )
Posted by Vishal Santoshi <vi...@gmail.com>.
I can verify that the above did take. ( kicking myself ) . It should be the
same for, these too ?
b.producer.batch.size = 1048576
b.producer.linger.ms = 30
b.producer.acks = 1
etc etc...
I also see that the properties can be overridden, so this routine
* kill 1 MM2
* change the mm2.properites
* restart mm2
should work provided we do not change the client_id
Thanks.
On Mon, Nov 4, 2019 at 3:08 PM Ryanne Dolan <ry...@gmail.com> wrote:
> > BTW any ideas when 2.4 is being released
>
> Looks like there are a few blockers still.
>
> On Mon, Nov 4, 2019 at 2:06 PM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
> > I bet I have tested the "b.producer.acks' route. I will test again and
> let
> > you know. Note that I resorted to hardcoding that value in the Sender and
> > that alleviated the throttle I was seeing on consumption. BTW any ideas
> > when 2.4 is being released ( I thought it was Oct 30th 2019 )...
> >
> > On Mon, Nov 4, 2019 at 1:49 PM Ryanne Dolan <ry...@gmail.com>
> wrote:
> >
> > > Vishal, b.producer.acks should work, as can be seen in the following
> unit
> > > test with similar producer property "client.id":
> > >
> > >
> > >
> >
> https://github.com/apache/kafka/blob/6b905ade0cdc7a5f6f746727ecfe4e7a7463a200/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorMakerConfigTest.java#L182
> > >
> > > Keep in mind this should be configured for the target cluster, and it
> > might
> > > not take effect until the workers finish rebalancing.
> > >
> > > Ryanne
> > >
> > > On Sun, Nov 3, 2019, 9:40 AM Vishal Santoshi <
> vishal.santoshi@gmail.com>
> > > wrote:
> > >
> > > > Hello folks,
> > > > Was doing stress tests and realized that the
> > > replication
> > > > to the target cluster and thus the configuration of the KafkaProducer
> > > has a
> > > > default acks of -1 ( all ) and that was prohibitively expensive. It
> > > should
> > > > have been a simple a->b.producer.acks = 1 ( or b.producer.acks = 1
> or
> > > acks
> > > > = 1, have tried all ) override in the mm2.properties, but it does not
> > > take.
> > > > I verified by turning on TRACE and a little instrumenting the code
> to
> > > > print out the acks .
> > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L806
> > > > .
> > > > Am I missing something?
> > > >
> > > > Regards
> > > >
> > >
> >
>
Re: MirrorMaker 2.0 ( working off origin/2.4 )
Posted by Ryanne Dolan <ry...@gmail.com>.
> BTW any ideas when 2.4 is being released
Looks like there are a few blockers still.
On Mon, Nov 4, 2019 at 2:06 PM Vishal Santoshi <vi...@gmail.com>
wrote:
> I bet I have tested the "b.producer.acks' route. I will test again and let
> you know. Note that I resorted to hardcoding that value in the Sender and
> that alleviated the throttle I was seeing on consumption. BTW any ideas
> when 2.4 is being released ( I thought it was Oct 30th 2019 )...
>
> On Mon, Nov 4, 2019 at 1:49 PM Ryanne Dolan <ry...@gmail.com> wrote:
>
> > Vishal, b.producer.acks should work, as can be seen in the following unit
> > test with similar producer property "client.id":
> >
> >
> >
> https://github.com/apache/kafka/blob/6b905ade0cdc7a5f6f746727ecfe4e7a7463a200/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorMakerConfigTest.java#L182
> >
> > Keep in mind this should be configured for the target cluster, and it
> might
> > not take effect until the workers finish rebalancing.
> >
> > Ryanne
> >
> > On Sun, Nov 3, 2019, 9:40 AM Vishal Santoshi <vi...@gmail.com>
> > wrote:
> >
> > > Hello folks,
> > > Was doing stress tests and realized that the
> > replication
> > > to the target cluster and thus the configuration of the KafkaProducer
> > has a
> > > default acks of -1 ( all ) and that was prohibitively expensive. It
> > should
> > > have been a simple a->b.producer.acks = 1 ( or b.producer.acks = 1 or
> > acks
> > > = 1, have tried all ) override in the mm2.properties, but it does not
> > take.
> > > I verified by turning on TRACE and a little instrumenting the code to
> > > print out the acks .
> > >
> > >
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L806
> > > .
> > > Am I missing something?
> > >
> > > Regards
> > >
> >
>
Re: MirrorMaker 2.0 ( working off origin/2.4 )
Posted by Vishal Santoshi <vi...@gmail.com>.
I bet I have tested the "b.producer.acks' route. I will test again and let
you know. Note that I resorted to hardcoding that value in the Sender and
that alleviated the throttle I was seeing on consumption. BTW any ideas
when 2.4 is being released ( I thought it was Oct 30th 2019 )...
On Mon, Nov 4, 2019 at 1:49 PM Ryanne Dolan <ry...@gmail.com> wrote:
> Vishal, b.producer.acks should work, as can be seen in the following unit
> test with similar producer property "client.id":
>
>
> https://github.com/apache/kafka/blob/6b905ade0cdc7a5f6f746727ecfe4e7a7463a200/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorMakerConfigTest.java#L182
>
> Keep in mind this should be configured for the target cluster, and it might
> not take effect until the workers finish rebalancing.
>
> Ryanne
>
> On Sun, Nov 3, 2019, 9:40 AM Vishal Santoshi <vi...@gmail.com>
> wrote:
>
> > Hello folks,
> > Was doing stress tests and realized that the
> replication
> > to the target cluster and thus the configuration of the KafkaProducer
> has a
> > default acks of -1 ( all ) and that was prohibitively expensive. It
> should
> > have been a simple a->b.producer.acks = 1 ( or b.producer.acks = 1 or
> acks
> > = 1, have tried all ) override in the mm2.properties, but it does not
> take.
> > I verified by turning on TRACE and a little instrumenting the code to
> > print out the acks .
> >
> >
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L806
> > .
> > Am I missing something?
> >
> > Regards
> >
>
Re: MirrorMaker 2.0 ( working off origin/2.4 )
Posted by Ryanne Dolan <ry...@gmail.com>.
Vishal, b.producer.acks should work, as can be seen in the following unit
test with similar producer property "client.id":
https://github.com/apache/kafka/blob/6b905ade0cdc7a5f6f746727ecfe4e7a7463a200/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorMakerConfigTest.java#L182
Keep in mind this should be configured for the target cluster, and it might
not take effect until the workers finish rebalancing.
Ryanne
On Sun, Nov 3, 2019, 9:40 AM Vishal Santoshi <vi...@gmail.com>
wrote:
> Hello folks,
> Was doing stress tests and realized that the replication
> to the target cluster and thus the configuration of the KafkaProducer has a
> default acks of -1 ( all ) and that was prohibitively expensive. It should
> have been a simple a->b.producer.acks = 1 ( or b.producer.acks = 1 or acks
> = 1, have tried all ) override in the mm2.properties, but it does not take.
> I verified by turning on TRACE and a little instrumenting the code to
> print out the acks .
>
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java#L806
> .
> Am I missing something?
>
> Regards
>