Simplifying standalone mm2-connect config

[Karan Kumar <ka...@gmail.com>]

Hi

One of the nice things about kafka is setting up in the local environment
is really simple. I was giving a try to the latest feature ie MM2 and found
it took me some time to get a minimal setup running.
Default config provided assumes that there will already be 3 brokers
running due to the default replication factor of the admin topics the mm2
connector creates.

This got me thinking that most of the people would follow the same approach
I followed.
1. Start a single broker cluster on 9092
2. Start another single cluster broker on, let's say, 10002
3. Start mm2 by"./bin/connect-mirror-maker.sh
./config/connect-mirror-maker.properties"

What happened was I had to supply a lot more configs

clusters = A, B

# connection information for each cluster
A.bootstrap.servers = localhost:9092
B.bootstrap.servers = localhost:10092

# enable and configure individual replication flows
A->B.enabled = *
A->B.topics = test
B->A.enabled = true
B->A.topics = *


A.heartbeats.topic.replication.factor=1
A.checkpoints.topic.replication.factor=1
A.offset-syncs.topic.replication.factor=1

B.heartbeats.topic.replication.factor=1
B.checkpoints.topic.replication.factor=1
B.offset-syncs.topic.replication.factor=1


A.offset.storage.replication.factor=1
B.offset.storage.replication.factor=1



A.status.storage.replication.factor=1
B.status.storage.replication.factor=1


A.config.storage.replication.factor=1
B.config.storage.replication.factor=1


The server.properties has bunch of properties like
"offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1"
which make it easier for people to start a local single broker kafka
cluster.


Does it make sense to have a similar config as the default mirror maker
config so that it becomes easier for people to use the MM2 feature.

-- 
Thanks
Karan

Re: Simplifying standalone mm2-connect config

[Karan Kumar <ka...@gmail.com>]

Hi Ryanne

Thanks for the response. Based on our discussion, raised a jira : KAFKA-9337
<https://issues.apache.org/jira/browse/KAFKA-9337>. Please find the PR here
<https://github.com/apache/kafka/pull/7872>.

On Tue, Dec 24, 2019 at 8:18 PM Ryanne Dolan <ry...@gmail.com> wrote:

> Hello Karan. I agree the initial experience could be a lot friendlier. Most
> of the complexity there is inherited from Connect, but it's compounded when
> multiple clusters are involved.
>
> I don't think we want to change Connect's (or MM2's) defaults to assume a
> single broker cluster -- it'd be too easy to overlook such defaults in a
> prod environment, which could be dangerous. It might be nice to add a
> high-level internal.replication.factor property that, if specified,
> overrides the replication factor for all internal topics. I actually had a
> prototype of this at one point but removed it, as it wasn't part of the
> KIP.
>
> I agree we should modify the example config to work out-of-the-box with
> single-broker clusters, and just call out that these should not be used in
> a production environment. Lemme know if you want to create a PR or if you'd
> like me to. It looks like you've already done most of the work :)
>
> Ryanne
>
>
> On Tue, Dec 24, 2019, 3:41 AM Karan Kumar <ka...@gmail.com>
> wrote:
>
> > Hi
> >
> > One of the nice things about kafka is setting up in the local environment
> > is really simple. I was giving a try to the latest feature ie MM2 and
> found
> > it took me some time to get a minimal setup running.
> > Default config provided assumes that there will already be 3 brokers
> > running due to the default replication factor of the admin topics the mm2
> > connector creates.
> >
> > This got me thinking that most of the people would follow the same
> approach
> > I followed.
> > 1. Start a single broker cluster on 9092
> > 2. Start another single cluster broker on, let's say, 10002
> > 3. Start mm2 by"./bin/connect-mirror-maker.sh
> > ./config/connect-mirror-maker.properties"
> >
> > What happened was I had to supply a lot more configs
> >
> > clusters = A, B
> >
> > # connection information for each cluster
> > A.bootstrap.servers = localhost:9092
> > B.bootstrap.servers = localhost:10092
> >
> > # enable and configure individual replication flows
> > A->B.enabled = *
> > A->B.topics = test
> > B->A.enabled = true
> > B->A.topics = *
> >
> >
> > A.heartbeats.topic.replication.factor=1
> > A.checkpoints.topic.replication.factor=1
> > A.offset-syncs.topic.replication.factor=1
> >
> > B.heartbeats.topic.replication.factor=1
> > B.checkpoints.topic.replication.factor=1
> > B.offset-syncs.topic.replication.factor=1
> >
> >
> > A.offset.storage.replication.factor=1
> > B.offset.storage.replication.factor=1
> >
> >
> >
> > A.status.storage.replication.factor=1
> > B.status.storage.replication.factor=1
> >
> >
> > A.config.storage.replication.factor=1
> > B.config.storage.replication.factor=1
> >
> >
> > The server.properties has bunch of properties like
> > "offsets.topic.replication.factor=1
> > transaction.state.log.replication.factor=1"
> > which make it easier for people to start a local single broker kafka
> > cluster.
> >
> >
> > Does it make sense to have a similar config as the default mirror maker
> > config so that it becomes easier for people to use the MM2 feature.
> >
> > --
> > Thanks
> > Karan
> >
>


-- 
Thanks
Karan

Re: Simplifying standalone mm2-connect config

[Ryanne Dolan <ry...@gmail.com>]

Hello Karan. I agree the initial experience could be a lot friendlier. Most
of the complexity there is inherited from Connect, but it's compounded when
multiple clusters are involved.

I don't think we want to change Connect's (or MM2's) defaults to assume a
single broker cluster -- it'd be too easy to overlook such defaults in a
prod environment, which could be dangerous. It might be nice to add a
high-level internal.replication.factor property that, if specified,
overrides the replication factor for all internal topics. I actually had a
prototype of this at one point but removed it, as it wasn't part of the KIP.

I agree we should modify the example config to work out-of-the-box with
single-broker clusters, and just call out that these should not be used in
a production environment. Lemme know if you want to create a PR or if you'd
like me to. It looks like you've already done most of the work :)

Ryanne


On Tue, Dec 24, 2019, 3:41 AM Karan Kumar <ka...@gmail.com> wrote:

> Hi
>
> One of the nice things about kafka is setting up in the local environment
> is really simple. I was giving a try to the latest feature ie MM2 and found
> it took me some time to get a minimal setup running.
> Default config provided assumes that there will already be 3 brokers
> running due to the default replication factor of the admin topics the mm2
> connector creates.
>
> This got me thinking that most of the people would follow the same approach
> I followed.
> 1. Start a single broker cluster on 9092
> 2. Start another single cluster broker on, let's say, 10002
> 3. Start mm2 by"./bin/connect-mirror-maker.sh
> ./config/connect-mirror-maker.properties"
>
> What happened was I had to supply a lot more configs
>
> clusters = A, B
>
> # connection information for each cluster
> A.bootstrap.servers = localhost:9092
> B.bootstrap.servers = localhost:10092
>
> # enable and configure individual replication flows
> A->B.enabled = *
> A->B.topics = test
> B->A.enabled = true
> B->A.topics = *
>
>
> A.heartbeats.topic.replication.factor=1
> A.checkpoints.topic.replication.factor=1
> A.offset-syncs.topic.replication.factor=1
>
> B.heartbeats.topic.replication.factor=1
> B.checkpoints.topic.replication.factor=1
> B.offset-syncs.topic.replication.factor=1
>
>
> A.offset.storage.replication.factor=1
> B.offset.storage.replication.factor=1
>
>
>
> A.status.storage.replication.factor=1
> B.status.storage.replication.factor=1
>
>
> A.config.storage.replication.factor=1
> B.config.storage.replication.factor=1
>
>
> The server.properties has bunch of properties like
> "offsets.topic.replication.factor=1
> transaction.state.log.replication.factor=1"
> which make it easier for people to start a local single broker kafka
> cluster.
>
>
> Does it make sense to have a similar config as the default mirror maker
> config so that it becomes easier for people to use the MM2 feature.
>
> --
> Thanks
> Karan
>