When using mirrormaker, how are people creating topics?

[James Cheng <wu...@gmail.com>]

Hi,

We are using mirrormaker to mirror topics from one cluster to another, and I wanted to get some advice from the community on how people are doing mirroring. In particular, how are people dealing with topic creation?

Do you turn on auto-topic creation in your destination clusters (auto.create.topics.enable=true)?

If not, do you manually create the individual destination topics?

If so, how does that work with mirroring based on a whitelist (regex)?

The way we are doing it right now is, we have our regex in a file somewhere. The regex is used in 2 ways:
1) Passed to mirrormaker, to do the mirroring.
2) Passed to a program which looks up all the topics on the source cluster, finds the ones that match the regex, and then creates them on the destination cluster. (We have auto-topic creation turned off auto.create.topics.enable=false)

One downside of that approach is there a potential race, where if the regex changes, then mirrormaker (in #1) might start trying to produce to a new destination topic before the topic was created (by #2).

Some other hand-wavy ideas that came to mind might be:
* handling topic creation in a MirrorMakerMessageHandler
* handling topic creation in an interceptor

Anyway, was hoping to get some thoughts from people who are already doing this.

Thanks!
-James

Re: When using mirrormaker, how are people creating topics?

[Todd Palino <tp...@gmail.com>]

For most of our clusters, we just use auto topic creation and it’s handled
that way. Periodically we’ll go through and clean up partition counts
across everything if there’s a new high-volume topic. We also have the
ability for people to pre-create topics using a central management system.

For the special mirror maker that we have that does 1-to-1 mappings between
partitions for clusters that do not have auto topic creation enabled, the
topic creation (or partition count changes) are taken care of in the
message handler.

-Todd


On Mon, Dec 5, 2016 at 12:42 AM, James Cheng <wu...@gmail.com> wrote:

> Hi,
>
> We are using mirrormaker to mirror topics from one cluster to another, and
> I wanted to get some advice from the community on how people are doing
> mirroring. In particular, how are people dealing with topic creation?
>
> Do you turn on auto-topic creation in your destination clusters
> (auto.create.topics.enable=true)?
>
> If not, do you manually create the individual destination topics?
>
> If so, how does that work with mirroring based on a whitelist (regex)?
>
> The way we are doing it right now is, we have our regex in a file
> somewhere. The regex is used in 2 ways:
> 1) Passed to mirrormaker, to do the mirroring.
> 2) Passed to a program which looks up all the topics on the source
> cluster, finds the ones that match the regex, and then creates them on the
> destination cluster. (We have auto-topic creation turned off
> auto.create.topics.enable=false)
>
> One downside of that approach is there a potential race, where if the
> regex changes, then mirrormaker (in #1) might start trying to produce to a
> new destination topic before the topic was created (by #2).
>
> Some other hand-wavy ideas that came to mind might be:
> * handling topic creation in a MirrorMakerMessageHandler
> * handling topic creation in an interceptor
>
> Anyway, was hoping to get some thoughts from people who are already doing
> this.
>
> Thanks!
> -James
>
>


-- 
*Todd Palino*
Staff Site Reliability Engineer
Data Infrastructure Streaming



linkedin.com/in/toddpalino