Mirror Maker for Avro Messages (with schema registry)

[Adithya Tirumale <at...@peak6.com>]

Hi,

We have a use case where we need to maintain a replicated/cloned kafka cluster in a lower environment mirroring the production kafka cluster. The messages in kafka are a mix of JSON and AVRO messages. The schemas for the avro messages are registered using schema registry.

Using Mirror maker works for JSON messages but doesn't work well with avro messages. Since the lower(cloned) environment has its own schema registry, the schema id for the messages might go out of sync between the production and the lower(cloned) environment.

To solve this problem, I can think of the below options :


  *   Add _schemas topic to the list of topics to be syncd by mirror maker. This way when the applications in the cloned environment try to look up schema for a schema id it gets the same thing as in production.
  *   Add a special message transformation to the mirror maker to basically translate the production schema ids to the cloned environment schema ids. For example if the schema id for a value for the foo-bar topic is 15 in production and 13 in the cloned environment, mirror maker would set schema id field to 13 before writing the message to the cloned env.

My Questions :

  *   Are the above 2 options acceptable or am I missing something here ?
  *   Are there any other options I'm missing ?

Thanks!



______________________________________________________________________
See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email