Identity Mirroring

[Henry Cai <hc...@pinterest.com.INVALID>]

I saw this feature mentioned in the cloudera blog post:
https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/

High Throughput Identity Mirroring
The batch does not need to be decompressed and compressed and deserialized
and serialized if nothing had to be changed.  Identity mirroring can have a
much higher throughput than the traditional approach. This is another
feature that will be coming soon in MM2.

Is this feature already implemented in MM2?  Is there a KIP or JIRA
associated with the feature?

Re: Identity Mirroring

[Ryanne Dolan <ry...@gmail.com>]

Hey Henry, this was done with MM1 at LinkedIn at one point, but it requires
support for shallow iteration in KafkaConsumer, which was removed from
Apache Kafka a long time ago. Given recent talk of breaking changes in
Kafka 3.0, this might be a good time to revisit this.

Ryanne


On Thu, May 7, 2020, 2:46 AM Henry Cai <hc...@pinterest.com> wrote:

> I saw this feature mentioned in the cloudera blog post:
> https://blog.cloudera.com/a-look-inside-kafka-mirrormaker-2/
>
> High Throughput Identity Mirroring
> The batch does not need to be decompressed and compressed and deserialized
> and serialized if nothing had to be changed.  Identity mirroring can have a
> much higher throughput than the traditional approach. This is another
> feature that will be coming soon in MM2.
>
> Is this feature already implemented in MM2?  Is there a KIP or JIRA
> associated with the feature?
>