MirrorConnectorsIntegrationTest's testOneWayReplicationWithAutoOffsetSync

[Aki Yoshida <el...@gmail.com>]

Hi,
I am new to MirrorMaker and I have been trying to understand its
behavior, in particular about its consumer offset synchronization.

I saw KIP-545 https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0,
which talks about this feature implemented in 2.7.0. I thought this
feature would allow a consumer to be able to switch from the source
cluster and to the mirrored cluster to continue consuming records.

I tried to test this feature with 2.7.0 but was unsuccessful. So, I
looked at testOneWayReplicationWithAutoOffsetSync of
MirrorConnectorsIntegrationTest which seems to test this offset
sychronization feature.
https://github.com/apache/kafka/blob/trunk/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorConnectorsIntegrationTest.java#L453

But I see that in this test, the consumer is switched to the new
cluster when its offset is already set at "latest" and the assertion
is only verifying the consumer to be at the latest offset (i.e., no
records are consumed) at the new cluster when it is switched.
https://github.com/apache/kafka/blob/trunk/connect/mirror/src/test/java/org/apache/kafka/connect/mirror/MirrorConnectorsIntegrationTest.java#L496

This doesn't distinguish the cases between whether the offset is
synchronized or the consumer started at the new cluster with "latest",
no?

So I modified this test to make the consumer's offset to have some lag
when it is switched so that the consumer can consume those remaining
records from the new cluster, as in this change

https://github.com/elakito/kafka/commit/9063db0f8c4a53f5d9764612af898981d499a7b7

But this test failed. So, I am confused. Is my test wrong or do I
misunderstand what KIP-545 is supposed to provide? Any help to
understand this matter would be greatly appreciated.

Regards, aki