[jira] [Assigned] (KAFKA-16622) Mirromaker2 first Checkpoint not emitted until consumer group fully catches up once

["Edoardo Comar (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Edoardo Comar reassigned KAFKA-16622:
-------------------------------------

    Assignee: Edoardo Comar

> Mirromaker2 first Checkpoint not emitted until consumer group fully catches up once
> -----------------------------------------------------------------------------------
>
>                 Key: KAFKA-16622
>                 URL: https://issues.apache.org/jira/browse/KAFKA-16622
>             Project: Kafka
>          Issue Type: Bug
>          Components: mirrormaker
>    Affects Versions: 3.7.0, 3.6.2, 3.8.0
>            Reporter: Edoardo Comar
>            Assignee: Edoardo Comar
>            Priority: Major
>         Attachments: connect.log.2024-04-26-10.zip, edo-connect-mirror-maker-sourcetarget.properties
>
>
> We observed an excessively delayed emission of the MM2 Checkpoint record.
> It only gets created when the source consumer reaches the end of a topic. This does not seem reasonable.
> In a very simple setup :
> Tested with a standalone single process MirrorMaker2 mirroring between two single-node kafka clusters(mirromaker config attached) with quick refresh intervals (eg 5 sec) and a small offset.lag.max (eg 10)
> create a single topic in the source cluster
> produce data to it (e.g. 10000 records)
> start a slow consumer - e.g. fetching 50records/poll and pausing 1 sec between polls which commits after each poll
> watch the Checkpoint topic in the target cluster
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 \
>   --topic source.checkpoints.internal \
>   --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter \
>    --from-beginning
> -> no record appears in the checkpoint topic until the consumer reaches the end of the topic (ie its consumer group lag gets down to 0).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)