How do I debug replication that never completes on Kraft/3.6.+

[Rinaldo Digiorgio <rd...@nabancard.com>]

Hello Kafka Community,

I'm reaching out for assistance with an issue I've encountered in our
Kafka cluster that's been running smoothly for over a year. Our setup
includes:

Kafka Version: 3.6.0
Cluster Configuration: 3-node cluster running in KRaft mode (without ZooKeeper).
Issue Description: Despite configuring replication for some topics to
enhance data redundancy and fault tolerance, AKHQ (our Kafka
monitoring tool) indicates that these topics are not replicating as
expected. This is puzzling because the cluster otherwise operates
normally, and we've been relying on this setup for a significant
period.

Given the following:

The cluster is in KRaft mode, aiming for a simplified architecture.
We've specifically added replication to certain topics, expecting this
to reflect in AKHQ.
Despite our efforts and the cluster's long-term stability, replication
issues persist according to AKHQ.

I'm wondering if anyone has encountered similar issues or could offer
insights into:

Potential misconfigurations with topic replication factors or broker
settings that could lead to this issue.
Any known quirks or bugs with KRaft mode that might affect replication
visibility or functionality.
Steps for diagnosing and resolving issues related to replication in a
KRaft mode cluster, especially regarding tooling compatibility with
Kafka 3.6.0.
Suggestions for verifying network connectivity between brokers or any
ACL/permission settings that might inadvertently prevent proper
replication.

Any advice, diagnostic tips, or shared experiences with similar setups
would be immensely appreciated. I'm particularly interested in
ensuring that our replication configurations are effective and
accurately monitored, maintaining the high availability and fault
tolerance our Kafka cluster is intended to provide.

Thank you in advance for your time and assistance!


-- 
Rinaldo DiGiorgio - 410-714-1099 www.nabancard.com
North American Bancard, LLC - 250 Stephenson Highway -Troy, MI 48083
CONFIDENTIALITY NOTICE

This e-mail message and any attachments are only for the use of the
intended recipient and may contain information that is privileged,
confidential or exempt from disclosure under applicable law. If you
are not the intended recipient, any disclosure, distribution or other
use of this e-mail message or attachments is prohibited. If you have
received this e-mail message in error, please delete and notify the
sender immediately. Thank you.