[jira] [Resolved] (ATLAS-5336) Upgrade Kafka to 3.9.1; embedded notification broker uses KRaft (no ZooKeeper))

["Ramachandran Krishnan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ATLAS-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ramachandran Krishnan resolved ATLAS-5336.
------------------------------------------
    Resolution: Fixed

>  Upgrade Kafka to 3.9.1; embedded notification broker uses KRaft (no ZooKeeper))
> --------------------------------------------------------------------------------
>
>                 Key: ATLAS-5336
>                 URL: https://issues.apache.org/jira/browse/ATLAS-5336
>             Project: Atlas
>          Issue Type: Task
>          Components:  atlas-core
>            Reporter: Ramachandran Krishnan
>            Assignee: Ramachandran Krishnan
>            Priority: Major
>             Fix For: 3.0.0
>
>          Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> h3. Summary
> Upgrade Apache Atlas notification Kafka from *2.8.2* (Scala 2.12) to *3.9.1* (Scala 2.13), and replace the ZooKeeper-based in-process embedded broker with a *KRaft* broker via {{{}KafkaClusterTestKit{}}}.
> h3. Background / Motivation
>  * Kafka 2.8.x clients are only partially compatible with modern 3.9.x / 4.x brokers.
>  * The legacy {{EmbeddedKafkaServer}} started an in-process {{ZooKeeperServer}} plus {{KafkaServer}} tied to ZK — unnecessary complexity and a maintenance burden as Kafka moves to KRaft-only.
>  * Atlas remains on Spring 5.3; Spring Kafka 3.x / {{EmbeddedKafkaKraftBroker}} requires Spring 6. Using Kafka's {{KafkaClusterTestKit}} provides the same KRaft engine without a Spring upgrade.
>  * Aligns Atlas with Kafka 3.9.x before external cluster KRaft migration and a future Kafka 4.x path.
> h3. Proposed Changes
> ||Area||Before||After||
> |{{kafka.version}} (root {{{}pom.xml{}}})|2.8.2|*3.9.1*|
> |{{kafka.scala.binary.version}}|2.12|*2.13*|
> |{{EmbeddedKafkaServer}}|In-process ZK + {{KafkaServer}}|{{KafkaClusterTestKit}} (combined KRaft controller+broker)|
> |Embedded ZK for notification|Required at Atlas startup when embedded=true|*Removed* — no {{ZooKeeperServer}} / {{zookeeper.connect}} in embedded path|
> |Docker {{atlas-kafka}} image|{{kafka_2.12-*}} tarball|{{kafka_2.13-3.9.1}} ({{{}.env{}}}, {{{}Dockerfile.atlas-kafka{}}}, {{{}download-archives.sh{}}})|
> |Default distro config|ZK-oriented embedded settings|{{{}atlas.notification.embedded=true{}}}; {{bootstrap.servers}} placeholder updated *at runtime* by {{EmbeddedKafkaServer}}|
> h3. Core code — {{EmbeddedKafkaServer}}
> *Path:* {{notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java}}
> When {{{}atlas.notification.embedded=true{}}}:
>  # Start KRaft broker via {{KafkaClusterTestKit}} (1 combined controller+broker node).
>  # Assign ephemeral {{bootstrap.servers}} (e.g. {{{}localhost:40111{}}}).
>  # Write live bootstrap URL into {{atlas.kafka.bootstrap.servers}} so {{KafkaNotification}} clients connect.
> When {{{}atlas.notification.embedded=false{}}}: no-op.
> *Removed (old 2.8 embedded path):*
>  * {{{}startZk(){}}}, {{{}ZooKeeperServer{}}}, {{ServerCnxnFactory}}
>  * Direct {{kafka.server.KafkaServer}} construction tied to ZK
>  * Port-kill retry loops for fixed ZK/Kafka ports
> h3.  
> h3. Acceptance Criteria
>  * {{kafka.version}} is 3.9.1 and Scala binary version is 2.13 across build.
>  * {{EmbeddedKafkaServer}} starts KRaft broker without ZooKeeper when {{{}atlas.notification.embedded=true{}}}.
>  * {{atlas.kafka.bootstrap.servers}} is rewritten at runtime to the live embedded broker address.
>  * Unit tests pass:
>  ** {{KafkaNotificationTest}} (notification module)
>  ** {{NotificationHookConsumerKafkaTest}} (webapp, with {{{}-am{}}})
>  * Manual: Atlas docker container with {{atlas.notification.embedded=true}} — KRaft log lines present, REST {{/api/atlas/admin/version}} returns 200.
>  * Manual: Coexist stack with external {{atlas-kafka}} 3.9.1 — Hive hook → Atlas, PII classification → TagSync → Ranger, Hive/Kafka audit paths work.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)