[jira] [Commented] (RANGER-5658) Tag Sync: remove atlas.kafka.zookeeper.connect and fix Atlas Kafka consumer for Kafka 3.9

["Ramachandran Krishnan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/RANGER-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093509#comment-18093509 ] 

Ramachandran Krishnan commented on RANGER-5658:
-----------------------------------------------

merged into master commit details:https://github.com/apache/ranger/commit/deaf13df3c62af8edde53e185890682fb711bec5


> Tag Sync: remove atlas.kafka.zookeeper.connect and fix Atlas Kafka consumer for Kafka 3.9
> -----------------------------------------------------------------------------------------
>
>                 Key: RANGER-5658
>                 URL: https://issues.apache.org/jira/browse/RANGER-5658
>             Project: Ranger
>          Issue Type: Task
>          Components: Ranger
>    Affects Versions: 3.0.0
>            Reporter: Ramachandran Krishnan
>            Assignee: Ramachandran Krishnan
>            Priority: Major
>          Time Spent: 50m
>  Remaining Estimate: 0h
>
> h3. Summary
> Ranger Tag Sync's Atlas Kafka source (ranger.tagsync.source.atlas) consumed Atlas entity notifications via kafka-clients but still required the legacy property atlas.kafka.zookeeper.connect at startup. That property was never used by Atlas KafkaNotification for consumer creation. With Kafka 3.9.x and KRaft brokers, clients use bootstrap.servers only.
> This change removes the ZK requirement and fixes Tag Sync Atlas Kafka setup so consumption works with both PLAINTEXT and Kerberos (SASL) Atlas Kafka, including when Ranger runs with is_secure=true but the Atlas metadata bus is PLAINTEXT.
> h3. Problem
> * Tag Sync validated atlas.kafka.zookeeper.connect even though Atlas 2.4+ KafkaNotification does not use it
> * setup.py wrote Kerberos JAAS for the Atlas Kafka client whenever is_secure=true, even when atlas.kafka.security.protocol=PLAINTEXT — breaking PLAINTEXT/KRaft metadata buses
> * AtlasTagSource loaded atlas-application.properties from the classpath only; installer-generated conf under TAGSYNC_CONF_DIR was not reliably used without -Datlas.conf
> h3. Changes
> * AtlasTagSource — remove ZK validation; load config via ApplicationProperties; validate bootstrap.servers + consumer group only
> * setup.py — atlas_kafka_uses_kerberos(): emit atlas.jaas.KafkaClient.* only when security.protocol starts with SASL; always write atlas.kafka.security.protocol
> * ranger-tagsync-services.sh — pass -Datlas.conf=${TAGSYNC_CONF_DIR}
> * install.properties / installprop2xml.properties — remove TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT
> * setup.py — use rangerTagsync_password for initial tagadmin jceks credential (fixes 401 on tag upload in basic-auth deployments)
> * dev-support/ranger-docker/scripts/tagsync/ranger-tagsync-install.properties — default Atlas REST on, Kafka off; keep Kafka sample settings for ranger-kafka.rangernw:9092 + SASL when enabled
> * dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md — operator guide (PLAINTEXT vs SASL, startup, verification)
> h3. Supported Atlas Kafka modes
> || Mode || TAG_SOURCE_ATLAS_KAFKA_SECURITY_PROTOCOL || Kerberos JAAS in atlas-application.properties ||
> | PLAINTEXT | PLAINTEXT | No |
> | Kerberos | SASL_PLAINTEXT / SASL_SSL | Yes (when is_secure=true + principal/keytab) |
> Atlas Kafka security is independent of Tag Sync → Ranger Admin authentication (basic auth or Ranger Kerberos for tag upload).
> h3. Out of scope
> * Tag Sync HA (ranger-tagsync.server.ha.zookeeper.*) — unchanged
> * Atlas REST tag source behavior — unchanged except jceks password fix
> h3. Upgrade note
> Existing installs may still have atlas.kafka.zookeeper.connect in conf/atlas-application.properties; harmless, can be removed on next config refresh.
> h3. Test plan
> * mvn test -pl tagsync -Dtest=AtlasTagSourceConfigTest
> * Fresh install: atlas-application.properties has no zookeeper.connect
> * PLAINTEXT: is_secure=true + security.protocol=PLAINTEXT → no Kerberos JAAS; consumer connects
> * SASL: security.protocol=SASL_PLAINTEXT + principal/keytab → JAAS written; consumer connects (e.g. ranger-kafka.rangernw:9092 in ranger-docker)
> * Tag Sync starts with -Datlas.conf and consumes ATLAS_ENTITIES from Kafka 3.9.x (KRaft)
> * Upgrade: legacy zookeeper.connect present → Tag Sync still starts
> * Tag Sync HA smoke test (if HA enabled)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)