You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ranger.apache.org by ra...@apache.org on 2026/06/29 13:45:59 UTC
(ranger) 01/01: Remove obsolete atlas.kafka.zookeeper.connect requirement from Tag Sync.
This is an automated email from the ASF dual-hosted git repository.
ramackri pushed a commit to branch tagsync-remove-atlas-kafka-zookeeper-connect
in repository https://gitbox.apache.org/repos/asf/ranger.git
commit 1f8dd4e5da53e7ada8aa5ed9d6235933f04a4b56
Author: ramk <ra...@cloudera.com>
AuthorDate: Mon Jun 29 19:15:45 2026 +0530
Remove obsolete atlas.kafka.zookeeper.connect requirement from Tag Sync.
Tag Sync consumes Atlas notifications via kafka-clients (bootstrap.servers only).
The legacy zookeeper.connect property was never used by Atlas KafkaNotification
but was still required at startup and in installer templates.
Co-authored-by: Cursor <cu...@cursor.com>
---
dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md | 136 +++++++++++++++++++++
tagsync/conf/templates/installprop2xml.properties | 1 -
tagsync/scripts/install.properties | 3 +-
tagsync/scripts/setup.py | 3 -
.../tagsync/source/atlas/AtlasTagSource.java | 9 +-
5 files changed, 139 insertions(+), 13 deletions(-)
diff --git a/dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md b/dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md
new file mode 100644
index 000000000..463b20571
--- /dev/null
+++ b/dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md
@@ -0,0 +1,136 @@
+# Tag Sync — Atlas Kafka configuration (Kafka 3.9+ / KRaft)
+
+Reference for operators and developers configuring Ranger Tag Sync with the **Atlas Kafka** tag source (`ranger.tagsync.source.atlas`).
+
+Related code change: remove obsolete `atlas.kafka.zookeeper.connect` requirement from Tag Sync startup and installer templates.
+
+---
+
+## Summary
+
+Tag Sync consumes Atlas entity notifications using the standard **`kafka-clients`** library (`bootstrap.servers`, consumer group, SASL/SSL as needed). It does **not** connect to Kafka via ZooKeeper.
+
+The property `atlas.kafka.zookeeper.connect` was legacy configuration carried forward from older Atlas/Kafka deployments. Ranger previously **required** it at startup even though:
+
+- Atlas 2.4 `KafkaNotification` never uses it for consumer/producer creation
+- Kafka 3.x clients (including 3.9.1) talk to brokers over `bootstrap.servers` only
+- KRaft-mode Kafka brokers do not expose ZooKeeper to clients
+
+That requirement has been **removed** from Tag Sync.
+
+---
+
+## Required configuration (Atlas Kafka source)
+
+Set these in `install.properties` (installer writes `conf/atlas-application.properties`):
+
+| Install property | Atlas property | Required |
+|------------------|----------------|----------|
+| `TAG_SOURCE_ATLAS_KAFKA_BOOTSTRAP_SERVERS` | `atlas.kafka.bootstrap.servers` | Yes |
+| `TAG_SOURCE_ATLAS_KAFKA_ENTITIES_GROUP_ID` | `atlas.kafka.entities.group.id` | Yes |
+| `TAG_SOURCE_ATLAS_KAFKA_SECURITY_PROTOCOL` | `atlas.kafka.security.protocol` | If secured |
+| `TAG_SOURCE_ATLAS_KAFKA_SERVICE_NAME` | `atlas.kafka.sasl.kerberos.service.name` | If Kerberos |
+| Kerberos principal/keytab | `atlas.jaas.KafkaClient.*` | If Kerberos |
+
+Example `conf/atlas-application.properties`:
+
+```properties
+atlas.kafka.bootstrap.servers=kafka-broker1:9092,kafka-broker2:9092
+atlas.kafka.entities.group.id=ranger_entities_consumer
+atlas.kafka.security.protocol=SASL_PLAINTEXT
+atlas.kafka.sasl.kerberos.service.name=kafka
+```
+
+---
+
+## Removed configuration
+
+| Property | Status |
+|----------|--------|
+| `atlas.kafka.zookeeper.connect` | **No longer required** — not generated by `setup.py`, not validated by `AtlasTagSource` |
+
+### Upgrading existing deployments
+
+If `conf/atlas-application.properties` still contains `atlas.kafka.zookeeper.connect` from an older install, it is harmless (Kafka client ignores unknown keys). You may delete the line during the next config refresh.
+
+---
+
+## ZooKeeper still used elsewhere (do not confuse)
+
+Tag Sync has **two unrelated** ZooKeeper touch points:
+
+| Config | Purpose | Needed for KRaft Kafka? |
+|--------|---------|-------------------------|
+| `atlas.kafka.zookeeper.connect` | Legacy Atlas/Kafka config | **No** (removed) |
+| `ranger-tagsync.server.ha.zookeeper.connect` | Tag Sync active/passive leader election | **Only when HA enabled** |
+
+When `ranger-tagsync.server.ha.enabled=true`, Tag Sync still needs a real ZooKeeper ensemble for HA — independent of how Kafka brokers run (ZooKeeper vs KRaft).
+
+Default in `ranger-tagsync-site.xml`: `ranger-tagsync.server.ha.enabled=false`.
+
+---
+
+## Alternative: Atlas REST source
+
+If Tag Sync reads tags over Atlas REST instead of Kafka, none of the Atlas Kafka properties apply:
+
+```properties
+TAG_SOURCE_ATLASREST_ENABLED=true
+TAG_SOURCE_ATLASREST_ENDPOINT=http://atlas-host:21000
+```
+
+Enable via `ranger.tagsync.source.atlasrest` in `ranger-tagsync-site.xml`.
+
+---
+
+## Code / config files changed
+
+| File | Change |
+|------|--------|
+| `tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java` | Drop `TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT` and startup validation |
+| `tagsync/scripts/install.properties` | Remove `TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT` |
+| `tagsync/conf/templates/installprop2xml.properties` | Remove install → Atlas mapping |
+| `tagsync/scripts/setup.py` | Stop writing `atlas.kafka.zookeeper.connect` |
+| `dev-support/ranger-docker/scripts/tagsync/ranger-tagsync-install.properties` | Remove sample ZK property |
+
+**Unchanged:** Tag Sync HA properties, `tagsync/pom.xml` ZooKeeper dependency (used by `ranger-common-ha` when HA is on).
+
+---
+
+## Jira ticket template
+
+**Title:** `RANGER-XXXX: Remove obsolete atlas.kafka.zookeeper.connect requirement from Tag Sync`
+
+**Description:**
+
+```
+Remove the legacy atlas.kafka.zookeeper.connect configuration requirement from
+Ranger Tag Sync's Atlas Kafka source.
+
+Tag Sync consumes Atlas notifications via kafka-clients (bootstrap.servers).
+The zookeeper.connect property was never used by Atlas 2.4 KafkaNotification
+for consumer creation; Ranger only validated that it was set. With Kafka 3.9.x
+and KRaft brokers, clients do not use ZooKeeper.
+
+Changes:
+- Drop validation from AtlasTagSource
+- Remove property from install.properties, installprop2xml.properties, setup.py,
+ and docker sample config
+
+Out of scope: Tag Sync HA (ranger-tagsync.server.ha.zookeeper.*) unchanged.
+```
+
+**Test plan:**
+
+- [ ] Fresh install: `atlas-application.properties` has no `zookeeper.connect`
+- [ ] Tag Sync starts and consumes from Kafka 3.9.x (KRaft)
+- [ ] Upgrade: existing installs with old property still start
+- [ ] Tag Sync HA smoke test (if HA enabled)
+
+---
+
+## References
+
+- Tag Sync Atlas source: `tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java`
+- Atlas notification (external): `org.apache.atlas.kafka.KafkaNotification` in `atlas-notification` 2.4.0
+- Root Kafka client version: `${kafka.version}` in root `pom.xml` (3.9.1)
diff --git a/tagsync/conf/templates/installprop2xml.properties b/tagsync/conf/templates/installprop2xml.properties
index d6f289a55..aea6537ee 100644
--- a/tagsync/conf/templates/installprop2xml.properties
+++ b/tagsync/conf/templates/installprop2xml.properties
@@ -24,7 +24,6 @@ TAG_SOURCE_ATLAS_ENABLED = ranger.tagsync.source.atlas
TAG_SOURCE_ATLAS_KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR = atlas.kafka.offsets.topic.replication.factor
TAG_SOURCE_ATLAS_KAFKA_BOOTSTRAP_SERVERS = atlas.kafka.bootstrap.servers
-TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT = atlas.kafka.zookeeper.connect
TAG_SOURCE_ATLAS_KAFKA_ENTITIES_GROUP_ID = atlas.kafka.entities.group.id
TAG_SOURCE_ATLAS_KAFKA_SERVICE_NAME = atlas.kafka.sasl.kerberos.service.name
diff --git a/tagsync/scripts/install.properties b/tagsync/scripts/install.properties
index ae2e555eb..278df0ff8 100644
--- a/tagsync/scripts/install.properties
+++ b/tagsync/scripts/install.properties
@@ -27,9 +27,8 @@ TAG_DEST_RANGER_SSL_CONFIG_FILENAME =
TAG_SOURCE_ATLAS_ENABLED = true
-# Endpoint specifications needed by Atlas
+# Kafka settings for Atlas notification consumer (bootstrap.servers only; no ZooKeeper)
TAG_SOURCE_ATLAS_KAFKA_BOOTSTRAP_SERVERS = localhost:6667
-TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT = localhost:2181
TAG_SOURCE_ATLAS_KAFKA_ENTITIES_GROUP_ID = ranger_entities_consumer
TAG_SOURCE_ATLAS_KAFKA_SERVICE_NAME = kafka
diff --git a/tagsync/scripts/setup.py b/tagsync/scripts/setup.py
index 1fbf5f181..df98f87c0 100755
--- a/tagsync/scripts/setup.py
+++ b/tagsync/scripts/setup.py
@@ -68,7 +68,6 @@
initPrefixList = ['S99', 'K00']
TAGSYNC_ATLAS_KAFKA_ENDPOINTS_KEY = 'TAG_SOURCE_ATLAS_KAFKA_BOOTSTRAP_SERVERS'
-TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT_KEY = 'TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT'
TAGSYNC_ATLAS_KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR_KEY = 'TAG_SOURCE_ATLAS_KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR'
TAGSYNC_ATLAS_CONSUMER_GROUP_KEY = 'TAG_SOURCE_ATLAS_KAFKA_ENTITIES_GROUP_ID'
@@ -231,8 +230,6 @@ def convertInstallPropsToXML(props):
newKey = directKeyMap[k]
if (k == TAGSYNC_ATLAS_KAFKA_ENDPOINTS_KEY):
atlasOutFile.write(newKey + "=" + v + "\n")
- elif (k == TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT_KEY):
- atlasOutFile.write(newKey + "=" + v + "\n")
elif (k == TAGSYNC_ATLAS_KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR_KEY):
atlasOutFile.write(newKey + "=" + v + "\n")
elif (k == TAGSYNC_ATLAS_CONSUMER_GROUP_KEY):
diff --git a/tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java b/tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java
index eeeccaf4a..382d40418 100644
--- a/tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java
+++ b/tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java
@@ -48,9 +48,8 @@ public class AtlasTagSource extends AbstractTagSource {
private static final Logger LOG = LoggerFactory.getLogger(AtlasTagSource.class);
public static final String TAGSYNC_ATLAS_PROPERTIES_FILE_NAME = "atlas-application.properties";
- public static final String TAGSYNC_ATLAS_KAFKA_ENDPOINTS = "atlas.kafka.bootstrap.servers";
- public static final String TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT = "atlas.kafka.zookeeper.connect";
- public static final String TAGSYNC_ATLAS_CONSUMER_GROUP = "atlas.kafka.entities.group.id";
+ public static final String TAGSYNC_ATLAS_KAFKA_ENDPOINTS = "atlas.kafka.bootstrap.servers";
+ public static final String TAGSYNC_ATLAS_CONSUMER_GROUP = "atlas.kafka.entities.group.id";
public static final int MAX_WAIT_TIME_IN_MILLIS = 1000;
private int maxBatchSize;
@@ -92,10 +91,6 @@ public boolean initialize(Properties properties) {
ret = false;
LOG.error("Value of property '{}' is not specified!", TAGSYNC_ATLAS_KAFKA_ENDPOINTS);
}
- if (StringUtils.isBlank(atlasProperties.getProperty(TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT))) {
- ret = false;
- LOG.error("Value of property '{}' is not specified!", TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT);
- }
if (StringUtils.isBlank(atlasProperties.getProperty(TAGSYNC_ATLAS_CONSUMER_GROUP))) {
ret = false;
LOG.error("Value of property '{}' is not specified!", TAGSYNC_ATLAS_CONSUMER_GROUP);