You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Radhika Kundam (Jira)" <ji...@apache.org> on 2026/06/12 19:34:00 UTC
[jira] [Created] (ATLAS-5320) Distributed Notification Processing
Radhika Kundam created ATLAS-5320:
-------------------------------------
Summary: Distributed Notification Processing
Key: ATLAS-5320
URL: https://issues.apache.org/jira/browse/ATLAS-5320
Project: Atlas
Issue Type: New Feature
Reporter: Radhika Kundam
Assignee: Radhika Kundam
Current entity and lineage processing in Atlas is mostly single-threaded to maintain message order, which limits scalability. HMS messages create entities, and HS2 messages create lineage based on those entities. To ensure correct lineage, we currently serialize all processing, which becomes a performance bottleneck.
As a solution for this, Introduce a proof-of-concept for scalable message processing in Apache Atlas by using multiple Kafka topics based on a key ({{{}dbName.tableName{}}}). This will enable parallel processing of HMS and HS2 messages, improve throughput, and reduce bottlenecks caused by single-threaded lineage creation.
Implementation details:
* HMS messages are routed to Kafka partitions based on {{{}dbName.tableName{}}}.
* HS2 messages are routed to *all* relevant partitions based on input/output tables.
* Messages are processed in parallel by multiple consumer threads.
* Deduplication and shell entity handling is incorporated.
Attached architectural design document.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)