You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by "Ashutosh Mestry (Jira)" <ji...@apache.org> on 2021/10/28 20:43:00 UTC

[jira] [Created] (ATLAS-4464) Ingest: Improve Rate of Ingest

Ashutosh Mestry created ATLAS-4464:
--------------------------------------

             Summary: Ingest: Improve Rate of Ingest
                 Key: ATLAS-4464
                 URL: https://issues.apache.org/jira/browse/ATLAS-4464
             Project: Atlas
          Issue Type: Improvement
          Components:  atlas-core
    Affects Versions: trunk
            Reporter: Ashutosh Mestry
            Assignee: Ashutosh Mestry
             Fix For: trunk


*Background*

Existing implementation of _NotificationHookConsumer_ has linear complexity for ingestion. This has several impacts:
 * Authorization policies will take longer to get enforced.
 * Unpredictable wait times for metadata showing up in Atlas.

*Solution*

Implement a mechanism for processing messages such that:
 * Determine dependencies within incoming messages.
 * Dependent messages should get processed serially.
 * Messages without dependencies are processed concurrently.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)