You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by "Ashutosh Mestry (Jira)" <ji...@apache.org> on 2021/10/28 20:44:00 UTC

[jira] [Updated] (ATLAS-4464) Ingest: Improve Rate of Ingest

     [ https://issues.apache.org/jira/browse/ATLAS-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ashutosh Mestry updated ATLAS-4464:
-----------------------------------
    Attachment: ATLAS-4464-concurrent-notification-processing-obj-sync-v3.patch

> Ingest: Improve Rate of Ingest
> ------------------------------
>
>                 Key: ATLAS-4464
>                 URL: https://issues.apache.org/jira/browse/ATLAS-4464
>             Project: Atlas
>          Issue Type: Improvement
>          Components:  atlas-core
>    Affects Versions: trunk
>            Reporter: Ashutosh Mestry
>            Assignee: Ashutosh Mestry
>            Priority: Major
>             Fix For: trunk
>
>         Attachments: ATLAS-4464-concurrent-notification-processing-obj-sync-v3.patch
>
>
> *Background*
> Existing implementation of _NotificationHookConsumer_ has linear complexity for ingestion. This has several impacts:
>  * Authorization policies will take longer to get enforced.
>  * Unpredictable wait times for metadata showing up in Atlas.
> *Solution*
> Implement a mechanism for processing messages such that:
>  * Determine dependencies within incoming messages.
>  * Dependent messages should get processed serially.
>  * Messages without dependencies are processed concurrently.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)