You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Radhika Kundam (Jira)" <ji...@apache.org> on 2022/10/25 19:11:00 UTC
[jira] [Assigned] (ATLAS-4335) Hook Notifications through Rest Interface
[ https://issues.apache.org/jira/browse/ATLAS-4335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radhika Kundam reassigned ATLAS-4335:
-------------------------------------
Assignee: Radhika Kundam (was: Deep Singh)
> Hook Notifications through Rest Interface
> -----------------------------------------
>
> Key: ATLAS-4335
> URL: https://issues.apache.org/jira/browse/ATLAS-4335
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
> Reporter: Deep Singh
> Assignee: Radhika Kundam
> Priority: Critical
>
> In Data-lake <-> Workload-cluster scenario, versions of the services available on clusters are not always the same. Especially when Kafka versions are different on the clusters, many a time the Kafka client (on workload-cluster) becomes incompatible with the Kafka broker (on data-lake). This leads to failure in the delivery of Atlas Hook notification as the mechanism is dependent on the Kafka client's capability to deliver messages.
> To resolve such an issue there is no other way but to make Kafka client compatible with the broker by upgrading or degrading the client or the broker. Often this is not straight-fwd and the easiest thing to do.
> We need another mechanism independent of the Kafka client to deliver hook notifications to the Atlas consumer.
> There are various REST API available today to perform CURD on Atlas entities. Those APIs process notifications in real-time as they arrive, since event processing is slow and time-consuming these APIs cannot give the throughput which hooks require.
> We need another REST interface that is lightweight and does not process messages as they arrive instead, it could dump the messages to the local kafka. Since Kafka will be available locally in the cluster, the interface will never face the problem of version mismatch of client-broker, as faced by service hooks running on a different cluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)