You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by "Radhika Kundam (Jira)" <ji...@apache.org> on 2021/07/08 05:56:00 UTC
[jira] [Updated] (ATLAS-4348) Atlas-Kafka Hook : When a producer
publishes messages to multiple topics, the latest relationship is marked
ACTIVE , rest are marked DELETED
[ https://issues.apache.org/jira/browse/ATLAS-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Radhika Kundam updated ATLAS-4348:
----------------------------------
Description:
*Steps to reproduce the issue*:
Created 2 topics : test1, test2
Using console consumer produced messages to test1 and test2.
Relationship between console-producer-lineage and test1 is ACTIVE now.
Relationship between console-producer-lineage and test1 is set to DELETED when messages are published to test2.
*Root cause of issue:*
When messages published on topic, Atlas hook receives partial_update request with only one topic as output and by default partial update deletes relationships between previous topics by considering the only topic(which is in partial_update request) as the active.
*Proposed Solution:*
Introducing typedef option "appendRelationshipsOnPartialUpdate" which will hold list of relationship attribute names. While Atlas startup, it will resolve entity type definition and all the specified relationship attributes definitions will be updated with typedef property "isAppendOnPartialUpdate" only for that entity type.
With this solution, we can keep all the previous relationship entries active along with latest relationship entry for the specific entity by providing typedef options as below.
"typeDefOptions": { "appendRelationshipsOnPartialUpdate": "[\"inputs\", \"outputs\"]"}
This solution can be used for any entity type to consider all the existing relationship entries as active along with new entries from partial_update.
was:
*Steps to reproduce the issue*:
Created 2 topics : test1, test2
Using console consumer produced messages to test1 and test2.
Relationship between console-producer-lineage and test1 is ACTIVE now.
Relationship between console-producer-lineage and test1 is set to DELETED when messages are published to test2.
*Root cause of issue:*
When messages published on topic, Atlas hook receives partial_update request with only one topic as output and by default partial update deletes relationships between previous topics by considering the only topic(which is in partial_update request) as the active.
*Proposed Solution:*
Introducing typedef option "appendRelationshipsOnPartialUpdate" which will hold list of relationship attribute names. While Atlas startup, it will resolve entity type definition and all the specified relationship attributes definitions will be updated with typedef property "isAppendOnPartialUpdate" only for that entity type.
With this solution, we can keep all the previous relationship entries active along with latest relationship entry for the specific entity by providing typedef options as below.
"typeDefOptions": \{ "appendRelationshipsOnPartialUpdate": "[\"inputs\", \"outputs\"]"}
> Atlas-Kafka Hook : When a producer publishes messages to multiple topics, the latest relationship is marked ACTIVE , rest are marked DELETED
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ATLAS-4348
> URL: https://issues.apache.org/jira/browse/ATLAS-4348
> Project: Atlas
> Issue Type: Task
> Components: atlas-core
> Affects Versions: 2.1.0
> Reporter: Radhika Kundam
> Assignee: Radhika Kundam
> Priority: Major
>
> *Steps to reproduce the issue*:
> Created 2 topics : test1, test2
> Using console consumer produced messages to test1 and test2.
> Relationship between console-producer-lineage and test1 is ACTIVE now.
> Relationship between console-producer-lineage and test1 is set to DELETED when messages are published to test2.
> *Root cause of issue:*
> When messages published on topic, Atlas hook receives partial_update request with only one topic as output and by default partial update deletes relationships between previous topics by considering the only topic(which is in partial_update request) as the active.
> *Proposed Solution:*
> Introducing typedef option "appendRelationshipsOnPartialUpdate" which will hold list of relationship attribute names. While Atlas startup, it will resolve entity type definition and all the specified relationship attributes definitions will be updated with typedef property "isAppendOnPartialUpdate" only for that entity type.
> With this solution, we can keep all the previous relationship entries active along with latest relationship entry for the specific entity by providing typedef options as below.
> "typeDefOptions": { "appendRelationshipsOnPartialUpdate": "[\"inputs\", \"outputs\"]"}
>
> This solution can be used for any entity type to consider all the existing relationship entries as active along with new entries from partial_update.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)