You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Nikhil Bonte <ni...@freestoneinfotech.com> on 2018/12/14 12:29:01 UTC
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/
-----------------------------------------------------------
(Updated Dec. 14, 2018, 12:29 p.m.)
Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
includes code refactorings.
Repository: atlas
Description (updated)
-------
Notifications for relationship mutations.
This will add message to kafka topic when relationship is added/updated/deleted.
Sample messages :
ADD
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
UPDATE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
DELETE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 8ca8c9a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java a8c3363
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java e0a60a1
Diff: https://reviews.apache.org/r/69392/diff/5/
Changes: https://reviews.apache.org/r/69392/diff/4-5/
Testing
-------
Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
## Add relationship - create hive_db then create hive_table. This will create new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST
hive_db
request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
hive_table
request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
## Update relationship -
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
## Delete relationship -
API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
Thanks,
Nikhil Bonte
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Graham Wallis <gr...@uk.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/#review211403
-----------------------------------------------------------
Hi Nikhil
I downloaded this and tested it and it works pretty well.
I'm not sure whether you intended it or not, but when a relationship is added it fires two notifications. I think this is because both the relationship store's createRelationship() and create() methods invoke sendNotifications(). You maybe only wanted one of them. I also tested the update and delete operations and they each generate one notification.
You may want to add a full implementation of toString() so that it provides more information in a log. For example, I temporarily modified it like this:
@Override
public StringBuilder toString(StringBuilder sb) {
if (sb == null) {
sb = new StringBuilder();
}
sb.append("AtlasRelationshipHeader{");
sb.append("guid='").append(guid).append('\'');
sb.append(", status=").append(status);
sb.append(", label=").append(label);
sb.append(", propagateTags=").append(propagateTags);
sb.append(", end1=").append(end1);
sb.append(", end2=").append(end2);
super.toString(sb);
sb.append('}');
return sb;
}
- Graham Wallis
On Dec. 17, 2018, 5:13 a.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69392/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2018, 5:13 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2810
> https://issues.apache.org/jira/browse/ATLAS-2810
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Notifications for relationship mutations.
> This will add message to kafka topic when relationship is added/updated/deleted.
>
> Sample messages :
>
> ADD
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
>
> UPDATE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
>
> DELETE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
> intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 8ca8c9a
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java a8c3363
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java e0a60a1
>
>
> Diff: https://reviews.apache.org/r/69392/diff/5/
>
>
> Testing
> -------
>
> Tested from REST client(Postman).
> 1) Added/updated/deleted relationship by hitting REST APIs(see details below)
> 2) Verified message is sent to kafka using kafka-console-consumer
> ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
>
>
> ## Add relationship - create hive_db then create hive_table. This will create new relationship
> API - http://localhost:21000/api/atlas/v2/entity - POST
>
> hive_db
> request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
>
> hive_table
> request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
>
> ## Update relationship -
> API - http://localhost:21000/api/atlas/v2/relationship - PUT
> request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
>
> ## Delete relationship -
> API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
> On Dec. 20, 2018, 7:42 p.m., Sarath Subramanian wrote:
> > conside including qualifiedName/unique attributes for end1 and end2 in the notification sent out.
done
- Nikhil
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/#review211448
-----------------------------------------------------------
On Dec. 20, 2018, 10:13 a.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69392/
> -----------------------------------------------------------
>
> (Updated Dec. 20, 2018, 10:13 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2810
> https://issues.apache.org/jira/browse/ATLAS-2810
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Notifications for relationship mutations.
> This will add message to kafka topic when relationship is added/updated/deleted.
>
> Sample messages :
>
> ADD
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
>
> UPDATE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
>
> DELETE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
> intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
>
>
> Diff: https://reviews.apache.org/r/69392/diff/8/
>
>
> Testing
> -------
>
> Tested from REST client(Postman).
> 1) Added/updated/deleted relationship by hitting REST APIs(see details below)
> 2) Verified message is sent to kafka using kafka-console-consumer
> ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
>
>
> ## Add relationship - create hive_db then create hive_table. This will create new relationship
> API - http://localhost:21000/api/atlas/v2/entity - POST
>
> hive_db
> request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
>
> hive_table
> request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
>
> ## Update relationship -
> API - http://localhost:21000/api/atlas/v2/relationship - PUT
> request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
>
> ## Delete relationship -
> API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/#review211448
-----------------------------------------------------------
conside including qualifiedName/unique attributes for end1 and end2 in the notification sent out.
intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java
Lines 105 (patched)
<https://reviews.apache.org/r/69392/#comment296583>
RELATIONSHIP_ADD => RELATIONSHIP_CREATE
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
Lines 117 (patched)
<https://reviews.apache.org/r/69392/#comment296584>
notifyRelationMutated => notifyRelationshipMutation
- Sarath Subramanian
On Dec. 20, 2018, 2:13 a.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69392/
> -----------------------------------------------------------
>
> (Updated Dec. 20, 2018, 2:13 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2810
> https://issues.apache.org/jira/browse/ATLAS-2810
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Notifications for relationship mutations.
> This will add message to kafka topic when relationship is added/updated/deleted.
>
> Sample messages :
>
> ADD
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
>
> UPDATE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
>
> DELETE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
> intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
>
>
> Diff: https://reviews.apache.org/r/69392/diff/8/
>
>
> Testing
> -------
>
> Tested from REST client(Postman).
> 1) Added/updated/deleted relationship by hitting REST APIs(see details below)
> 2) Verified message is sent to kafka using kafka-console-consumer
> ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
>
>
> ## Add relationship - create hive_db then create hive_table. This will create new relationship
> API - http://localhost:21000/api/atlas/v2/entity - POST
>
> hive_db
> request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
>
> hive_table
> request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
>
> ## Update relationship -
> API - http://localhost:21000/api/atlas/v2/relationship - PUT
> request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
>
> ## Delete relationship -
> API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/#review212010
-----------------------------------------------------------
Ship it!
Ship It!
- Nixon Rodrigues
On Jan. 4, 2019, 1:21 p.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69392/
> -----------------------------------------------------------
>
> (Updated Jan. 4, 2019, 1:21 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2810
> https://issues.apache.org/jira/browse/ATLAS-2810
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Notifications for relationship mutations.
>
> To enable notifications, set property "atlas.notification.relationships.enabled" to true.
> If property is not specified, notifications will be disabled by default.
>
> If enabled, this will add message to kafka topic when relationship is added/updated/deleted.
>
> Sample messages :
>
> ADD
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
>
> UPDATE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
>
> DELETE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
> intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
>
>
> Diff: https://reviews.apache.org/r/69392/diff/10/
>
>
> Testing
> -------
>
> Tested from REST client(Postman).
> 1) Added/updated/deleted relationship by hitting REST APIs(see details below)
> 2) Verified message is sent to kafka using kafka-console-consumer
> ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
>
>
> ## Add relationship - create hive_db then create hive_table. This will create new relationship
> API - http://localhost:21000/api/atlas/v2/entity - POST
>
> hive_db
> request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
>
> hive_table
> request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
>
> ## Update relationship -
> API - http://localhost:21000/api/atlas/v2/relationship - PUT
> request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
>
> ## Delete relationship -
> API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/#review212015
-----------------------------------------------------------
Failed tests:
GlossaryServiceTest.testAddTermsToCategory:712 Glossary category update should've succeeded
GlossaryServiceTest.testTermAssignmentAndDissociation:793 Term update should've succeeded
GlossaryServiceTest.testUpdateGlossaryCategory:521 Customer category fetch should've succeeded
Tests run: 594, Failures: 3, Errors: 0, Skipped: 25
- Nixon Rodrigues
On Jan. 4, 2019, 1:21 p.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69392/
> -----------------------------------------------------------
>
> (Updated Jan. 4, 2019, 1:21 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2810
> https://issues.apache.org/jira/browse/ATLAS-2810
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Notifications for relationship mutations.
>
> To enable notifications, set property "atlas.notification.relationships.enabled" to true.
> If property is not specified, notifications will be disabled by default.
>
> If enabled, this will add message to kafka topic when relationship is added/updated/deleted.
>
> Sample messages :
>
> ADD
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
>
> UPDATE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
>
> DELETE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
> intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
>
>
> Diff: https://reviews.apache.org/r/69392/diff/10/
>
>
> Testing
> -------
>
> Tested from REST client(Postman).
> 1) Added/updated/deleted relationship by hitting REST APIs(see details below)
> 2) Verified message is sent to kafka using kafka-console-consumer
> ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
>
>
> ## Add relationship - create hive_db then create hive_table. This will create new relationship
> API - http://localhost:21000/api/atlas/v2/entity - POST
>
> hive_db
> request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
>
> hive_table
> request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
>
> ## Update relationship -
> API - http://localhost:21000/api/atlas/v2/relationship - PUT
> request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
>
> ## Delete relationship -
> API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/#review212408
-----------------------------------------------------------
Ship it!
Ship It!
- Nixon Rodrigues
On Jan. 24, 2019, 10:34 a.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69392/
> -----------------------------------------------------------
>
> (Updated Jan. 24, 2019, 10:34 a.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2810
> https://issues.apache.org/jira/browse/ATLAS-2810
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Notifications for relationship mutations.
>
> To enable notifications, set property "atlas.notification.relationships.enabled" to true.
> If property is not specified, notifications will be disabled by default.
>
> If enabled, this will add message to kafka topic when relationship is added/updated/deleted.
>
> Sample messages :
>
> ADD
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
>
> UPDATE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
>
> DELETE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
> intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
>
>
> Diff: https://reviews.apache.org/r/69392/diff/11/
>
>
> Testing
> -------
>
> Tested from REST client(Postman).
> 1) Added/updated/deleted relationship by hitting REST APIs(see details below)
> 2) Verified message is sent to kafka using kafka-console-consumer
> ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
>
>
> ## Add relationship - create hive_db then create hive_table. This will create new relationship
> API - http://localhost:21000/api/atlas/v2/entity - POST
>
> hive_db
> request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
>
> hive_table
> request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
>
> ## Update relationship -
> API - http://localhost:21000/api/atlas/v2/relationship - PUT
> request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
>
> ## Delete relationship -
> API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/
-----------------------------------------------------------
(Updated Jan. 24, 2019, 10:34 a.m.)
Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Fixed test case failure issues.
Bugs: ATLAS-2810
https://issues.apache.org/jira/browse/ATLAS-2810
Repository: atlas
Description
-------
Notifications for relationship mutations.
To enable notifications, set property "atlas.notification.relationships.enabled" to true.
If property is not specified, notifications will be disabled by default.
If enabled, this will add message to kafka topic when relationship is added/updated/deleted.
Sample messages :
ADD
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
UPDATE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
DELETE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5
intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
Diff: https://reviews.apache.org/r/69392/diff/11/
Changes: https://reviews.apache.org/r/69392/diff/10-11/
Testing
-------
Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
## Add relationship - create hive_db then create hive_table. This will create new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST
hive_db
request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
hive_table
request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
## Update relationship -
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
## Delete relationship -
API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
Thanks,
Nikhil Bonte
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Graham Wallis <gr...@uk.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/#review211828
-----------------------------------------------------------
Ship it!
Ship It!
- Graham Wallis
On Jan. 4, 2019, 1:21 p.m., Nikhil Bonte wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69392/
> -----------------------------------------------------------
>
> (Updated Jan. 4, 2019, 1:21 p.m.)
>
>
> Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2810
> https://issues.apache.org/jira/browse/ATLAS-2810
>
>
> Repository: atlas
>
>
> Description
> -------
>
> Notifications for relationship mutations.
>
> To enable notifications, set property "atlas.notification.relationships.enabled" to true.
> If property is not specified, notifications will be disabled by default.
>
> If enabled, this will add message to kafka topic when relationship is added/updated/deleted.
>
> Sample messages :
>
> ADD
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
>
> UPDATE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
>
> DELETE
> {"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5
> intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
> intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
> repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
> webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
>
>
> Diff: https://reviews.apache.org/r/69392/diff/10/
>
>
> Testing
> -------
>
> Tested from REST client(Postman).
> 1) Added/updated/deleted relationship by hitting REST APIs(see details below)
> 2) Verified message is sent to kafka using kafka-console-consumer
> ./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
>
>
> ## Add relationship - create hive_db then create hive_table. This will create new relationship
> API - http://localhost:21000/api/atlas/v2/entity - POST
>
> hive_db
> request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
>
> hive_table
> request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
>
> ## Update relationship -
> API - http://localhost:21000/api/atlas/v2/relationship - PUT
> request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
>
> ## Delete relationship -
> API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
>
>
> Thanks,
>
> Nikhil Bonte
>
>
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/
-----------------------------------------------------------
(Updated Jan. 4, 2019, 1:21 p.m.)
Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Added support for disabling/enabling relationship notifications by specifying property.
Bugs: ATLAS-2810
https://issues.apache.org/jira/browse/ATLAS-2810
Repository: atlas
Description (updated)
-------
Notifications for relationship mutations.
To enable notifications, set property "atlas.notification.relationships.enabled" to true.
If property is not specified, notifications will be disabled by default.
If enabled, this will add message to kafka topic when relationship is added/updated/deleted.
Sample messages :
ADD
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
UPDATE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
DELETE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/AtlasConfiguration.java c5357f5
intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
Diff: https://reviews.apache.org/r/69392/diff/10/
Changes: https://reviews.apache.org/r/69392/diff/9-10/
Testing
-------
Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
## Add relationship - create hive_db then create hive_table. This will create new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST
hive_db
request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
hive_table
request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
## Update relationship -
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
## Delete relationship -
API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
Thanks,
Nikhil Bonte
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/
-----------------------------------------------------------
(Updated Dec. 21, 2018, 3:45 p.m.)
Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
addressed comments and updated Description
Bugs: ATLAS-2810
https://issues.apache.org/jira/browse/ATLAS-2810
Repository: atlas
Description
-------
Notifications for relationship mutations.
This will add message to kafka topic when relationship is added/updated/deleted.
Sample messages :
ADD
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
UPDATE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
DELETE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table","uniqueAttributes":{"qualifiedName":"tbl2@qn"}},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db","uniqueAttributes":{"qualifiedName":"db2@qn"}}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
Diff: https://reviews.apache.org/r/69392/diff/9/
Changes: https://reviews.apache.org/r/69392/diff/8-9/
Testing
-------
Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
## Add relationship - create hive_db then create hive_table. This will create new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST
hive_db
request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
hive_table
request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
## Update relationship -
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
## Delete relationship -
API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
Thanks,
Nikhil Bonte
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/
-----------------------------------------------------------
(Updated Dec. 20, 2018, 10:13 a.m.)
Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Added unique attributes for relationship end definitions in message. Added metric recording for relationship notifications also.
Bugs: ATLAS-2810
https://issues.apache.org/jira/browse/ATLAS-2810
Repository: atlas
Description
-------
Notifications for relationship mutations.
This will add message to kafka topic when relationship is added/updated/deleted.
Sample messages :
ADD
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
UPDATE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
DELETE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 066abc1
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
Diff: https://reviews.apache.org/r/69392/diff/8/
Changes: https://reviews.apache.org/r/69392/diff/7-8/
Testing
-------
Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
## Add relationship - create hive_db then create hive_table. This will create new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST
hive_db
request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
hive_table
request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
## Update relationship -
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
## Delete relationship -
API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
Thanks,
Nikhil Bonte
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/
-----------------------------------------------------------
(Updated Dec. 19, 2018, 12:45 p.m.)
Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
resolved apply patch issue
Bugs: ATLAS-2810
https://issues.apache.org/jira/browse/ATLAS-2810
Repository: atlas
Description
-------
Notifications for relationship mutations.
This will add message to kafka topic when relationship is added/updated/deleted.
Sample messages :
ADD
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
UPDATE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
DELETE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
Diff: https://reviews.apache.org/r/69392/diff/7/
Changes: https://reviews.apache.org/r/69392/diff/6-7/
Testing
-------
Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
## Add relationship - create hive_db then create hive_table. This will create new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST
hive_db
request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
hive_table
request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
## Update relationship -
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
## Delete relationship -
API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
Thanks,
Nikhil Bonte
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/
-----------------------------------------------------------
(Updated Dec. 19, 2018, 10:16 a.m.)
Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Addressed comment.
Bugs: ATLAS-2810
https://issues.apache.org/jira/browse/ATLAS-2810
Repository: atlas
Description
-------
Notifications for relationship mutations.
This will add message to kafka topic when relationship is added/updated/deleted.
Sample messages :
ADD
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
UPDATE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
DELETE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 16d8879
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java ca3179a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java ef9ebab
Diff: https://reviews.apache.org/r/69392/diff/6/
Changes: https://reviews.apache.org/r/69392/diff/5-6/
Testing
-------
Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
## Add relationship - create hive_db then create hive_table. This will create new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST
hive_db
request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
hive_table
request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
## Update relationship -
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
## Delete relationship -
API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
Thanks,
Nikhil Bonte
Re: Review Request 69392: ATLAS-2810 - Add notifications from
RelationshipStore
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69392/
-----------------------------------------------------------
(Updated Dec. 17, 2018, 5:13 a.m.)
Review request for atlas, Ashutosh Mestry, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
added bugs link
Bugs: ATLAS-2810
https://issues.apache.org/jira/browse/ATLAS-2810
Repository: atlas
Description
-------
Notifications for relationship mutations.
This will add message to kafka topic when relationship is added/updated/deleted.
Sample messages :
ADD
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787083448,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_ADD","eventTime":1544787072298}}
UPDATE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787404692,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"ACTIVE","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_UPDATE","eventTime":1544787404679}}
DELETE
{"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"127.0.1.1","msgCreatedBy":"","msgCreationTime":1544787460081,"message":{"type":"ENTITY_NOTIFICATION_V2","relationship":{"typeName":"hive_table_db","guid":"e4949325-562a-43bf-a980-c0c562c8759e","status":"DELETED","label":"__hive_table.db","end1":{"guid":"c8792b2f-b550-47c2-baf9-1ddc0f25bac1","typeName":"hive_table"},"end2":{"guid":"414b7277-06f5-4a50-869c-3af62655e799","typeName":"hive_db"}},"operationType":"RELATIONSHIP_DELETE","eventTime":1544787459708}}
Diffs
-----
intg/src/main/java/org/apache/atlas/listener/EntityChangeListenerV2.java cccf0d4
intg/src/main/java/org/apache/atlas/model/instance/AtlasRelationshipHeader.java PRE-CREATION
intg/src/main/java/org/apache/atlas/model/notification/EntityNotification.java 1eae100
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java 8ca8c9a
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java a8c3363
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasRelationshipStoreV2.java 86cc98c
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java e0a60a1
Diff: https://reviews.apache.org/r/69392/diff/5/
Testing
-------
Tested from REST client(Postman).
1) Added/updated/deleted relationship by hitting REST APIs(see details below)
2) Verified message is sent to kafka using kafka-console-consumer
./kafka-console-consumer.sh --bootstrap-server localhost:9027 --topic ATLAS_ENTITIES --from-beginning
## Add relationship - create hive_db then create hive_table. This will create new relationship
API - http://localhost:21000/api/atlas/v2/entity - POST
hive_db
request body - {"entity":{"typeName":"hive_db","attributes":{"description":null,"name":"db2","owner":null,"qualifiedName":"db2@qn","replicatedFrom":[],"replicatedTo":[],"clusterName":"cluster0","location":null,"ownerType":null},"guid":-1},"referredEntities":{}}
hive_table
request body - {"entity":{"typeName":"hive_table","attributes":{"description":null,"name":"tbl2","owner":null,"qualifiedName":"tbl2@qn","replicatedFrom":[],"replicatedTo":[],"aliases":[],"columns":[],"comment":null,"createTime":1544553000000,"db":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"lastAccessTime":1544553000000,"partitionKeys":[],"retention":null,"sd":null,"tableType":null,"temporary":null,"viewExpandedText":null,"viewOriginalText":null},"guid":-1},"referredEntities":{}}
## Update relationship -
API - http://localhost:21000/api/atlas/v2/relationship - PUT
request body - {"typeName":"hive_table_db","guid":"8a8b8dc2-a163-42f0-b706-01c30cb0a688","provenanceType":0,"end1":{"guid":"6da63233-d449-42d5-8449-b6ad243f6af2","typeName":"hive_table"},"end2":{"guid":"cea76431-af27-43ae-ab11-98a2ef989bb8","typeName":"hive_db"},"label":"__hive_table.db","propagateTags":"NONE","status":"ACTIVE","createdBy":"admin","updatedBy":"admin","createTime":1542630658904,"updateTime":1542630658904,"version":0,"propagatedClassifications":[],"blockedPropagatedClassifications":[]}
## Delete relationship -
API - http://localhost:21000/api/atlas/v2/relationship/guid/8a8b8dc2-a163-42f0-b706-01c30cb0a688 - DELETE
Thanks,
Nikhil Bonte