You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@atlas.apache.org by Jo MailingList <jo...@gmail.com> on 2018/02/06 10:44:51 UTC
Re: Relating to an entity using uniqueAttributes
Hello,
On the same topic,
Now I realised that when refering to an object alredy created, is not
enough to add the qualifiedName, but it also needs to include the name
attribute since it is a required attribute. The point is that no matter
which name I put, the actual referenced entity is updated to have that name.
Is there a way to put the reference without adding the name attribute in
the values of the "referenced entity"?
Thank you very much,
Best regards,
Joan
2018-01-05 14:24 GMT+01:00 Jo MailingList <jo...@gmail.com>:
> Hello,
>
> Thank you very much,
>
> This has been of great help.
>
> Best regards,
>
> Joan
>
> 2018-01-05 9:05 GMT+01:00 Madhan Neethiraj <ma...@apache.org>:
>
>> Joan,
>>
>>
>>
>> A slightly better approach is to provide repository entity as a separate
>> entry in “entities” and refer to the repository in database entity using
>> its temp-id, as shown below.
>>
>>
>>
>> This approach will allow an entity be referenced from multiple entities
>> using its temp-id (instead of adding the entire entity details in each such
>> reference).
>>
>>
>>
>> Madhan
>>
>>
>>
>> {
>>
>> "message": {
>>
>> "type": "ENTITY_FULL_UPDATE",
>>
>> "user": "user",
>>
>> "entities": [
>>
>> {
>>
>> "jsonClass": "org.apache.atlas.typesystem.j
>> son.InstanceSerialization$_Reference",
>>
>> "id": {
>>
>> "jsonClass": "org.apache.atlas.typesystem.j
>> son.InstanceSerialization$_Id",
>>
>> "id": "-14568879465268",
>>
>> "typeName": "database",
>>
>> "state": "ACTIVE",
>>
>> "version": 0
>>
>> },
>>
>> "typeName": "database",
>>
>> "values": {
>>
>> "dbName": "Test7",
>>
>> "description": "Test db7",
>>
>> "name": "Testdb7",
>>
>> "origin_app": "origin",
>>
>> "origin_username": "externalusername",
>>
>> "owner": "owner",
>>
>> "qualifiedName": "Testdb5",
>>
>> "repository": {
>>
>> "jsonClass": "org.apache.atlas.typesystem.j
>> son.InstanceSerialization$_Id",
>>
>> "id": "-14568879465269",
>>
>> "typeName": "repository",
>>
>> "state": "ACTIVE",
>>
>> "version": 0
>>
>> }
>>
>> },
>>
>> "traitNames": [],
>>
>> "traits": {}
>>
>> },
>>
>> {
>>
>> "jsonClass": "org.apache.atlas.typesystem.j
>> son.InstanceSerialization$_Reference",
>>
>> "id": {
>>
>> "jsonClass": "org.apache.atlas.typesystem.j
>> son.InstanceSerialization$_Id",
>>
>> "id": "-14568879465269",
>>
>> "typeName": "repository",
>>
>> "state": "ACTIVE",
>>
>> "version": 0
>>
>> },
>>
>> "typeName": "repository",
>>
>> "values": {
>>
>> "qualifiedName": "repository01"
>>
>> },
>>
>> "traitNames": [],
>>
>> "traits": {}
>>
>> }
>>
>> ]
>>
>> },
>>
>> "version": {
>>
>> "version": "1.0.0"
>>
>> }
>>
>> }
>>
>>
>>
>> *From: *Madhan Neethiraj <ma...@apache.org>
>> *Date: *Thursday, January 4, 2018 at 11:18 PM
>> *To: *"user@atlas.apache.org" <us...@atlas.apache.org>
>> *Subject: *Re: Relating to an entity using uniqueAttributes
>>
>>
>>
>> Joan,
>>
>>
>>
>> To refer to another entity In Atlas hook notifications, add the referred
>> entity details as shown below. Also, I would suggest using
>> ENTITY_FULL_UPDATE as message type (instead of ENTITY_CREATE).
>>
>>
>>
>> {
>>
>> "message": {
>>
>> "type": "ENTITY_FULL_UPDATE",
>>
>> "user": "user",
>>
>> "entities": [
>>
>> {
>>
>> "jsonClass": "org.apache.atlas.typesystem.j
>> son.InstanceSerialization$_Reference",
>>
>> "id": {
>>
>> "jsonClass": "org.apache.atlas.typesystem.j
>> son.InstanceSerialization$_Id",
>>
>> "id": "-14568879465268",
>>
>> "typeName": "database",
>>
>> "state": "ACTIVE",
>>
>> "version": 0
>>
>> },
>>
>> "typeName": "database",
>>
>> "values": {
>>
>> "dbName": "Test7",
>>
>> "description": "Test db7",
>>
>> "name": "Testdb7",
>>
>> "origin_app": "origin",
>>
>> "origin_username": "externalusername",
>>
>> "owner": "owner",
>>
>> "qualifiedName": "Testdb5",
>>
>> "repository": {
>>
>> "jsonClass": "org.apache.atlas.typesystem.j
>> son.InstanceSerialization$_Reference",
>>
>> "id": {
>>
>> "jsonClass": "org.apache.atlas.typesystem.j
>> son.InstanceSerialization$_Id",
>>
>> "id": "-14568879465269",
>>
>> "typeName": "repository",
>>
>> "state": "ACTIVE",
>>
>> "version": 0
>>
>> },
>>
>> "typeName": "repository",
>>
>> "values": {
>>
>> "qualifiedName": "repository01"
>>
>> },
>>
>> "traitNames": [],
>>
>> "traits": {}
>>
>> }
>>
>> },
>>
>> "traitNames": [],
>>
>> "traits": {}
>>
>> }
>>
>> ]
>>
>> },
>>
>> "version": {
>>
>> "version": "1.0.0"
>>
>> }
>>
>> }
>>
>>
>>
>> Hope this helps.
>>
>>
>>
>> Madhan
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From: *Jo MailingList <jo...@gmail.com>
>> *Reply-To: *"user@atlas.apache.org" <us...@atlas.apache.org>
>> *Date: *Tuesday, January 2, 2018 at 9:54 AM
>> *To: *"user@atlas.apache.org" <us...@atlas.apache.org>
>> *Subject: *Relating to an entity using uniqueAttributes
>>
>>
>>
>> Hello everyone,
>>
>>
>>
>> I am trying to create a database that in my type model has a reference to
>> a repository. I want to create it by sending a message to the ATLAS_HOOK
>> topic to the Kafka broker as follows:
>>
>>
>>
>> {
>>
>> "version": {
>>
>> "version": "1.0.0"
>>
>> },
>>
>> "message": {
>>
>> "entities": [{
>>
>> "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Ref
>> erence",
>>
>> "id": {
>>
>> "jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id"
>> ,
>>
>> "id": "-14568879465268",
>>
>> "version": 0,
>>
>> "typeName": "database",
>>
>> "state": "ACTIVE"
>>
>> },
>>
>> "typeName": "database",
>>
>> "values": {
>>
>> "qualifiedName": "Testdb5",
>>
>> *"repository": {*
>>
>> *"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id"*
>>
>> *"version": 0,*
>>
>> *"typeName": "repository",*
>>
>> *"uniqueAttributes": {*
>>
>> *"qualifiedName": "repository01"*
>>
>> *},*
>>
>> *"state": "ACTIVE"*
>>
>> *},*
>>
>> "owner": "owner",
>>
>> "description": "Test db7",
>>
>> "name": "Testdb7",
>>
>> "dbName": "Test7",
>>
>> "origin_app": "origin",
>>
>> "origin_username": "externalusername"
>>
>> },
>>
>> "traitNames": [],
>>
>> "traits": {}
>>
>> }],
>>
>> "type": "ENTITY_CREATE",
>>
>> "user": "user"
>>
>> }
>>
>> }
>>
>>
>>
>> The database type expects a repository attribute of the type repository
>> as a reference. It works when I give the guid of the reference directly but
>> when trying my example it fails and it seems to me that it does not process
>> the information I provide as uniqueAttributes. What I see in the
>> application.log is the following stacktrace:
>>
>>
>>
>> *org.apache.atlas.exception.AtlasBaseException: ObjectId is not valid
>> AtlasObjectId{guid='null', typeName='repository', uniqueAttributes={}}*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityGraphDiscoveryV1.visitReference(AtlasEntityGraphDiscoveryV1.java:186)*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityGraphDiscoveryV1.visitAttribute(AtlasEntityGraphDiscoveryV1.java:234)*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityGraphDiscoveryV1.visitEntity(AtlasEntityGraphDiscoveryV1.java:327)*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityGraphDiscoveryV1.walkEntityGraph(AtlasEntityGraphDiscoveryV1.java:354)*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:142)*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:523)*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:174)*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:193)*
>>
>> * at org.apache.atlas.repository.st
>> <http://org.apache.atlas.repository.st>ore.graph.v1.AtlasEntityStoreV1$$FastClassBySpringCGLIB$$80c00649.invoke(<generated>)*
>>
>>
>>
>> This message makes me suspicious that Atlas is not processing my input
>> message correctly and not reading my uniqueAttributes information.
>>
>>
>>
>> Does someone have a hint of what I am doing wrong or if this is a
>> currrent issue inside Atlas or it is not a supported feature right now?
>>
>>
>>
>> Thank you very much,
>>
>>
>>
>> Best regards,
>>
>>
>>
>> Joan
>>
>>
>>
>>
>>
>
>
Re: Relating to an entity using uniqueAttributes
Posted by Madhan Neethiraj <ma...@apache.org>.
Joan,
> Is there a way to put the reference without adding the name attribute in the values of the "referenced entity"?
Atlas notification module was enhanced recently, with addition of V2 style notifications (ATLAS-2421). Data structures used in V2 notification would allow referencing an entity using its unique-name (as shown below), without having to provide other attributes of the entity. This enhancement was included in Apache Atlas 0.8.2, which was released on last month.
Please review the sample V2 notification below, for the usecase discussed earlier in this thread. Hope this helps.
Sorry for the delay in responding to your question.
Regards,
Madhan
{
"message": {
"type": "ENTITY_CREATE_V2",
"user": "user",
"entities": {
"entities": [
{
"guid": "-14568879465268",
"typeName": "database",
"attributes": {
"dbName": "Test7",
"description": "Test db7",
"name": "Testdb7",
"origin_app": "origin",
"origin_username": "externalusername",
"owner": "owner",
"qualifiedName": "Testdb5",
"repository": {
"typeName": "repository",
"qualifiedName": "repository01"
}
}
}
],
"referredEntities": {
}
}
},
"version": {
"version": "1.0.0"
}
}
From: Jo MailingList <jo...@gmail.com>
Reply-To: "user@atlas.apache.org" <us...@atlas.apache.org>
Date: Tuesday, February 6, 2018 at 2:45 AM
To: "user@atlas.apache.org" <us...@atlas.apache.org>
Subject: Re: Relating to an entity using uniqueAttributes
Hello,
On the same topic,
Now I realised that when refering to an object alredy created, is not enough to add the qualifiedName, but it also needs to include the name attribute since it is a required attribute. The point is that no matter which name I put, the actual referenced entity is updated to have that name.
Is there a way to put the reference without adding the name attribute in the values of the "referenced entity"?
Thank you very much,
Best regards,
Joan
2018-01-05 14:24 GMT+01:00 Jo MailingList <jo...@gmail.com>:
Hello,
Thank you very much,
This has been of great help.
Best regards,
Joan
2018-01-05 9:05 GMT+01:00 Madhan Neethiraj <ma...@apache.org>:
Joan,
A slightly better approach is to provide repository entity as a separate entry in “entities” and refer to the repository in database entity using its temp-id, as shown below.
This approach will allow an entity be referenced from multiple entities using its temp-id (instead of adding the entire entity details in each such reference).
Madhan
{
"message": {
"type": "ENTITY_FULL_UPDATE",
"user": "user",
"entities": [
{
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id": "-14568879465268",
"typeName": "database",
"state": "ACTIVE",
"version": 0
},
"typeName": "database",
"values": {
"dbName": "Test7",
"description": "Test db7",
"name": "Testdb7",
"origin_app": "origin",
"origin_username": "externalusername",
"owner": "owner",
"qualifiedName": "Testdb5",
"repository": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id": "-14568879465269",
"typeName": "repository",
"state": "ACTIVE",
"version": 0
}
},
"traitNames": [],
"traits": {}
},
{
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id": "-14568879465269",
"typeName": "repository",
"state": "ACTIVE",
"version": 0
},
"typeName": "repository",
"values": {
"qualifiedName": "repository01"
},
"traitNames": [],
"traits": {}
}
]
},
"version": {
"version": "1.0.0"
}
}
From: Madhan Neethiraj <ma...@apache.org>
Date: Thursday, January 4, 2018 at 11:18 PM
To: "user@atlas.apache.org" <us...@atlas.apache.org>
Subject: Re: Relating to an entity using uniqueAttributes
Joan,
To refer to another entity In Atlas hook notifications, add the referred entity details as shown below. Also, I would suggest using ENTITY_FULL_UPDATE as message type (instead of ENTITY_CREATE).
{
"message": {
"type": "ENTITY_FULL_UPDATE",
"user": "user",
"entities": [
{
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id": "-14568879465268",
"typeName": "database",
"state": "ACTIVE",
"version": 0
},
"typeName": "database",
"values": {
"dbName": "Test7",
"description": "Test db7",
"name": "Testdb7",
"origin_app": "origin",
"origin_username": "externalusername",
"owner": "owner",
"qualifiedName": "Testdb5",
"repository": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id": "-14568879465269",
"typeName": "repository",
"state": "ACTIVE",
"version": 0
},
"typeName": "repository",
"values": {
"qualifiedName": "repository01"
},
"traitNames": [],
"traits": {}
}
},
"traitNames": [],
"traits": {}
}
]
},
"version": {
"version": "1.0.0"
}
}
Hope this helps.
Madhan
From: Jo MailingList <jo...@gmail.com>
Reply-To: "user@atlas.apache.org" <us...@atlas.apache.org>
Date: Tuesday, January 2, 2018 at 9:54 AM
To: "user@atlas.apache.org" <us...@atlas.apache.org>
Subject: Relating to an entity using uniqueAttributes
Hello everyone,
I am trying to create a database that in my type model has a reference to a repository. I want to create it by sending a message to the ATLAS_HOOK topic to the Kafka broker as follows:
{
"version": {
"version": "1.0.0"
},
"message": {
"entities": [{
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Reference",
"id": {
"jsonClass": "org.apache.atlas.typesystem.json.InstanceSerialization$_Id",
"id": "-14568879465268",
"version": 0,
"typeName": "database",
"state": "ACTIVE"
},
"typeName": "database",
"values": {
"qualifiedName": "Testdb5",
"repository": {
"jsonClass":"org.apache.atlas.typesystem.json.InstanceSerialization$_Id"
"version": 0,
"typeName": "repository",
"uniqueAttributes": {
"qualifiedName": "repository01"
},
"state": "ACTIVE"
},
"owner": "owner",
"description": "Test db7",
"name": "Testdb7",
"dbName": "Test7",
"origin_app": "origin",
"origin_username": "externalusername"
},
"traitNames": [],
"traits": {}
}],
"type": "ENTITY_CREATE",
"user": "user"
}
}
The database type expects a repository attribute of the type repository as a reference. It works when I give the guid of the reference directly but when trying my example it fails and it seems to me that it does not process the information I provide as uniqueAttributes. What I see in the application.log is the following stacktrace:
org.apache.atlas.exception.AtlasBaseException: ObjectId is not valid AtlasObjectId{guid='null', typeName='repository', uniqueAttributes={}}
at org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.visitReference(AtlasEntityGraphDiscoveryV1.java:186)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.visitAttribute(AtlasEntityGraphDiscoveryV1.java:234)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.visitEntity(AtlasEntityGraphDiscoveryV1.java:327)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.walkEntityGraph(AtlasEntityGraphDiscoveryV1.java:354)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discover(AtlasEntityGraphDiscoveryV1.java:142)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityGraphDiscoveryV1.discoverEntities(AtlasEntityGraphDiscoveryV1.java:69)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.preCreateOrUpdate(AtlasEntityStoreV1.java:523)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:174)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1.createOrUpdate(AtlasEntityStoreV1.java:193)
at org.apache.atlas.repository.store.graph.v1.AtlasEntityStoreV1$$FastClassBySpringCGLIB$$80c00649.invoke(<generated>)
This message makes me suspicious that Atlas is not processing my input message correctly and not reading my uniqueAttributes information.
Does someone have a hint of what I am doing wrong or if this is a currrent issue inside Atlas or it is not a supported feature right now?
Thank you very much,
Best regards,
Joan