You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@atlas.apache.org by Cade Parker <ca...@civitaslearning.com> on 2018/11/28 23:04:25 UTC
Exporting/Importing Relationships
Is it possible to export relationships along with the entities when using
the export API? My use case is trying to make a backwards incompatible
change to the type system by exporting all entities of the type I want to
change, delete all of the entities of that type (including the
relationships that use that type), modify the entities and the typedef(s)
from the export locally, and import the modified entities with the import
API. However, I can't seem to find an option or see any behavior that
exporting/importing relationships is possible. Is the expectation that when
you export from Atlas and re-import, that the relationships need to be
recreated? How would a data migration from one Atlas instance to another
work using the Export/Import APIs, and how does it affect the relationships
of those entities?
Thanks,
Cade Parker, Civitas Learning Inc.
Re: Exporting/Importing Relationships
Posted by Madhan Neethiraj <ma...@apache.org>.
Cade,
Thanks for the details. I could reproduce the issue – of missing relationship-def in atlas-typedefs.json.
I field ATLAS-2984 to track fix for this issue. Fix should be out for review shortly.
Thanks,
Madhan
From: Cade Parker <ca...@civitaslearning.com>
Reply-To: "user@atlas.apache.org" <us...@atlas.apache.org>
Date: Thursday, November 29, 2018 at 12:39 PM
To: "user@atlas.apache.org" <us...@atlas.apache.org>
Subject: Re: Exporting/Importing Relationships
Thanks for the reply Madhan. I think my initial assumptions about the import functionality were wrong. I assumed that the relationships weren't being rebuilt on the import and it was only creating the entities themselves.
In my first attempt, I exported a type (TypeA) and then removed all instances of that entity. Once all the entities were removed, all of the relationships were deleted automatically by Atlas. Then I removed the relationshipDefs that were referencing TypeA and the TypeA entitydef itself. On my local machine, I made the backwards incompatible changes of TypeA in the same atlas-typedefs.json file that was produced by the export operation, and I edited the entity attributes in the guid.json files of the TypeA entities. I zipped the changes and ran the import operation with the new entities/typedefs, and all of the entities were recreated as expected, but the relationships for TypeA were not recreated. After reading your email, I tried again, but I recreated the relationship definitions that used TypeA with the typedefs API before importing (I did not recreate the TypeA entityDef via the typedef API, and instead let the import operation handle it), and the relationships were rebuilt as expected. Success! Finally, I tried the whole process again, but included the TypeA relationshipDefs in the atlas-typedefs.json file used in the import, and it also gave me the desired result.
The process is a little tedious, but it's useful to have in case we want to migrate all entities to a new typedef without having to handle two entity types in the dependent services or keep track of which type is the latest. To make this process a little easier, I would suggest including the relationshipDefs that are used by the exported entities in the atlas-typedefs.json output file, so they are recreated automatically and it minimizes the changes needed in the typedefs file.
Thanks again!
Cade Parker, Civitas Learning Inc.
On Thu, Nov 29, 2018 at 1:10 PM Madhan Neethiraj <ma...@apache.org> wrote:
Cade,
In addition to including entity data, Atlas export operation includes details of entity-relationships, entity-classifications and all typedefs referenced by the entities.
Here is the sample export command line:
curl -X POST -u user:password -H "Accept: application/zip" -H "Content-Type: application/json" http://atlas-host:port/api/atlas/admin/export -d '{ "itemsToExport": [ { "typeName": "hive_db", "uniqueAttributes": { "qualifiedName": "hortoniabank@cl1" } } ], "options": { "matchType":"equals", "fetchType":"connected" }}' > export-hive_db-hortoniabank.zip
Here is the exported data for ‘hortoniabank’ database entity, which includes details of its relationships with 3 tables – please see inside relationshipAttributes:
{
"entity": {
"guid": "7bede7f3-3ad6-4051-a79a-6d97c8e42de2",
"typeName": "hive_db",
"attributes": {
"qualifiedName": "hortoniabank@cl1",
...
},
"classifications": [
{
"typeName": "FINANCE",
...
}
],
"relationshipAttributes": {
"tables": [
{
"displayText": "ww_customers",
"entityStatus": "ACTIVE",
"guid": "247887d3-bf6b-4ae7-8ba8-cb3ed9fdbc9d",
"relationshipAttributes": {
"typeName": "hive_table_db"
},
"relationshipGuid": "aacba4bb-9cbd-4b50-b3e0-4d27270c90a3",
"relationshipStatus": "ACTIVE",
"typeName": "hive_table"
},
{
"displayText": "us_customers",
"entityStatus": "ACTIVE",
"guid": "707b380b-3524-4a61-bfde-696f8002ff22",
"relationshipAttributes": {
"typeName": "hive_table_db"
},
"relationshipGuid": "8a5c8185-1b99-4f95-9d02-7d06c303dce0",
"relationshipStatus": "ACTIVE",
"typeName": "hive_table"
},
{
"displayText": "eu_countries",
"entityStatus": "ACTIVE",
"guid": "1bb2aa5e-d589-4350-afa6-b5fd75a41ccc",
"relationshipAttributes": {
"typeName": "hive_table_db"
},
"relationshipGuid": "0b7fddbd-3d0a-417d-aeb0-dd678fb62e01",
"relationshipStatus": "ACTIVE",
"typeName": "hive_table"
}
]
},
...
}
}
Hope this helps.
Getting back to your requirement on type-system updates: Atlas doesn’t allow certain changes due to complexities, like:
- addition of mandatory attribute would require populating existing entities with default value
- removal of attributes might require cleaning up the index stored in Solr (or whatever the index backend is); there may not be a clean way to remove index.
Such changes can take a very long time for Atlas instances having large amount of data.
However, I understand this is a very critical usecase for you, hence you are taking a tedious approach of export/cleanup/import. If you can enumerate specific changes you are looking for, we can look at the feasibility of adding support for these cases.
Thanks,
Madhan
From: Cade Parker <ca...@civitaslearning.com>
Reply-To: "user@atlas.apache.org" <us...@atlas.apache.org>
Date: Wednesday, November 28, 2018 at 3:04 PM
To: "user@atlas.apache.org" <us...@atlas.apache.org>
Subject: Exporting/Importing Relationships
Is it possible to export relationships along with the entities when using the export API? My use case is trying to make a backwards incompatible change to the type system by exporting all entities of the type I want to change, delete all of the entities of that type (including the relationships that use that type), modify the entities and the typedef(s) from the export locally, and import the modified entities with the import API. However, I can't seem to find an option or see any behavior that exporting/importing relationships is possible. Is the expectation that when you export from Atlas and re-import, that the relationships need to be recreated? How would a data migration from one Atlas instance to another work using the Export/Import APIs, and how does it affect the relationships of those entities?
Thanks,
Cade Parker, Civitas Learning Inc.
Re: Exporting/Importing Relationships
Posted by Cade Parker <ca...@civitaslearning.com>.
Thanks for the reply Madhan. I think my initial assumptions about the
import functionality were wrong. I assumed that the relationships weren't
being rebuilt on the import and it was only creating the entities
themselves.
In my first attempt, I exported a type (TypeA) and then removed all
instances of that entity. Once all the entities were removed, all of the
relationships were deleted automatically by Atlas. Then I removed the
relationshipDefs that were referencing TypeA and the TypeA entitydef
itself. On my local machine, I made the backwards incompatible changes of
TypeA in the same atlas-typedefs.json file that was produced by the export
operation, and I edited the entity attributes in the guid.json files of the
TypeA entities. I zipped the changes and ran the import operation with the
new entities/typedefs, and all of the entities were recreated as expected,
but the relationships for TypeA were not recreated. After reading your
email, I tried again, but I recreated the relationship definitions that
used TypeA with the typedefs API before importing (I did not recreate the
TypeA entityDef via the typedef API, and instead let the import operation
handle it), and the relationships were rebuilt as expected. Success!
Finally, I tried the whole process again, but included the TypeA
relationshipDefs in the atlas-typedefs.json file used in the import, and it
also gave me the desired result.
The process is a little tedious, but it's useful to have in case we want to
migrate all entities to a new typedef without having to handle two entity
types in the dependent services or keep track of which type is the latest.
To make this process a little easier, I would suggest including the
relationshipDefs that are used by the exported entities in the
atlas-typedefs.json output file, so they are recreated automatically and it
minimizes the changes needed in the typedefs file.
Thanks again!
Cade Parker, Civitas Learning Inc.
On Thu, Nov 29, 2018 at 1:10 PM Madhan Neethiraj <ma...@apache.org> wrote:
> Cade,
>
>
>
> In addition to including entity data, Atlas export operation includes
> details of entity-relationships, entity-classifications and all typedefs
> referenced by the entities.
>
>
>
> Here is the sample export command line:
>
> curl -X POST -u user:password -H "Accept: application/zip" -H
> "Content-Type: application/json" http://atlas-host:port/api/atlas/admin/export
> -d '{ "itemsToExport": [ { "typeName": "hive_db", "uniqueAttributes": {
> "qualifiedName": "hortoniabank@cl1" } } ], "options": {
> "matchType":"equals", "fetchType":"connected" }}' >
> export-hive_db-hortoniabank.zip
>
>
>
> Here is the exported data for ‘hortoniabank’ database entity, which
> includes details of its relationships with 3 tables – please see inside
> relationshipAttributes:
>
> {
>
> "entity": {
>
> "guid": "7bede7f3-3ad6-4051-a79a-6d97c8e42de2",
>
> "typeName": "hive_db",
>
> "attributes": {
>
> "qualifiedName": "hortoniabank@cl1",
>
> ...
>
> },
>
> "classifications": [
>
> {
>
> "typeName": "FINANCE",
>
> ...
>
> }
>
> ],
>
> "relationshipAttributes": {
>
> "tables": [
>
> {
>
> "displayText": "ww_customers",
>
> "entityStatus": "ACTIVE",
>
> "guid": "247887d3-bf6b-4ae7-8ba8-cb3ed9fdbc9d",
>
> "relationshipAttributes": {
>
> "typeName": "hive_table_db"
>
> },
>
> "relationshipGuid":
> "aacba4bb-9cbd-4b50-b3e0-4d27270c90a3",
>
> "relationshipStatus": "ACTIVE",
>
> "typeName": "hive_table"
>
> },
>
> {
>
> "displayText": "us_customers",
>
> "entityStatus": "ACTIVE",
>
> "guid": "707b380b-3524-4a61-bfde-696f8002ff22",
>
> "relationshipAttributes": {
>
> "typeName": "hive_table_db"
>
> },
>
> "relationshipGuid":
> "8a5c8185-1b99-4f95-9d02-7d06c303dce0",
>
> "relationshipStatus": "ACTIVE",
>
> "typeName": "hive_table"
>
> },
>
> {
>
> "displayText": "eu_countries",
>
> "entityStatus": "ACTIVE",
>
> "guid": "1bb2aa5e-d589-4350-afa6-b5fd75a41ccc",
>
> "relationshipAttributes": {
>
> "typeName": "hive_table_db"
>
> },
>
> "relationshipGuid":
> "0b7fddbd-3d0a-417d-aeb0-dd678fb62e01",
>
> "relationshipStatus": "ACTIVE",
>
> "typeName": "hive_table"
>
> }
>
> ]
>
> },
>
> ...
>
> }
>
> }
>
>
>
> Hope this helps.
>
>
>
> Getting back to your requirement on type-system updates: Atlas doesn’t
> allow certain changes due to complexities, like:
>
> - addition of mandatory attribute would require populating existing
> entities with default value
>
> - removal of attributes might require cleaning up the index stored in Solr
> (or whatever the index backend is); there may not be a clean way to remove
> index.
>
>
>
> Such changes can take a very long time for Atlas instances having large
> amount of data.
>
>
>
> However, I understand this is a very critical usecase for you, hence you
> are taking a tedious approach of export/cleanup/import. If you can
> enumerate specific changes you are looking for, we can look at the
> feasibility of adding support for these cases.
>
>
>
> Thanks,
>
> Madhan
>
>
>
>
>
> *From: *Cade Parker <ca...@civitaslearning.com>
> *Reply-To: *"user@atlas.apache.org" <us...@atlas.apache.org>
> *Date: *Wednesday, November 28, 2018 at 3:04 PM
> *To: *"user@atlas.apache.org" <us...@atlas.apache.org>
> *Subject: *Exporting/Importing Relationships
>
>
>
> Is it possible to export relationships along with the entities when using
> the export API? My use case is trying to make a backwards incompatible
> change to the type system by exporting all entities of the type I want to
> change, delete all of the entities of that type (including the
> relationships that use that type), modify the entities and the typedef(s)
> from the export locally, and import the modified entities with the import
> API. However, I can't seem to find an option or see any behavior that
> exporting/importing relationships is possible. Is the expectation that when
> you export from Atlas and re-import, that the relationships need to be
> recreated? How would a data migration from one Atlas instance to another
> work using the Export/Import APIs, and how does it affect the relationships
> of those entities?
>
>
>
> Thanks,
>
> Cade Parker, Civitas Learning Inc.
>
Re: Exporting/Importing Relationships
Posted by Madhan Neethiraj <ma...@apache.org>.
Cade,
In addition to including entity data, Atlas export operation includes details of entity-relationships, entity-classifications and all typedefs referenced by the entities.
Here is the sample export command line:
curl -X POST -u user:password -H "Accept: application/zip" -H "Content-Type: application/json" http://atlas-host:port/api/atlas/admin/export -d '{ "itemsToExport": [ { "typeName": "hive_db", "uniqueAttributes": { "qualifiedName": "hortoniabank@cl1" } } ], "options": { "matchType":"equals", "fetchType":"connected" }}' > export-hive_db-hortoniabank.zip
Here is the exported data for ‘hortoniabank’ database entity, which includes details of its relationships with 3 tables – please see inside relationshipAttributes:
{
"entity": {
"guid": "7bede7f3-3ad6-4051-a79a-6d97c8e42de2",
"typeName": "hive_db",
"attributes": {
"qualifiedName": "hortoniabank@cl1",
...
},
"classifications": [
{
"typeName": "FINANCE",
...
}
],
"relationshipAttributes": {
"tables": [
{
"displayText": "ww_customers",
"entityStatus": "ACTIVE",
"guid": "247887d3-bf6b-4ae7-8ba8-cb3ed9fdbc9d",
"relationshipAttributes": {
"typeName": "hive_table_db"
},
"relationshipGuid": "aacba4bb-9cbd-4b50-b3e0-4d27270c90a3",
"relationshipStatus": "ACTIVE",
"typeName": "hive_table"
},
{
"displayText": "us_customers",
"entityStatus": "ACTIVE",
"guid": "707b380b-3524-4a61-bfde-696f8002ff22",
"relationshipAttributes": {
"typeName": "hive_table_db"
},
"relationshipGuid": "8a5c8185-1b99-4f95-9d02-7d06c303dce0",
"relationshipStatus": "ACTIVE",
"typeName": "hive_table"
},
{
"displayText": "eu_countries",
"entityStatus": "ACTIVE",
"guid": "1bb2aa5e-d589-4350-afa6-b5fd75a41ccc",
"relationshipAttributes": {
"typeName": "hive_table_db"
},
"relationshipGuid": "0b7fddbd-3d0a-417d-aeb0-dd678fb62e01",
"relationshipStatus": "ACTIVE",
"typeName": "hive_table"
}
]
},
...
}
}
Hope this helps.
Getting back to your requirement on type-system updates: Atlas doesn’t allow certain changes due to complexities, like:
- addition of mandatory attribute would require populating existing entities with default value
- removal of attributes might require cleaning up the index stored in Solr (or whatever the index backend is); there may not be a clean way to remove index.
Such changes can take a very long time for Atlas instances having large amount of data.
However, I understand this is a very critical usecase for you, hence you are taking a tedious approach of export/cleanup/import. If you can enumerate specific changes you are looking for, we can look at the feasibility of adding support for these cases.
Thanks,
Madhan
From: Cade Parker <ca...@civitaslearning.com>
Reply-To: "user@atlas.apache.org" <us...@atlas.apache.org>
Date: Wednesday, November 28, 2018 at 3:04 PM
To: "user@atlas.apache.org" <us...@atlas.apache.org>
Subject: Exporting/Importing Relationships
Is it possible to export relationships along with the entities when using the export API? My use case is trying to make a backwards incompatible change to the type system by exporting all entities of the type I want to change, delete all of the entities of that type (including the relationships that use that type), modify the entities and the typedef(s) from the export locally, and import the modified entities with the import API. However, I can't seem to find an option or see any behavior that exporting/importing relationships is possible. Is the expectation that when you export from Atlas and re-import, that the relationships need to be recreated? How would a data migration from one Atlas instance to another work using the Export/Import APIs, and how does it affect the relationships of those entities?
Thanks,
Cade Parker, Civitas Learning Inc.