You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@atlas.apache.org by Ernie Ostic <eo...@us.ibm.com> on 2017/06/23 16:07:35 UTC
Looking for recommended practices for generic retrieval of types
Hi all.
Am working on several solutions for pulling general types from Atlas for
(near term) inclusion in other repositories. Getting all instances of
"hive_table" for example. Using the v2 api, earlier research shows that
there are two effective ways to do this, one using "basic" and the other
using "dsl"...
http://hostname:21000/api/atlas/v2/search/basic?query=hive_table
http://hostname.157:21000/api/atlas/v2/search/dsl?query=hive_table
The legacy API had a way to do this directly, but of course, we want to use
only the v2. I haven't tested yet with any massive number of instances,
so this is a question of "is this the best approach?".....are there
performance concerns of one over the other? Some other method I have
overlooked?
Ultimately, requests like this with additional facets will be more common,
in which case future improvements in indexing or the need for more detailed
syntax/etc. may clearly point to one method or the other. But the
question remains for primitive "dump everything" strategies that will be
requested in the future.
Thanks!
Ernie
Ernie Ostic
Worldwide Technical Sales
InfoSphere Information Server
IBM Analytics
Cell: (617) 331 8238
---------------------------------------------------------------
Open IGC is here!
Extend the Catalog with custom objects and lineage definitions!
https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/
Re: Looking for recommended practices for generic retrieval of types
Posted by Ernie Ostic <eo...@us.ibm.com>.
Thanks Madhan. I will look into it. Is there a Swagger page or other
outline of the API calls available for export? ....is this a POST? Thx!
Ernie
Ernie Ostic
Worldwide Technical Sales
InfoSphere Information Server
IBM Analytics
Cell: (617) 331 8238
---------------------------------------------------------------
Open IGC is here!
Extend the Catalog with custom objects and lineage definitions!
https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/
From: Madhan Neethiraj <ma...@apache.org>
To: "user@atlas.apache.org" <us...@atlas.apache.org>,
"user@atlas.incubator.apache.org"
<us...@atlas.incubator.apache.org>, Ashutosh Mestry
<am...@hortonworks.com>
Date: 06/24/2017 02:52 AM
Subject: Re: Looking for recommended practices for generic retrieval of
types
Sent by: Madhan Neethiraj <mn...@hortonworks.com>
Ernie,
Given there could be a large number of entities of the given type, I would
suggest you consider using export REST API at api/atlas/admin/export. This
API returns a zip file containing the entities that qualify for the given
criteria. For example, the following request exports all hive_table
entities, along with classifications and entities that are connected to
hive_table entities (like hive_clolumns, hive_processes, etc):
{
"itemsToExport": [
{
"typeName": "hive_table",
"uniqueAttributes": {
"name": ".*"
}
}
],
"options": {
"fetchType": "connected",
"matchType": "matches"
}
}
If you need more details, Ashutosh would be able to help.
Thanks,
Madhan
From: Ernie Ostic <eo...@us.ibm.com>
Reply-To: "user@atlas.apache.org" <us...@atlas.apache.org>
Date: Friday, June 23, 2017 at 9:07 AM
To: "user@atlas.incubator.apache.org" <us...@atlas.incubator.apache.org>
Subject: Looking for recommended practices for generic retrieval of types
Hi all.
Am working on several solutions for pulling general types from Atlas for
(near term) inclusion in other repositories. Getting all instances of
"hive_table" for example. Using the v2 api, earlier research shows that
there are two effective ways to do this, one using "basic" and the other
using "dsl"...
http://hostname:21000/api/atlas/v2/search/basic?query=hive_table
http://hostname.157:21000/api/atlas/v2/search/dsl?query=hive_table
The legacy API had a way to do this directly, but of course, we want to use
only the v2. I haven't tested yet with any massive number of instances, so
this is a question of "is this the best approach?".....are there
performance concerns of one over the other? Some other method I have
overlooked?
Ultimately, requests like this with additional facets will be more common,
in which case future improvements in indexing or the need for more detailed
syntax/etc. may clearly point to one method or the other. But the question
remains for primitive "dump everything" strategies that will be requested
in the future.
Thanks!
Ernie
Ernie Ostic
Worldwide Technical Sales
InfoSphere Information Server
IBM Analytics
Cell: (617) 331 8238
---------------------------------------------------------------
Open IGC is here!
Extend the Catalog with custom objects and lineage definitions!
https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/
Re: Looking for recommended practices for generic retrieval of types
Posted by Madhan Neethiraj <ma...@apache.org>.
Ernie,
Given there could be a large number of entities of the given type, I would suggest you consider using export REST API at api/atlas/admin/export. This API returns a zip file containing the entities that qualify for the given criteria. For example, the following request exports all hive_table entities, along with classifications and entities that are connected to hive_table entities (like hive_clolumns, hive_processes, etc):
{
"itemsToExport": [
{
"typeName": "hive_table",
"uniqueAttributes": {
"name": ".*"
}
}
],
"options": {
"fetchType": "connected",
"matchType": "matches"
}
}
If you need more details, Ashutosh would be able to help.
Thanks,
Madhan
From: Ernie Ostic <eo...@us.ibm.com>
Reply-To: "user@atlas.apache.org" <us...@atlas.apache.org>
Date: Friday, June 23, 2017 at 9:07 AM
To: "user@atlas.incubator.apache.org" <us...@atlas.incubator.apache.org>
Subject: Looking for recommended practices for generic retrieval of types
Hi all.
Am working on several solutions for pulling general types from Atlas for (near term) inclusion in other repositories. Getting all instances of "hive_table" for example. Using the v2 api, earlier research shows that there are two effective ways to do this, one using "basic" and the other using "dsl"...
http://hostname:21000/api/atlas/v2/search/basic?query=hive_table
http://hostname.157:21000/api/atlas/v2/search/dsl?query=hive_table
The legacy API had a way to do this directly, but of course, we want to use only the v2. I haven't tested yet with any massive number of instances, so this is a question of "is this the best approach?".....are there performance concerns of one over the other? Some other method I have overlooked?
Ultimately, requests like this with additional facets will be more common, in which case future improvements in indexing or the need for more detailed syntax/etc. may clearly point to one method or the other. But the question remains for primitive "dump everything" strategies that will be requested in the future.
Thanks!
Ernie
Ernie Ostic
Worldwide Technical Sales
InfoSphere Information Server
IBM Analytics
Cell: (617) 331 8238
---------------------------------------------------------------
Open IGC is here!
Extend the Catalog with custom objects and lineage definitions!
https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/