You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Madhan Neethiraj <ma...@apache.org> on 2018/02/01 18:10:37 UTC
Re: Review Request 65412: Export: Support Full Export Using Type Names
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65412/#review196616
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
Lines 227 (patched)
<https://reviews.apache.org/r/65412/#comment276307>
getQuerty ==> getQuery
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
Lines 292 (patched)
<https://reviews.apache.org/r/65412/#comment276345>
This method seems to retrieve entities for the given 'guids'. Consider renaming this method as:
- List<AtlasEntityWithExtInfo> getEntitites(List<String> guids)
If possible, consider using existing method EntityGraphRetriever.toAtlasEntitiesWithExtInfo(List<String> guids).
repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java
Lines 46 (patched)
<https://reviews.apache.org/r/65412/#comment276346>
Instead of returning list of guids, wouldn't it be efficient to retrieve list of vertices from all export queries - given that further processing would retrieve vertices for each guid returned here.
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformerJSONTest.java
Lines 20 (patched)
<https://reviews.apache.org/r/65412/#comment276344>
Only change is addition of these 'import' statements. Please review and revert if unnecessary.
- Madhan Neethiraj
On Jan. 30, 2018, 12:52 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65412/
> -----------------------------------------------------------
>
> (Updated Jan. 30, 2018, 12:52 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-2422
> https://issues.apache.org/jira/browse/ATLAS-2422
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Existing implementation of _ExportService_, uses the starting entity that is fetched using a gremlin query. This implementation expands on that.
>
> **Approach**
> - Additional query (switch-case) added to _AtlasGremlinQueryProvider_.
> - Modify _ExportService_ to process additional flavor of starting entity by looking at the new option specified in _AtlasExportRequest_.
>
> **Usage**
> Note the _typeName_ and _options_ parameters.
>
> Contents of _exportRequest.json_:
> ```
> {"itemsToExport": [ { "typeName": "hive_db,hdfs_path" } ], "options": { "fetchType": "FULL", "matchType": "forType"}}
>
> ```
> CURL:
> ```
> curl -X POST -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d @../docs/exportRequest.json http://localhost:21000/api/atlas/admin/export > ../docs/full-export.zip
> ```
>
> Import is not impacted.
> ```
> curl -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@../docs/full-export.zip http://localhost:21000/api/atlas/admin/import
> ```
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java a015e9b
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java a88c09e
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java 9cffdb9
> repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java e4898bd
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 7901ef6
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceReportingTest.java 0aaaa70
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java 7289512
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformerJSONTest.java 7044e71
>
>
> Diff: https://reviews.apache.org/r/65412/diff/1/
>
>
> Testing
> -------
>
> **Unit test**
> - Added new test to verify the behavior.
>
>
> **Functional test**
> - Via CURL calls.
>
> **Accuracy testing**
> - Using Export-import-export compare.
>
>
> File Attachments
> ----------------
>
> Export 1
> https://reviews.apache.org/media/uploaded/files/2018/01/30/813f8af8-8b38-4896-8f6b-dc72c3652d3e__full-export-pre.zip
> Export After Import
> https://reviews.apache.org/media/uploaded/files/2018/01/30/967aff68-2320-4f85-b10e-9500060ae118__full-export-post.zip
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 65412: Export: Support Full Export Using Type Names
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On Feb. 1, 2018, 6:10 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java
> > Lines 292 (patched)
> > <https://reviews.apache.org/r/65412/diff/1/?file=1950174#file1950174line311>
> >
> > This method seems to retrieve entities for the given 'guids'. Consider renaming this method as:
> > - List<AtlasEntityWithExtInfo> getEntitites(List<String> guids)
> >
> > If possible, consider using existing method EntityGraphRetriever.toAtlasEntitiesWithExtInfo(List<String> guids).
This method is no longer necessary after refactoring.
> On Feb. 1, 2018, 6:10 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java
> > Lines 46 (patched)
> > <https://reviews.apache.org/r/65412/diff/1/?file=1950175#file1950175line46>
> >
> > Instead of returning list of guids, wouldn't it be efficient to retrieve list of vertices from all export queries - given that further processing would retrieve vertices for each guid returned here.
Based on discussion.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65412/#review196616
-----------------------------------------------------------
On Feb. 2, 2018, 1:17 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65412/
> -----------------------------------------------------------
>
> (Updated Feb. 2, 2018, 1:17 a.m.)
>
>
> Review request for atlas.
>
>
> Bugs: ATLAS-2422
> https://issues.apache.org/jira/browse/ATLAS-2422
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Existing implementation of _ExportService_, uses the starting entity that is fetched using a gremlin query. This implementation expands on that.
>
> **Approach**
> - Additional query (switch-case) added to _AtlasGremlinQueryProvider_.
> - Modify _ExportService_ to process additional flavor of starting entity by looking at the new option specified in _AtlasExportRequest_.
>
> **Usage**
> Note the _typeName_ and _options_ parameters.
>
> Contents of _exportRequest.json_:
> ```
> {"itemsToExport": [ { "typeName": "hive_db,hdfs_path" } ], "options": { "fetchType": "FULL", "matchType": "forType"}}
>
> ```
> CURL:
> ```
> curl -X POST -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" -d @../docs/exportRequest.json http://localhost:21000/api/atlas/admin/export > ../docs/full-export.zip
> ```
>
> Import is not impacted.
> ```
> curl -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@../docs/full-export.zip http://localhost:21000/api/atlas/admin/import
> ```
>
>
> Diffs
> -----
>
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportRequest.java a015e9bf0
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java a88c09e84
> repository/src/main/java/org/apache/atlas/util/AtlasGremlin2QueryProvider.java 9cffdb9dc
> repository/src/main/java/org/apache/atlas/util/AtlasGremlinQueryProvider.java e4898bde1
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 7901ef687
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceReportingTest.java 0aaaa7048
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTestUtils.java 72895125c
>
>
> Diff: https://reviews.apache.org/r/65412/diff/3/
>
>
> Testing
> -------
>
> **Unit test**
> - Added new test to verify the behavior.
>
>
> **Functional test**
> - Via CURL calls.
>
> **Accuracy testing**
> - Using Export-import-export compare.
>
>
> File Attachments
> ----------------
>
> Export 1
> https://reviews.apache.org/media/uploaded/files/2018/01/30/813f8af8-8b38-4896-8f6b-dc72c3652d3e__full-export-pre.zip
> Export After Import
> https://reviews.apache.org/media/uploaded/files/2018/01/30/967aff68-2320-4f85-b10e-9500060ae118__full-export-post.zip
>
>
> Thanks,
>
> Ashutosh Mestry
>
>