You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2019/05/06 23:34:19 UTC
Review Request 70599: Import Service: Improvement to Handle Importing
to Existing Entity
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/
-----------------------------------------------------------
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
Bugs: ATLAS-3193
https://issues.apache.org/jira/browse/ATLAS-3193
Repository: atlas
Description
-------
**Background**
Please look at bug description.
**Approach**
_ExportService_: Refactor to extract starting entity logic in a separate reusable class.
(New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
_BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
_GraphBackedSearchIndexer_: Additional property to index.
**CURL**
Import:
```
curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
```
Diffs
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java 91ff89a00
Diff: https://reviews.apache.org/r/70599/diff/1/
Testing
-------
**Unit tests**
New tests added.
**Functional tests**
Existing export and import flow via CURL calls.
Thanks,
Ashutosh Mestry
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On May 7, 2019, 8:59 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
> > Lines 171 (patched)
> > <https://reviews.apache.org/r/70599/diff/2/?file=2143593#file2143593line171>
> >
> > why do we need to maintain historical guids? it doesn't seem to be read anywhere
This is for debugging and tracability.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215103
-----------------------------------------------------------
On May 8, 2019, 3:10 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
>
> (Updated May 8, 2019, 3:10 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3193
> https://issues.apache.org/jira/browse/ATLAS-3193
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Please look at bug description.
>
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
>
> **CURL**
>
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
> repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42
>
>
> Diff: https://reviews.apache.org/r/70599/diff/5/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Existing export and import flow via CURL calls.
>
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215103
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
Lines 171 (patched)
<https://reviews.apache.org/r/70599/#comment301535>
why do we need to maintain historical guids? it doesn't seem to be read anywhere
- Sarath Subramanian
On May 6, 2019, 4:34 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
>
> (Updated May 6, 2019, 4:34 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3193
> https://issues.apache.org/jira/browse/ATLAS-3193
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Please look at bug description.
>
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
>
> **CURL**
>
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
> repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
> tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java 91ff89a00
>
>
> Diff: https://reviews.apache.org/r/70599/diff/2/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Existing export and import flow via CURL calls.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215124
-----------------------------------------------------------
Ship it!
Ship It!
- Nixon Rodrigues
On May 8, 2019, 1:42 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
>
> (Updated May 8, 2019, 1:42 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3193
> https://issues.apache.org/jira/browse/ATLAS-3193
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Please look at bug description.
>
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
>
> **CURL**
>
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
> repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42
>
>
> Diff: https://reviews.apache.org/r/70599/diff/4/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Existing export and import flow via CURL calls.
>
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215122
-----------------------------------------------------------
Ship it!
Ship It!
- Nikhil Bonte
On May 8, 2019, 1:42 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
>
> (Updated May 8, 2019, 1:42 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3193
> https://issues.apache.org/jira/browse/ATLAS-3193
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Please look at bug description.
>
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
>
> **CURL**
>
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
> repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42
>
>
> Diff: https://reviews.apache.org/r/70599/diff/4/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Existing export and import flow via CURL calls.
>
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Sridhar K <Sr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215108
-----------------------------------------------------------
common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 41 (patched)
<https://reviews.apache.org/r/70599/#comment301558>
I understand what you are doing this for? But, do we have to replace the existing id? Why not keep using the existing id?
- Sridhar K
On May 8, 2019, 3:10 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
>
> (Updated May 8, 2019, 3:10 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3193
> https://issues.apache.org/jira/browse/ATLAS-3193
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Please look at bug description.
>
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
>
> **CURL**
>
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
> repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42
>
>
> Diff: https://reviews.apache.org/r/70599/diff/6/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Existing export and import flow via CURL calls.
>
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/
-----------------------------------------------------------
(Updated May 8, 2019, 3:10 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
Changes
-------
Updates include: Minor refactoring.
Bugs: ATLAS-3193
https://issues.apache.org/jira/browse/ATLAS-3193
Repository: atlas
Description
-------
**Background**
Please look at bug description.
**Approach**
_ExportService_: Refactor to extract starting entity logic in a separate reusable class.
(New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
_BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
_GraphBackedSearchIndexer_: Additional property to index.
**CURL**
Import:
```
curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42
Diff: https://reviews.apache.org/r/70599/diff/5/
Changes: https://reviews.apache.org/r/70599/diff/4-5/
Testing
-------
**Unit tests**
New tests added.
**Functional tests**
Existing export and import flow via CURL calls.
**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
Thanks,
Ashutosh Mestry
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/
-----------------------------------------------------------
(Updated May 8, 2019, 1:42 a.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
Changes
-------
Updates include: Refactoring for simplicity.
Bugs: ATLAS-3193
https://issues.apache.org/jira/browse/ATLAS-3193
Repository: atlas
Description
-------
**Background**
Please look at bug description.
**Approach**
_ExportService_: Refactor to extract starting entity logic in a separate reusable class.
(New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
_BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
_GraphBackedSearchIndexer_: Additional property to index.
**CURL**
Import:
```
curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42
Diff: https://reviews.apache.org/r/70599/diff/4/
Changes: https://reviews.apache.org/r/70599/diff/3-4/
Testing
-------
**Unit tests**
New tests added.
**Functional tests**
Existing export and import flow via CURL calls.
**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
Thanks,
Ashutosh Mestry
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/
-----------------------------------------------------------
(Updated May 7, 2019, 9:19 p.m.)
Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
Changes
-------
Updates include:
- Addressed review comments.
- Added pre-commit build details.
Bugs: ATLAS-3193
https://issues.apache.org/jira/browse/ATLAS-3193
Repository: atlas
Description
-------
**Background**
Please look at bug description.
**Approach**
_ExportService_: Refactor to extract starting entity logic in a separate reusable class.
(New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
_BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
_GraphBackedSearchIndexer_: Additional property to index.
**CURL**
Import:
```
curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/70599/diff/3/
Changes: https://reviews.apache.org/r/70599/diff/2-3/
Testing (updated)
-------
**Unit tests**
New tests added.
**Functional tests**
Existing export and import flow via CURL calls.
**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
Thanks,
Ashutosh Mestry
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On May 7, 2019, 8:16 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
> > Lines 182 (patched)
> > <https://reviews.apache.org/r/70599/diff/2/?file=2143593#file2143593line182>
> >
> > consider having delimited separated values -
> > guid1:guid2:guid3 instead of braces ([]);
This is storing in JSON format. The intent was that we would be able to retrieve this at some point in future.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215093
-----------------------------------------------------------
On May 6, 2019, 11:34 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
>
> (Updated May 6, 2019, 11:34 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3193
> https://issues.apache.org/jira/browse/ATLAS-3193
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Please look at bug description.
>
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
>
> **CURL**
>
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
> repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
> tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java 91ff89a00
>
>
> Diff: https://reviews.apache.org/r/70599/diff/2/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Existing export and import flow via CURL calls.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 70599: Import Service: Improvement to Handle
Importing to Existing Entity
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215093
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
Lines 69 (patched)
<https://reviews.apache.org/r/70599/#comment301512>
typeRegistry is never used. consider removing it.
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
Lines 188 (patched)
<https://reviews.apache.org/r/70599/#comment301513>
getUniqueAttributes() is never used; consider removing it.
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
Lines 182 (patched)
<https://reviews.apache.org/r/70599/#comment301522>
consider having delimited separated values -
guid1:guid2:guid3 instead of braces ([]);
- Sarath Subramanian
On May 6, 2019, 4:34 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
>
> (Updated May 6, 2019, 4:34 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
>
>
> Bugs: ATLAS-3193
> https://issues.apache.org/jira/browse/ATLAS-3193
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> Please look at bug description.
>
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
>
> **CURL**
>
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9
> repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986
> repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df
> repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697
> repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c
> repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION
> tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java 91ff89a00
>
>
> Diff: https://reviews.apache.org/r/70599/diff/2/
>
>
> Testing
> -------
>
> **Unit tests**
> New tests added.
>
> **Functional tests**
> Existing export and import flow via CURL calls.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>