You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2019/05/06 23:34:19 UTC

Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/
-----------------------------------------------------------

Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.


Bugs: ATLAS-3193
    https://issues.apache.org/jira/browse/ATLAS-3193


Repository: atlas


Description
-------

**Background**
Please look at bug description.

**Approach**
_ExportService_: Refactor to extract starting entity logic in a separate reusable class.
(New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
_BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
_GraphBackedSearchIndexer_: Additional property to index.

**CURL**

Import:
```
curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
```


Diffs
-----

  common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
  repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
  repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
  repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
  repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
  tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java 91ff89a00 


Diff: https://reviews.apache.org/r/70599/diff/1/


Testing
-------

**Unit tests**
New tests added.

**Functional tests**
Existing export and import flow via CURL calls.


Thanks,

Ashutosh Mestry


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Ashutosh Mestry <am...@hortonworks.com>.

> On May 7, 2019, 8:59 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
> > Lines 171 (patched)
> > <https://reviews.apache.org/r/70599/diff/2/?file=2143593#file2143593line171>
> >
> >     why do we need to maintain historical guids? it doesn't seem to be read anywhere

This is for debugging and tracability.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215103
-----------------------------------------------------------


On May 8, 2019, 3:10 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
> 
> (Updated May 8, 2019, 3:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3193
>     https://issues.apache.org/jira/browse/ATLAS-3193
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> Please look at bug description.
> 
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
> 
> **CURL**
> 
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
>   repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
>   repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42 
> 
> 
> Diff: https://reviews.apache.org/r/70599/diff/5/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests added.
> 
> **Functional tests**
> Existing export and import flow via CURL calls.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215103
-----------------------------------------------------------




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
Lines 171 (patched)
<https://reviews.apache.org/r/70599/#comment301535>

    why do we need to maintain historical guids? it doesn't seem to be read anywhere


- Sarath Subramanian


On May 6, 2019, 4:34 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
> 
> (Updated May 6, 2019, 4:34 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3193
>     https://issues.apache.org/jira/browse/ATLAS-3193
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> Please look at bug description.
> 
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
> 
> **CURL**
> 
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
>   repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
>   repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
>   tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java 91ff89a00 
> 
> 
> Diff: https://reviews.apache.org/r/70599/diff/2/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests added.
> 
> **Functional tests**
> Existing export and import flow via CURL calls.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Nixon Rodrigues <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215124
-----------------------------------------------------------


Ship it!




Ship It!

- Nixon Rodrigues


On May 8, 2019, 1:42 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
> 
> (Updated May 8, 2019, 1:42 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3193
>     https://issues.apache.org/jira/browse/ATLAS-3193
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> Please look at bug description.
> 
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
> 
> **CURL**
> 
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
>   repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
>   repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42 
> 
> 
> Diff: https://reviews.apache.org/r/70599/diff/4/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests added.
> 
> **Functional tests**
> Existing export and import flow via CURL calls.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Nikhil Bonte <ni...@freestoneinfotech.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215122
-----------------------------------------------------------


Ship it!




Ship It!

- Nikhil Bonte


On May 8, 2019, 1:42 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
> 
> (Updated May 8, 2019, 1:42 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3193
>     https://issues.apache.org/jira/browse/ATLAS-3193
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> Please look at bug description.
> 
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
> 
> **CURL**
> 
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
>   repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
>   repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42 
> 
> 
> Diff: https://reviews.apache.org/r/70599/diff/4/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests added.
> 
> **Functional tests**
> Existing export and import flow via CURL calls.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Sridhar K <Sr...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215108
-----------------------------------------------------------




common/src/main/java/org/apache/atlas/repository/Constants.java
Lines 41 (patched)
<https://reviews.apache.org/r/70599/#comment301558>

    I understand what you are doing this for? But, do we have to replace the existing id? Why not keep using the existing id?


- Sridhar K


On May 8, 2019, 3:10 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
> 
> (Updated May 8, 2019, 3:10 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3193
>     https://issues.apache.org/jira/browse/ATLAS-3193
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> Please look at bug description.
> 
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
> 
> **CURL**
> 
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
>   repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
>   repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42 
> 
> 
> Diff: https://reviews.apache.org/r/70599/diff/6/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests added.
> 
> **Functional tests**
> Existing export and import flow via CURL calls.
> 
> **Pre-commit**
> https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/
-----------------------------------------------------------

(Updated May 8, 2019, 3:10 p.m.)


Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.


Changes
-------

Updates include: Minor refactoring.


Bugs: ATLAS-3193
    https://issues.apache.org/jira/browse/ATLAS-3193


Repository: atlas


Description
-------

**Background**
Please look at bug description.

**Approach**
_ExportService_: Refactor to extract starting entity logic in a separate reusable class.
(New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
_BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
_GraphBackedSearchIndexer_: Additional property to index.

**CURL**

Import:
```
curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
  repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
  repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
  repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
  repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42 


Diff: https://reviews.apache.org/r/70599/diff/5/

Changes: https://reviews.apache.org/r/70599/diff/4-5/


Testing
-------

**Unit tests**
New tests added.

**Functional tests**
Existing export and import flow via CURL calls.

**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/


Thanks,

Ashutosh Mestry


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/
-----------------------------------------------------------

(Updated May 8, 2019, 1:42 a.m.)


Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.


Changes
-------

Updates include: Refactoring for simplicity.


Bugs: ATLAS-3193
    https://issues.apache.org/jira/browse/ATLAS-3193


Repository: atlas


Description
-------

**Background**
Please look at bug description.

**Approach**
_ExportService_: Refactor to extract starting entity logic in a separate reusable class.
(New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
_BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
_GraphBackedSearchIndexer_: Additional property to index.

**CURL**

Import:
```
curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
  repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
  repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
  repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
  repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2BulkImportPercentTest.java e432d4e42 


Diff: https://reviews.apache.org/r/70599/diff/4/

Changes: https://reviews.apache.org/r/70599/diff/3-4/


Testing
-------

**Unit tests**
New tests added.

**Functional tests**
Existing export and import flow via CURL calls.

**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/


Thanks,

Ashutosh Mestry


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/
-----------------------------------------------------------

(Updated May 7, 2019, 9:19 p.m.)


Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.


Changes
-------

Updates include: 
- Addressed review comments.
- Added pre-commit build details.


Bugs: ATLAS-3193
    https://issues.apache.org/jira/browse/ATLAS-3193


Repository: atlas


Description
-------

**Background**
Please look at bug description.

**Approach**
_ExportService_: Refactor to extract starting entity logic in a separate reusable class.
(New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
_BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
_GraphBackedSearchIndexer_: Additional property to index.

**CURL**

Import:
```
curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
  repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
  repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
  repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
  repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
  repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 


Diff: https://reviews.apache.org/r/70599/diff/3/

Changes: https://reviews.apache.org/r/70599/diff/2-3/


Testing (updated)
-------

**Unit tests**
New tests added.

**Functional tests**
Existing export and import flow via CURL calls.

**Pre-commit**
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/1081/


Thanks,

Ashutosh Mestry


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Ashutosh Mestry <am...@hortonworks.com>.

> On May 7, 2019, 8:16 p.m., Sarath Subramanian wrote:
> > repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
> > Lines 182 (patched)
> > <https://reviews.apache.org/r/70599/diff/2/?file=2143593#file2143593line182>
> >
> >     consider having delimited separated values - 
> >     guid1:guid2:guid3 instead of braces ([]);

This is storing in JSON format. The intent was that we would be able to retrieve this at some point in future.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215093
-----------------------------------------------------------


On May 6, 2019, 11:34 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
> 
> (Updated May 6, 2019, 11:34 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3193
>     https://issues.apache.org/jira/browse/ATLAS-3193
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> Please look at bug description.
> 
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
> 
> **CURL**
> 
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
>   repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
>   repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
>   tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java 91ff89a00 
> 
> 
> Diff: https://reviews.apache.org/r/70599/diff/2/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests added.
> 
> **Functional tests**
> Existing export and import flow via CURL calls.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Re: Review Request 70599: Import Service: Improvement to Handle Importing to Existing Entity

Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/70599/#review215093
-----------------------------------------------------------




repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
Lines 69 (patched)
<https://reviews.apache.org/r/70599/#comment301512>

    typeRegistry is never used. consider removing it.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
Lines 188 (patched)
<https://reviews.apache.org/r/70599/#comment301513>

    getUniqueAttributes() is never used; consider removing it.



repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java
Lines 182 (patched)
<https://reviews.apache.org/r/70599/#comment301522>

    consider having delimited separated values - 
    guid1:guid2:guid3 instead of braces ([]);


- Sarath Subramanian


On May 6, 2019, 4:34 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/70599/
> -----------------------------------------------------------
> 
> (Updated May 6, 2019, 4:34 p.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj, Nikhil Bonte, and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-3193
>     https://issues.apache.org/jira/browse/ATLAS-3193
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> Please look at bug description.
> 
> **Approach**
> _ExportService_: Refactor to extract starting entity logic in a separate reusable class.
> (New) _StartEntityFetchByExportRequest_: Extracted logic for fetching start entity.
> _BulkImporterImpl_: Handle _AtlasSchemaViolation_ to update guid of existing entity. Uses start entity logic (from _StartEntityFetchByExportRequest_ to check and fetch existing entities).
> _GraphBackedSearchIndexer_: Additional property to index.
> 
> **CURL**
> 
> Import:
> ```
> curl -X POST -u admin:passW0rd -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F data=@./stocks.zip http://localhost:21000/api/atlas/admin/import
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 5055a0426 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 1cfa202a9 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java 62298f9f1 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java a52c3f986 
>   repository/src/main/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequest.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/BulkImporterImpl.java dc8bd19df 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphRetriever.java 14f991697 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java a1d6cef5c 
>   repository/src/test/java/org/apache/atlas/repository/impexp/StartEntityFetchByExportRequestTest.java PRE-CREATION 
>   tools/classification-updater/src/main/java/org/apache/atlas/tools/BulkFetchAndUpdate.java 91ff89a00 
> 
> 
> Diff: https://reviews.apache.org/r/70599/diff/2/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> New tests added.
> 
> **Functional tests**
> Existing export and import flow via CURL calls.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>