You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/01/18 17:29:28 UTC

Review Request 65220: Import: Support for Types with Relationship Attributes

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65220/
-----------------------------------------------------------

Review request for atlas, Madhan Neethiraj and Sarath Subramanian.

Bugs: ATLAS-2372
https://issues.apache.org/jira/browse/ATLAS-2372

Repository: atlas

Description
-------

**Background**
After introduction of _relationshipAttributes_ the import of types with relationship attributes started failing.

**Analysis**
Root cause: The _AtlasEntityGraphDiscoveryV1_ started using _relationshipAttribute_ to discover entities. This caused more entities to be discovered than what was happening before. This caused the single stream entity not to lookup entities correctly. Failures started occuring when these additional entities started getting connected by edges.

Example: _hive_db_ has relationship attribute that now has relationship pointing to _hive_table_. This caused many more entities to be discovered. This also has a side effect of bringing-in way more entities than needed.

With relationshipAttributes, edges started receiving GUIDs. Import process did not handle creating of edges with incoming guids.

**Approach**
- The new _AtlasEntityGraphDiscoveryV1_ entitiy discovery approach using relationships is not being used during import. Additional discussion are needed to guage validity of this approach.
- _EntityMutationContext_ now has additional getter to know of import is in progress.
- Edge creation (new method _AtlasRelationshipStoreV1.createEdgeWithGuid_) has been modified to support edge creation using specific guid.

Diffs
-----

repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java c5e6534c
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityGraphDiscoveryV1.java 450fa875
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1.java 9301cfaa
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 7474a8a4
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityMutationContext.java 8a6a0e3f
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java ab25faaa
repository/src/test/resources/stocks-rel-2.zip PRE-CREATION

Diff: https://reviews.apache.org/r/65220/diff/1/

Testing
-------

**Unit tests**
- Additional tests for testing entities with relationships.

**Functional tests**
- Suite of tests that verify export-import-export and diff comparision for accuracy testing.

Thanks,

Ashutosh Mestry

Re: Review Request 65220: Import: Support for Types with Relationship Attributes

Posted by Sarath Subramanian <sa...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65220/#review195841
-----------------------------------------------------------


Ship it!




Ship It!

- Sarath Subramanian


On Jan. 19, 2018, 9:29 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65220/
> -----------------------------------------------------------
> 
> (Updated Jan. 19, 2018, 9:29 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Sarath Subramanian.
> 
> 
> Bugs: ATLAS-2372
>     https://issues.apache.org/jira/browse/ATLAS-2372
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> After introduction of _relationshipAttributes_ the import of types with relationship attributes started failing.
> 
> **Analysis**
> Root cause: The _AtlasEntityGraphDiscoveryV1_ started using _relationshipAttribute_ to discover entities. This caused more entities to be discovered than what was happening before. This caused the single stream entity not to lookup entities correctly. Failures started occuring when these additional entities started getting connected by edges.
> 
> Example: _hive_db_ has relationship attribute that now has relationship pointing to _hive_table_. This caused many more entities to be discovered. This also has a side effect of bringing-in way more entities than needed.
> 
> With relationshipAttributes, edges started receiving GUIDs. Import process did not handle creating of edges with incoming guids. 
> 
> **Approach**
> - The new _AtlasEntityGraphDiscoveryV1_ entitiy discovery approach using relationships is not being used during import. Additional discussion are needed to guage validity of this approach.
> - _EntityMutationContext_ now has additional getter to know of import is in progress.
> - Edge creation (new method _AtlasRelationshipStoreV1.createEdgeWithGuid_) has been modified to support edge creation using specific guid.
> 
> 
> Diffs
> -----
> 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java c5e6534c 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityGraphDiscoveryV1.java 450fa875 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 7474a8a4 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityMutationContext.java 8a6a0e3f 
>   repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java ab25faaa 
> 
> 
> Diff: https://reviews.apache.org/r/65220/diff/3/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> - Additional tests for testing entities with relationships.
> - (To execute tests locally, place the attached zip at: _./repository/src/test/resources_
> 
> **Functional tests**
> - Suite of tests that verify export-import-export and diff comparision for accuracy testing.
> 
> 
> File Attachments
> ----------------
> 
> Additional resource
>   https://reviews.apache.org/media/uploaded/files/2018/01/18/6edce43a-b1df-415a-98ea-1ab43359a9d7__stocks-rel-2.zip
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 65220: Import: Support for Types with Relationship Attributes

Posted by Ashutosh Mestry <am...@hortonworks.com>.

(Updated Jan. 19, 2018, 5:29 p.m.)

Review request for atlas, Madhan Neethiraj and Sarath Subramanian.

Changes
-------

Updates include: Addressed review comments.

Bugs: ATLAS-2372
https://issues.apache.org/jira/browse/ATLAS-2372

Repository: atlas

Description
-------

**Background**
After introduction of _relationshipAttributes_ the import of types with relationship attributes started failing.

With relationshipAttributes, edges started receiving GUIDs. Import process did not handle creating of edges with incoming guids.

Diffs (updated)
-----

repository/src/main/java/org/apache/atlas/repository/impexp/ExportService.java c5e6534c
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityGraphDiscoveryV1.java 450fa875
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 7474a8a4
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityMutationContext.java 8a6a0e3f
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java ab25faaa

Diff: https://reviews.apache.org/r/65220/diff/3/

Changes: https://reviews.apache.org/r/65220/diff/2-3/

Testing
-------

**Unit tests**
- Additional tests for testing entities with relationships.
- (To execute tests locally, place the attached zip at: _./repository/src/test/resources_

**Functional tests**
- Suite of tests that verify export-import-export and diff comparision for accuracy testing.

File Attachments
----------------

Additional resource
https://reviews.apache.org/media/uploaded/files/2018/01/18/6edce43a-b1df-415a-98ea-1ab43359a9d7__stocks-rel-2.zip

Thanks,

Ashutosh Mestry

Re: Review Request 65220: Import: Support for Types with Relationship Attributes

Posted by Ashutosh Mestry <am...@hortonworks.com>.

(Updated Jan. 18, 2018, 9:38 p.m.)

Review request for atlas, Madhan Neethiraj and Sarath Subramanian.

Changes
-------

Updates include: Removed zip resource. This allows build machine to load the patch and run tests.

Bugs: ATLAS-2372
https://issues.apache.org/jira/browse/ATLAS-2372

Repository: atlas

Description
-------

**Background**
After introduction of _relationshipAttributes_ the import of types with relationship attributes started failing.

With relationshipAttributes, edges started receiving GUIDs. Import process did not handle creating of edges with incoming guids.

Diffs (updated)
-----

Diff: https://reviews.apache.org/r/65220/diff/2/

Changes: https://reviews.apache.org/r/65220/diff/1-2/

Testing
-------

**Unit tests**
- Additional tests for testing entities with relationships.

**Functional tests**
- Suite of tests that verify export-import-export and diff comparision for accuracy testing.

File Attachments
----------------

Additional resource
https://reviews.apache.org/media/uploaded/files/2018/01/18/6edce43a-b1df-415a-98ea-1ab43359a9d7__stocks-rel-2.zip

Thanks,

Ashutosh Mestry

Re: Review Request 65220: Import: Support for Types with Relationship Attributes

Posted by Ashutosh Mestry <am...@hortonworks.com>.

(Updated Jan. 18, 2018, 6:37 p.m.)

Review request for atlas, Madhan Neethiraj and Sarath Subramanian.

Changes
-------

Updates include:
- Additional resource attached in case of difficulty running the patch locally.

Bugs: ATLAS-2372
https://issues.apache.org/jira/browse/ATLAS-2372

Repository: atlas

Description
-------

**Background**
After introduction of _relationshipAttributes_ the import of types with relationship attributes started failing.

With relationshipAttributes, edges started receiving GUIDs. Import process did not handle creating of edges with incoming guids.

Diffs
-----

Diff: https://reviews.apache.org/r/65220/diff/1/

Testing
-------

**Unit tests**
- Additional tests for testing entities with relationships.

**Functional tests**
- Suite of tests that verify export-import-export and diff comparision for accuracy testing.

File Attachments (updated)
----------------

Additional resource
https://reviews.apache.org/media/uploaded/files/2018/01/18/6edce43a-b1df-415a-98ea-1ab43359a9d7__stocks-rel-2.zip

Thanks,

Ashutosh Mestry