You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/05/24 23:33:05 UTC

Review Request 67302: Data Migration: Handle Legacy Types that Reference Classifications

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67302/
-----------------------------------------------------------

Review request for atlas, Madhan Neethiraj and Ruchi Solani.


Bugs: ATLAS-2717
    https://issues.apache.org/jira/browse/ATLAS-2717


Repository: atlas


Description
-------

**Approach**
- Attributes of classification types are pre-processed (before incoming types are processed).
- New _STRUCT_ types are created with the classification name prefixed with _legacy_. So, classification of the name PII is used to create a _STRUCT_ type named _legacy_PII_.
- Entity definitions are updated to reference these newly created types.
- During edge processing, the type name of the in vertex is replaced with this new type.
- Rest of the processing continues as is.


Diffs
-----

  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java e5316d80a 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java b4d6b33ea 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java c9d6067db 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java f51080a61 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesDefScrubber.java PRE-CREATION 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java aee1b698c 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 049fd6418 
  graphdb/janus/src/test/resources/entity-with-trait-type.json PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 081376ff5 
  repository/src/test/java/org/apache/atlas/repository/migration/TypesDefScrubberTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/TypesWithClassificationTest.java PRE-CREATION 
  repository/src/test/resources/classification_defs/atlas-migration-data.json PRE-CREATION 
  repository/src/test/resources/classification_defs/atlas-migration-typesdef.json PRE-CREATION 
  repository/src/test/resources/legacy-typesdef.json PRE-CREATION 


Diff: https://reviews.apache.org/r/67302/diff/1/


Testing
-------

**Unit tests**
- Added tests to verify the new flow.
- Executed existing tests.

**Test data**
- In versions 2.5 onwards (post 2/13/2017) it is not possible to create entities of such types. Entity creation API throws a validation error.
- The test data was thus hand-created.
- This data verifies the cases where attribute type is:
  - Classification type
  - Array of classification type.
  - Map of classification type.


Thanks,

Ashutosh Mestry

Re: Review Request 67302: Data Migration: Handle Legacy Types that Reference Classifications

Posted by Madhan Neethiraj <ma...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67302/#review203840
-----------------------------------------------------------




graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java
Lines 329 (patched)
<https://reviews.apache.org/r/67302/#comment286150>

    If getScrubbedTypesDef() is not dependent on specific implementation of AtlasGraph, consider moving this out of this interface. Perhaps TypesDefScrubber can directly be referenced from caller of this method.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesDefScrubber.java
Lines 39 (patched)
<https://reviews.apache.org/r/67302/#comment286151>

    For better readability, and to be consistent with rest of the code - please organize source files as below:
    
    - private static fields
    - protected static fields
    - public static fields
    - private fields
    - protected fields
    - constructors
    - @Overrides
    - public static methods
    - public methods
    - protected static methods
    - protected methods
    - private static methods
    - private methods
    - inner/static classes


- Madhan Neethiraj


On May 25, 2018, 2:23 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67302/
> -----------------------------------------------------------
> 
> (Updated May 25, 2018, 2:23 a.m.)
> 
> 
> Review request for atlas, Madhan Neethiraj and Ruchi Solani.
> 
> 
> Bugs: ATLAS-2717
>     https://issues.apache.org/jira/browse/ATLAS-2717
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Approach**
> - Attributes of classification types are pre-processed (before incoming types are processed).
> - New _STRUCT_ types are created with the classification name prefixed with _legacy_. So, classification of the name PII is used to create a _STRUCT_ type named _legacy_PII_.
> - Entity definitions are updated to reference these newly created types.
> - During edge processing, the type name of the in vertex is replaced with this new type.
> - Rest of the processing continues as is.
> 
> 
> Diffs
> -----
> 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java e5316d80a 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java b4d6b33ea 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java c9d6067db 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java f51080a61 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesDefScrubber.java PRE-CREATION 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java aee1b698c 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 049fd6418 
>   graphdb/janus/src/test/resources/entity-with-trait-type.json PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 081376ff5 
>   repository/src/test/java/org/apache/atlas/repository/migration/TypesDefScrubberTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/migration/TypesWithClassificationTest.java PRE-CREATION 
>   repository/src/test/resources/classification_defs/atlas-migration-data.json PRE-CREATION 
>   repository/src/test/resources/classification_defs/atlas-migration-typesdef.json PRE-CREATION 
>   repository/src/test/resources/legacy-typesdef.json PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/67302/diff/2/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> - Added tests to verify the new flow.
> - Executed existing tests.
> 
> **Test data**
> - In versions 2.5 onwards (post 2/13/2017) it is not possible to create entities of such types. Entity creation API throws a validation error.
> - The test data was thus hand-created.
> - This data verifies the cases where attribute type is:
>   - Classification type
>   - Array of classification type.
>   - Map of classification type.
> 
> **Build**
> [Pre-commit](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/451/)
> 
> 
> File Attachments
> ----------------
> 
> atlas-migration-typesdef.json
>   https://reviews.apache.org/media/uploaded/files/2018/05/24/c5cc2171-27f5-4d26-acd9-08aac9e1e1ef__atlas-migration-typesdef.json
> atlas-migration-data.json
>   https://reviews.apache.org/media/uploaded/files/2018/05/24/8230c44b-9f2e-4ae2-bc1a-04424dafc08c__atlas-migration-data.json
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>