You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/05/03 04:18:47 UTC
Review Request 66928: Data Migration: Import: Infer Types that Store
Edge Ids
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
Review request for atlas, Madhan Neethiraj and Ruchi Solani.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Diffs
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 86cd29900
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java d191b5549
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/66928/diff/1/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203643
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On May 23, 2018, 1:30 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> -----------------------------------------------------------
>
> (Updated May 23, 2018, 1:30 a.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
>
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/AtlasConstants.java 2b92e6ed6
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java e4b6ee2e9
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java cca72ad11
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipCacheGenerator.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesWithCollectionsFinder.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsersTest.java 170b1af60
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
> graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
> graphdb/janus/src/test/resources/col-3-legacy.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-process.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-tag.json PRE-CREATION
> graphdb/janus/src/test/resources/lineage-v-98312.json PRE-CREATION
> graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
> graphdb/janus/src/test/resources/tag-163856752.json PRE-CREATION
> intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
> intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
> repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java 2550dafc1
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
> repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
> repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java 48e539100
> repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
> repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
> repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
> repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
>
>
> Diff: https://reviews.apache.org/r/66928/diff/21/
>
>
> Testing
> -------
>
> **Unit tests**
>
> Additional tests added.
>
> **Functional tests**
>
> Regular flow verified.
>
> **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/435/)**
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 23, 2018, 1:30 a.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include:
- Addressed exception 'IllegalStateException' during edge creation.
- Fixed bug: Where lineage did not have correct propagate direction.
- Clean up of unit tests.
- Minor updates to unit tests.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java 2b92e6ed6
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java e4b6ee2e9
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java cca72ad11
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipCacheGenerator.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesWithCollectionsFinder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsersTest.java 170b1af60
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/col-3-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-process.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-tag.json PRE-CREATION
graphdb/janus/src/test/resources/lineage-v-98312.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
graphdb/janus/src/test/resources/tag-163856752.json PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java 2550dafc1
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java 48e539100
repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
Diff: https://reviews.apache.org/r/66928/diff/21/
Changes: https://reviews.apache.org/r/66928/diff/20-21/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/435/)**
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 22, 2018, 10:03 p.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include:
- Default propagation value now uses the relationship value.
- Unit test refactoring.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java 2b92e6ed6
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ElementProcessors.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java cca72ad11
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipCacheGenerator.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesWithCollectionsFinder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsersTest.java 170b1af60
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/col-3-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-process.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-tag.json PRE-CREATION
graphdb/janus/src/test/resources/lineage-v-98312.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
graphdb/janus/src/test/resources/tag-163856752.json PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java 2550dafc1
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java 48e539100
repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
Diff: https://reviews.apache.org/r/66928/diff/20/
Changes: https://reviews.apache.org/r/66928/diff/19-20/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/423/)**
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 21, 2018, 11:49 p.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include:
- Default propagation set to ONE_TO_TWO for lineages edges.
- Refactoring to use _AtlasTypeRegistry_ instead of complex maps.
- Refactoring to move logic to _RelationshipTypeCache_ from _GraphSONUtility_.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java 2b92e6ed6
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java cca72ad11
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/TypesWithCollectionsFinder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/col-3-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-Process.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-tag.json PRE-CREATION
graphdb/janus/src/test/resources/lineage-v-98312.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
graphdb/janus/src/test/resources/tag-163856752.json PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java 2550dafc1
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java 48e539100
repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
Diff: https://reviews.apache.org/r/66928/diff/19/
Changes: https://reviews.apache.org/r/66928/diff/18-19/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/423/)**
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 21, 2018, 2:15 a.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include:
- Optimization for post processing.
- Refactoring of _RelationshipTypeCache_.
- Addressed review comments.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
Diff: https://reviews.apache.org/r/66928/diff/18/
Changes: https://reviews.apache.org/r/66928/diff/17-18/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/423/)**
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 20, 2018, 4:30 p.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include:
- Addressed review comments.
- Updated pre-commit build #.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
Diff: https://reviews.apache.org/r/66928/diff/17/
Changes: https://reviews.apache.org/r/66928/diff/16-17/
Testing (updated)
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/423/)**
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On May 19, 2018, 12:36 a.m., Madhan Neethiraj wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
> > Line 67 (original), 74 (patched)
> > <https://reviews.apache.org/r/66928/diff/16/?file=2025013#file2025013line74>
> >
> > This one handles removal/updte of properties for map type attributes. Aren't similar updates needed for array type attributes as well (primitive/struct/entity-ref)?
Yes, only that the indexes are known only after all edges are created. I considered doing this during creation itself, it eded up becoming messy and number of computations creases drastically. There is an option that is possible if it is some how possible to store a temporary map of edge ids to indexes.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203462
-----------------------------------------------------------
On May 17, 2018, 8:48 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> -----------------------------------------------------------
>
> (Updated May 17, 2018, 8:48 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
>
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
>
>
> Diffs
> -----
>
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
> graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
> graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
> intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
> intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
> repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
> repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
> repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
> repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
> repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
> repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
> repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
>
>
> Diff: https://reviews.apache.org/r/66928/diff/16/
>
>
> Testing
> -------
>
> **Unit tests**
>
> Additional tests added.
>
> **Functional tests**
>
> Regular flow verified.
>
> **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/416/)**
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On May 19, 2018, 12:36 a.m., Madhan Neethiraj wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
> > Lines 152 (patched)
> > <https://reviews.apache.org/r/66928/diff/16/?file=2025013#file2025013line152>
> >
> > edges for all array elements will have __index set to 0. Does this get updated in subsequent processing?
Yes. This is just initialized here.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203462
-----------------------------------------------------------
On May 17, 2018, 8:48 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> -----------------------------------------------------------
>
> (Updated May 17, 2018, 8:48 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
>
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
>
>
> Diffs
> -----
>
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
> graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
> graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
> intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
> intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
> repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
> repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
> repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
> repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
> repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
> repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
> repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
>
>
> Diff: https://reviews.apache.org/r/66928/diff/16/
>
>
> Testing
> -------
>
> **Unit tests**
>
> Additional tests added.
>
> **Functional tests**
>
> Regular flow verified.
>
> **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/416/)**
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203462
-----------------------------------------------------------
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java
Line 315 (original), 332 (patched)
<https://reviews.apache.org/r/66928/#comment285691>
shouldn't this return 'this.propertiesToPostProcess', instead of creating a new instance? If this method is used only to initialize, either rename it or avoid this method by directly initializing in line #252.
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Line 67 (original), 74 (patched)
<https://reviews.apache.org/r/66928/#comment285696>
This one handles removal/updte of properties for map type attributes. Aren't similar updates needed for array type attributes as well (primitive/struct/entity-ref)?
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Lines 152 (patched)
<https://reviews.apache.org/r/66928/#comment285693>
edges for all array elements will have __index set to 0. Does this get updated in subsequent processing?
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Lines 340 (patched)
<https://reviews.apache.org/r/66928/#comment285694>
Method replaceReferencedEdgeIdForList() and postProcessListProperty don't seem to be used. Please review and remove.
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java
Lines 43 (patched)
<https://reviews.apache.org/r/66928/#comment285703>
Should newIdPositionMap be cleared here?
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
Lines 80 (patched)
<https://reviews.apache.org/r/66928/#comment285688>
doesNotHaveProperty() - the name doesn't seem to match the condition "return v.property(Constants.TYPENAME_PROPERTY_KEY).isPresent()", which will return true for all entity/classification vertices. Please review.
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java
Lines 84 (patched)
<https://reviews.apache.org/r/66928/#comment285689>
isVertexOfType() ==> isInstanceVertexOfType()
- Madhan Neethiraj
On May 17, 2018, 8:48 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> -----------------------------------------------------------
>
> (Updated May 17, 2018, 8:48 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
>
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
>
>
> Diffs
> -----
>
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
> graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
> graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
> intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
> intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
> repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
> repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
> repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
> repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
> repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
> repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
> repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
>
>
> Diff: https://reviews.apache.org/r/66928/diff/16/
>
>
> Testing
> -------
>
> **Unit tests**
>
> Additional tests added.
>
> **Functional tests**
>
> Regular flow verified.
>
> **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/416/)**
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203388
-----------------------------------------------------------
Ship it!
Ship It!
- Sarath Subramanian
On May 17, 2018, 1:48 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> -----------------------------------------------------------
>
> (Updated May 17, 2018, 1:48 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
>
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
>
>
> Diffs
> -----
>
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
> graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
> graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
> intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
> intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
> repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
> repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
> repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
> repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
> repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
> repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
> repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
>
>
> Diff: https://reviews.apache.org/r/66928/diff/16/
>
>
> Testing
> -------
>
> **Unit tests**
>
> Additional tests added.
>
> **Functional tests**
>
> Regular flow verified.
>
> **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/416/)**
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 17, 2018, 8:48 p.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include: Addressed review comments.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
Diff: https://reviews.apache.org/r/66928/diff/15/
Changes: https://reviews.apache.org/r/66928/diff/14-15/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/416/)**
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Sarath Subramanian <sa...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/#review203360
-----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
Lines 66 (patched)
<https://reviews.apache.org/r/66928/#comment285509>
getProperties() method in 66 can be replaced with one in line 82. AtlasStructDef is supertype of AtlasEntityDef.
repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
Lines 109 (patched)
<https://reviews.apache.org/r/66928/#comment285517>
rename 109 and 113 with getArrayElementType and getMapValueType for better readability.
repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java
Lines 117 (patched)
<https://reviews.apache.org/r/66928/#comment285524>
update() and updatePrimitive() looks duplicate of each other with only "_PRIMTITIVE" string difference, can be refactored into a single method.
- Sarath Subramanian
On May 15, 2018, 10:58 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66928/
> -----------------------------------------------------------
>
> (Updated May 15, 2018, 10:58 p.m.)
>
>
> Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
>
>
> Bugs: ATLAS-2637
> https://issues.apache.org/jira/browse/ATLAS-2637
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
> Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
> Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
> Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
>
> Added PostProcess framework.
> Added logic for handling new Array and Map representation.
>
>
> Diffs
> -----
>
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
> graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
> graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col3.json PRE-CREATION
> graphdb/janus/src/test/resources/edge-legacy-col4.json PRE-CREATION
> graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
> intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
> intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
> repository/src/main/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinder.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
> repository/src/test/java/org/apache/atlas/repository/migration/ComplexAttributesTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionTest.java ac0b79d38
> repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksTest.java ffbf3200b
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
> repository/src/test/java/org/apache/atlas/repository/migration/TypesWithCollectionsFinderTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityTestBase.java d810a1166
> repository/src/test/resources/complex-attr_db/atlas-migration-data.json PRE-CREATION
> repository/src/test/resources/complex-attr_db/atlas-migration-typesdef.json PRE-CREATION
> repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
>
>
> Diff: https://reviews.apache.org/r/66928/diff/14/
>
>
> Testing
> -------
>
> **Unit tests**
>
> Additional tests added.
>
> **Functional tests**
>
> Regular flow verified.
>
> **[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/416/)**
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 16, 2018, 5:58 a.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include:
- Removed commented code block from _GraphBackedIndexer_.
- Updated logic for _TypeStoringEdgeIds_.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java ae119b0bc
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java e863d9fae
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION
repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
Diff: https://reviews.apache.org/r/66928/diff/10/
Changes: https://reviews.apache.org/r/66928/diff/9-10/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/404/)**
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 15, 2018, 4:18 a.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include:
- Handling primitive and non-primitive Map.
- Handling non-primitive Arrays.
- Refactoring.
- Fixed failing UTs.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 16aecd5e2
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java de43e0f41
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
intg/src/test/java/org/apache/atlas/TestUtilsV2.java 886ce77f5
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java f289365f5
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION
repository/src/test/resources/parts_db/atlas-migration-data.json 1414ea160
Diff: https://reviews.apache.org/r/66928/diff/9/
Changes: https://reviews.apache.org/r/66928/diff/8-9/
Testing (updated)
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/404/)**
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 11, 2018, 10:27 p.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include:
- Refactoring.
- Minor modification to post processing.
- Fix to new attribute assignment.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 3636ca65a
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java e4e82649b
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java de43e0f41
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityTest.java 794b5471e
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/66928/diff/8/
Changes: https://reviews.apache.org/r/66928/diff/7-8/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
**[Pre-commit build](https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/382/)**
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 11, 2018, 1:50 a.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include: Fixes based on testing.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 3636ca65a
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java de43e0f41
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCacheTest.java cac09d229
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/66928/diff/7/
Changes: https://reviews.apache.org/r/66928/diff/6-7/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 11, 2018, 12:40 a.m.)
Review request for atlas, Madhan Neethiraj, Ruchi Solani, and Sarath Subramanian.
Changes
-------
Updates include: Array and Map type handling.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description (updated)
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Added PostProcess framework.
Added logic for handling new Array and Map representation.
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 3636ca65a
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessListProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessMapProperty.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/postProcess/PostProcessor.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/BaseUtils.java de43e0f41
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessListPropertyTest.java PRE-CREATION
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessMapPropertyTest.java PRE-CREATION
graphdb/janus/src/test/resources/col-2-legacy.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col.json PRE-CREATION
graphdb/janus/src/test/resources/edge-legacy-col2.json PRE-CREATION
graphdb/janus/src/test/resources/table-v-147504.json 898dce5df
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/66928/diff/5/
Changes: https://reviews.apache.org/r/66928/diff/4-5/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
Thanks,
Ashutosh Mestry
Re: Review Request 66928: Data Migration: Import: Infer Types that
Store Edge Ids
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66928/
-----------------------------------------------------------
(Updated May 3, 2018, 5:39 a.m.)
Review request for atlas, Madhan Neethiraj and Ruchi Solani.
Changes
-------
Updates include:
- Refactored _AtlasGraphSONReader.Builder.setDefaults_.
Bugs: ATLAS-2637
https://issues.apache.org/jira/browse/ATLAS-2637
Repository: atlas
Description
-------
**Approach**
New class: _TypesStoringEdgeIds_: Navigates through all the types in the _typeRegistry_, returns map of entity type and properties that store edge ids.
Modified: _DataMigrationService_: Uses output from class above and passes it down to migration.
Modified: _PostProcessManager_: Uses the map generated above and uses it for post processing.
Modified: _GraphSONUtility_: Improvement to check for vertex of type. This avoids potential exeception when a non-existent property is checked for presence.
Diffs (updated)
-----
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 607baf664
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java c0b9c1741
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java 86cd29900
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java aa0b6375d
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java ec320b03e
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java d0a65f7b1
graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtilityPostProcessTest.java 4d73c78ef
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java d191b5549
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java b05754f4b
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java 22cd55217
repository/src/main/java/org/apache/atlas/repository/migration/TypesStoringEdgeIds.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 3c84e3c22
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java ec6e64a25
repository/src/test/java/org/apache/atlas/repository/migration/TypesStoringEdgeIdsTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/66928/diff/2/
Changes: https://reviews.apache.org/r/66928/diff/1-2/
Testing
-------
**Unit tests**
Additional tests added.
**Functional tests**
Regular flow verified.
Thanks,
Ashutosh Mestry