You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/03/23 19:59:03 UTC

Review Request 66253: Migration: GraphSON-based Import

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs
-----

  addons/models/0000-Area0/0010-base_model.json aebe955d 
  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml ee2746a9 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/ogm/DTORegistry.java 818960d9 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java b5461d4b 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/1/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.


> On March 31, 2018, 7:39 p.m., Madhan Neethiraj wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java
> > Lines 57 (patched)
> > <https://reviews.apache.org/r/66253/diff/6/?file=1990665#file1990665line57>
> >
> >     avgDuration is in seconds; the wait is in minutes - intentional? Please review.

This is used to give additional time than just what is available.


> On March 31, 2018, 7:39 p.m., Madhan Neethiraj wrote:
> > repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java
> > Lines 21 (patched)
> > <https://reviews.apache.org/r/66253/diff/6/?file=1990675#file1990675line21>
> >
> >     I would suggest to avoid adding a new dependency - unless it is critical.

I couldn't find another implementation of expiring cache. We can certaily look at a different implementation.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/#review200259
-----------------------------------------------------------


On March 31, 2018, 2:18 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66253/
> -----------------------------------------------------------
> 
> (Updated March 31, 2018, 2:18 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
> 
> 
> Bugs: ATLAS-2460
>     https://issues.apache.org/jira/browse/ATLAS-2460
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> This implementation deals with the 'import into' part of the data migration process. 
> 
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
> 
> **Implementation**
> 
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
> 
> Here's are the udpates:
> - New configuration parameter has been added:
>     _atlas.migration.mode.filename=<name of the file to be imported>_
>   This configuration parameter is set by Ambari as part of its migration orchestration. 
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _MigrationProgressService_: Added. Get progress of import.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _Services_ modified for special handling of _DataMigrationService_.
> 
> 
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
> 
> **Migration Status**
> The above URL in migration mode yields JSON like:
> ```
> {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
>   common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeManagerTest.java PRE-CREATION 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManagerTest.java PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
>   intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
>   pom.xml bfbb9535 
>   repository/pom.xml b1d6b1f9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
>   repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
>   repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
>   repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
>   repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
>   repository/src/test/resources/stocks-2.zip PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
>   webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 
> 
> 
> Diff: https://reviews.apache.org/r/66253/diff/6/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Unit tests for _AtlasGraphSONReader_ added.
> 
> **Functional tests**
> Steps to test file-based import:
> - Place the exported file say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> Steps to test directory-based import:
> - Place the exported files say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Madhan Neethiraj <ma...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/#review200259
-----------------------------------------------------------




graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java
Lines 107 (patched)
<https://reviews.apache.org/r/66253/#comment280933>

    These constants are defined in too many places. Please consolidate to a single defintion.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java
Lines 156 (patched)
<https://reviews.apache.org/r/66253/#comment280932>

    These constants are defined in too many places. Please consolidate to a single defintion.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java
Lines 44 (patched)
<https://reviews.apache.org/r/66253/#comment280931>

    For better readability, please add comments on what 'rGraph' and 'bGraph' represent.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java
Lines 38 (patched)
<https://reviews.apache.org/r/66253/#comment280930>

    what does 'bGraph' mean?



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java
Lines 92 (patched)
<https://reviews.apache.org/r/66253/#comment280929>

    in line #84, END_TIME is set to Date. Here it is set to a long value. Please review and update.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java
Lines 56 (patched)
<https://reviews.apache.org/r/66253/#comment280928>

    relationship can be between:
     - inputTypeName and a superType of outputTypeName
     - outputTypeName and a superType of inputTypeName
    
    So, getSuperType() should be getTypeAndSuperTypes(). Please review.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java
Lines 85 (patched)
<https://reviews.apache.org/r/66253/#comment280926>

    It seems 'commitDuration' tracks the maxCommitTime taken by this consumer. In that case, please consider renaming.



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java
Lines 57 (patched)
<https://reviews.apache.org/r/66253/#comment280927>

    avgDuration is in seconds; the wait is in minutes - intentional? Please review.



graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeManagerTest.java
Lines 20 (patched)
<https://reviews.apache.org/r/66253/#comment280925>

    If this empty class is not used/needed, please remove.



graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManagerTest.java
Lines 28 (patched)
<https://reviews.apache.org/r/66253/#comment280924>

    If this empty class is not used/needed, please remove.



repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
Lines 289 (patched)
<https://reviews.apache.org/r/66253/#comment280922>

    VERTEX_ORIGINAL_ID_KEY => VERTEX_ID_IN_IMPORT_KEY



repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java
Lines 298 (patched)
<https://reviews.apache.org/r/66253/#comment280923>

    EDGE_ORIGINAL_ID_KEY => EDGE_ID_IN_IMPORT_KEY



repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java
Lines 21 (patched)
<https://reviews.apache.org/r/66253/#comment280920>

    I would suggest to avoid adding a new dependency - unless it is critical.



repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java
Lines 98 (patched)
<https://reviews.apache.org/r/66253/#comment280921>

    Consider using:
    
    GraphHelper.getSingleValuedProperty(vertex, START_TIME_PROPERTY, Date.class)



repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java
Lines 37 (patched)
<https://reviews.apache.org/r/66253/#comment280919>

    this seems to assume that there can be only one relationship type between a given 2 entity-types. This is not true. For example, between hive_table and hive_column types, following relationship types exist:
     - hive_table_columns
     - hive_table_partitionkeys
    
    Please review.



repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Lines 95 (patched)
<https://reviews.apache.org/r/66253/#comment280917>

    When migration is enabled, all instances of Atlas will enter this block. This may not be desirable. Please review.



repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Lines 104 (patched)
<https://reviews.apache.org/r/66253/#comment280918>

    Why eat all exceptions here?



repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java
Lines 71 (patched)
<https://reviews.apache.org/r/66253/#comment280916>

    migrationInProgress doesn't seem to be used. Please review and remove.



repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java
Lines 66 (patched)
<https://reviews.apache.org/r/66253/#comment280914>

    It is not clear what this assert looks for?
     - 'key' is formed by the test in getKey() method
     - this assert is validating that getKey() returns the right value?



repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java
Lines 86 (patched)
<https://reviews.apache.org/r/66253/#comment280913>

    'configuration' doesn't seem to be used in this class. If this is not needed, please remove.



webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java
Lines 51 (patched)
<https://reviews.apache.org/r/66253/#comment280912>

    MIGRATION => MIGRATING


- Madhan Neethiraj


On March 31, 2018, 2:18 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66253/
> -----------------------------------------------------------
> 
> (Updated March 31, 2018, 2:18 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
> 
> 
> Bugs: ATLAS-2460
>     https://issues.apache.org/jira/browse/ATLAS-2460
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> This implementation deals with the 'import into' part of the data migration process. 
> 
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
> 
> **Implementation**
> 
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
> 
> Here's are the udpates:
> - New configuration parameter has been added:
>     _atlas.migration.mode.filename=<name of the file to be imported>_
>   This configuration parameter is set by Ambari as part of its migration orchestration. 
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _MigrationProgressService_: Added. Get progress of import.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _Services_ modified for special handling of _DataMigrationService_.
> 
> 
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
> 
> **Migration Status**
> The above URL in migration mode yields JSON like:
> ```
> {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
>   common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeManagerTest.java PRE-CREATION 
>   graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManagerTest.java PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
>   intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
>   pom.xml bfbb9535 
>   repository/pom.xml b1d6b1f9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
>   repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
>   repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
>   repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
>   repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
>   repository/src/test/resources/stocks-2.zip PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
>   webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 
> 
> 
> Diff: https://reviews.apache.org/r/66253/diff/6/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Unit tests for _AtlasGraphSONReader_ added.
> 
> **Functional tests**
> Steps to test file-based import:
> - Place the exported file say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> Steps to test directory-based import:
> - Place the exported files say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.


> On April 1, 2018, 7:25 a.m., Madhan Neethiraj wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java
> > Lines 61 (patched)
> > <https://reviews.apache.org/r/66253/diff/7/?file=1990770#file1990770line61>
> >
> >     wait upto 120 times the avgCommitTimeSeconds?

There is safety factor. Normally all commits adhere to the regular times. Occasionally I see delays from ZK/Solr where the commit time goes out of wack. This is just to account for those cases.


- Ashutosh


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/#review200270
-----------------------------------------------------------


On April 3, 2018, 7:44 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66253/
> -----------------------------------------------------------
> 
> (Updated April 3, 2018, 7:44 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
> 
> 
> Bugs: ATLAS-2460
>     https://issues.apache.org/jira/browse/ATLAS-2460
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> This implementation deals with the 'import into' part of the data migration process. 
> 
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
> 
> **Implementation**
> 
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
> 
> Here's are the udpates:
> - New configuration parameter has been added:
>     _atlas.migration.mode.filename=<name of the file to be imported>_
>   This configuration parameter is set by Ambari as part of its migration orchestration. 
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _MigrationProgressService_: Added. Get progress of import.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _Services_ modified for special handling of _DataMigrationService_.
> 
> 
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
> 
> **Migration Status**
> The above URL in migration mode yields JSON like:
> ```
> {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
>   common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
>   intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
>   pom.xml bfbb9535 
>   repository/pom.xml b1d6b1f9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
>   repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
>   repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
>   repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
>   repository/src/test/resources/stocks-2.zip PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
>   webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 
> 
> 
> Diff: https://reviews.apache.org/r/66253/diff/11/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Unit tests for _AtlasGraphSONReader_ added.
> 
> **Functional tests**
> Steps to test file-based import:
> - Place the exported file say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> Steps to test directory-based import:
> - Place the exported files say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> **Performance Tests**
> Single threaded average commits: 600,000 per minute.
> Number of threads: 8, average commit: 2 million per minute.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Madhan Neethiraj <ma...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/#review200270
-----------------------------------------------------------




graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Lines 233 (patched)
<https://reviews.apache.org/r/66253/#comment280940>

    Please review how the following types in Atlas are handled here:
     - BigInteger
     - BigDecimal
     - Date



graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java
Lines 61 (patched)
<https://reviews.apache.org/r/66253/#comment280939>

    wait upto 120 times the avgCommitTimeSeconds?



repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Line 92 (original), 94 (patched)
<https://reviews.apache.org/r/66253/#comment280938>

    When in HA mode, is it necessary to inialize typeDefStore here when migration is enabled? Shouldn't it be initialized only in active instance, in AtlasTypeDefStoreInitializer.instanceIsActive()?
    
    Running typeDefStore initialzation from multiple instances can result in incorrect store contents.



webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
Lines 90 (patched)
<https://reviews.apache.org/r/66253/#comment280937>

    missing "else" before "if"



webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
Lines 94 (patched)
<https://reviews.apache.org/r/66253/#comment280936>

    this 'if' block is a duplicate of one above, in #90.


- Madhan Neethiraj


On April 1, 2018, 6:21 a.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66253/
> -----------------------------------------------------------
> 
> (Updated April 1, 2018, 6:21 a.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
> 
> 
> Bugs: ATLAS-2460
>     https://issues.apache.org/jira/browse/ATLAS-2460
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> This implementation deals with the 'import into' part of the data migration process. 
> 
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
> 
> **Implementation**
> 
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
> 
> Here's are the udpates:
> - New configuration parameter has been added:
>     _atlas.migration.mode.filename=<name of the file to be imported>_
>   This configuration parameter is set by Ambari as part of its migration orchestration. 
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _MigrationProgressService_: Added. Get progress of import.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _Services_ modified for special handling of _DataMigrationService_.
> 
> 
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
> 
> **Migration Status**
> The above URL in migration mode yields JSON like:
> ```
> {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
>   common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
>   intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
>   pom.xml bfbb9535 
>   repository/pom.xml b1d6b1f9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
>   repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
>   repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
>   repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
>   repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
>   repository/src/test/resources/stocks-2.zip PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
>   repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
>   webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 
> 
> 
> Diff: https://reviews.apache.org/r/66253/diff/8/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Unit tests for _AtlasGraphSONReader_ added.
> 
> **Functional tests**
> Steps to test file-based import:
> - Place the exported file say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> Steps to test directory-based import:
> - Place the exported files say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Madhan Neethiraj <ma...@apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/#review200530
-----------------------------------------------------------


Fix it, then Ship it!





graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Lines 268 (patched)
<https://reviews.apache.org/r/66253/#comment281267>

    How are bigint, bigdecimal and date types handled - in readProperty() and getTypedValueFromJsonNode()?



repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Line 92 (original), 94 (patched)
<https://reviews.apache.org/r/66253/#comment281268>

    When migraiton is enabled, multiple instances can attempt to initialize the typeDef store at the same time. This is not desired. Please review and update.


- Madhan Neethiraj


On April 4, 2018, 10:04 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66253/
> -----------------------------------------------------------
> 
> (Updated April 4, 2018, 10:04 p.m.)
> 
> 
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
> 
> 
> Bugs: ATLAS-2460
>     https://issues.apache.org/jira/browse/ATLAS-2460
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Background**
> This implementation deals with the 'import into' part of the data migration process. 
> 
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
> 
> **Implementation**
> 
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
> 
> Here's are the udpates:
> - New configuration parameter has been added:
>     _atlas.migration.mode.filename=<name of the file to be imported>_
>   This configuration parameter is set by Ambari as part of its migration orchestration. 
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _MigrationProgressService_: Added. Get progress of import.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _Services_ modified for special handling of _DataMigrationService_.
> 
> 
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
> 
> **Migration Status**
> The above URL in migration mode yields JSON like:
> ```
> {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
> ```
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
>   common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
>   common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
>   graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
>   graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
>   graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
>   intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
>   intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
>   intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
>   pom.xml bfbb9535 
>   repository/pom.xml b1d6b1f9 
>   repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
>   repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
>   repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
>   repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
>   repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionIT.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksIT.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
>   repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
>   repository/src/test/java/org/apache/atlas/utils/TestResourceFileUtils.java 1a01e3a5 
>   repository/src/test/resources/parts_db/atlas-migration-data.json PRE-CREATION 
>   repository/src/test/resources/parts_db/atlas-migration-typesdef.json PRE-CREATION 
>   repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
>   repository/src/test/resources/stocks-2.zip PRE-CREATION 
>   webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
>   webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
>   webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
>   webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
>   webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 
> 
> 
> Diff: https://reviews.apache.org/r/66253/diff/13/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Unit tests for _AtlasGraphSONReader_ added.
> 
> **Functional tests**
> Steps to test file-based import:
> - Place the exported file say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> Steps to test directory-based import:
> - Place the exported files say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
>     _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
> 
> **Performance Tests**
> Single threaded average commits: 10K per minute (~ 600K per hour).
> 
> Configuration:
>   - Number of worker threads: 8
>   - Batch Size: 3000
>   - Number of nodes in cluster: 5, (Solr & Atlas servers on same node)
> Total duration: ~11 hrs (2018-04-03 16:20:52 to 2018-04-04 03:03:12)
> Vertex processing: ~ 6 hrs (2018-04-03 16:20:57 to 2018-04-03 22:30:39)
>   - Total vertices: 8.7 million (8700227)
>   - Rate: ~ 1.45 million per hour
> Edge processing: ~2 hrs (2018-04-03 22:30:40 to 2018-04-04 00:17:54)
>   - Total edges: 17 million (17149967)
>   - Rate: 8.5 million per hour
> Post processing:  ~ 3 hrs (2018-04-04 00:18:16 to 2018-04-04 03:03:12)
>   - Total vertices: 8.7 million.
>   - Rate: ~3 million per hour
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated April 4, 2018, 10:04 p.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include:
- Additional unit tests.
- Refactoring for unit tests.
- Added exception handling in _PostProcessing_.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionIT.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksIT.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/java/org/apache/atlas/utils/TestResourceFileUtils.java 1a01e3a5 
  repository/src/test/resources/parts_db/atlas-migration-data.json PRE-CREATION 
  repository/src/test/resources/parts_db/atlas-migration-typesdef.json PRE-CREATION 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/13/

Changes: https://reviews.apache.org/r/66253/diff/12-13/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

**Performance Tests**
Single threaded average commits: 10K per minute (~ 600K per hour).

Configuration:
  - Number of worker threads: 8
  - Batch Size: 3000
  - Number of nodes in cluster: 5, (Solr & Atlas servers on same node)
Total duration: ~11 hrs (2018-04-03 16:20:52 to 2018-04-04 03:03:12)
Vertex processing: ~ 6 hrs (2018-04-03 16:20:57 to 2018-04-03 22:30:39)
  - Total vertices: 8.7 million (8700227)
  - Rate: ~ 1.45 million per hour
Edge processing: ~2 hrs (2018-04-03 22:30:40 to 2018-04-04 00:17:54)
  - Total edges: 17 million (17149967)
  - Rate: 8.5 million per hour
Post processing:  ~ 3 hrs (2018-04-04 00:18:16 to 2018-04-04 03:03:12)
  - Total vertices: 8.7 million.
  - Rate: ~3 million per hour


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated April 4, 2018, 4:23 a.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include: Metrics from large import.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/12/


Testing (updated)
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

**Performance Tests**
Single threaded average commits: 10K per minute (~ 600K per hour).

Configuration:
  - Number of worker threads: 8
  - Batch Size: 3000
  - Number of nodes in cluster: 5, (Solr & Atlas servers on same node)
Total duration: ~11 hrs (2018-04-03 16:20:52 to 2018-04-04 03:03:12)
Vertex processing: ~ 6 hrs (2018-04-03 16:20:57 to 2018-04-03 22:30:39)
  - Total vertices: 8.7 million (8700227)
  - Rate: ~ 1.45 million per hour
Edge processing: ~2 hrs (2018-04-03 22:30:40 to 2018-04-04 00:17:54)
  - Total edges: 17 million (17149967)
  - Rate: 8.5 million per hour
Post processing:  ~ 3 hrs (2018-04-04 00:18:16 to 2018-04-04 03:03:12)
  - Total vertices: 8.7 million.
  - Rate: ~3 million per hour


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated April 3, 2018, 10:49 p.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include:
- Refactoring to _AtlasGraphSONReader.Builder_.
- Parameterized post process properties list.
- Added _AtlasUserProfile.savedSearches_ to post process.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/12/

Changes: https://reviews.apache.org/r/66253/diff/11-12/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

**Performance Tests**
Single threaded average commits: 600,000 per minute.
Number of threads: 8, average commit: 2 million per minute.


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated April 3, 2018, 7:44 p.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include:
- Improved accuracy data within of _migrationStatus_ vertex. 
- Added _classification_ entity guid to classification vertex.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/11/

Changes: https://reviews.apache.org/r/66253/diff/10-11/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

**Performance Tests**
Single threaded average commits: 600,000 per minute.
Number of threads: 8, average commit: 2 million per minute.


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated April 3, 2018, 4:58 a.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include:
- Simpler approach to assigning relationship types.
- Updated _RelationshipCache_ implementation.
- Addressed review comments.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/10/

Changes: https://reviews.apache.org/r/66253/diff/9-10/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

**Performance Tests**
Single threaded average commits: 600,000 per minute.
Number of threads: 8, average commit: 2 million per minute.


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated April 2, 2018, 3:41 a.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include: 
- Added performance test numbers.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/9/


Testing (updated)
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

**Performance Tests**
Single threaded average commits: 600,000 per minute.
Number of threads: 8, average commit: 2 million per minute.


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated April 2, 2018, 3:39 a.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include: 
- Refactoring.
- Addressed review comments.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/9/

Changes: https://reviews.apache.org/r/66253/diff/8-9/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated April 1, 2018, 6:21 a.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include:
- Additional refactoring.
- Updated logging.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/8/

Changes: https://reviews.apache.org/r/66253/diff/7-8/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated April 1, 2018, 1:38 a.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include:
- Addressed review comments.
- Added _startIndex_ to _reader builder_, gives ability to override index from _migrationStatus_.
- Additional log messages.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/7/

Changes: https://reviews.apache.org/r/66253/diff/6-7/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated March 31, 2018, 2:18 a.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include:
- Scalable migration import.
- Significant refactoring.
- Updated status reporting using _status_ REST end point.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs (updated)
-----

  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeManagerTest.java PRE-CREATION 
  graphdb/janus/src/test/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManagerTest.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml bfbb9535 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/6/

Changes: https://reviews.apache.org/r/66253/diff/5-6/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.


Thanks,

Ashutosh Mestry

Re: Review Request 66253: Migration: GraphSON-based Import

Posted by Ashutosh Mestry <am...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------

(Updated March 23, 2018, 9:21 p.m.)


Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.


Changes
-------

Updates include: Minor refactoring.


Bugs: ATLAS-2460
    https://issues.apache.org/jira/browse/ATLAS-2460


Repository: atlas


Description
-------

**Background**
This implementation deals with the 'import into' part of the data migration process. 

It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.

**Implementation**

During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.

Here's are the udpates:
- New configuration parameter has been added:
    _atlas.migration.mode.filename=<name of the file to be imported>_
  This configuration parameter is set by Ambari as part of its migration orchestration. 
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.


**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```

**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```


Diffs (updated)
-----

  addons/models/0000-Area0/0010-base_model.json aebe955d 
  common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3 
  common/src/main/java/org/apache/atlas/service/Services.java 1267dc92 
  graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION 
  graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION 
  graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097 
  intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8 
  intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION 
  intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a 
  pom.xml ee2746a9 
  repository/pom.xml b1d6b1f9 
  repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc 
  repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753 
  repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION 
  repository/src/main/java/org/apache/atlas/repository/ogm/DTORegistry.java 818960d9 
  repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java b5461d4b 
  repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a 
  repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1 
  repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d 
  repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION 
  repository/src/test/resources/stocks-2.zip PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION 
  repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION 
  webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372 
  webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86 
  webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f 
  webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c 
  webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f 


Diff: https://reviews.apache.org/r/66253/diff/2/

Changes: https://reviews.apache.org/r/66253/diff/1-2/


Testing
-------

**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.

**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.

Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
    _atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.


Thanks,

Ashutosh Mestry