You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/04/01 01:38:58 UTC
Re: Review Request 66253: Migration: GraphSON-based Import
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------
(Updated April 1, 2018, 1:38 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
Changes
-------
Updates include:
- Addressed review comments.
- Added _startIndex_ to _reader builder_, gives ability to override index from _migrationStatus_.
- Additional log messages.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
pom.xml bfbb9535
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/66253/diff/7/
Changes: https://reviews.apache.org/r/66253/diff/6-7/
Testing
-------
**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.
**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Thanks,
Ashutosh Mestry
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On April 1, 2018, 7:25 a.m., Madhan Neethiraj wrote:
> > graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java
> > Lines 61 (patched)
> > <https://reviews.apache.org/r/66253/diff/7/?file=1990770#file1990770line61>
> >
> > wait upto 120 times the avgCommitTimeSeconds?
There is safety factor. Normally all commits adhere to the regular times. Occasionally I see delays from ZK/Solr where the commit time goes out of wack. This is just to account for those cases.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/#review200270
-----------------------------------------------------------
On April 3, 2018, 7:44 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66253/
> -----------------------------------------------------------
>
> (Updated April 3, 2018, 7:44 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
>
>
> Bugs: ATLAS-2460
> https://issues.apache.org/jira/browse/ATLAS-2460
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> This implementation deals with the 'import into' part of the data migration process.
>
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
>
> **Implementation**
>
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
>
> Here's are the udpates:
> - New configuration parameter has been added:
> _atlas.migration.mode.filename=<name of the file to be imported>_
> This configuration parameter is set by Ambari as part of its migration orchestration.
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _MigrationProgressService_: Added. Get progress of import.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _Services_ modified for special handling of _DataMigrationService_.
>
>
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
>
> **Migration Status**
> The above URL in migration mode yields JSON like:
> ```
> {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
> common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
> common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
> intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
> pom.xml bfbb9535
> repository/pom.xml b1d6b1f9
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
> repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
> repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
> repository/src/test/resources/stocks-2.zip PRE-CREATION
> repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
> repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
> repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
> webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
> webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
>
>
> Diff: https://reviews.apache.org/r/66253/diff/11/
>
>
> Testing
> -------
>
> **Unit tests**
> Unit tests for _AtlasGraphSONReader_ added.
>
> **Functional tests**
> Steps to test file-based import:
> - Place the exported file say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
> _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
>
> Steps to test directory-based import:
> - Place the exported files say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
> _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
>
> **Performance Tests**
> Single threaded average commits: 600,000 per minute.
> Number of threads: 8, average commit: 2 million per minute.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/#review200270
-----------------------------------------------------------
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Lines 233 (patched)
<https://reviews.apache.org/r/66253/#comment280940>
Please review how the following types in Atlas are handled here:
- BigInteger
- BigDecimal
- Date
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java
Lines 61 (patched)
<https://reviews.apache.org/r/66253/#comment280939>
wait upto 120 times the avgCommitTimeSeconds?
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Line 92 (original), 94 (patched)
<https://reviews.apache.org/r/66253/#comment280938>
When in HA mode, is it necessary to inialize typeDefStore here when migration is enabled? Shouldn't it be initialized only in active instance, in AtlasTypeDefStoreInitializer.instanceIsActive()?
Running typeDefStore initialzation from multiple instances can result in incorrect store contents.
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
Lines 90 (patched)
<https://reviews.apache.org/r/66253/#comment280937>
missing "else" before "if"
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java
Lines 94 (patched)
<https://reviews.apache.org/r/66253/#comment280936>
this 'if' block is a duplicate of one above, in #90.
- Madhan Neethiraj
On April 1, 2018, 6:21 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66253/
> -----------------------------------------------------------
>
> (Updated April 1, 2018, 6:21 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
>
>
> Bugs: ATLAS-2460
> https://issues.apache.org/jira/browse/ATLAS-2460
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> This implementation deals with the 'import into' part of the data migration process.
>
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
>
> **Implementation**
>
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
>
> Here's are the udpates:
> - New configuration parameter has been added:
> _atlas.migration.mode.filename=<name of the file to be imported>_
> This configuration parameter is set by Ambari as part of its migration orchestration.
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _MigrationProgressService_: Added. Get progress of import.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _Services_ modified for special handling of _DataMigrationService_.
>
>
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
>
> **Migration Status**
> The above URL in migration mode yields JSON like:
> ```
> {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
> common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
> common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
> intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
> pom.xml bfbb9535
> repository/pom.xml b1d6b1f9
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
> repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
> repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
> repository/src/test/resources/stocks-2.zip PRE-CREATION
> repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
> repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
> repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
> webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
> webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
>
>
> Diff: https://reviews.apache.org/r/66253/diff/8/
>
>
> Testing
> -------
>
> **Unit tests**
> Unit tests for _AtlasGraphSONReader_ added.
>
> **Functional tests**
> Steps to test file-based import:
> - Place the exported file say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
> _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
>
> Steps to test directory-based import:
> - Place the exported files say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
> _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/#review200530
-----------------------------------------------------------
Fix it, then Ship it!
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java
Lines 268 (patched)
<https://reviews.apache.org/r/66253/#comment281267>
How are bigint, bigdecimal and date types handled - in readProperty() and getTypedValueFromJsonNode()?
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java
Line 92 (original), 94 (patched)
<https://reviews.apache.org/r/66253/#comment281268>
When migraiton is enabled, multiple instances can attempt to initialize the typeDef store at the same time. This is not desired. Please review and update.
- Madhan Neethiraj
On April 4, 2018, 10:04 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66253/
> -----------------------------------------------------------
>
> (Updated April 4, 2018, 10:04 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
>
>
> Bugs: ATLAS-2460
> https://issues.apache.org/jira/browse/ATLAS-2460
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> This implementation deals with the 'import into' part of the data migration process.
>
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
>
> **Implementation**
>
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
>
> Here's are the udpates:
> - New configuration parameter has been added:
> _atlas.migration.mode.filename=<name of the file to be imported>_
> This configuration parameter is set by Ambari as part of its migration orchestration.
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _MigrationProgressService_: Added. Get progress of import.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _Services_ modified for special handling of _DataMigrationService_.
>
>
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
>
> **Migration Status**
> The above URL in migration mode yields JSON like:
> ```
> {"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
> common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
> common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
> graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
> graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
> graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
> intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
> intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
> pom.xml bfbb9535
> repository/pom.xml b1d6b1f9
> repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
> repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
> repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionIT.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksIT.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
> repository/src/test/java/org/apache/atlas/utils/TestResourceFileUtils.java 1a01e3a5
> repository/src/test/resources/parts_db/atlas-migration-data.json PRE-CREATION
> repository/src/test/resources/parts_db/atlas-migration-typesdef.json PRE-CREATION
> repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
> repository/src/test/resources/stocks-2.zip PRE-CREATION
> webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
> webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
> webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
>
>
> Diff: https://reviews.apache.org/r/66253/diff/13/
>
>
> Testing
> -------
>
> **Unit tests**
> Unit tests for _AtlasGraphSONReader_ added.
>
> **Functional tests**
> Steps to test file-based import:
> - Place the exported file say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
> _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
>
> Steps to test directory-based import:
> - Place the exported files say _/root/atlas-data_
> - Add to _Atlas_ Ambari's customer property:
> _atlas.migration.mode.filename=/root/atlas-data_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
>
> **Performance Tests**
> Single threaded average commits: 10K per minute (~ 600K per hour).
>
> Configuration:
> - Number of worker threads: 8
> - Batch Size: 3000
> - Number of nodes in cluster: 5, (Solr & Atlas servers on same node)
> Total duration: ~11 hrs (2018-04-03 16:20:52 to 2018-04-04 03:03:12)
> Vertex processing: ~ 6 hrs (2018-04-03 16:20:57 to 2018-04-03 22:30:39)
> - Total vertices: 8.7 million (8700227)
> - Rate: ~ 1.45 million per hour
> Edge processing: ~2 hrs (2018-04-03 22:30:40 to 2018-04-04 00:17:54)
> - Total edges: 17 million (17149967)
> - Rate: 8.5 million per hour
> Post processing: ~ 3 hrs (2018-04-04 00:18:16 to 2018-04-04 03:03:12)
> - Total vertices: 8.7 million.
> - Rate: ~3 million per hour
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------
(Updated April 4, 2018, 10:04 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
Changes
-------
Updates include:
- Additional unit tests.
- Refactoring for unit tests.
- Added exception handling in _PostProcessing_.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
pom.xml bfbb9535
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
repository/src/test/java/org/apache/atlas/repository/migration/HiveParititionIT.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/HiveStocksIT.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/MigrationBaseAsserts.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/java/org/apache/atlas/utils/TestResourceFileUtils.java 1a01e3a5
repository/src/test/resources/parts_db/atlas-migration-data.json PRE-CREATION
repository/src/test/resources/parts_db/atlas-migration-typesdef.json PRE-CREATION
repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
repository/src/test/resources/stocks-2.zip PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/66253/diff/13/
Changes: https://reviews.apache.org/r/66253/diff/12-13/
Testing
-------
**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.
**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
**Performance Tests**
Single threaded average commits: 10K per minute (~ 600K per hour).
Configuration:
- Number of worker threads: 8
- Batch Size: 3000
- Number of nodes in cluster: 5, (Solr & Atlas servers on same node)
Total duration: ~11 hrs (2018-04-03 16:20:52 to 2018-04-04 03:03:12)
Vertex processing: ~ 6 hrs (2018-04-03 16:20:57 to 2018-04-03 22:30:39)
- Total vertices: 8.7 million (8700227)
- Rate: ~ 1.45 million per hour
Edge processing: ~2 hrs (2018-04-03 22:30:40 to 2018-04-04 00:17:54)
- Total edges: 17 million (17149967)
- Rate: 8.5 million per hour
Post processing: ~ 3 hrs (2018-04-04 00:18:16 to 2018-04-04 03:03:12)
- Total vertices: 8.7 million.
- Rate: ~3 million per hour
Thanks,
Ashutosh Mestry
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------
(Updated April 4, 2018, 4:23 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
Changes
-------
Updates include: Metrics from large import.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```
Diffs
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
pom.xml bfbb9535
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/66253/diff/12/
Testing (updated)
-------
**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.
**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
**Performance Tests**
Single threaded average commits: 10K per minute (~ 600K per hour).
Configuration:
- Number of worker threads: 8
- Batch Size: 3000
- Number of nodes in cluster: 5, (Solr & Atlas servers on same node)
Total duration: ~11 hrs (2018-04-03 16:20:52 to 2018-04-04 03:03:12)
Vertex processing: ~ 6 hrs (2018-04-03 16:20:57 to 2018-04-03 22:30:39)
- Total vertices: 8.7 million (8700227)
- Rate: ~ 1.45 million per hour
Edge processing: ~2 hrs (2018-04-03 22:30:40 to 2018-04-04 00:17:54)
- Total edges: 17 million (17149967)
- Rate: 8.5 million per hour
Post processing: ~ 3 hrs (2018-04-04 00:18:16 to 2018-04-04 03:03:12)
- Total vertices: 8.7 million.
- Rate: ~3 million per hour
Thanks,
Ashutosh Mestry
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------
(Updated April 3, 2018, 10:49 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
Changes
-------
Updates include:
- Refactoring to _AtlasGraphSONReader.Builder_.
- Parameterized post process properties list.
- Added _AtlasUserProfile.savedSearches_ to post process.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
pom.xml bfbb9535
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/66253/diff/12/
Changes: https://reviews.apache.org/r/66253/diff/11-12/
Testing
-------
**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.
**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
**Performance Tests**
Single threaded average commits: 600,000 per minute.
Number of threads: 8, average commit: 2 million per minute.
Thanks,
Ashutosh Mestry
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------
(Updated April 3, 2018, 7:44 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
Changes
-------
Updates include:
- Improved accuracy data within of _migrationStatus_ vertex.
- Added _classification_ entity guid to classification vertex.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
pom.xml bfbb9535
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/66253/diff/11/
Changes: https://reviews.apache.org/r/66253/diff/10-11/
Testing
-------
**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.
**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
**Performance Tests**
Single threaded average commits: 600,000 per minute.
Number of threads: 8, average commit: 2 million per minute.
Thanks,
Ashutosh Mestry
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------
(Updated April 3, 2018, 4:58 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
Changes
-------
Updates include:
- Simpler approach to assigning relationship types.
- Updated _RelationshipCache_ implementation.
- Addressed review comments.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
pom.xml bfbb9535
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/66253/diff/10/
Changes: https://reviews.apache.org/r/66253/diff/9-10/
Testing
-------
**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.
**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
**Performance Tests**
Single threaded average commits: 600,000 per minute.
Number of threads: 8, average commit: 2 million per minute.
Thanks,
Ashutosh Mestry
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------
(Updated April 2, 2018, 3:41 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
Changes
-------
Updates include:
- Added performance test numbers.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```
Diffs
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
pom.xml bfbb9535
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/66253/diff/9/
Testing (updated)
-------
**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.
**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
**Performance Tests**
Single threaded average commits: 600,000 per minute.
Number of threads: 8, average commit: 2 million per minute.
Thanks,
Ashutosh Mestry
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------
(Updated April 2, 2018, 3:39 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
Changes
-------
Updates include:
- Refactoring.
- Addressed review comments.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
pom.xml bfbb9535
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/66253/diff/9/
Changes: https://reviews.apache.org/r/66253/diff/8-9/
Testing
-------
**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.
**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Thanks,
Ashutosh Mestry
Re: Review Request 66253: Migration: GraphSON-based Import
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66253/
-----------------------------------------------------------
(Updated April 1, 2018, 6:21 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, Ruchi Solani, Sarath Subramanian, and Vishal Suvagia.
Changes
-------
Updates include:
- Additional refactoring.
- Updated logging.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _MigrationProgressService_: Added. Get progress of import.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
**Migration Status**
The above URL in migration mode yields JSON like:
```
{"Status":"MIGRATION","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1521738357947,"endTime":1521738359272,"currentIndex":48544}}
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/repository/Constants.java 310dddb4
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
graphdb/api/src/main/java/org/apache/atlas/repository/graphdb/AtlasGraph.java 31d20855
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraph.java 6820a93c
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/AtlasJanusGraphDatabase.java a0060200
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/AtlasGraphSONReader.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONTokensTP2.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/GraphSONUtility.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeParsers.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/JsonNodeProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/MappedElementCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/PostProcessManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/ReaderStatusManager.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/RelationshipTypeCache.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemBuilder.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemConsumer.java PRE-CREATION
graphdb/janus/src/main/java/org/apache/atlas/repository/graphdb/janus/migration/pc/WorkItemManager.java PRE-CREATION
graphdb/titan0/src/main/java/org/apache/atlas/repository/graphdb/titan0/Titan0Graph.java 44090097
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/MigrationStatus.java PRE-CREATION
intg/src/main/java/org/apache/atlas/store/AtlasTypeDefStore.java c63dc24a
pom.xml bfbb9535
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/graph/GraphBackedSearchIndexer.java 5672d9dc
repository/src/main/java/org/apache/atlas/repository/impexp/MigrationProgressService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipExportFileNames.java 351b4753
repository/src/main/java/org/apache/atlas/repository/migration/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/migration/RelationshipCacheGenerator.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/store/bootstrap/AtlasTypeDefStoreInitializer.java 66762001
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java 5bec16ed
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasTypeDefGraphStoreV1.java 1a04418a
repository/src/test/java/org/apache/atlas/repository/migration/MigrationServiceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/migration/RelationshipMappingTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2-0.8-extended-tag.json PRE-CREATION
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/web/filters/ActiveServerFilter.java 6681a372
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/security/AtlasSecurityConfig.java f1760e7f
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/66253/diff/8/
Changes: https://reviews.apache.org/r/66253/diff/7-8/
Testing
-------
**Unit tests**
Unit tests for _AtlasGraphSONReader_ added.
**Functional tests**
Steps to test file-based import:
- Place the exported file say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Steps to test directory-based import:
- Place the exported files say _/root/atlas-data_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/atlas-data_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Thanks,
Ashutosh Mestry