You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/02/23 22:38:05 UTC
Review Request 65779: Data Migration: Moving Data to New Version of
Atlas
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/
-----------------------------------------------------------
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _ImportProgressTracker_: Added to be able to query progress of import.
- _ImportService_: Modified to send updates to _ImportProgressTracker_.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _AtlasEntityStoreV1_ modified to block sending out notifications during migration.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
Diffs
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java b97cbb3f
repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java c976c593
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java bf417dd0
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java af310954
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 0ba06b94
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 07524d08
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java f4b70531
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java 8de978c8
webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 86485fba
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java fd96fc3d
Diff: https://reviews.apache.org/r/65779/diff/1/
Testing
-------
**Unit tests**
None added.
**Functional tests**
Steps to test:
- Place the exported file say _export-1.zip_ at _/root_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/export-1.zip_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Thanks,
Ashutosh Mestry
Re: Review Request 65779: Data Migration: Moving Data to New Version
of Atlas
Posted by Ashutosh Mestry <am...@hortonworks.com>.
> On Feb. 24, 2018, 3:06 a.m., Apoorv Naik wrote:
> > common/src/main/java/org/apache/atlas/service/Services.java
> > Line 52 (original), 61 (patched)
> > <https://reviews.apache.org/r/65779/diff/1/?file=1963965#file1963965line62>
> >
> > Wouldn't it be easier to add another state called MIGRATION in ServiceState enum and the Service.start() would be a no-op when Atlas is in migration state.
ServiceState has MIGRATION as new enum. Don't fully understand this comment.
> On Feb. 24, 2018, 3:06 a.m., Apoorv Naik wrote:
> > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
> > Lines 419 (patched)
> > <https://reviews.apache.org/r/65779/diff/1/?file=1963978#file1963978line423>
> >
> > Might be a better idea to send a java object instead of String.
This method is not used. I just forgot to remove it.
- Ashutosh
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/#review198242
-----------------------------------------------------------
On Feb. 24, 2018, 5:08 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65779/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2018, 5:08 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
>
>
> Bugs: ATLAS-2460
> https://issues.apache.org/jira/browse/ATLAS-2460
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> This implementation deals with the 'import into' part of the data migration process.
>
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
>
> **Implementation**
>
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
>
> Here's are the udpates:
> - New configuration parameter has been added:
> _atlas.migration.mode.filename=<name of the file to be imported>_
> This configuration parameter is set by Ambari as part of its migration orchestration.
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _ImportProgressTracker_: Added to be able to query progress of import.
> - _ImportService_: Modified to send updates to _ImportProgressTracker_.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _AtlasEntityStoreV1_ modified to block sending out notifications during migration.
> - _Services_ modified for special handling of _DataMigrationService_.
>
>
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
> common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java b97cbb3f
> repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java c976c593
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java bf417dd0
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java af310954
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 0ba06b94
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 07524d08
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java f4b70531
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java 8de978c8
> webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 86485fba
> webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java fd96fc3d
>
>
> Diff: https://reviews.apache.org/r/65779/diff/2/
>
>
> Testing
> -------
>
> **Unit tests**
> None added.
>
> **Functional tests**
> Steps to test:
> - Place the exported file say _export-1.zip_ at _/root_
> - Add to _Atlas_ Ambari's customer property:
> _atlas.migration.mode.filename=/root/export-1.zip_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 65779: Data Migration: Moving Data to New Version
of Atlas
Posted by Apoorv Naik <na...@gmail.com>.
> On Feb. 24, 2018, 3:06 a.m., Apoorv Naik wrote:
> > common/src/main/java/org/apache/atlas/service/Services.java
> > Line 52 (original), 61 (patched)
> > <https://reviews.apache.org/r/65779/diff/1/?file=1963965#file1963965line62>
> >
> > Wouldn't it be easier to add another state called MIGRATION in ServiceState enum and the Service.start() would be a no-op when Atlas is in migration state.
>
> Ashutosh Mestry wrote:
> ServiceState has MIGRATION as new enum. Don't fully understand this comment.
Every implementation of Service interface should check the current ServiceState and only do stuff when it's an ACTIVE instance and not in migration mode.
eg. KafkaNotification and NotificationHookConsumer start can be no-op during migration, HBaseAuditRepository might be needed to generate audits.
- Apoorv
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/#review198242
-----------------------------------------------------------
On Feb. 24, 2018, 5:08 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65779/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2018, 5:08 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
>
>
> Bugs: ATLAS-2460
> https://issues.apache.org/jira/browse/ATLAS-2460
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> This implementation deals with the 'import into' part of the data migration process.
>
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
>
> **Implementation**
>
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
>
> Here's are the udpates:
> - New configuration parameter has been added:
> _atlas.migration.mode.filename=<name of the file to be imported>_
> This configuration parameter is set by Ambari as part of its migration orchestration.
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _ImportProgressTracker_: Added to be able to query progress of import.
> - _ImportService_: Modified to send updates to _ImportProgressTracker_.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _AtlasEntityStoreV1_ modified to block sending out notifications during migration.
> - _Services_ modified for special handling of _DataMigrationService_.
>
>
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
> common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java b97cbb3f
> repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java c976c593
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java bf417dd0
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java af310954
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 0ba06b94
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 07524d08
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java f4b70531
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java 8de978c8
> webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 86485fba
> webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java fd96fc3d
>
>
> Diff: https://reviews.apache.org/r/65779/diff/2/
>
>
> Testing
> -------
>
> **Unit tests**
> None added.
>
> **Functional tests**
> Steps to test:
> - Place the exported file say _export-1.zip_ at _/root_
> - Add to _Atlas_ Ambari's customer property:
> _atlas.migration.mode.filename=/root/export-1.zip_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 65779: Data Migration: Moving Data to New Version
of Atlas
Posted by Apoorv Naik <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/#review198242
-----------------------------------------------------------
common/src/main/java/org/apache/atlas/service/Services.java
Line 52 (original), 61 (patched)
<https://reviews.apache.org/r/65779/#comment278347>
Wouldn't it be easier to add another state called MIGRATION in ServiceState enum and the Service.start() would be a no-op when Atlas is in migration state.
repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java
Lines 89 (patched)
<https://reviews.apache.org/r/65779/#comment278348>
AtlasException is old class, use AtlasBaseException instead.
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
Lines 53 (patched)
<https://reviews.apache.org/r/65779/#comment278349>
Constructor injection over field injection. Will be easier to test
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
Lines 419 (patched)
<https://reviews.apache.org/r/65779/#comment278351>
Might be a better idea to send a java object instead of String.
- Apoorv Naik
On Feb. 23, 2018, 10:38 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65779/
> -----------------------------------------------------------
>
> (Updated Feb. 23, 2018, 10:38 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
>
>
> Bugs: ATLAS-2460
> https://issues.apache.org/jira/browse/ATLAS-2460
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> This implementation deals with the 'import into' part of the data migration process.
>
> It assumes:
> - Export from older cluster is done.
> - Generated file has been moved to newer cluster.
>
> **Implementation**
>
> During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
>
> Here's are the udpates:
> - New configuration parameter has been added:
> _atlas.migration.mode.filename=<name of the file to be imported>_
> This configuration parameter is set by Ambari as part of its migration orchestration.
> - _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
> - _ImportProgressTracker_: Added to be able to query progress of import.
> - _ImportService_: Modified to send updates to _ImportProgressTracker_.
> - _AdminResource.getStatus()_ Now supplies additional status about migration.
> - _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
> - _AtlasEntityStoreV1_ modified to block sending out notifications during migration.
> - _Services_ modified for special handling of _DataMigrationService_.
>
>
> **CURL**
> Check status using:
> ```
> curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
> ```
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
> common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
> intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java b97cbb3f
> repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java PRE-CREATION
> repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java c976c593
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java bf417dd0
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java af310954
> repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 0ba06b94
> repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 07524d08
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java f4b70531
> repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java 8de978c8
> webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea
> webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 86485fba
> webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
> webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java fd96fc3d
>
>
> Diff: https://reviews.apache.org/r/65779/diff/1/
>
>
> Testing
> -------
>
> **Unit tests**
> None added.
>
> **Functional tests**
> Steps to test:
> - Place the exported file say _export-1.zip_ at _/root_
> - Add to _Atlas_ Ambari's customer property:
> _atlas.migration.mode.filename=/root/export-1.zip_
> - Ambari will prompt for a restart. Restart Atlas.
> - On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
> - Use the CURL call mentioned above and view the status and the progress of the import.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 65779: Data Migration: Moving Data to New Version
of Atlas
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/
-----------------------------------------------------------
(Updated Feb. 28, 2018, 9:20 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include:
- Fixed place where notification sending is blocked.
- Updated location of _log.info_ message in _Services_.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _ImportProgressTracker_: Added to be able to query progress of import.
- _ImportService_: Modified to send updates to _ImportProgressTracker_.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _AtlasEntityStoreV1_ modified to block sending out notifications during migration.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java b97cbb3f
pom.xml 50476262
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/DirectorySource.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java c976c593
repository/src/main/java/org/apache/atlas/repository/impexp/ImportSource.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java ecfbd242
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityChangeNotifier.java b9945d4f
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java b5461d4b
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java af310954
repository/src/test/java/org/apache/atlas/repository/impexp/DirectorySourceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 6fb6b454
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 080a96ac
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java d8e916d7
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/4e13b36b-9c54-4616-9001-1058221165d0.json PRE-CREATION
repository/src/test/resources/stocks-2/5cfc2540-9947-40e0-8905-367e07481774.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/65779/diff/8/
Changes: https://reviews.apache.org/r/65779/diff/7-8/
Testing
-------
**Unit tests**
Unit tests for _DirectorySource_ added.
**Functional tests**
Steps to test:
- Place the exported file say _export-1.zip_ at _/root_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/export-1.zip_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Thanks,
Ashutosh Mestry
Re: Review Request 65779: Data Migration: Moving Data to New Version
of Atlas
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/
-----------------------------------------------------------
(Updated Feb. 27, 2018, 7:57 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include:
- Re-enabled disabled unit tests from _ImportService_.
- Fixed indvertently introduced bug related to relationship mapping.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _ImportProgressTracker_: Added to be able to query progress of import.
- _ImportService_: Modified to send updates to _ImportProgressTracker_.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _AtlasEntityStoreV1_ modified to block sending out notifications during migration.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java b97cbb3f
pom.xml 50476262
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/DirectorySource.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java c976c593
repository/src/main/java/org/apache/atlas/repository/impexp/ImportSource.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java ecfbd242
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java b5461d4b
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java af310954
repository/src/test/java/org/apache/atlas/repository/impexp/DirectorySourceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 6fb6b454
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 080a96ac
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java d8e916d7
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2.zip PRE-CREATION
repository/src/test/resources/stocks-2/4e13b36b-9c54-4616-9001-1058221165d0.json PRE-CREATION
repository/src/test/resources/stocks-2/5cfc2540-9947-40e0-8905-367e07481774.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-info.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-export-order.json PRE-CREATION
repository/src/test/resources/stocks-2/atlas-typesdef.json PRE-CREATION
webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java aab2bb8f
Diff: https://reviews.apache.org/r/65779/diff/6/
Changes: https://reviews.apache.org/r/65779/diff/5-6/
Testing
-------
**Unit tests**
Unit tests for _DirectorySource_ added.
**Functional tests**
Steps to test:
- Place the exported file say _export-1.zip_ at _/root_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/export-1.zip_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Thanks,
Ashutosh Mestry
Re: Review Request 65779: Data Migration: Moving Data to New Version
of Atlas
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/
-----------------------------------------------------------
(Updated Feb. 27, 2018, 6:13 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include:
- New import source added: _DirectorySource_.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _ImportProgressTracker_: Added to be able to query progress of import.
- _ImportService_: Modified to send updates to _ImportProgressTracker_.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _AtlasEntityStoreV1_ modified to block sending out notifications during migration.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java b97cbb3f
repository/pom.xml b1d6b1f9
repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/DirectorySource.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java c976c593
repository/src/main/java/org/apache/atlas/repository/impexp/ImportSource.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java ecfbd242
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java b5461d4b
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java af310954
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 0ba06b94
repository/src/test/java/org/apache/atlas/repository/impexp/DirectorySourceTest.java PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 6fb6b454
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 080a96ac
repository/src/test/java/org/apache/atlas/repository/impexp/ZipFileResourceTestUtils.java d8e916d7
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java 8257faa1
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java ac35860d
repository/src/test/resources/stocks-2.zip PRE-CREATION
webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 1b3f2c86
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
Diff: https://reviews.apache.org/r/65779/diff/3/
Changes: https://reviews.apache.org/r/65779/diff/2-3/
Testing (updated)
-------
**Unit tests**
Unit tests for _DirectorySource_ added.
**Functional tests**
Steps to test:
- Place the exported file say _export-1.zip_ at _/root_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/export-1.zip_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Thanks,
Ashutosh Mestry
Re: Review Request 65779: Data Migration: Moving Data to New Version
of Atlas
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65779/
-----------------------------------------------------------
(Updated Feb. 24, 2018, 5:08 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include:
- Addressed review comments.
Bugs: ATLAS-2460
https://issues.apache.org/jira/browse/ATLAS-2460
Repository: atlas
Description
-------
**Background**
This implementation deals with the 'import into' part of the data migration process.
It assumes:
- Export from older cluster is done.
- Generated file has been moved to newer cluster.
**Implementation**
During _Atlas_ server startup, the configuration parameter is checked, if that parameter exists, all services except _DataMigrationService_ is started. Migration is started. Atlas server is available in _MIGRATION_ mode. It processes REST calls made only to the _AdminResource_.
Here's are the udpates:
- New configuration parameter has been added:
_atlas.migration.mode.filename=<name of the file to be imported>_
This configuration parameter is set by Ambari as part of its migration orchestration.
- _DataMigrationService_: New service that performs async migration as soon as Atlas server starts up.
- _ImportProgressTracker_: Added to be able to query progress of import.
- _ImportService_: Modified to send updates to _ImportProgressTracker_.
- _AdminResource.getStatus()_ Now supplies additional status about migration.
- _ServiceState_ Modified to carry additional status _MIGRATION_. This status is set by looking at the configuration parameter above.
- _AtlasEntityStoreV1_ modified to block sending out notifications during migration.
- _Services_ modified for special handling of _DataMigrationService_.
**CURL**
Check status using:
```
curl -X GET -u admin:admin -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://localhost:21000/api/atlas/admin/status
```
Diffs (updated)
-----
common/src/main/java/org/apache/atlas/AtlasConstants.java f5de1df3
common/src/main/java/org/apache/atlas/service/Services.java 1267dc92
intg/src/main/java/org/apache/atlas/model/impexp/AtlasExportResult.java 1ea961d8
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportResult.java b97cbb3f
repository/src/main/java/org/apache/atlas/repository/impexp/DataMigrationService.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportProgressTracker.java PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java c976c593
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1.java bf417dd0
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/BulkImporterImpl.java af310954
repository/src/main/java/org/apache/atlas/repository/store/graph/v1/EntityGraphMapper.java 0ba06b94
repository/src/test/java/org/apache/atlas/repository/impexp/ExportServiceTest.java 07524d08
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasEntityStoreV1Test.java f4b70531
repository/src/test/java/org/apache/atlas/repository/store/graph/v1/AtlasRelationshipStoreV1Test.java 8de978c8
webapp/src/main/java/org/apache/atlas/Atlas.java 7cf6e3ea
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 86485fba
webapp/src/main/java/org/apache/atlas/web/service/ServiceState.java 3fe8d18c
webapp/src/test/java/org/apache/atlas/web/resources/AdminResourceTest.java fd96fc3d
Diff: https://reviews.apache.org/r/65779/diff/2/
Changes: https://reviews.apache.org/r/65779/diff/1-2/
Testing
-------
**Unit tests**
None added.
**Functional tests**
Steps to test:
- Place the exported file say _export-1.zip_ at _/root_
- Add to _Atlas_ Ambari's customer property:
_atlas.migration.mode.filename=/root/export-1.zip_
- Ambari will prompt for a restart. Restart Atlas.
- On the server view the progress in the logs using: _tail -f /var/log/atlas/application.log_
- Use the CURL call mentioned above and view the status and the progress of the import.
Thanks,
Ashutosh Mestry