You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/02/24 00:18:47 UTC
Review Request 65790: Data Migration Utility: Export
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/
-----------------------------------------------------------
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Bugs: ATLAS-2461
https://issues.apache.org/jira/browse/ATLAS-2461
Repository: atlas
Description
-------
**Background**
The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
**Approach**
The migration is a new utility that will perform export of data using a pre-defined export request file.
The approach used by this application:
- Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
- Create _ImportService_ using Spring's framework classes.
Here are the pieces:
- _MigrationApp_: Contains _main_. It is the entry point for the app.
- _Interactors_: Contains plumbing needed to use the new _migrationContext.xml_ and create _applicationContext_ for use.
- Dengenerated classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application.
- _atlas-migration-setup.sh_ Installs the JAR at _/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib_
- _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
- _export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
- _atlas-migration-shutdown.sh_ Removes the installed migration app and restarts Atlas.
**Build**
The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
_mvn install package_ generates a ZIP file in _target_ directory.
**Usage**
- Copy the ZIP file to the server location.
- Use _unzip atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip_
- Contents will be unzipped to _atlas-migration-kit-0.8.3-SNAPSHOT_ directory.
- cd _atlas-migration-kit-0.8.3-SNAPSHOT_
- _bash ./atlas-migration-setup.sh_
- _python /usr/hdp/current/atlas-server/bin/atlas_migration.py export export-2.zip_
- Watch the progress using _tail -f /var/log/atlas/application.log_
- When done use: _bash ./atlas-migration-shutdown.sh_
- Move the generated ZIP file to the new clusters server location.
Diffs
-----
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 655252cb
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java a428b92e
tools/atlas-migration-utility/pom.xml PRE-CREATION
tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotification.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotificationChangeListener.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Interactors.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/MigrationApp.java PRE-CREATION
tools/atlas-migration-utility/src/main/resources/README PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas-migration-setup.sh PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas-migration-shutdown.sh PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION
tools/atlas-migration-utility/src/main/resources/export-request.json PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION
Diff: https://reviews.apache.org/r/65790/diff/1/
Testing
-------
**Unit tests**
None.
**Functional tests**
Exports from existing clusters.
Thanks,
Ashutosh Mestry
Re: Review Request 65790: Data Migration Utility: Export
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/#review198585
-----------------------------------------------------------
- update pom.xml to enable build of this utility from the root
- this utility deals with only export; consider naming the utilty something like 'migration-export'.
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotification.java
Lines 29 (patched)
<https://reviews.apache.org/r/65790/#comment278789>
EmptyNotification ==> NoOpNotification
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotificationChangeListener.java
Lines 31 (patched)
<https://reviews.apache.org/r/65790/#comment278790>
EmptyNotificationChangeListener ==> NoOpNotificationChangeListener
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/MigrationApp.java
Lines 23 (patched)
<https://reviews.apache.org/r/65790/#comment278791>
If this tool only handles export, consider folding "MigrationApp" into "Exporter" i.e. move the main() method to Exporter.
tools/atlas-migration-utility/src/main/resources/export-request.json
Lines 1 (patched)
<https://reviews.apache.org/r/65790/#comment278792>
contents of "export-request.json" and "migration-export-request.json" are the same. Consider removing one of these files.
- Madhan Neethiraj
On March 1, 2018, 11:09 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65790/
> -----------------------------------------------------------
>
> (Updated March 1, 2018, 11:09 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
>
>
> Bugs: ATLAS-2461
> https://issues.apache.org/jira/browse/ATLAS-2461
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
>
> **Approach**
> The migration is a new utility that will perform export of data using a pre-defined export request file.
>
> The approach used by this application:
> - Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
> - Create _ImportService_ using Spring's framework classes.
>
> Here are the pieces:
> - _MigrationApp_: Contains _main_. It is the entry point for the app.
> - _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ and create _ApplicationContext_ for use.
> - Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application in a way that no notifications are sent out.
> - _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
> - _migration-export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
> - _README_: Instructions on usage.
>
> **Build**
> The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
>
> _mvn install package_ generates a ZIP file in _target_ directory.
>
> **Usage (from README)**
> - Use Ambari to turn shutdown Atlas.
> - Unzip contents of the ZIP to a directory on the server using: 'unzip atlas-migration-kit.zip'
> - cd atlas-migration-kit
> - Run 'python atlas_migration.py'. Use 'python atlas_migration.py <path-file.zip>' to export to a specific file.
> - To watch the progress: 'tail -f /var/log/atlas/application.log'.
>
>
> Diffs
> -----
>
> notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 655252cb
> repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java a428b92e
> tools/atlas-migration-utility/pom.xml PRE-CREATION
> tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotification.java PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotificationChangeListener.java PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/MigrationApp.java PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/README PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/export-request.json PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/migration-export-request.json PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/65790/diff/3/
>
>
> Testing
> -------
>
> **Unit tests**
> None.
>
> **Functional tests**
> Exports from existing clusters.
>
>
> File Attachments
> ----------------
>
> Migration Utility
> https://reviews.apache.org/media/uploaded/files/2018/02/24/e8090ed0-13b6-4253-a59c-d3f2098943af__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 65790: Data Migration Utility: Export
Posted by Apoorv Naik <na...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/#review198595
-----------------------------------------------------------
tools/atlas-migration-utility/pom.xml
Lines 41 (patched)
<https://reviews.apache.org/r/65790/#comment278809>
Remove duplicate dependencies. Since we're inherting all versions from parent, the version override is not necessary here.
tools/atlas-migration-utility/pom.xml
Lines 65 (patched)
<https://reviews.apache.org/r/65790/#comment278810>
Typesystem is removed from 1.0.0-SNAPSHOT so this seems incorrect.
- Apoorv Naik
On March 4, 2018, 5:25 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65790/
> -----------------------------------------------------------
>
> (Updated March 4, 2018, 5:25 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
>
>
> Bugs: ATLAS-2461
> https://issues.apache.org/jira/browse/ATLAS-2461
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
>
> **Approach**
> The migration is a new utility that will perform export of data using a pre-defined export request file.
>
> The approach used by this application:
> - Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
> - Create _ImportService_ using Spring's framework classes.
>
> Here are the pieces:
> - _MigrationApp_: Contains _main_. It is the entry point for the app.
> - _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ and create _ApplicationContext_ for use.
> - Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application in a way that no notifications are sent out.
> - _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
> - _migration-export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
> - _README_: Instructions on usage.
>
> **Build**
> The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
>
> _mvn install package_ generates a ZIP file in _target_ directory.
>
> **Usage (from README)**
> - Use Ambari to turn shutdown Atlas.
> - Unzip contents of the ZIP to a directory on the server using: 'unzip atlas-migration-kit.zip'
> - cd atlas-migration-kit
> - Run 'python atlas_migration.py'. Use 'python atlas_migration.py <path-file.zip>' to export to a specific file.
> - To watch the progress: 'tail -f /var/log/atlas/application.log'.
>
>
> Diffs
> -----
>
> pom.xml 7db1be78
> tools/atlas-migration-utility/pom.xml PRE-CREATION
> tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotification.java PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotificationChangeListener.java PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/README PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/migration-export-request.json PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/65790/diff/4/
>
>
> Testing
> -------
>
> **Unit tests**
> None.
>
> **Functional tests**
> Exports from existing clusters.
>
>
> File Attachments
> ----------------
>
> Migration Utility
> https://reviews.apache.org/media/uploaded/files/2018/02/24/e8090ed0-13b6-4253-a59c-d3f2098943af__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 65790: Data Migration Utility: Export
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/#review198859
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On March 7, 2018, 6:34 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65790/
> -----------------------------------------------------------
>
> (Updated March 7, 2018, 6:34 p.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
>
>
> Bugs: ATLAS-2461
> https://issues.apache.org/jira/browse/ATLAS-2461
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
>
> **Approach**
> The migration is a new utility that will perform export of data using a pre-defined export request file.
>
> The approach used by this application:
> - Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
> - Create _ImportService_ using Spring's framework classes.
>
> Here are the pieces:
> - _MigrationApp_: Contains _main_. It is the entry point for the app.
> - _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ and create _ApplicationContext_ for use.
> - Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application in a way that no notifications are sent out.
> - _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
> - _migration-export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
> - _README_: Instructions on usage.
>
> **Build**
> The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
>
> _mvn install package_ generates a ZIP file in _target_ directory.
>
> **Usage (from README)**
> - Use Ambari to turn shutdown Atlas.
> - Unzip contents of the ZIP to a directory on the server using: 'unzip atlas-migration-kit.zip'
> - cd atlas-migration-kit
> - Run 'python atlas_migration.py'. Use 'python atlas_migration.py <path-file.zip>' to export to a specific file.
> - To watch the progress: 'tail -f /var/log/atlas/application.log'.
>
>
> Diffs
> -----
>
> distro/pom.xml 68b967d8
> distro/src/main/assemblies/migration-exporter.xml PRE-CREATION
> pom.xml 7db1be78
> tools/atlas-migration-exporter/pom.xml PRE-CREATION
> tools/atlas-migration-exporter/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION
> tools/atlas-migration-exporter/src/main/java/org/apache/atlas/migration/NoOpNotification.java PRE-CREATION
> tools/atlas-migration-exporter/src/main/java/org/apache/atlas/migration/NoOpNotificationChangeListener.java PRE-CREATION
> tools/atlas-migration-exporter/src/main/resources/README PRE-CREATION
> tools/atlas-migration-exporter/src/main/resources/atlas_migration.py PRE-CREATION
> tools/atlas-migration-exporter/src/main/resources/migration-export-request.json PRE-CREATION
> tools/atlas-migration-exporter/src/main/resources/migrationContext.xml PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/65790/diff/6/
>
>
> Testing
> -------
>
> **Unit tests**
> None.
>
> **Functional tests**
> Exports from existing clusters.
>
>
> File Attachments
> ----------------
>
> apache-atlas-0.8.3-SNAPSHOT-migration-exporter.zip
> https://reviews.apache.org/media/uploaded/files/2018/03/07/177943b6-df32-4296-99e7-76e00dc783dd__apache-atlas-0.8.3-SNAPSHOT-migration-exporter.zip
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 65790: Data Migration Utility: Export
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/
-----------------------------------------------------------
(Updated March 7, 2018, 6:34 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include:
- Included ZIP file creation to _dist_ profile.
- Moved assembly profile to _distro_ directory.
- Addressed review comments.
- Renamed to _atlas-migration-exporter_ since it is apt with its function.
Bugs: ATLAS-2461
https://issues.apache.org/jira/browse/ATLAS-2461
Repository: atlas
Description
-------
**Background**
The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
**Approach**
The migration is a new utility that will perform export of data using a pre-defined export request file.
The approach used by this application:
- Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
- Create _ImportService_ using Spring's framework classes.
Here are the pieces:
- _MigrationApp_: Contains _main_. It is the entry point for the app.
- _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ and create _ApplicationContext_ for use.
- Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application in a way that no notifications are sent out.
- _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
- _migration-export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
- _README_: Instructions on usage.
**Build**
The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
_mvn install package_ generates a ZIP file in _target_ directory.
**Usage (from README)**
- Use Ambari to turn shutdown Atlas.
- Unzip contents of the ZIP to a directory on the server using: 'unzip atlas-migration-kit.zip'
- cd atlas-migration-kit
- Run 'python atlas_migration.py'. Use 'python atlas_migration.py <path-file.zip>' to export to a specific file.
- To watch the progress: 'tail -f /var/log/atlas/application.log'.
Diffs (updated)
-----
distro/pom.xml 68b967d8
distro/src/main/assemblies/migration-exporter.xml PRE-CREATION
pom.xml 7db1be78
tools/atlas-migration-exporter/pom.xml PRE-CREATION
tools/atlas-migration-exporter/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION
tools/atlas-migration-exporter/src/main/java/org/apache/atlas/migration/NoOpNotification.java PRE-CREATION
tools/atlas-migration-exporter/src/main/java/org/apache/atlas/migration/NoOpNotificationChangeListener.java PRE-CREATION
tools/atlas-migration-exporter/src/main/resources/README PRE-CREATION
tools/atlas-migration-exporter/src/main/resources/atlas_migration.py PRE-CREATION
tools/atlas-migration-exporter/src/main/resources/migration-export-request.json PRE-CREATION
tools/atlas-migration-exporter/src/main/resources/migrationContext.xml PRE-CREATION
Diff: https://reviews.apache.org/r/65790/diff/6/
Changes: https://reviews.apache.org/r/65790/diff/5-6/
Testing
-------
**Unit tests**
None.
**Functional tests**
Exports from existing clusters.
File Attachments
----------------
Migration Utility
https://reviews.apache.org/media/uploaded/files/2018/03/05/4d42eda3-cba8-441d-8505-12271cffc569__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
Thanks,
Ashutosh Mestry
Re: Review Request 65790: Data Migration Utility: Export
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/#review198672
-----------------------------------------------------------
Fix it, then Ship it!
tools/atlas-migration-utility/pom.xml
Lines 59 (patched)
<https://reviews.apache.org/r/65790/#comment278915>
'atlas-repository' is included twice.
tools/atlas-migration-utility/pom.xml
Lines 67 (patched)
<https://reviews.apache.org/r/65790/#comment278921>
Assembly is handled for other modules from distro/pom.xml. Consider moving this (and tools/atlas-migration-utility/src/assembly/bin.xml) into distro module.
- Madhan Neethiraj
On March 5, 2018, 5:39 a.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65790/
> -----------------------------------------------------------
>
> (Updated March 5, 2018, 5:39 a.m.)
>
>
> Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
>
>
> Bugs: ATLAS-2461
> https://issues.apache.org/jira/browse/ATLAS-2461
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Background**
> The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
>
> **Approach**
> The migration is a new utility that will perform export of data using a pre-defined export request file.
>
> The approach used by this application:
> - Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
> - Create _ImportService_ using Spring's framework classes.
>
> Here are the pieces:
> - _MigrationApp_: Contains _main_. It is the entry point for the app.
> - _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ and create _ApplicationContext_ for use.
> - Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application in a way that no notifications are sent out.
> - _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
> - _migration-export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
> - _README_: Instructions on usage.
>
> **Build**
> The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
>
> _mvn install package_ generates a ZIP file in _target_ directory.
>
> **Usage (from README)**
> - Use Ambari to turn shutdown Atlas.
> - Unzip contents of the ZIP to a directory on the server using: 'unzip atlas-migration-kit.zip'
> - cd atlas-migration-kit
> - Run 'python atlas_migration.py'. Use 'python atlas_migration.py <path-file.zip>' to export to a specific file.
> - To watch the progress: 'tail -f /var/log/atlas/application.log'.
>
>
> Diffs
> -----
>
> pom.xml 7db1be78
> tools/atlas-migration-utility/pom.xml PRE-CREATION
> tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotification.java PRE-CREATION
> tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotificationChangeListener.java PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/README PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/migration-export-request.json PRE-CREATION
> tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION
>
>
> Diff: https://reviews.apache.org/r/65790/diff/5/
>
>
> Testing
> -------
>
> **Unit tests**
> None.
>
> **Functional tests**
> Exports from existing clusters.
>
>
> File Attachments
> ----------------
>
> Migration Utility
> https://reviews.apache.org/media/uploaded/files/2018/03/05/4d42eda3-cba8-441d-8505-12271cffc569__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 65790: Data Migration Utility: Export
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/
-----------------------------------------------------------
(Updated March 5, 2018, 5:39 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include:
- Addressed review comments.
- Attached latest migration utility.
Bugs: ATLAS-2461
https://issues.apache.org/jira/browse/ATLAS-2461
Repository: atlas
Description
-------
**Background**
The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
**Approach**
The migration is a new utility that will perform export of data using a pre-defined export request file.
The approach used by this application:
- Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
- Create _ImportService_ using Spring's framework classes.
Here are the pieces:
- _MigrationApp_: Contains _main_. It is the entry point for the app.
- _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ and create _ApplicationContext_ for use.
- Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application in a way that no notifications are sent out.
- _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
- _migration-export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
- _README_: Instructions on usage.
**Build**
The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
_mvn install package_ generates a ZIP file in _target_ directory.
**Usage (from README)**
- Use Ambari to turn shutdown Atlas.
- Unzip contents of the ZIP to a directory on the server using: 'unzip atlas-migration-kit.zip'
- cd atlas-migration-kit
- Run 'python atlas_migration.py'. Use 'python atlas_migration.py <path-file.zip>' to export to a specific file.
- To watch the progress: 'tail -f /var/log/atlas/application.log'.
Diffs (updated)
-----
pom.xml 7db1be78
tools/atlas-migration-utility/pom.xml PRE-CREATION
tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotification.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotificationChangeListener.java PRE-CREATION
tools/atlas-migration-utility/src/main/resources/README PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migration-export-request.json PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION
Diff: https://reviews.apache.org/r/65790/diff/5/
Changes: https://reviews.apache.org/r/65790/diff/4-5/
Testing
-------
**Unit tests**
None.
**Functional tests**
Exports from existing clusters.
File Attachments (updated)
----------------
Migration Utility
https://reviews.apache.org/media/uploaded/files/2018/03/05/4d42eda3-cba8-441d-8505-12271cffc569__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
Thanks,
Ashutosh Mestry
Re: Review Request 65790: Data Migration Utility: Export
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/
-----------------------------------------------------------
(Updated March 4, 2018, 5:25 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include:
- Addressed review comments.
- Added entry to main _pom.xml_ to be included as part of build.
Bugs: ATLAS-2461
https://issues.apache.org/jira/browse/ATLAS-2461
Repository: atlas
Description
-------
**Background**
The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
**Approach**
The migration is a new utility that will perform export of data using a pre-defined export request file.
The approach used by this application:
- Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
- Create _ImportService_ using Spring's framework classes.
Here are the pieces:
- _MigrationApp_: Contains _main_. It is the entry point for the app.
- _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ and create _ApplicationContext_ for use.
- Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application in a way that no notifications are sent out.
- _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
- _migration-export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
- _README_: Instructions on usage.
**Build**
The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
_mvn install package_ generates a ZIP file in _target_ directory.
**Usage (from README)**
- Use Ambari to turn shutdown Atlas.
- Unzip contents of the ZIP to a directory on the server using: 'unzip atlas-migration-kit.zip'
- cd atlas-migration-kit
- Run 'python atlas_migration.py'. Use 'python atlas_migration.py <path-file.zip>' to export to a specific file.
- To watch the progress: 'tail -f /var/log/atlas/application.log'.
Diffs (updated)
-----
pom.xml 7db1be78
tools/atlas-migration-utility/pom.xml PRE-CREATION
tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotification.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/NoOpNotificationChangeListener.java PRE-CREATION
tools/atlas-migration-utility/src/main/resources/README PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migration-export-request.json PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION
Diff: https://reviews.apache.org/r/65790/diff/4/
Changes: https://reviews.apache.org/r/65790/diff/3-4/
Testing
-------
**Unit tests**
None.
**Functional tests**
Exports from existing clusters.
File Attachments
----------------
Migration Utility
https://reviews.apache.org/media/uploaded/files/2018/02/24/e8090ed0-13b6-4253-a59c-d3f2098943af__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
Thanks,
Ashutosh Mestry
Re: Review Request 65790: Data Migration Utility: Export
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/
-----------------------------------------------------------
(Updated March 1, 2018, 11:09 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include: Minor refactoring.
Bugs: ATLAS-2461
https://issues.apache.org/jira/browse/ATLAS-2461
Repository: atlas
Description
-------
**Background**
The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
**Approach**
The migration is a new utility that will perform export of data using a pre-defined export request file.
The approach used by this application:
- Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
- Create _ImportService_ using Spring's framework classes.
Here are the pieces:
- _MigrationApp_: Contains _main_. It is the entry point for the app.
- _Exporter_: Contains plumbing needed to use the new _migrationContext.xml_ and create _ApplicationContext_ for use.
- Dengenerate classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application in a way that no notifications are sent out.
- _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
- _migration-export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
- _README_: Instructions on usage.
**Build**
The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
_mvn install package_ generates a ZIP file in _target_ directory.
**Usage (from README)**
- Use Ambari to turn shutdown Atlas.
- Unzip contents of the ZIP to a directory on the server using: 'unzip atlas-migration-kit.zip'
- cd atlas-migration-kit
- Run 'python atlas_migration.py'. Use 'python atlas_migration.py <path-file.zip>' to export to a specific file.
- To watch the progress: 'tail -f /var/log/atlas/application.log'.
Diffs (updated)
-----
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 655252cb
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java a428b92e
tools/atlas-migration-utility/pom.xml PRE-CREATION
tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotification.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotificationChangeListener.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/MigrationApp.java PRE-CREATION
tools/atlas-migration-utility/src/main/resources/README PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION
tools/atlas-migration-utility/src/main/resources/export-request.json PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migration-export-request.json PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION
Diff: https://reviews.apache.org/r/65790/diff/3/
Changes: https://reviews.apache.org/r/65790/diff/2-3/
Testing
-------
**Unit tests**
None.
**Functional tests**
Exports from existing clusters.
File Attachments
----------------
Migration Utility
https://reviews.apache.org/media/uploaded/files/2018/02/24/e8090ed0-13b6-4253-a59c-d3f2098943af__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
Thanks,
Ashutosh Mestry
Re: Review Request 65790: Data Migration Utility: Export
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/
-----------------------------------------------------------
(Updated March 1, 2018, 10:51 p.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include:
- Removed bash scripts for setup and shutdown.
- Simplified flow.
- Refactoring.
- Additional logging.
- Updated _atlas_migration.py_ for more diagnostics.
Bugs: ATLAS-2461
https://issues.apache.org/jira/browse/ATLAS-2461
Repository: atlas
Description
-------
**Background**
The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
**Approach**
The migration is a new utility that will perform export of data using a pre-defined export request file.
The approach used by this application:
- Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
- Create _ImportService_ using Spring's framework classes.
Here are the pieces:
- _MigrationApp_: Contains _main_. It is the entry point for the app.
- _Interactors_: Contains plumbing needed to use the new _migrationContext.xml_ and create _applicationContext_ for use.
- Dengenerated classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application.
- _atlas-migration-setup.sh_ Installs the JAR at _/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib_
- _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
- _export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
- _atlas-migration-shutdown.sh_ Removes the installed migration app and restarts Atlas.
**Build**
The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
_mvn install package_ generates a ZIP file in _target_ directory.
**Usage**
- Copy the ZIP file to the server location.
- Use _unzip atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip_
- Contents will be unzipped to _atlas-migration-kit-0.8.3-SNAPSHOT_ directory.
- cd _atlas-migration-kit-0.8.3-SNAPSHOT_
- _bash ./atlas-migration-setup.sh_
- _python /usr/hdp/current/atlas-server/bin/atlas_migration.py export export-2.zip_
- Watch the progress using _tail -f /var/log/atlas/application.log_
- When done use: _bash ./atlas-migration-shutdown.sh_
- Move the generated ZIP file to the new clusters server location.
Diffs (updated)
-----
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 655252cb
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java a428b92e
tools/atlas-migration-utility/pom.xml PRE-CREATION
tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotification.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotificationChangeListener.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Exporter.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Interactors.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/MigrationApp.java PRE-CREATION
tools/atlas-migration-utility/src/main/resources/README PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION
tools/atlas-migration-utility/src/main/resources/export-request.json PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migration-export-request.json PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION
Diff: https://reviews.apache.org/r/65790/diff/2/
Changes: https://reviews.apache.org/r/65790/diff/1-2/
Testing
-------
**Unit tests**
None.
**Functional tests**
Exports from existing clusters.
File Attachments
----------------
Migration Utility
https://reviews.apache.org/media/uploaded/files/2018/02/24/e8090ed0-13b6-4253-a59c-d3f2098943af__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
Thanks,
Ashutosh Mestry
Re: Review Request 65790: Data Migration Utility: Export
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65790/
-----------------------------------------------------------
(Updated Feb. 24, 2018, 12:20 a.m.)
Review request for atlas, Apoorv Naik, Madhan Neethiraj, and Ruchi Solani.
Changes
-------
Updates include: Migration utility zip.
Bugs: ATLAS-2461
https://issues.apache.org/jira/browse/ATLAS-2461
Repository: atlas
Description
-------
**Background**
The data migration utility allows for exporting data from an Atlas instance without the server running. This is important as it will prevent Atlas from processing any requests.
**Approach**
The migration is a new utility that will perform export of data using a pre-defined export request file.
The approach used by this application:
- Create an application context (_migrationContext.xml_). This context, prevents instantiation of number of classes, most notably _webapp_, _notifications_, _listeners_.
- Create _ImportService_ using Spring's framework classes.
Here are the pieces:
- _MigrationApp_: Contains _main_. It is the entry point for the app.
- _Interactors_: Contains plumbing needed to use the new _migrationContext.xml_ and create _applicationContext_ for use.
- Dengenerated classes _EmptyNotification_, _EmptyNotificationChangeListener_. These are necessary to launch the application.
- _atlas-migration-setup.sh_ Installs the JAR at _/usr/hdp/current/atlas-server/server/webapp/atlas/WEB-INF/lib_
- _atlas_migration.py_ Creates environment for the app to execute. It launches the application.
- _export-request.json_ This can be modified for environments that do not have latest improvements to _Export_.
- _atlas-migration-shutdown.sh_ Removes the installed migration app and restarts Atlas.
**Build**
The project maven's assembly building plugin. The build will create a ZIP file with the necessary files.
_mvn install package_ generates a ZIP file in _target_ directory.
**Usage**
- Copy the ZIP file to the server location.
- Use _unzip atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip_
- Contents will be unzipped to _atlas-migration-kit-0.8.3-SNAPSHOT_ directory.
- cd _atlas-migration-kit-0.8.3-SNAPSHOT_
- _bash ./atlas-migration-setup.sh_
- _python /usr/hdp/current/atlas-server/bin/atlas_migration.py export export-2.zip_
- Watch the progress using _tail -f /var/log/atlas/application.log_
- When done use: _bash ./atlas-migration-shutdown.sh_
- Move the generated ZIP file to the new clusters server location.
Diffs
-----
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 655252cb
repository/src/main/java/org/apache/atlas/repository/userprofile/UserProfileService.java a428b92e
tools/atlas-migration-utility/pom.xml PRE-CREATION
tools/atlas-migration-utility/src/assembly/bin.xml PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotification.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/EmptyNotificationChangeListener.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/Interactors.java PRE-CREATION
tools/atlas-migration-utility/src/main/java/org/apache/atlas/migration/MigrationApp.java PRE-CREATION
tools/atlas-migration-utility/src/main/resources/README PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas-migration-setup.sh PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas-migration-shutdown.sh PRE-CREATION
tools/atlas-migration-utility/src/main/resources/atlas_migration.py PRE-CREATION
tools/atlas-migration-utility/src/main/resources/export-request.json PRE-CREATION
tools/atlas-migration-utility/src/main/resources/migrationContext.xml PRE-CREATION
Diff: https://reviews.apache.org/r/65790/diff/1/
Testing
-------
**Unit tests**
None.
**Functional tests**
Exports from existing clusters.
File Attachments (updated)
----------------
Migration Utility
https://reviews.apache.org/media/uploaded/files/2018/02/24/e8090ed0-13b6-4253-a59c-d3f2098943af__atlas-migration-kit-0.8.3-SNAPSHOT-bin.zip
Thanks,
Ashutosh Mestry