You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@atlas.apache.org by am...@apache.org on 2018/05/25 04:53:12 UTC
atlas git commit: ATLAS-2704: documentation on migration from 0.8 to
1.0
Repository: atlas
Updated Branches:
refs/heads/master 3d5b4880f -> 84196cfc5
ATLAS-2704: documentation on migration from 0.8 to 1.0
Signed-off-by: Ashutosh Mestry <am...@hortonworks.com>
Project: http://git-wip-us.apache.org/repos/asf/atlas/repo
Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/84196cfc
Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/84196cfc
Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/84196cfc
Branch: refs/heads/master
Commit: 84196cfc5cf9ef18db52caa16ebca8542da89acf
Parents: 3d5b488
Author: Ashutosh Mestry <am...@hortonworks.com>
Authored: Thu May 24 18:42:07 2018 -0700
Committer: Ashutosh Mestry <am...@hortonworks.com>
Committed: Thu May 24 21:52:40 2018 -0700
----------------------------------------------------------------------
docs/src/site/twiki/Migration-0.8-to-1.0.twiki | 133 ++++++++++++++++++++
docs/src/site/twiki/index.twiki | 1 +
2 files changed, 134 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/atlas/blob/84196cfc/docs/src/site/twiki/Migration-0.8-to-1.0.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/Migration-0.8-to-1.0.twiki b/docs/src/site/twiki/Migration-0.8-to-1.0.twiki
new file mode 100644
index 0000000..c145af7
--- /dev/null
+++ b/docs/src/site/twiki/Migration-0.8-to-1.0.twiki
@@ -0,0 +1,133 @@
+---+ Migrating data from Apache Atlas 0.8 to Apache Atlas 1.0
+
+Apache Atlas 1.0 uses !JanusGraph graph database to store its type and entity details. Prior versions of Apache Atlas
+use Titan 0.5.4 graph database. The two databases use different formats for storage. For deployments upgrading from
+earlier version Apache Atlas, the data in Titan 0.5.4 graph database should be migrated to !JanusGraph graph database.
+
+In addition to the change to the graph database, Apache Atlas 1.0 introduces few optimizations that require different internal
+representation compared to previous versions. Migration steps detailed below will transform data to be compliant with
+the new internal representation.
+
+---+++ Migration Steps
+
+Migration of data is done in following steps:
+ * Planning the migration.
+ * Export Apache Atlas 0.8 data to a directory on the file system.
+ * Import data from exported files into Apache Atlas 1.0.
+
+---++++ Planning the migration
+
+The duration of migration of data from Apache Atlas 0.8 to Apache Atlas 1.0 can be significant, depending upon the
+amount of data present in Apache Atlas. This section helps you to estimate the time to migrate, so that you can plan the
+upgrade process better.
+
+To estimate the time needed to export data, first you need to find the number of entities in Apache Atlas 0.8. This can
+be done by running the following DSL query:
+<verbatim>
+Referenceable select count()
+</verbatim>
+
+Assuming Apache Atlas is deployed in a quad-core CPU with 4 GB of RAM allocated:
+ * Export from Apache Atlas 0.8 will process approximately 2 million entities per hour.
+ * Import into Apache Atlas 1.0 will process approximately 0.7 million entities per hour.
+
+---++++ Exporting data from Apache Atlas 0.8
+_Atlas Migration Export Utility_ from Apache Atlas branch-0.8 should be used to export the data from Apache Atlas 0.8
+deployments. The implementation of which can be found [[https://github.com/apache/atlas/tree/branch-0.8/tools/atlas-migration-exporter][here]].
+
+To build this utility:
+ * Navigate to root directory of Apache Atlas branch-0.8 local repo (say, _/home/atlas/_)
+ * Build using the command: _mvn clean -DskipTests package -Pdist_.
+ * After successful build, the _Atlas Migration Utility_ can be found in this _distro/target_ directory. (Say, _/home/atlas/distro/target/apache-atlas-0.8.3-SNAPSHOT-bin/apache-atlas-0.8.3-SNAPSHOT/tools/migration-exporter_)
+
+Move the _Atlas Migration Utility_ directory to the Apache Atlas 0.8 cluster.
+
+Follow these steps to export the data:
+ * Shutdown _Apache Atlas 0.8_, so that the database is not updated while the migration is in progress.
+ * Execute the following command to export Apache Atlas data in Titan graph database to the specified directory:
+<verbatim>
+ atlas_migration_export.py -d <output directory></verbatim>
+
+Example:
+<verbatim>
+ /home/atlas-migration-utility/atlas_migration_export.py -d /home/atlas-0.8-data
+</verbatim>
+
+On successful execution, _Atlas Migration Utility_ tool will display messages like these:
+<verbatim>
+atlas-migration-export: starting migration export. Log file location /var/log/atlas/atlas-migration-exporter.log
+atlas-migration-export: initializing
+atlas-migration-export: initialized
+atlas-migration-export: exporting typesDef to file /home/atlas-0.8-data/atlas-migration-typesdef.json
+atlas-migration-export: exported typesDef to file /home/atlas-0.8-data/atlas-migration-typesdef.json
+atlas-migration-export: exporting data to file /home/atlas-0.8-data/atlas-migration-data.json
+atlas-migration-export: exported data to file /home/atlas-0.8-data/atlas-migration-data.json
+atlas-migration-export: completed migration export!
+</verbatim>
+
+More details on the progress of export can be found in a log file named _atlas-migration-exporter.log_, in the log directory
+specified in _atlas-log4j.xml_.
+
+---+++ Before importing into Apache Atlas 1.0
+ * For Apache Atlas deployments that use Solr as index store, please ensure that existing Apache Atlas specific collections are deleted or renamed before installing Apache Atlas 1.0.
+
+Apache Atlas specific Solr collections can be deleted using CURL commands shown below:
+<verbatim>
+curl 'http://<solrHost:port>/solr/admin/collections?action=DELETE&name=vertex_index'
+curl 'http://<solrHost:port>/solr/admin/collections?action=DELETE&name=edge_index'
+curl 'http://<solrHost:port>/solr/admin/collections?action=DELETE&name=fulltext_index'</verbatim>
+
+ * Create Solr collections for Apache Atlas 1.0
+Apache Atlas specific Solr collections can be created using CURL commands shown below:
+<verbatim>
+curl 'http://<solrHost:port>/solr/admin/collections?action=CREATE&name=vertex_index&numShards=1&replicationFactor=1&collection.configName=atlas_configs'
+curl 'http://<solrHost:port>/solr/admin/collections?action=CREATE&name=edge_index&numShards=1&replicationFactor=1&collection.configName=atlas_configs'
+curl 'http://<solrHost:port>/solr/admin/collections?action=CREATE&name=fulltext_index&numShards=1&replicationFactor=1&collection.configName=atlas_configs'</verbatim>
+
+ * For Apache Atlas deployments that use HBase as backend store, please note that HBase table used by earlier version can't be used by Apache Atlas 1.0. If you are constrained on disk storage space, the table used by earlier version can be removed after successful export of data.
+ * Apache Atlas 0.8 uses HBase table named 'atlas_titan' (by default)
+ * Apache Atlas 1.0 uses HBase table named 'atlas_janus' (by default)
+
+ * Install Apache Atlas 1.0. Do not start yet!
+
+ * Make sure the directory containing exported data is accessible to Apache Atlas 1.0 instance.
+
+
+---++++ Importing Data into Apache Atlas 1.0
+Please follow the steps below to import the data exported above into Apache Atlas 1.0:
+ * Specify the location of the directory containing exported data in following property to _atlas-application.properties_:
+<verbatim>
+atlas.migration.data.filename=<location of the directory containing exported data></verbatim>
+
+ * Start Apache Atlas 1.0. Apache Atlas will start in migration mode. It will start importing data from the specified directory.
+
+ * Monitor the progress of import process with the following curl command:
+<verbatim>
+curl -X GET -u admin:<password> -H "Content-Type: application/json" -H "Cache-Control: no-cache" http://<atlasHost>:port/api/atlas/admin/status</verbatim>
+
+Progress of import will be indicated by a message like this:
+<verbatim>
+{"Status":"MIGRATING","MigrationStatus":{"operationStatus":"IN_PROGRESS","startTime":1526512275110,"endTime":1526512302750,"currentIndex":10,"currentCounter":101,"totalCount":0}}
+</verbatim>
+
+Successful completion of the operation will show a message like this:
+<verbatim>
+{"Status":"MIGRATING","MigrationStatus":{"operationStatus":"SUCCESS","startTime":1526512275110,"endTime":1526512302750,"currentIndex":0,"currentCounter":0,"totalCount":371}}
+</verbatim>
+
+Once migration import is complete, i.e. _operationStatus_ is _SUCCESS_, follow the steps given below to restart Apache Atlas
+in _ACTIVE_ mode for regular use:
+ * Stop Apache Atlas 1.0.
+ * Remove property _atlas.migration.data.filename_ from _atlas-application.properties_.
+ * Start Apache Atlas 1.0.
+
+---+++ Atlas Entity Defaults for Migrated Data
+
+Apache Atlas 1.0 introduces number of new features. For data that is migrated, the following defaults are set:
+ * All classifications will have _isPropagate_ set to _false_.
+ * Taxonomy terms present in Apache Atlas 0.8, if any, will be converted to classification.
+
+---++++ Handling of Entity Definitions that use Classifications as Types
+
+This features is no longer supported. Classifications that are used as types in _attribute definitions_ (_AttributeDefs_) are converted in to new types whose name has _legacy_ prefix. These are then handled like any other type.
+Creation of such types was prevented in an earlier release, hence only type definitions have potential to exist. Care has been taken to handle entities of this type as well.
http://git-wip-us.apache.org/repos/asf/atlas/blob/84196cfc/docs/src/site/twiki/index.twiki
----------------------------------------------------------------------
diff --git a/docs/src/site/twiki/index.twiki b/docs/src/site/twiki/index.twiki
index ed14615..7d8c982 100755
--- a/docs/src/site/twiki/index.twiki
+++ b/docs/src/site/twiki/index.twiki
@@ -71,6 +71,7 @@ capabilities around these data assets for data scientists, analysts and the data
* [[Hook-Storm][Storm Hook]]
* [[Bridge-Kafka][Kafka Bridge]]
* [[HighAvailability][Fault Tolerance And High Availability Options]]
+ * [[Migration-0.8-to-1.0][Migration from Apache Atlas 0.8]]
---++ API Documentation