kylin git commit: KYLIN-1462 Document on Upgrade from 1.x to 2.0

[li...@apache.org]

Repository: kylin
Updated Branches:
  refs/heads/document dbd3538e0 -> e878eb7d3


KYLIN-1462 Document on Upgrade from 1.x to 2.0


Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/e878eb7d
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/e878eb7d
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/e878eb7d

Branch: refs/heads/document
Commit: e878eb7d3dee897697c65d8348d54ce54020c773
Parents: dbd3538
Author: lidongsjtu <li...@apache.org>
Authored: Tue Mar 1 23:32:36 2016 +0800
Committer: lidongsjtu <li...@apache.org>
Committed: Tue Mar 1 23:32:45 2016 +0800

----------------------------------------------------------------------
 website/_docs2/howto/howto_upgrade.md | 110 +++++++++++++----------------
 1 file changed, 50 insertions(+), 60 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kylin/blob/e878eb7d/website/_docs2/howto/howto_upgrade.md
----------------------------------------------------------------------
diff --git a/website/_docs2/howto/howto_upgrade.md b/website/_docs2/howto/howto_upgrade.md
index 709f94d..8d80d3a 100644
--- a/website/_docs2/howto/howto_upgrade.md
+++ b/website/_docs2/howto/howto_upgrade.md
@@ -3,101 +3,91 @@ layout: docs2
 title:  How to Upgrade
 categories: howto
 permalink: /docs2/howto/howto_upgrade.html
-version: v1.2
-since: v0.7.1
+version: v2.0
+since: v2.0
 ---
 
-## Upgrade among v0.7.x and v1.x 
+## Upgrade from v1.x to v2.0 
 
-From v0.7.1 to latest v1.2, Kylin's metadata is backward compatible, the upgrade can be finished in couple of minutes:
+From v1.x to v2.0, Kylin's cube data is backward compatible, but metadata has been refactored as new schema, to support new features on cubing and query enhancement. So if you want to deploy v2.0 on your v1.x base, you need to upgrade the metadata as following steps:
 
-#### 1. Backup metadata
-Backup the Kylin metadata peridically is a good practice, and is highly suggested before upgrade; 
+#### 1. Backup metadata on v1.x
+To avoid data loss during the upgrade, a backup at the very beginning is always suggested. In case of upgrade failure, you can roll back to original state with the backup.
 
 ```
-cd $KYLIN_HOME
-./bin/metastore.sh backup
+$KYLIN_HOME/bin/metastore.sh backup
 ``` 
-It will print the backup folder, take it down and make sure it will not be deleted before the upgrade finished. If there is no "metastore.sh", use HBase's snapshot command to do backup:
+It will print the backup folder, take it down and make sure it will not be deleted before the upgrade finished. If there is no "metastore.sh", you can use HBase's snapshot command to do backup:
 
 ```
 hbase shell
-snapshot 'kylin_metadata', 'kylin_metadata_backup20150610'
+snapshot 'kylin_metadata', 'kylin_metadata_backup20160101'
 ```
-Here 'kylin_metadata' is the default kylin metadata table name, replace it with the right table name of your Kylin;
+Here 'kylin_metadata' is the default kylin metadata table name, replace it with the right table name of your Kylin metastore.
 
-#### 2. Install new Kylin and copy back "conf"
-Download the new Kylin binary package from Kylin's download page; Extract it to a different folder other than current KYLIN_HOME; Before copy back the "conf" folder, do a compare and merge between the old and new kylin.properties to ensure newly introduced property will be kept.
+#### 2. Stop Kylin v1.x instance
+Before deploying Kylin v2.0 instance, you need to stop the old instance. Note that end users cannot access kylin service from this point.
 
-#### 3. Stop old and start new Kylin instance
 ```
-cd $KYLIN_HOME
-./bin/kylin.sh stop
-export KYLIN_HOME="<path_of_new_installation>"
-cd $KYLIN_HOME
-./bin/kylin.sh start
-```
-
-#### 4. Back-port if the upgrade is failed
-If the new version couldn't startup and need back-port, shutdown it and then switch to the old KYLIN_HOME to start. Idealy that would return to the origin state. If the metadata is broken, restore it from the backup folder.
-
+$KYLIN_HOME/bin/kylin.sh stop
 ```
-./bin/metastore.sh restore <path_of_metadata_backup>
-```
-
-## Upgrade from v0.6.x to v0.7.x 
+#### 3. Install Kylin v2.0 and copy back "conf"
+Download the new Kylin v2.0 binary package from Kylin's download page; Extract it to a different folder other than current KYLIN_HOME; Before copy back the "conf" folder, do a compare and merge between the old and new kylin.properties to ensure newly introduced property will be kept.
 
-In v0.7, Kylin refactored the metadata structure, for the new features like inverted-index and streaming; If you have cube created with v0.6 and want to keep in v0.7, a migration is needed; (Please skip v0.7.1 as
-it has several compatible issues and the fix will be included in v0.7.2) Below is the steps;
+#### (Optional) 4. Upgrading metadata will not bring new features of v2.0 to existing cube built with v1.x engine. If you want to leverage those features, please refer to [Highlight]() part.
 
-#### 1. Backup v0.6 metadata
-To avoid data loss in the migration, a backup at the very beginning is always suggested; You can use HBase's backup or snapshot command to achieve this; Here is a sample with snapshot:
+#### 5. Automaticly upgrade metadata
+Kylin v2.0 package provides a script for metadata automaticly upgrade. In this upgrade, empty cubes will be updated to v2.0 version and all new features are enabled for them. But those non-empty cubes are not able to use those new features.
 
 ```
-hbase shell
-snapshot 'kylin_metadata', 'kylin_metadata_backup20150610'
+export KYLIN_HOME="<path_of_new_installation>"
+$KYLIN_HOME/bin/upgrade_v2.sh
 ```
+After this, the metadata in hbase table has been applied with new metadata schema.
 
-'kylin_metadata' is the default kylin metadata table name, replace it with the right table name of your Kylin;
-
-#### 2. Dump v0.6 metadata to local file
-This is also a backup method; As the migration tool is only tested with local file system, this step is must; All metadata need be downloaded, including snapshot, dictionary, etc;
-
+#### 6. Start Kylin v2.0 instance
 ```
-hbase  org.apache.hadoop.util.RunJar  ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar  org.apache.kylin.common.persistence.ResourceTool  download  ./meta_dump
+$KYLIN_HOME/bin/kylin.sh start
 ```
+Check the log and open web UI to see if the upgrade succeeded.
 
-(./meta_dump is the local folder that the metadata will be downloaded, change to name you preferred)
+## Rollback if the upgrade is failed
+If the new version couldn't startup normally, you need to roll back to orignal v1.x version. The steps are as followed:
 
-#### 3. Run CubeMetadataUpgrade to migrate the metadata
-This step is to run the migration tool to parse the v0.6 metadata and then convert to v0.7 format; A verification will be performed in the last, and report error if some cube couldn't be migrated;
+#### 1. Stop Kylin v2.0 instance
 
 ```
-hbase org.apache.hadoop.util.RunJar  ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar org.apache.kylin.job.CubeMetadataUpgrade ./meta_dump
+$KYLIN_HOME/bin/kylin.sh stop
 ```
+#### 2. Rstore 1.x metadata from backup folder
 
-1. The tool will not overwrite v0.6 metadata; It will create a new folder with "_v2" suffix in the same folder, in this case the "./meta_dump_v2" will be created;
-2. By default this tool will only migrate the job history in last 30 days; If you want to keep elder job history, please tweak upgradeJobInstance() method by your own;
-3. If you see _No error or warning messages; The migration is success_ , that's good; Otherwise please check the error/warning messages carefully;
-4. For some problem you may need manually update the JSON file, to check whether the problem is gone, you can run a verify against the new metadata:
+```
+export KYLIN_HOME="<path_of_1.x_installation>"
+$KYLIN_HOME/bin/metastore.sh restore <backup_folder>
+``` 
+#### 3. Deploy coprocessor of v1.x
+Since coprocessor of used HTable are upgraded as v2.0, you need to manually downgrade them with this command.
 
 ```
-hbase org.apache.hadoop.util.RunJar  ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar org.apache.kylin.job.CubeMetadataUpgrade ./meta_dump2 verify
+$KYLIN_HOME/bin/kylin.sh org.apache.kylin.job.tools.DeployCoprocessorCLI $KYLIN_HOME/lib/kylin-coprocessor*.jar -all
 ```
 
-#### 4. Upload the new metadata to HBase
-Now the new format of metadata will be upload to the HBase to replace the old format; Stop Kylin, and then:
-
+#### 4. Start Kylin v1.x instance
+ 
 ```
-hbase org.apache.hadoop.util.RunJar  ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar  org.apache.kylin.common.persistence.ResourceTool  reset
-hbase org.apache.hadoop.util.RunJar  ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar  org.apache.kylin.common.persistence.ResourceTool  upload  ./meta_dump_v2
+$KYLIN_HOME/bin/kylin.sh start
 ```
 
-#### 5. Update HTables to use new coprocessor
-Kylin uses HBase coprocessor to do server side aggregation; When Kylin instance upgrades to V0.7, the HTables that created in V0.6 should also be updated to use the new coprocessor:
+## Highlights
+Since old cubes built with v1.x cannot leverage new features of v2.0. But if you must have them on your cubes, you can choose one of these solutions:
+#### 1. Rebuilt cubes
+If the cost of rebuilding is acceptable, if you purge the cube before Step 4(Running Upgrade Scripts). After upgrade done, you need to manually rebuilt those segments by yourself.
+#### 2. Use hybrid model
+If you don't want to rebuild any cube, but want to leverage new features for  new data. You can use hybrid model, which contains not only your old cube, but also an empty cube which has same model with the old one. For the empty cube, you can do incremental building with v2 features. For the old cube, you can refresh existing segments only.
 
-```
-hbase org.apache.hadoop.util.RunJar  ${KYLIN_HOME}/lib/kylin-job-x.x.x-job.jar  org.apache.kylin.job.tools.DeployCoprocessorCLI ${KYLIN_HOME}/lib/kylin-coprocessor-x.x.x.jar
-```
+Here is the command to create hybrid model:
 
-Done; Update your v0.7 Kylin configure to point to the same metadata HBase table, then start Kylin server; Check whether all cubes and other information are kept;
\ No newline at end of file
+```
+export KYLIN_HOME="<path_of_v2.0_installation>"
+$KYLIN_HOME/bin/kylin.sh org.apache.kylin.storage.hbase.util.ExtendCubeToHybridCLI <project_name> <cube_name>
+```
\ No newline at end of file