incubator-kylin git commit: Update document for storage cleanup

[sh...@apache.org]

Repository: incubator-kylin
Updated Branches:
  refs/heads/0.7-staging 64e0ad5fe -> a925cdab1


Update document for storage cleanup

Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/a925cdab
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/a925cdab
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/a925cdab

Branch: refs/heads/0.7-staging
Commit: a925cdab183947c9122701d8eea242be72d49c38
Parents: 64e0ad5
Author: shaofengshi <sh...@apache.org>
Authored: Sun Sep 6 14:23:20 2015 +0800
Committer: shaofengshi <sh...@apache.org>
Committed: Sun Sep 6 14:23:32 2015 +0800

----------------------------------------------------------------------
 website/_docs/howto/howto_backup_hbase.md    | 26 ---------------------
 website/_docs/howto/howto_cleanup_storage.md | 28 +++++++++++++++++++++++
 2 files changed, 28 insertions(+), 26 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a925cdab/website/_docs/howto/howto_backup_hbase.md
----------------------------------------------------------------------
diff --git a/website/_docs/howto/howto_backup_hbase.md b/website/_docs/howto/howto_backup_hbase.md
deleted file mode 100644
index 5c54ca5..0000000
--- a/website/_docs/howto/howto_backup_hbase.md
+++ /dev/null
@@ -1,26 +0,0 @@
----
-layout: docs
-title:  How to Clean/Backup HBase Tables
-categories: howto
-permalink: /docs/howto/howto_backup_hbase.html
-version: v0.7.2
-since: v0.7.1
----
-
-Kylin persists all data (meta data and cube) in HBase; You may want to export the data sometimes for whatever purposes (backup, migration, troubleshotting etc); This page describes the steps to do this and also there is a Java app for you to do this easily.
-
-Steps:
-
-1. Cleanup unused cubes to save storage space (be cautious on production!): run the following command in hbase CLI: 
-{% highlight Groff markup %}
-hbase org.apache.hadoop.util.RunJar /${KYLIN_HOME}/lib/kylin-job-(version).jar org.apache.kylin.job.hadoop.cube.StorageCleanupJob --delete true
-{% endhighlight %}
-2. List all HBase tables, iterate and then export each Kylin table to HDFS; See [https://hbase.apache.org/book/ops_mgt.html#export](https://hbase.apache.org/book/ops_mgt.html#export)
-
-3. Copy the export folder from HDFS to local file system, and then archive it;
-
-4. (optional) Download the archive from Hadoop CLI to local;
-
-5. Cleanup the export folder from CLI HDFS and local file system;
-
-Kylin provide the "ExportHBaseData.java" (currently only exist in "minicluster" branch) for you to do the step 2-5 in one run; Please ensure the correct path of "kylin.properties" has been set in the sys env; This Java uses the sandbox config by default;

http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a925cdab/website/_docs/howto/howto_cleanup_storage.md
----------------------------------------------------------------------
diff --git a/website/_docs/howto/howto_cleanup_storage.md b/website/_docs/howto/howto_cleanup_storage.md
new file mode 100644
index 0000000..f8ebb6f
--- /dev/null
+++ b/website/_docs/howto/howto_cleanup_storage.md
@@ -0,0 +1,28 @@
+---
+layout: docs
+title:  How to Cleanup Storage (HDFS & HBase Tables)
+categories: howto
+permalink: /docs/howto/howto_cleanup_storage.html
+version: v0.7.2
+since: v0.7.1
+---
+
+Kylin will generate intermediate files in HDFS during the cube building; Besides, when purge/drop/merge cubes, some HBase tables may be left in HBase and will no longer be queried; Although Kylin has started to do some 
+automated garbage collection, it might not cover all cases; You can do an offline storage cleanup periodically:
+
+Steps:
+
+1. Check which resources can be cleanup, this will not remove anything: 
+{% highlight Groff markup %}
+hbase org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-(version).jar org.apache.kylin.job.hadoop.cube.StorageCleanupJob --delete true
+{% endhighlight %}
+
+Here please replace (version) with the specific Kylin jar version in your installation;
+
+2. You can pickup 1 or 2 resources to check whether they're no longer be referred; Then add the "--delete true" option to start the cleanup:  
+
+{% highlight Groff markup %}
+hbase org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-(version).jar org.apache.kylin.job.hadoop.cube.StorageCleanupJob --delete true
+{% endhighlight %}
+
+On finish, the intermediate HDFS location and HTables will be dropped;
\ No newline at end of file