You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by lu...@apache.org on 2015/09/06 09:59:22 UTC
[50/50] [abbrv] incubator-kylin git commit: Merge branch
'0.7-staging' of https://git-wip-us.apache.org/repos/asf/incubator-kylin into
0.7-staging
Merge branch '0.7-staging' of https://git-wip-us.apache.org/repos/asf/incubator-kylin into 0.7-staging
Conflicts:
website/_docs/howto/howto_backup_hbase.md
website/_docs/install/advance_settings.md
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/78dba126
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/78dba126
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/78dba126
Branch: refs/heads/0.7-staging
Commit: 78dba126cc3c35bbe3a61b6592c7c23bcd44f018
Parents: 412a150 a6b0df9
Author: Luke Han <lu...@apache.org>
Authored: Sun Sep 6 15:57:28 2015 +0800
Committer: Luke Han <lu...@apache.org>
Committed: Sun Sep 6 15:57:28 2015 +0800
----------------------------------------------------------------------
.../apache/kylin/dict/DictionaryManager.java | 12 +--
.../org/apache/kylin/dict/TrieDictionary.java | 87 +++++++-------------
website/_data/docs.yml | 2 +-
website/_docs/howto/howto_backup_hbase.md | 9 +-
website/_docs/howto/howto_cleanup_storage.md | 28 +++++++
website/_docs/install/advance_settings.md | 11 ++-
6 files changed, 77 insertions(+), 72 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/78dba126/website/_docs/howto/howto_backup_hbase.md
----------------------------------------------------------------------
diff --cc website/_docs/howto/howto_backup_hbase.md
index 5c54ca5,0000000..cc43f27
mode 100644,000000..100644
--- a/website/_docs/howto/howto_backup_hbase.md
+++ b/website/_docs/howto/howto_backup_hbase.md
@@@ -1,26 -1,0 +1,29 @@@
+---
+layout: docs
+title: How to Clean/Backup HBase Tables
+categories: howto
+permalink: /docs/howto/howto_backup_hbase.html
+version: v0.7.2
+since: v0.7.1
+---
+
- Kylin persists all data (meta data and cube) in HBase; You may want to export the data sometimes for whatever purposes (backup, migration, troubleshotting etc); This page describes the steps to do this and also there is a Java app for you to do this easily.
++Kylin persists all data (meta data and cube) in HBase; You may want to export the data sometimes for whatever purposes
++(backup, migration, troubleshotting etc); This page describes the steps to do this and also there is a Java app for you to do this easily.
+
+Steps:
+
+1. Cleanup unused cubes to save storage space (be cautious on production!): run the following command in hbase CLI:
+{% highlight Groff markup %}
+hbase org.apache.hadoop.util.RunJar /${KYLIN_HOME}/lib/kylin-job-(version).jar org.apache.kylin.job.hadoop.cube.StorageCleanupJob --delete true
+{% endhighlight %}
- 2. List all HBase tables, iterate and then export each Kylin table to HDFS; See [https://hbase.apache.org/book/ops_mgt.html#export](https://hbase.apache.org/book/ops_mgt.html#export)
++2. List all HBase tables, iterate and then export each Kylin table to HDFS;
++See [https://hbase.apache.org/book/ops_mgt.html#export](https://hbase.apache.org/book/ops_mgt.html#export)
+
+3. Copy the export folder from HDFS to local file system, and then archive it;
+
+4. (optional) Download the archive from Hadoop CLI to local;
+
+5. Cleanup the export folder from CLI HDFS and local file system;
+
- Kylin provide the "ExportHBaseData.java" (currently only exist in "minicluster" branch) for you to do the step 2-5 in one run; Please ensure the correct path of "kylin.properties" has been set in the sys env; This Java uses the sandbox config by default;
++Kylin provide the "ExportHBaseData.java" (currently only exist in "minicluster" branch) for you to do the
++step 2-5 in one run; Please ensure the correct path of "kylin.properties" has been set in the sys env; This Java uses the sandbox config by default;
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/78dba126/website/_docs/install/advance_settings.md
----------------------------------------------------------------------
diff --cc website/_docs/install/advance_settings.md
index b52b3d6,5eb9e90..f9704ea
--- a/website/_docs/install/advance_settings.md
+++ b/website/_docs/install/advance_settings.md
@@@ -29,12 -29,19 +29,21 @@@ To test if the hbase cluster is ready t
create 'lzoTable', {NAME => 'colFam',COMPRESSION => 'LZO'}
{% endhighlight %}
- ### Modify kylin_job_conf.xml
+ ### Use LZO for HBase compression
+You'll need to stop Kylin first by running `./kylin.sh stop`, and then modify $KYLIN_HOME/conf/kylin_job_conf.xml by uncommenting some configuration entries related to LZO compression.
+After this, you need to run `./kylin.sh start` to start Kylin again. Now Kylin will use LZO to compress MR outputs and hbase tables.
+
+ Goto $KYLIN_HOME/conf/kylin.properties, change kylin.hbase.default.compression.codec=snappy to kylin.hbase.default.compression.codec=lzo
+
+ ### Use LZO for MR jobs
+
+ Modify $KYLIN_HOME/conf/kylin_job_conf.xml by changing all org.apache.hadoop.io.compress.SnappyCodec to com.hadoop.compression.lzo.LzoCodec.
+
+ ### Restart Kylin
+
+ Start Kylin again. Now Kylin will use LZO to compress MR outputs and HBase tables.
-
## Enable LDAP authentication
Kylin supports LDAP authentication for enterprise or production deployment; This is implemented based on Spring Security framework; Before enable LDAP, please contact your LDAP administrator to get necessary information, like LDAP server URL, username/password, search patterns, etc;