You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2015/03/28 01:04:00 UTC
[12/37] incubator-kylin git commit: KYLIN-650 add operations sections
KYLIN-650 add operations sections
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/284ef260
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/284ef260
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/284ef260
Branch: refs/heads/master
Commit: 284ef260140d0cae845dfb6075a1e679f9e9300c
Parents: 4bd5407
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 23:07:09 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 23:07:09 2015 -0700
----------------------------------------------------------------------
README.md | 14 ++++++++-
.../Advanced settings of Kylin environment.md | 28 ++++++++++++++++++
docs/Operations/Export Kylin HBase data.md | 19 ++++++++++++
docs/Operations/Kylin Metadata Store.md | 31 ++++++++++++++++++++
4 files changed, 91 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/284ef260/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 2489576..fa180b0 100644
--- a/README.md
+++ b/README.md
@@ -17,8 +17,20 @@ Tutorial
Please follow this installation tutorial to start with Kylin: [Installation Tutorial](docs/Installation/Installation.md)
-Advance Usage
+Advance Topics
-------
+####REST
+
+####Operations
+1.[Kylin Metadata Store](docs/Operations/Kylin Metadata Store.md)
+
+2.[Export Kylin HBase data](docs/Operations/Export Kylin HBase data.md)
+
+3.[Advanced settings of Kylin environment](docs/Operations/Advanced settings of Kylin environment.md)
+
+####Test
+
+####Technial Details
Get Help
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/284ef260/docs/Operations/Advanced settings of Kylin environment.md
----------------------------------------------------------------------
diff --git a/docs/Operations/Advanced settings of Kylin environment.md b/docs/Operations/Advanced settings of Kylin environment.md
new file mode 100644
index 0000000..eabfa43
--- /dev/null
+++ b/docs/Operations/Advanced settings of Kylin environment.md
@@ -0,0 +1,28 @@
+Advanced settings of Kylin environment
+----
+
+### A. Enabling LZO compression
+
+LZO compression can be leveraged to compress the output of MR jobs, as well as hbase table storage, reducing the storage overhead. By default we do not enable LZO compression in Kylin because hadoop sandbox venders tend to not include LZO in their distributions due to license(GPL) issues.
+
+To enable LZO in Kylin, there are three steps:
+
+#### Make sure LZO is working in your environment
+
+we have a simple tool to test whether LZO is well installed and configured in your environment(we only test it on the hadoop CLI that you deployed Kylin), Just run
+
+`hbase org.apache.hadoop.util.RunJar kylin-job-latest.jar com.kylinolap.job.tools.LZOSupportnessChecker`
+
+If the program prints "LZO supported by current env? true", you're good to go. Otherwise you'll need to first install LZO properly.
+
+#### Modify kylin_job_conf.xml
+
+You'll need to stop Kylin first by running `${KYLIN_HOME}/bin/kylin.sh stop`, and then modify /etc ${KYLIN_HOME}/conf/kylin_job_conf.xml by uncommenting some configuration entries related to LZO compression.
+
+#### export KYLIN_LD_LIBRARY_PATH to carry your native library paths
+
+Before starting Kylin again, you'll need to uncomment KYLIN_LD_LIBRARY_PATH in ${KYLIN_HOME}/bin/setenv.sh to carry your LZO native library paths. Here's an example for hdp 2.1:
+
+ export KYLIN_LD_LIBRARY_PATH=::/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
+
+After exporting, you need to run `${KYLIN_HOME}/bin/kylin.sh start` to start Kylin again. Now Kylin will use LZO to compress MR outputs and hbase tables.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/284ef260/docs/Operations/Export Kylin HBase data.md
----------------------------------------------------------------------
diff --git a/docs/Operations/Export Kylin HBase data.md b/docs/Operations/Export Kylin HBase data.md
new file mode 100644
index 0000000..90c4778
--- /dev/null
+++ b/docs/Operations/Export Kylin HBase data.md
@@ -0,0 +1,19 @@
+# Export Kylin HBase data
+
+Kylin persists all data (meta data and cube) in HBase; You may want to export the data sometimes for whatever purposes (backup, migration, troubleshotting etc); This page describes the steps to do this and also there is a Java app for you to do this easily;
+
+Steps:
+
+1. Cleanup unused cubes to save storage space (be cautious on production!): run the following command in hbase CLI:
+```
+hbase org.apache.hadoop.util.RunJar /${KYLIN_HOME}/lib/kylin-job-(version).jar org.apache.kylin.job.hadoop.cube.StorageCleanupJob --delete true
+```
+2. List all HBase tables, iterate and then export each Kylin table to HDFS; See https://hbase.apache.org/book/ops_mgt.html#export
+
+3. Copy the export folder from HDFS to local file system, and then archive it;
+
+4. (optional) Download the archive from Hadoop CLI to local;
+
+5. Cleanup the export folder from CLI HDFS and local file system;
+
+Kylin provide the "ExportHBaseData.java" (currently only exist in "minicluster" branch) for you to do the step 2-5 in one run; Please ensure the correct path of "kylin.properties" has been set in the sys env; This Java uses the sandbox config by default;
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/284ef260/docs/Operations/Kylin Metadata Store.md
----------------------------------------------------------------------
diff --git a/docs/Operations/Kylin Metadata Store.md b/docs/Operations/Kylin Metadata Store.md
new file mode 100644
index 0000000..33c1afe
--- /dev/null
+++ b/docs/Operations/Kylin Metadata Store.md
@@ -0,0 +1,31 @@
+Kylin organizes all of its metadata(including cube descriptions and instances, projects, inverted index description and instances, jobs, tables and dictionaries) as a hierarchy file system. However, Kylin uses hbase to store it, rather than normal file system. If you check your kylin configuration file(kylin.properties) you will find such a line:
+
+`# The metadata store in hbase`
+`kylin.metadata.url=kylin_metadata_qa@hbase:sandbox.hortonworks.com:2181:/hbase-unsecure`
+
+This indicates that the metadata will be saved as a htable called `kylin_metadata_qa`. You can scan the htable in hbase shell to check it out.
+
+# Backup Metadata Store
+
+Sometimes you need to backup the Kylin's Metadata Store from hbase to your disk file system.
+In such cases, assuming you're on the hadoop CLI(or sandbox) where you deployed Kylin, you can use:
+
+ mkdir ~/meta_dump
+
+ hbase org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-SNAPSHOT-job.jar com.kylinolap.common.persistence.ResourceTool copy ${KYLIN_HOME}/conf/kylin.properties ~/meta_dump
+
+to dump your metadata to your local folder ~/meta_dump.
+
+# Restore Metadata Store
+
+In case you find your metadata store messed up, and you want to restore to a previous backup:
+
+first clean up the metadata store:
+
+ hbase org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-SNAPSHOT-job.jar com.kylinolap.common.persistence.ResourceTool reset
+
+then upload the backup metadata in ~/meta_dump to Kylin's metadata store:
+
+ hbase org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-SNAPSHOT-job.jar com.kylinolap.common.persistence.ResourceTool copy ~/meta_dump ${KYLIN_HOME}/conf/kylin.properties
+
+