You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2018/01/14 12:28:37 UTC
[1/4] kylin git commit: APACHE-KYLIN-3153: Create a document for
system cube creation
Repository: kylin
Updated Branches:
refs/heads/document af6c05328 -> 2a282eeb8
APACHE-KYLIN-3153: Create a document for system cube creation
Signed-off-by: lidongsjtu <li...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/5c5914b6
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/5c5914b6
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/5c5914b6
Branch: refs/heads/document
Commit: 5c5914b6a9ff87efc3734fd424ab82cdfed43d3b
Parents: af6c053
Author: xueyzhang <xu...@ebay.com>
Authored: Mon Dec 18 16:18:53 2017 +0800
Committer: lidongsjtu <li...@apache.org>
Committed: Sun Jan 14 19:47:06 2018 +0800
----------------------------------------------------------------------
website/_docs21/howto/howto_setup_systemcube.md | 437 +++++++++++++++++++
website/images/SystemCube/hive_table.png | Bin 0 -> 38911 bytes
website/images/SystemCube/kylin_system.png | Bin 0 -> 297495 bytes
website/images/SystemCube/metadata.png | Bin 0 -> 66489 bytes
website/images/SystemCube/reload_metadata.png | Bin 0 -> 138984 bytes
5 files changed, 437 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kylin/blob/5c5914b6/website/_docs21/howto/howto_setup_systemcube.md
----------------------------------------------------------------------
diff --git a/website/_docs21/howto/howto_setup_systemcube.md b/website/_docs21/howto/howto_setup_systemcube.md
new file mode 100644
index 0000000..e8a46d2
--- /dev/null
+++ b/website/_docs21/howto/howto_setup_systemcube.md
@@ -0,0 +1,437 @@
+---
+layout: docs21
+title: Set Up System Cube
+categories: howto
+permalink: /docs21/howto/howto_setup_systemcube.html
+---
+
+# Set Up System Cube
+
+## What is System Cube
+
+For better supporting self-monitoring, a set of system cubes are created under the system project, called "KYLIN_SYSTEM". Currently, there are five cubes. Three are for query metrics, "METRICS_QUERY", "METRICS_QUERY_CUBE", "METRICS_QUERY_RPC". And the other two are for job metrics, "METRICS_JOB", "METRICS_JOB_EXCEPTION".
+
+## How to Set Up System Cube
+
+### Prepare
+Create a configuration file SCSinkTools.json in KYLIN_HOME directory.
+
+For example:
+
+```
+[
+ [
+ "org.apache.kylin.tool.metrics.systemcube.util.HiveSinkTool",
+ {
+ "storage_type": 2,
+ "cube_desc_override_properties": [
+ "java.util.HashMap",
+ {
+ "kylin.cube.algorithm": "INMEM",
+ "kylin.cube.max-building-segments": "1"
+ }
+ ]
+ }
+ ]
+]
+```
+
+### 1. Generate Metadata
+Run the following command in KYLIN_HOME folder to generate related metadata:
+
+```
+./bin/kylin.sh org.apache.kylin.tool.metrics.systemcube.SCCreator \
+-inputConfig SCSinkTools.json \
+-output <output_forder>
+```
+
+By this command, the related metadata will be generated and its location is under the directory `<output_forder>`. The details are as follows, system_cube is our `<output_forder>`:
+
+
+
+### 2. Set Up Datasource
+Running the following command to create source hive tables:
+
+```
+hive -f <output_forder>/create_hive_tables_for_system_cubes.sql
+```
+
+By this command, the related hive table will be created.
+
+
+
+### 3. Upload Metadata for System Cubes
+Then we need to upload metadata to hbase by the following command:
+
+```
+./bin/metastore.sh restore <output_forder>
+```
+
+### 4. Reload Metadata
+Finally, we need to reload metadata in Kylin web UI.
+
+
+
+Then, a set of system cubes will be created under the system project, called "KYLIN_SYSTEM".
+
+
+
+### 5. System Cube build
+When the system cube is created, we need to build the cube regularly.
+
+1. Create a shell script that builds the system cube by calling org.apache.kylin.tool.job.CubeBuildingCLI
+
+ For example:
+
+ ```
+ #!/bin/bash
+
+ dir=$(dirname ${0})
+ export KYLIN_HOME=${dir}/../
+
+ CUBE=$1
+ INTERVAL=$2
+ DELAY=$3
+ CURRENT_TIME_IN_SECOND=`date +%s`
+ CURRENT_TIME=$((CURRENT_TIME_IN_SECOND * 1000))
+ END_TIME=$((CURRENT_TIME-DELAY))
+ END=$((END_TIME - END_TIME%INTERVAL))
+
+ ID="$END"
+ echo "building for ${CUBE}_${ID}" >> ${KYLIN_HOME}/logs/build_trace.log
+ sh ${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.job.CubeBuildingCLI --cube ${CUBE} --endTime ${END} > ${KYLIN_HOME}/logs/system_cube_${CUBE}_${END}.log 2>&1 &
+ ```
+
+2. Then run this shell script regularly
+
+ For example, add a cron job as follows:
+
+ ```
+ 0 */2 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_QUERY_DEV 3600000 1200000
+
+ 20 */2 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_QUERY_CUBE_DEV 3600000 1200000
+
+ 40 */4 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_QUERY_RPC_DEV 3600000 1200000
+
+ 30 */4 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_JOB_DEV 3600000 1200000
+
+ 50 */12 * * * sh ${KYLIN_HOME}/bin/system_cube_build.sh KYLIN_HIVE_METRICS_JOB_EXCEPTION_DEV 3600000 12000
+ ```
+
+## Details of System Cube
+### Common Dimension
+For all of these cube, admins can query at four time granularities. From higher level to lower, it's as follows:
+
+<table>
+ <tr>
+ <td>KYEAR_BEGIN_DATE</td>
+ <td>year</td>
+ </tr>
+ <tr>
+ <td>KMONTH_BEGIN_DATE</td>
+ <td>month</td>
+ </tr>
+ <tr>
+ <td>KWEEK_BEGIN_DATE</td>
+ <td>week</td>
+ </tr>
+ <tr>
+ <td>KDAY_DATE</td>
+ <td>date</td>
+ </tr>
+</table>
+
+### METRICS_QUERY
+This cube is for collecting query metrics at the highest level. The details are as follows:
+
+<table>
+ <tr>
+ <th colspan="2">Dimension</th>
+ </tr>
+ <tr>
+ <td>HOST</td>
+ <td>the host of server for query engine</td>
+ </tr>
+ <tr>
+ <td>PROJECT</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>REALIZATION</td>
+ <td>in kylin, there are two OLAP realizations: cube & hybrid of cubes</td>
+ </tr>
+ <tr>
+ <td>REALIZATION_TYPE</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>QUERY_TYPE</td>
+ <td>users can query on different data sources, CACHE, OLAP, LOOKUP_TABLE, HIVE</td>
+ </tr>
+ <tr>
+ <td>EXCEPTION</td>
+ <td>when doing query, exceptions may happen. It's for classifying different exception types</td>
+ </tr>
+</table>
+
+<table>
+ <tr>
+ <th colspan="2">Measure</th>
+ </tr>
+ <tr>
+ <td>COUNT</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>MIN, MAX, SUM of QUERY_TIME_COST</td>
+ <td>the time cost for the whole query</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of CALCITE_SIZE_RETURN</td>
+ <td>the row count of the result Calcite returns</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of STORAGE_SIZE_RETURN</td>
+ <td>the row count of the input to Calcite</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of CALCITE_SIZE_AGGREGATE_FILTER</td>
+ <td>the row count of Calcite aggregates and filters</td>
+ </tr>
+ <tr>
+ <td>COUNT DISTINCT of QUERY_HASH_CODE</td>
+ <td>the number of different queries</td>
+ </tr>
+</table>
+
+### METRICS_QUERY_RPC
+This cube is for collecting query metrics at the lowest level. For a query, the related aggregation and filter can be pushed down to each rpc target server. The robustness of rpc target servers is the foundation for better serving queries. The details are as follows:
+
+<table>
+ <tr>
+ <th colspan="2">Dimension</th>
+ </tr>
+ <tr>
+ <td>HOST</td>
+ <td>the host of server for query engine</td>
+ </tr>
+ <tr>
+ <td>PROJECT</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>REALIZATION</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>RPC_SERVER</td>
+ <td>the rpc related target server</td>
+ </tr>
+ <tr>
+ <td>EXCEPTION</td>
+ <td>the exception of a rpc call. If no exception, "NULL" is used</td>
+ </tr>
+</table>
+
+<table>
+ <tr>
+ <th colspan="2">Measure</th>
+ </tr>
+ <tr>
+ <td>COUNT</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of CALL_TIME</td>
+ <td>the time cost of a rpc all</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of COUNT_SKIP</td>
+ <td>based on fuzzy filters or else, a few rows will be skiped. This indicates the skipped row count</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of SIZE_SCAN</td>
+ <td>the row count actually scanned</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of SIZE_RETURN</td>
+ <td>the row count actually returned</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of SIZE_AGGREGATE</td>
+ <td>the row count actually aggregated</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of SIZE_AGGREGATE_FILTER</td>
+ <td>the row count actually aggregated and filtered, = SIZE_SCAN - SIZE_RETURN</td>
+ </tr>
+</table>
+
+### METRICS_QUERY_CUBE
+This cube is for collecting query metrics at the cube level. The most important are cuboids related, which will serve for cube planner. The details are as follows:
+
+<table>
+ <tr>
+ <th colspan="2">Dimension</th>
+ </tr>
+ <tr>
+ <td>CUBE_NAME</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>CUBOID_SOURCE</td>
+ <td>source cuboid parsed based on query and cube design</td>
+ </tr>
+ <tr>
+ <td>CUBOID_TARGET</td>
+ <td>target cuboid already precalculated and served for source cuboid</td>
+ </tr>
+ <tr>
+ <td>IF_MATCH</td>
+ <td>whether source cuboid and target cuboid are equal</td>
+ </tr>
+ <tr>
+ <td>IF_SUCCESS</td>
+ <td>whether a query on this cube is successful or not</td>
+ </tr>
+</table>
+
+<table>
+ <tr>
+ <th colspan="2">Measure</th>
+ </tr>
+ <tr>
+ <td>COUNT</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of STORAGE_CALL_COUNT</td>
+ <td>the number of rpc calls for a query hit on this cube</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of STORAGE_CALL_TIME_SUM</td>
+ <td>sum of time cost for the rpc calls of a query</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of STORAGE_CALL_TIME_MAX</td>
+ <td>max of time cost among the rpc calls of a query</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of STORAGE_COUNT_SKIP</td>
+ <td>the sum of row count skipped for the related rpc calls</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of STORAGE_SIZE_SCAN</td>
+ <td>the sum of row count scanned for the related rpc calls</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of STORAGE_SIZE_RETURN</td>
+ <td>the sum of row count returned for the related rpc calls</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of STORAGE_SIZE_AGGREGATE</td>
+ <td>the sum of row count aggregated for the related rpc calls</td>
+ </tr>
+ <tr>
+ <td>MAX, SUM of STORAGE_SIZE_AGGREGATE_FILTER</td>
+ <td>the sum of row count aggregated and filtered for the related rpc calls, = STORAGE_SIZE_SCAN - STORAGE_SIZE_RETURN</td>
+ </tr>
+</table>
+
+### METRICS_JOB
+In kylin, there are mainly three types of job:
+- "BUILD",for building cube segments from **HIVE**.
+- "MERGE",for merging cube segments in **HBASE**.
+- "OPTIMIZE",for dynamically adjusting the precalculated cuboid tree base on the **base cuboid** in **HBASE**.
+
+This cube is for collecting job metrics. The details are as follows:
+
+<table>
+ <tr>
+ <th colspan="2">Dimension</th>
+ </tr>
+ <tr>
+ <td>PROJECT</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>CUBE_NAME</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>JOB_TYPE</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>CUBING_TYPE</td>
+ <td>in kylin, there are two cubing algorithms, Layered & Fast(InMemory)</td>
+ </tr>
+</table>
+
+<table>
+ <tr>
+ <th colspan="2">Measure</th>
+ </tr>
+ <tr>
+ <td>COUNT</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>MIN, MAX, SUM of DURATION</td>
+ <td>the duration from a job start to finish</td>
+ </tr>
+ <tr>
+ <td>MIN, MAX, SUM of TABLE_SIZE</td>
+ <td>the size of data source in bytes</td>
+ </tr>
+ <tr>
+ <td>MIN, MAX, SUM of CUBE_SIZE</td>
+ <td>the size of created cube segment in bytes</td>
+ </tr>
+ <tr>
+ <td>MIN, MAX, SUM of PER_BYTES_TIME_COST</td>
+ <td>= DURATION / TABLE_SIZE</td>
+ </tr>
+ <tr>
+ <td>MIN, MAX, SUM of WAIT_RESOURCE_TIME</td>
+ <td>a job may includes serveral MR(map reduce) jobs. Those MR jobs may wait because of lack of Hadoop resources.</td>
+ </tr>
+</table>
+
+### METRICS_JOB_EXCEPTION
+This cube is for collecting job exception metrics. The details are as follows:
+
+<table>
+ <tr>
+ <th colspan="2">Dimension</th>
+ </tr>
+ <tr>
+ <td>PROJECT</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>CUBE_NAME</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>JOB_TYPE</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>CUBING_TYPE</td>
+ <td></td>
+ </tr>
+ <tr>
+ <td>EXCEPTION</td>
+ <td>when running a job, exceptions may happen. It's for classifying different exception types</td>
+ </tr>
+</table>
+
+<table>
+ <tr>
+ <th>Measure</th>
+ </tr>
+ <tr>
+ <td>COUNT</td>
+ </tr>
+</table>
http://git-wip-us.apache.org/repos/asf/kylin/blob/5c5914b6/website/images/SystemCube/hive_table.png
----------------------------------------------------------------------
diff --git a/website/images/SystemCube/hive_table.png b/website/images/SystemCube/hive_table.png
new file mode 100644
index 0000000..533de8c
Binary files /dev/null and b/website/images/SystemCube/hive_table.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/5c5914b6/website/images/SystemCube/kylin_system.png
----------------------------------------------------------------------
diff --git a/website/images/SystemCube/kylin_system.png b/website/images/SystemCube/kylin_system.png
new file mode 100644
index 0000000..e726c29
Binary files /dev/null and b/website/images/SystemCube/kylin_system.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/5c5914b6/website/images/SystemCube/metadata.png
----------------------------------------------------------------------
diff --git a/website/images/SystemCube/metadata.png b/website/images/SystemCube/metadata.png
new file mode 100644
index 0000000..d2303ab
Binary files /dev/null and b/website/images/SystemCube/metadata.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/5c5914b6/website/images/SystemCube/reload_metadata.png
----------------------------------------------------------------------
diff --git a/website/images/SystemCube/reload_metadata.png b/website/images/SystemCube/reload_metadata.png
new file mode 100644
index 0000000..0046f92
Binary files /dev/null and b/website/images/SystemCube/reload_metadata.png differ
[3/4] kylin git commit: APACHE-KYLIN-3155: Create a document for how
to use dashboard
Posted by li...@apache.org.
APACHE-KYLIN-3155: Create a document for how to use dashboard
Signed-off-by: lidongsjtu <li...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/3decd849
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/3decd849
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/3decd849
Branch: refs/heads/document
Commit: 3decd84956063e900926976556f233d27b5badd1
Parents: 7175ca9
Author: Kalin <qi...@ebay.com>
Authored: Mon Jan 8 04:39:24 2018 +0800
Committer: lidongsjtu <li...@apache.org>
Committed: Sun Jan 14 19:54:34 2018 +0800
----------------------------------------------------------------------
website/_docs21/howto/howto_use_dashboard.md | 102 +++++++++++++++++++
website/images/Dashboard/AVGBuildTimePerMB.jpg | Bin 0 -> 149647 bytes
website/images/Dashboard/AVGQueryLatency.jpg | Bin 0 -> 163229 bytes
website/images/Dashboard/Cube-Info-Page.png | Bin 0 -> 104311 bytes
.../images/Dashboard/Job-Link-Page-Waiting.png | Bin 0 -> 70926 bytes
website/images/Dashboard/Job-Link-Page.png | Bin 0 -> 73430 bytes
website/images/Dashboard/JobCount.jpg | Bin 0 -> 144877 bytes
website/images/Dashboard/Query-Link-Page.png | Bin 0 -> 74150 bytes
website/images/Dashboard/QueryCount.jpg | Bin 0 -> 162002 bytes
website/images/Dashboard/SelectPeriod.png | Bin 0 -> 28878 bytes
10 files changed, 102 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/_docs21/howto/howto_use_dashboard.md
----------------------------------------------------------------------
diff --git a/website/_docs21/howto/howto_use_dashboard.md b/website/_docs21/howto/howto_use_dashboard.md
new file mode 100644
index 0000000..120f4e2
--- /dev/null
+++ b/website/_docs21/howto/howto_use_dashboard.md
@@ -0,0 +1,102 @@
+---
+layout: docs21
+title: Use Dashboard
+categories: howto
+permalink: /docs21/howto/howto_use_dashboard.html
+---
+
+# Dashboard
+
+As a project owner, cube admin, do you want to know your cube usage metrics? Do you want to know how many queries are against your cube every day? What is the AVG query latency? Do you want to know the AVG cube build time per GB source data, which is very helpful to foresee the time cost of a coming cube build job? You can find all information from Kylin Dashboard.
+
+Kylin Dashboard shows useful cube usage statistics, which are very important to our customers.
+
+## How to use it
+
+#### Step 1:
+
+ Click the '**Dashboard**' button on the navigation bar.
+
+ There are 9 boxes on this page which you can operate.
+
+ The boxes represent different attributes, including '**Time Period**','**Total Cube Count**', '**Avg Cube Expansion**', '**Query Count**', '**Average Query Latency**', '**Job Count**', '**Average Build Time per MB**', '**Data grouped by Project**' and '**Data grouped by Time**'.
+
+
+
+#### Step 2:
+
+You should now click on the calender to modify the '**Time Period**'.
+
+
+
+- '**Time period**' is set deafult to **'Last 7 Days**'.
+
+- There are **2** ways to modify the time period, one is *using standard time period*s and the other is *customizing your time period*.
+
+ 1. If you want to *use standard time periods*, you can click on '**Last 7 Days**' to choose data only from last 7 days, or click on '**This Month**' to choose data only from this month, or click on '**Last Month**' to choose data only from last month.
+
+ 2. If you want to *customize your time period*, you can click on '**Custom Range**'.
+
+ There are **2** ways to customize the time period, one is *typing dates in the textfield* and the other is *selecting dates in the calender*.
+
+ 1. If you want to *type dates in the textfield*, please make sure that both dates are valid.
+ 2. If you want to *select dates in the calender*, please make sure that you have clicked on two specific dates.
+
+- After you have modified the time period, click '**Apply**' to apply the changes, click '**Cancel**' to give up the changes.
+
+#### Step 3:
+
+Now the data analysis will be changed and shown on the same page. (Important information has been pixilated.)
+
+- Numbers in '**Total Cube Count**' and '**Avg Cube Expansion**' are in **Blue**.
+
+ You can click the '**More Details**' in these two boxes and you will be led to the '**Model**' page.
+
+ 
+
+
+- Numbers in '**Query Count**', '**Average Query Latency**', '**Job Count**' and '**Average Build Time per MB**' are in **Green**.
+
+ You can click on these four rectangles to get detail infomation about the data you selected. The detail information will then be shown as diagrams and displayed in '**Data grouped by Project**' and '**Data grouped by Time**' boxes.
+
+ 1. '**Query Count**' and '**Average Query Latency**'
+
+ You can click on '**Query Count**' to get detail infomation.
+
+ 
+
+ You can click on '**Average Query Latency**' to get detail infomation.
+
+ 
+
+ You can click the '**More Details**' in these two boxes and you will be led to the '**Insight**' page.
+
+ 
+
+ 2. '**Job Count**' and '**Average Build Time per MB**'
+
+ You can click on '**Job Count**' to get detail infomation.
+
+ 
+
+ You can click on '**Average Build Time per MB**' to get detail information.
+
+ 
+
+ You can click the '**More Details**' in these two boxes and you will be led to the '**Monitor**' page.
+
+ 
+
+ It is common to see the browser showing 'Please wait...'.
+
+ 
+
+#### Step 4:
+
+**Advanced Operations**
+
+'**Data grouped by Project**' and '**Data grouped by Time**' displayed data in the form of diagram.
+
+There is a radio button called '**showValue**' in '**Data grouped by Project**', you can choose to show number in the diagram.
+
+There is a radio drop-down menu in '**Data grouped by Time**', you can choose to show the diagram in different timelines.
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/images/Dashboard/AVGBuildTimePerMB.jpg
----------------------------------------------------------------------
diff --git a/website/images/Dashboard/AVGBuildTimePerMB.jpg b/website/images/Dashboard/AVGBuildTimePerMB.jpg
new file mode 100644
index 0000000..e9b46a5
Binary files /dev/null and b/website/images/Dashboard/AVGBuildTimePerMB.jpg differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/images/Dashboard/AVGQueryLatency.jpg
----------------------------------------------------------------------
diff --git a/website/images/Dashboard/AVGQueryLatency.jpg b/website/images/Dashboard/AVGQueryLatency.jpg
new file mode 100644
index 0000000..a84fc86
Binary files /dev/null and b/website/images/Dashboard/AVGQueryLatency.jpg differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/images/Dashboard/Cube-Info-Page.png
----------------------------------------------------------------------
diff --git a/website/images/Dashboard/Cube-Info-Page.png b/website/images/Dashboard/Cube-Info-Page.png
new file mode 100644
index 0000000..f2b2cd5
Binary files /dev/null and b/website/images/Dashboard/Cube-Info-Page.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/images/Dashboard/Job-Link-Page-Waiting.png
----------------------------------------------------------------------
diff --git a/website/images/Dashboard/Job-Link-Page-Waiting.png b/website/images/Dashboard/Job-Link-Page-Waiting.png
new file mode 100644
index 0000000..95b0512
Binary files /dev/null and b/website/images/Dashboard/Job-Link-Page-Waiting.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/images/Dashboard/Job-Link-Page.png
----------------------------------------------------------------------
diff --git a/website/images/Dashboard/Job-Link-Page.png b/website/images/Dashboard/Job-Link-Page.png
new file mode 100644
index 0000000..4b00459
Binary files /dev/null and b/website/images/Dashboard/Job-Link-Page.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/images/Dashboard/JobCount.jpg
----------------------------------------------------------------------
diff --git a/website/images/Dashboard/JobCount.jpg b/website/images/Dashboard/JobCount.jpg
new file mode 100644
index 0000000..945eff4
Binary files /dev/null and b/website/images/Dashboard/JobCount.jpg differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/images/Dashboard/Query-Link-Page.png
----------------------------------------------------------------------
diff --git a/website/images/Dashboard/Query-Link-Page.png b/website/images/Dashboard/Query-Link-Page.png
new file mode 100644
index 0000000..eee9493
Binary files /dev/null and b/website/images/Dashboard/Query-Link-Page.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/images/Dashboard/QueryCount.jpg
----------------------------------------------------------------------
diff --git a/website/images/Dashboard/QueryCount.jpg b/website/images/Dashboard/QueryCount.jpg
new file mode 100644
index 0000000..18f8c32
Binary files /dev/null and b/website/images/Dashboard/QueryCount.jpg differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/3decd849/website/images/Dashboard/SelectPeriod.png
----------------------------------------------------------------------
diff --git a/website/images/Dashboard/SelectPeriod.png b/website/images/Dashboard/SelectPeriod.png
new file mode 100755
index 0000000..4fa0940
Binary files /dev/null and b/website/images/Dashboard/SelectPeriod.png differ
[2/4] kylin git commit: APACHE-KYLIN-3154: Create a document for cube
planner
Posted by li...@apache.org.
APACHE-KYLIN-3154: Create a document for cube planner
Signed-off-by: Zhong <nj...@apache.org>
Signed-off-by: lidongsjtu <li...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/7175ca94
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/7175ca94
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/7175ca94
Branch: refs/heads/document
Commit: 7175ca94aae712d49a83c1eb90dcc38e57706e5f
Parents: 5c5914b
Author: Kalin <qi...@ebay.com>
Authored: Mon Jan 8 04:26:22 2018 +0800
Committer: lidongsjtu <li...@apache.org>
Committed: Sun Jan 14 19:54:29 2018 +0800
----------------------------------------------------------------------
website/_docs21/howto/howto_use_cube_planner.md | 127 +++++++++++++++++++
website/images/CubePlanner/CP.png | Bin 0 -> 95460 bytes
website/images/CubePlanner/CPRecom.png | Bin 0 -> 142956 bytes
website/images/CubePlanner/CubePlanner.png | Bin 0 -> 272336 bytes
.../images/CubePlanner/CubePlanner_Optimize.png | Bin 0 -> 69277 bytes
website/images/CubePlanner/DISABLED2READY.png | Bin 0 -> 12155 bytes
.../images/CubePlanner/Leaf-Specify-Parent.png | Bin 0 -> 56938 bytes
website/images/CubePlanner/Leaf-Specify.png | Bin 0 -> 57581 bytes
website/images/CubePlanner/Leaf.png | Bin 0 -> 96579 bytes
website/images/CubePlanner/Recommending.png | Bin 0 -> 74217 bytes
website/images/CubePlanner/Root.png | Bin 0 -> 96658 bytes
website/images/CubePlanner/Rowkeys.png | Bin 0 -> 42367 bytes
website/images/CubePlanner/Running.png | Bin 0 -> 2903 bytes
website/images/CubePlanner/Status_Disabled.png | Bin 0 -> 2435 bytes
website/images/CubePlanner/Status_Ready.png | Bin 0 -> 2209 bytes
.../CubePlanner/column_name+optimize_time.png | Bin 0 -> 365858 bytes
website/images/CubePlanner/export_cuboids.png | Bin 0 -> 343238 bytes
website/images/CubePlanner/optimize_email.png | Bin 0 -> 155299 bytes
18 files changed, 127 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/_docs21/howto/howto_use_cube_planner.md
----------------------------------------------------------------------
diff --git a/website/_docs21/howto/howto_use_cube_planner.md b/website/_docs21/howto/howto_use_cube_planner.md
new file mode 100644
index 0000000..3ae412c
--- /dev/null
+++ b/website/_docs21/howto/howto_use_cube_planner.md
@@ -0,0 +1,127 @@
+---
+layout: docs21
+title: Use Cube Planner
+categories: howto
+permalink: /docs21/howto/howto_use_cube_planner.html
+---
+
+# Cube Planner
+
+## What is Cube Planner
+
+OLAP solution trades off online query speed with offline cube build cost (compute resource to build cube and storage resource to save the cube data). Resource efficiency is the most important competency of OLAP engine. To be resource efficient, It is critical to just pre-build the most valuable cuboids.
+
+Cube Planner makes Apache Kylin to be more resource efficient. It intelligently build a partial cube to minimize the cost of building a cube and at the same time maximize the benefit of serving end user queries, then learn patterns from queries at runtime and dynamically recommend cuboids accordingly.
+
+
+
+## How to use it
+
+*Note: Cube planner optimization is not suitable for new cube. Cube should be online on production for a while (like 3 months) before optimizing it. So that Kylin platform collects enough real queries from end user and use them to optimize the cube.*
+
+#### Step 1:
+
+ Select a cube
+
+#### Step 2:
+
+1. Click the '**Planner**' button to view the '**Current Cuboid Distribution**' of the cube.
+
+ You should make sure the status of the cube is '**READY**'.
+
+ If the status of the cube is '**DISABLED**', you will not be able to use the cube planner.
+
+
+
+ You should change the status of the cube from '**DISABLED**' to '**READY**' by clicking the '**Enable**' button.
+
+ 
+
+#### Step 3:
+
+a. Click the '**Planner**' button to view the '**Current Cuboid Distribution**' of the cube.
+
+- The data will be displayed in *[Sunburst Chart](https://en.wikipedia.org/wiki/Pie_chart#Ring_chart_.2F_Sunburst_chart_.2F_Multilevel_pie_chart)*.
+
+- Each part refers to a cuboid, is shown in different colors determined by the query **frequency** against this cuboid.
+
+ 
+
+
+- You can move the cursor over the chart and it will display the detail information of the cuboid.
+
+ The detail information contains 5 attributes, '**Name**', '**ID**', '**Query Count**', '**Exactly Match Count**', '**Row Count**' and '**Rollup Rate**'.
+
+ Cuboid **Name** is composed of several '0' or '1'. It means a combination of dimensions. '0' means the dimension doesn't exist in this combination, while '1' means the dimension exist in the combination. All the dimensions are ordered by the HBase row keys in advanced settings.
+
+ Here is an example:
+
+ 
+
+ Name:1111111110000000 means the dimension combination is ["MONTH_BEG_DT","USER_CNTRY_SITE_CD","RPRTD_SGMNT_VAL","RPRTD_IND","SRVY_TYPE_ID","QSTN_ID","L1_L2_IND","PRNT_L1_ID","TRANCHE_ID"] based on the row key orders below:
+
+ 
+
+ **ID** is the unique id of the cuboid.
+
+ **Query Count** is the total count of the queries that are served by this cuboid, including those queries that against other un-precalculated cuboids, but on line aggregated from this cuboid.
+
+ **Exactly Match Count** is the query count that the query is actually against this cuboid.
+
+ **Row Count** is the total row count of all the segments for this cuboid.
+
+ **Rollup Rate** = (Cuboid's Row Count/its parent cuboid's Row Count) * 100%
+
+- The center of the sunburst chart contains the combined information of basic cuboid. its '**Name**' is composed of several '1's.
+
+
+
+As for a leaf, its '**Name**' is composed of several '0's and 1's.
+
+
+
+- If you want to **specify** a leaf, just **click on** it. The view will change automatically.
+
+ 
+
+- If you want to specify the parent leaf of a leaf, click on the **center circle** (the part marked yellow).
+
+
+
+b. Click the '**Recommend**' button to view the '**Recommend Cuboid Distribution**' of the cube.
+
+If the cube is currently under building, the cube planner '**Recommend**' function will not be able to perform correctly. Please first **stop the building progress** of the cube.
+
+- The data will be calculated by unique algorithms. It is common to see this window.
+
+ 
+
+- The data will be displayed in *[Sunburst Chart](https://en.wikipedia.org/wiki/Pie_chart#Ring_chart_.2F_Sunburst_chart_.2F_Multilevel_pie_chart)*.
+
+ - Each part is shown in different colors determined by the **frequency**.
+
+
+
+- Detailed operation of the '**Recommend Cuboid Distribution**' chart is the same as '**Current Cuboid Distribution**' chart.
+- User is able to tell the dimension names from a cuboid when mouse hovers over the sunburst chart as figure shown below.
+- User is able to click **'Export'** to export hot dimension combinations (TopN cuboids, currently including options of Top 10, Top 50, Top 100) from an existing cube as a json file, which will be downloaded to your local file system for recording or future import of dimension combinations when creating cube.
+
+
+
+c. Click the '**Optimize**' button to optimize the cube.
+
+- A window will jump up to ensure your decision.
+
+ 
+
+ Click '**Yes**' to start the optimization.
+
+ Click '**Cancel**' to give up the optimization.
+
+- User is able to get to know the last optimized time of the cube in Cube Planner tab page.
+
+
+
+- User is able to receive an email notification for a cube optimization job.
+
+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/CP.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/CP.png b/website/images/CubePlanner/CP.png
new file mode 100755
index 0000000..4576a36
Binary files /dev/null and b/website/images/CubePlanner/CP.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/CPRecom.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/CPRecom.png b/website/images/CubePlanner/CPRecom.png
new file mode 100755
index 0000000..586956a
Binary files /dev/null and b/website/images/CubePlanner/CPRecom.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/CubePlanner.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/CubePlanner.png b/website/images/CubePlanner/CubePlanner.png
new file mode 100644
index 0000000..03055a4
Binary files /dev/null and b/website/images/CubePlanner/CubePlanner.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/CubePlanner_Optimize.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/CubePlanner_Optimize.png b/website/images/CubePlanner/CubePlanner_Optimize.png
new file mode 100755
index 0000000..a95c33f
Binary files /dev/null and b/website/images/CubePlanner/CubePlanner_Optimize.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/DISABLED2READY.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/DISABLED2READY.png b/website/images/CubePlanner/DISABLED2READY.png
new file mode 100755
index 0000000..023b3ca
Binary files /dev/null and b/website/images/CubePlanner/DISABLED2READY.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/Leaf-Specify-Parent.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/Leaf-Specify-Parent.png b/website/images/CubePlanner/Leaf-Specify-Parent.png
new file mode 100755
index 0000000..e4de0df
Binary files /dev/null and b/website/images/CubePlanner/Leaf-Specify-Parent.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/Leaf-Specify.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/Leaf-Specify.png b/website/images/CubePlanner/Leaf-Specify.png
new file mode 100755
index 0000000..e128d53
Binary files /dev/null and b/website/images/CubePlanner/Leaf-Specify.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/Leaf.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/Leaf.png b/website/images/CubePlanner/Leaf.png
new file mode 100755
index 0000000..1acc00d
Binary files /dev/null and b/website/images/CubePlanner/Leaf.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/Recommending.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/Recommending.png b/website/images/CubePlanner/Recommending.png
new file mode 100755
index 0000000..022952f
Binary files /dev/null and b/website/images/CubePlanner/Recommending.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/Root.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/Root.png b/website/images/CubePlanner/Root.png
new file mode 100755
index 0000000..f7a3478
Binary files /dev/null and b/website/images/CubePlanner/Root.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/Rowkeys.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/Rowkeys.png b/website/images/CubePlanner/Rowkeys.png
new file mode 100644
index 0000000..8dfbcac
Binary files /dev/null and b/website/images/CubePlanner/Rowkeys.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/Running.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/Running.png b/website/images/CubePlanner/Running.png
new file mode 100755
index 0000000..a9f0e96
Binary files /dev/null and b/website/images/CubePlanner/Running.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/Status_Disabled.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/Status_Disabled.png b/website/images/CubePlanner/Status_Disabled.png
new file mode 100755
index 0000000..74a1228
Binary files /dev/null and b/website/images/CubePlanner/Status_Disabled.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/Status_Ready.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/Status_Ready.png b/website/images/CubePlanner/Status_Ready.png
new file mode 100755
index 0000000..71fcbd5
Binary files /dev/null and b/website/images/CubePlanner/Status_Ready.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/column_name+optimize_time.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/column_name+optimize_time.png b/website/images/CubePlanner/column_name+optimize_time.png
new file mode 100644
index 0000000..b17385d
Binary files /dev/null and b/website/images/CubePlanner/column_name+optimize_time.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/export_cuboids.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/export_cuboids.png b/website/images/CubePlanner/export_cuboids.png
new file mode 100644
index 0000000..e4077cc
Binary files /dev/null and b/website/images/CubePlanner/export_cuboids.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/7175ca94/website/images/CubePlanner/optimize_email.png
----------------------------------------------------------------------
diff --git a/website/images/CubePlanner/optimize_email.png b/website/images/CubePlanner/optimize_email.png
new file mode 100644
index 0000000..37812b8
Binary files /dev/null and b/website/images/CubePlanner/optimize_email.png differ
[4/4] kylin git commit: KYLIN-3154 & KYLIN-3155 review
Posted by li...@apache.org.
KYLIN-3154 & KYLIN-3155 review
Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/2a282eeb
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/2a282eeb
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/2a282eeb
Branch: refs/heads/document
Commit: 2a282eeb87cd44f9398755c0f67d35a761b08a77
Parents: 3decd84
Author: lidongsjtu <li...@apache.org>
Authored: Tue Jan 9 20:06:30 2018 +0800
Committer: lidongsjtu <li...@apache.org>
Committed: Sun Jan 14 20:27:13 2018 +0800
----------------------------------------------------------------------
website/_data/docs21.yml | 3 +++
website/_docs21/howto/howto_setup_systemcube.md | 2 +-
website/_docs21/howto/howto_use_cube_planner.md | 8 +++++++-
website/_docs21/howto/howto_use_dashboard.md | 12 ++++++++++--
4 files changed, 21 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kylin/blob/2a282eeb/website/_data/docs21.yml
----------------------------------------------------------------------
diff --git a/website/_data/docs21.yml b/website/_data/docs21.yml
index 9a6e4de..1d5e8db 100644
--- a/website/_data/docs21.yml
+++ b/website/_data/docs21.yml
@@ -74,3 +74,6 @@
- howto/howto_update_coprocessor
- howto/howto_install_ranger_kylin_plugin
- howto/howto_enable_zookeeper_acl
+ - howto/howto_setup_systemcube
+ - howto/howto_use_cube_planner
+ - howto/howto_use_dashboard
http://git-wip-us.apache.org/repos/asf/kylin/blob/2a282eeb/website/_docs21/howto/howto_setup_systemcube.md
----------------------------------------------------------------------
diff --git a/website/_docs21/howto/howto_setup_systemcube.md b/website/_docs21/howto/howto_setup_systemcube.md
index e8a46d2..8f05554 100644
--- a/website/_docs21/howto/howto_setup_systemcube.md
+++ b/website/_docs21/howto/howto_setup_systemcube.md
@@ -5,7 +5,7 @@ categories: howto
permalink: /docs21/howto/howto_setup_systemcube.html
---
-# Set Up System Cube
+> Available since Apache Kylin v2.3.x
## What is System Cube
http://git-wip-us.apache.org/repos/asf/kylin/blob/2a282eeb/website/_docs21/howto/howto_use_cube_planner.md
----------------------------------------------------------------------
diff --git a/website/_docs21/howto/howto_use_cube_planner.md b/website/_docs21/howto/howto_use_cube_planner.md
index 3ae412c..ab216c2 100644
--- a/website/_docs21/howto/howto_use_cube_planner.md
+++ b/website/_docs21/howto/howto_use_cube_planner.md
@@ -5,6 +5,8 @@ categories: howto
permalink: /docs21/howto/howto_use_cube_planner.html
---
+> Available since Apache Kylin v2.3.0
+
# Cube Planner
## What is Cube Planner
@@ -15,6 +17,10 @@ Cube Planner makes Apache Kylin to be more resource efficient. It intelligently
+## Prerequisites
+
+To enable Dashboard on WebUI, you need to set **kylin.cube.cubeplanner.enabled=true** in **kylin.properties**.
+
## How to use it
*Note: Cube planner optimization is not suitable for new cube. Cube should be online on production for a while (like 3 months) before optimizing it. So that Kylin platform collects enough real queries from end user and use them to optimize the cube.*
@@ -124,4 +130,4 @@ c. Click the '**Optimize**' button to optimize the cube.
- User is able to receive an email notification for a cube optimization job.
-
\ No newline at end of file
+
http://git-wip-us.apache.org/repos/asf/kylin/blob/2a282eeb/website/_docs21/howto/howto_use_dashboard.md
----------------------------------------------------------------------
diff --git a/website/_docs21/howto/howto_use_dashboard.md b/website/_docs21/howto/howto_use_dashboard.md
index 120f4e2..b90b9f9 100644
--- a/website/_docs21/howto/howto_use_dashboard.md
+++ b/website/_docs21/howto/howto_use_dashboard.md
@@ -5,11 +5,19 @@ categories: howto
permalink: /docs21/howto/howto_use_dashboard.html
---
+> Available since Apache Kylin v2.3.0
+
# Dashboard
-As a project owner, cube admin, do you want to know your cube usage metrics? Do you want to know how many queries are against your cube every day? What is the AVG query latency? Do you want to know the AVG cube build time per GB source data, which is very helpful to foresee the time cost of a coming cube build job? You can find all information from Kylin Dashboard.
+As a project owner, do you want to know your cube usage metrics? Do you want to know how many queries are against your cube every day? What is the AVG query latency? Do you want to know the AVG cube build time per GB source data, which is very helpful to foresee the time cost of a coming cube build job? You can find all information from Kylin Dashboard.
+
+Kylin Dashboard shows useful cube usage statistics, which are very important to users.
+
+## Prerequisites
-Kylin Dashboard shows useful cube usage statistics, which are very important to our customers.
+To enable Dashboard on WebUI, you need to ensure these are all set:
+* Set **kylin.web.dashboard-enabled=true** in **kylin.properties**.
+* Setup system cubes according to .
## How to use it