You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by xx...@apache.org on 2021/09/06 02:19:56 UTC
[kylin] branch document updated: Modify docs40 to docs and some fix
This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a commit to branch document
in repository https://gitbox.apache.org/repos/asf/kylin.git
The following commit(s) were added to refs/heads/document by this push:
new c3dbf35 Modify docs40 to docs and some fix
c3dbf35 is described below
commit c3dbf35aaa13dd0182879f552470e8f1a65b8d19
Author: yaqian.zhang <59...@qq.com>
AuthorDate: Fri Sep 3 15:07:29 2021 +0800
Modify docs40 to docs and some fix
---
website/_data/docs-cn.yml | 18 +-
website/_data/docs.yml | 23 +-
website/_data/{docs-cn.yml => docs31-cn.yml} | 0
website/_data/{docs.yml => docs31.yml} | 0
website/_data/docs40-cn.yml | 66 -
website/_data/docs40.yml | 85 -
website/_docs/gettingstarted/concepts.md | 72 +-
website/_docs/gettingstarted/faq.cn.md | 197 +-
website/_docs/gettingstarted/faq.md | 290 +--
website/_docs/gettingstarted/quickstart.cn.md | 157 +-
website/_docs/gettingstarted/quickstart.md | 146 +-
website/_docs/gettingstarted/terminology.md | 2 +-
website/_docs/howto/howto_backup_metadata.cn.md | 37 +-
website/_docs/howto/howto_backup_metadata.md | 35 +-
.../howto/howto_build_cube_with_restapi.cn.md | 3 +
.../_docs/howto/howto_build_cube_with_restapi.md | 3 +
website/_docs/howto/howto_cleanup_storage.cn.md | 20 +-
website/_docs/howto/howto_cleanup_storage.md | 21 +-
.../howto/howto_config_spark_pool.md | 4 +-
.../howto/howto_optimize_build_and_query.cn.md | 4 +-
.../howto/howto_optimize_build_and_query.md | 4 +-
website/_docs/howto/howto_upgrade.md | 118 +-
website/_docs/howto/howto_use_cli.cn.md | 37 +-
website/_docs/howto/howto_use_cli.md | 41 +-
website/_docs/howto/howto_use_restapi.cn.md | 2169 +-------------------
website/_docs/howto/howto_use_restapi.md | 2169 +-------------------
website/_docs/index.cn.md | 179 +-
website/_docs/index.md | 184 +-
website/_docs/install/advance_settings.cn.md | 92 -
website/_docs/install/advance_settings.md | 96 +-
website/_docs/install/configuration.cn.md | 834 +-------
website/_docs/install/configuration.md | 822 +-------
.../install/deploy_without_hadoop.cn.md | 4 +-
.../install/deploy_without_hadoop.md | 4 +-
website/_docs/install/index.cn.md | 91 +-
website/_docs/install/index.md | 86 +-
website/_docs/install/kylin_cluster.cn.md | 28 +-
website/_docs/install/kylin_cluster.md | 26 +-
website/_docs/install/kylin_docker.cn.md | 23 +-
website/_docs/install/kylin_docker.md | 21 +-
website/_docs/release_notes.md | 55 +-
website/_docs/security.md | 26 -
website/_docs/tutorial/create_cube.cn.md | 29 +-
website/_docs/tutorial/create_cube.md | 12 +-
website/_docs/tutorial/cube_build_job.cn.md | 30 +-
website/_docs/tutorial/cube_build_job.md | 31 +-
website/_docs/tutorial/query_pushdown.cn.md | 91 +-
website/_docs/tutorial/query_pushdown.md | 99 +-
website/_docs/tutorial/setup_systemcube.cn.md | 412 +---
website/_docs/tutorial/setup_systemcube.md | 402 +---
website/_docs/tutorial/sql_reference.cn.md | 2 +-
website/_docs/tutorial/use_cube_planner.cn.md | 95 +-
website/_docs/tutorial/use_cube_planner.md | 95 +-
website/_docs/tutorial/use_dashboard.cn.md | 1 -
website/_docs/tutorial/use_dashboard.md | 2 -
.../gettingstarted/best_practices.md | 4 +-
.../{_docs => _docs31}/gettingstarted/concepts.md | 4 +-
.../{_docs40 => _docs31}/gettingstarted/events.md | 4 +-
.../{_docs => _docs31}/gettingstarted/faq.cn.md | 6 +-
website/{_docs => _docs31}/gettingstarted/faq.md | 8 +-
.../gettingstarted/quickstart.cn.md | 163 +-
.../gettingstarted/quickstart.md | 152 +-
.../gettingstarted/terminology.md | 6 +-
.../howto/howto_backup_metadata.cn.md | 41 +-
.../howto/howto_backup_metadata.md | 39 +-
.../howto/howto_build_cube_with_restapi.cn.md | 7 +-
.../howto/howto_build_cube_with_restapi.md | 7 +-
.../howto/howto_cleanup_storage.cn.md | 4 +-
.../howto/howto_cleanup_storage.md | 4 +-
.../howto/howto_enable_zookeeper_acl.md | 4 +-
.../howto/howto_install_ranger_kylin_plugin.md | 4 +-
.../{_docs40 => _docs31}/howto/howto_jdbc.cn.md | 4 +-
website/{_docs40 => _docs31}/howto/howto_jdbc.md | 4 +-
.../howto/howto_ldap_and_sso.md | 6 +-
.../howto/howto_optimize_cubes.cn.md | 4 +-
.../howto/howto_optimize_cubes.md | 4 +-
.../howto/howto_update_coprocessor.md | 4 +-
website/{_docs => _docs31}/howto/howto_upgrade.md | 4 +-
.../{_docs => _docs31}/howto/howto_use_beeline.md | 4 +-
.../{_docs40 => _docs31}/howto/howto_use_cli.cn.md | 41 +-
.../{_docs40 => _docs31}/howto/howto_use_cli.md | 45 +-
.../howto/howto_use_distributed_scheduler.md | 4 +-
.../howto/howto_use_mr_hive_dict.cn.md | 4 +-
.../howto/howto_use_mr_hive_dict.md | 4 +-
.../howto/howto_use_restapi.cn.md | 4 +-
.../{_docs => _docs31}/howto/howto_use_restapi.md | 4 +-
.../howto/howto_use_restapi_in_js.md | 4 +-
.../{_docs => _docs31}/howto/sample_dataset.cn.md | 4 +-
website/{_docs => _docs31}/howto/sample_dataset.md | 4 +-
website/{_docs => _docs31}/index.cn.md | 4 +-
website/{_docs => _docs31}/index.md | 4 +-
.../install/Kylin_kubernetes.cn.md | 4 +-
.../{_docs => _docs31}/install/Kylin_kubernetes.md | 4 +-
.../install/advance_settings.cn.md | 4 +-
.../{_docs => _docs31}/install/advance_settings.md | 4 +-
.../{_docs => _docs31}/install/configuration.cn.md | 18 +-
.../{_docs => _docs31}/install/configuration.md | 18 +-
website/{_docs => _docs31}/install/index.cn.md | 10 +-
website/{_docs => _docs31}/install/index.md | 10 +-
.../{_docs => _docs31}/install/kylin_aws_emr.cn.md | 4 +-
.../{_docs => _docs31}/install/kylin_aws_emr.md | 4 +-
.../install/kylin_cluster.cn.md | 32 +-
.../{_docs40 => _docs31}/install/kylin_cluster.md | 30 +-
.../install/kylin_docker.cn.md | 25 +-
.../{_docs40 => _docs31}/install/kylin_docker.md | 25 +-
website/{_docs40 => _docs31}/release_notes.md | 71 +-
website/{_docs40 => _docs31}/security.md | 30 +-
website/{_docs40 => _docs31}/tutorial/Qlik.cn.md | 6 +-
website/{_docs40 => _docs31}/tutorial/Qlik.md | 6 +-
website/{_docs40 => _docs31}/tutorial/acl.cn.md | 6 +-
website/{_docs40 => _docs31}/tutorial/acl.md | 6 +-
.../tutorial/create_cube.cn.md | 33 +-
.../{_docs40 => _docs31}/tutorial/create_cube.md | 16 +-
.../tutorial/cube_build_job.cn.md | 34 +-
.../tutorial/cube_build_job.md | 35 +-
.../tutorial/cube_build_performance.cn.md | 4 +-
.../tutorial/cube_build_performance.md | 4 +-
.../{_docs => _docs31}/tutorial/cube_flink.cn.md | 4 +-
website/{_docs => _docs31}/tutorial/cube_flink.md | 4 +-
.../tutorial/cube_migration.cn.md | 4 +-
.../{_docs => _docs31}/tutorial/cube_migration.md | 4 +-
.../{_docs => _docs31}/tutorial/cube_spark.cn.md | 4 +-
website/{_docs => _docs31}/tutorial/cube_spark.md | 4 +-
.../tutorial/cube_streaming.cn.md | 6 +-
.../{_docs => _docs31}/tutorial/cube_streaming.md | 6 +-
website/{_docs => _docs31}/tutorial/flink.md | 4 +-
website/{_docs => _docs31}/tutorial/hue.md | 8 +-
website/{_docs => _docs31}/tutorial/hybrid.cn.md | 4 +-
website/{_docs => _docs31}/tutorial/hybrid.md | 4 +-
website/{_docs40 => _docs31}/tutorial/jdbc.cn.md | 4 +-
website/{_docs40 => _docs31}/tutorial/jdbc.md | 4 +-
.../tutorial/kylin_client_tool.cn.md | 4 +-
.../tutorial/kylin_client_tool.md | 4 +-
.../{_docs => _docs31}/tutorial/kylin_sample.cn.md | 4 +-
.../{_docs => _docs31}/tutorial/kylin_sample.md | 4 +-
.../lambda_mode_and_timezone_realtime_olap.md | 6 +-
.../{_docs40 => _docs31}/tutorial/microstrategy.md | 6 +-
.../tutorial/mysql_metastore.cn.md | 4 +-
.../tutorial/mysql_metastore.md | 4 +-
website/{_docs40 => _docs31}/tutorial/odbc.cn.md | 4 +-
website/{_docs40 => _docs31}/tutorial/odbc.md | 4 +-
.../{_docs40 => _docs31}/tutorial/powerbi.cn.md | 4 +-
website/{_docs40 => _docs31}/tutorial/powerbi.md | 4 +-
.../tutorial/project_level_acl.cn.md | 4 +-
.../tutorial/project_level_acl.md | 4 +-
.../tutorial/query_pushdown.cn.md | 4 +-
.../{_docs => _docs31}/tutorial/query_pushdown.md | 4 +-
.../{_docs => _docs31}/tutorial/real_time_olap.md | 10 +-
.../tutorial/setup_jdbc_datasource.cn.md | 4 +-
.../tutorial/setup_jdbc_datasource.md | 6 +-
.../tutorial/setup_systemcube.cn.md | 4 +-
.../tutorial/setup_systemcube.md | 4 +-
website/{_docs => _docs31}/tutorial/spark.cn.md | 4 +-
website/{_docs => _docs31}/tutorial/spark.md | 4 +-
.../tutorial/sql_reference.cn.md | 6 +-
.../{_docs40 => _docs31}/tutorial/sql_reference.md | 4 +-
.../{_docs40 => _docs31}/tutorial/squirrel.cn.md | 4 +-
website/{_docs40 => _docs31}/tutorial/squirrel.md | 4 +-
.../{_docs40 => _docs31}/tutorial/superset.cn.md | 4 +-
website/{_docs40 => _docs31}/tutorial/superset.md | 4 +-
.../{_docs40 => _docs31}/tutorial/tableau.cn.md | 4 +-
website/{_docs40 => _docs31}/tutorial/tableau.md | 4 +-
.../{_docs40 => _docs31}/tutorial/tableau_91.cn.md | 4 +-
.../{_docs40 => _docs31}/tutorial/tableau_91.md | 4 +-
.../tutorial/use_cube_planner.cn.md | 4 +-
.../tutorial/use_cube_planner.md | 4 +-
.../tutorial/use_dashboard.cn.md | 5 +-
.../{_docs40 => _docs31}/tutorial/use_dashboard.md | 6 +-
website/{_docs40 => _docs31}/tutorial/web.cn.md | 4 +-
website/{_docs40 => _docs31}/tutorial/web.md | 4 +-
website/_docs40/gettingstarted/concepts.md | 64 -
website/_docs40/gettingstarted/faq.cn.md | 18 -
website/_docs40/gettingstarted/faq.md | 16 -
website/_docs40/howto/howto_cleanup_storage.cn.md | 8 -
website/_docs40/howto/howto_cleanup_storage.md | 8 -
website/_docs40/howto/howto_upgrade.md | 10 -
website/_docs40/howto/howto_use_restapi.cn.md | 10 -
website/_docs40/howto/howto_use_restapi.md | 10 -
website/_docs40/index.cn.md | 147 --
website/_docs40/index.md | 149 --
website/_docs40/install/advance_settings.cn.md | 98 -
website/_docs40/install/advance_settings.md | 95 -
website/_docs40/install/configuration.cn.md | 10 -
website/_docs40/install/configuration.md | 9 -
website/_docs40/install/index.cn.md | 133 --
website/_docs40/install/index.md | 137 --
website/_docs40/tutorial/query_pushdown.cn.md | 28 -
website/_docs40/tutorial/query_pushdown.md | 29 -
website/_docs40/tutorial/setup_systemcube.cn.md | 147 --
website/_docs40/tutorial/setup_systemcube.md | 146 --
website/_docs40/tutorial/use_cube_planner.cn.md | 38 -
website/_docs40/tutorial/use_cube_planner.md | 39 -
website/_includes/header.cn.html | 12 +-
website/_includes/header.html | 10 +-
website/_includes/home_header.cn.html | 12 +-
website/_includes/home_header.html | 10 +-
website/download/index.cn.md | 8 +-
website/download/index.md | 6 +-
198 files changed, 1526 insertions(+), 10771 deletions(-)
diff --git a/website/_data/docs-cn.yml b/website/_data/docs-cn.yml
index 764477c..f0de5b3 100644
--- a/website/_data/docs-cn.yml
+++ b/website/_data/docs-cn.yml
@@ -21,11 +21,11 @@
- title: 安装
docs:
- install/index
+ - install/deploy_without_hadoop
- install/kylin_cluster
- - install/configuration
- - install/kylin_aws_emr
- install/kylin_docker
- - install/kylin_on_kubernetes
+ - install/configuration
+ - install/advance_settings
- title: 教程
docs:
@@ -33,22 +33,14 @@
- tutorial/web
- tutorial/create_cube
- tutorial/cube_build_job
- - tutorial/cube_migration
- tutorial/sql_reference
- tutorial/project_level_acl
- - tutorial/cube_spark
- - tutorial/cube_flink
- - tutorial/cube_streaming
- - tutorial/realtime_olap
- - tutorial/cube_build_performance
- tutorial/kylin_client_tool
- tutorial/setup_systemcube
- tutorial/use_cube_planner
- tutorial/use_dashboard
- - tutorial/setup_jdbc_datasource
- tutorial/hybrid
- tutorial/mysql_metastore
- - tutorial/query_pushdown
- title: 工具集成
docs:
@@ -68,9 +60,7 @@
- howto/howto_use_restapi
- howto/howto_build_cube_with_restapi
- howto/howto_optimize_cubes
- - howto/howto_optimize_build
+ - howto/howto_optimize_build_and_query
- howto/howto_backup_metadata
- howto/howto_cleanup_storage
- howto/howto_use_cli
- - howto/howto_use_hive_mr_dict
- - howto/sample_dataset
diff --git a/website/_data/docs.yml b/website/_data/docs.yml
index 90af718..2c07665 100644
--- a/website/_data/docs.yml
+++ b/website/_data/docs.yml
@@ -29,11 +29,11 @@
- title: Installation
docs:
- install/index
+ - install/deploy_without_hadoop
- install/kylin_cluster
- - install/configuration
- - install/kylin_aws_emr
- install/kylin_docker
- - install/kylin_on_kubernetes
+ - install/configuration
+ - install/advance_settings
- title: Tutorial
docs:
@@ -41,23 +41,14 @@
- tutorial/web
- tutorial/create_cube
- tutorial/cube_build_job
- - tutorial/cube_migration
- tutorial/sql_reference
- tutorial/project_level_acl
- - tutorial/cube_spark
- - tutorial/cube_flink
- - tutorial/cube_streaming
- - tutorial/realtime_olap
- - tutorial/cube_build_performance
- tutorial/kylin_client_tool
- tutorial/setup_systemcube
- tutorial/use_cube_planner
- tutorial/use_dashboard
- - tutorial/setup_jdbc_datasource
- tutorial/hybrid
- tutorial/mysql_metastore
- - tutorial/lambda_mode_and_timezone_realtime_olap
- - tutorial/query_pushdown
- title: Integration
docs:
@@ -78,20 +69,16 @@
docs:
- howto/howto_use_restapi
- howto/howto_build_cube_with_restapi
- - howto/howto_optimize_cubes
- - howto/howto_optimize_build
- howto/howto_backup_metadata
- howto/howto_cleanup_storage
+ - howto/howto_optimize_build_and_query
+ - howto/howto_config_spark_pool
- howto/howto_upgrade
- howto/howto_use_cli
- howto/howto_ldap_and_sso
- howto/howto_use_beeline
- - howto/howto_update_coprocessor
- howto/howto_install_ranger_kylin_plugin
- howto/howto_enable_zookeeper_acl
- - howto/howto_use_health_check_cli
- - howto/howto_use_hive_mr_dict
- - howto/sample_dataset
- title: Security
docs:
diff --git a/website/_data/docs-cn.yml b/website/_data/docs31-cn.yml
similarity index 100%
copy from website/_data/docs-cn.yml
copy to website/_data/docs31-cn.yml
diff --git a/website/_data/docs.yml b/website/_data/docs31.yml
similarity index 100%
copy from website/_data/docs.yml
copy to website/_data/docs31.yml
diff --git a/website/_data/docs40-cn.yml b/website/_data/docs40-cn.yml
deleted file mode 100644
index f0de5b3..0000000
--- a/website/_data/docs40-cn.yml
+++ /dev/null
@@ -1,66 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to you under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-- title: 开始
- docs:
- - index
- - gettingstarted/faq
- - gettingstarted/kylin-quickstart
-
-- title: 安装
- docs:
- - install/index
- - install/deploy_without_hadoop
- - install/kylin_cluster
- - install/kylin_docker
- - install/configuration
- - install/advance_settings
-
-- title: 教程
- docs:
- - tutorial/kylin_sample
- - tutorial/web
- - tutorial/create_cube
- - tutorial/cube_build_job
- - tutorial/sql_reference
- - tutorial/project_level_acl
- - tutorial/kylin_client_tool
- - tutorial/setup_systemcube
- - tutorial/use_cube_planner
- - tutorial/use_dashboard
- - tutorial/hybrid
- - tutorial/mysql_metastore
-
-- title: 工具集成
- docs:
- - tutorial/odbc
- - tutorial/jdbc
- - tutorial/tableau
- - tutorial/tableau_91
- - tutorial/powerbi
- - tutorial/microstrategy
- - tutorial/squirrel
- - tutorial/Qlik
- - tutorial/superset
-
-
-- title: 帮助
- docs:
- - howto/howto_use_restapi
- - howto/howto_build_cube_with_restapi
- - howto/howto_optimize_cubes
- - howto/howto_optimize_build_and_query
- - howto/howto_backup_metadata
- - howto/howto_cleanup_storage
- - howto/howto_use_cli
diff --git a/website/_data/docs40.yml b/website/_data/docs40.yml
deleted file mode 100644
index 2c07665..0000000
--- a/website/_data/docs40.yml
+++ /dev/null
@@ -1,85 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to you under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Docs menu items, for English one, docs20-cn.yml is for Chinese one
-# The docs menu is constructed in docs20_nav.html with these data
-- title: Getting Started
- docs:
- - index
- - release_notes
- - gettingstarted/concepts
- - gettingstarted/terminology
- - gettingstarted/faq
- - gettingstarted/events
- - gettingstarted/best_practices
- - gettingstarted/kylin-quickstart
-
-- title: Installation
- docs:
- - install/index
- - install/deploy_without_hadoop
- - install/kylin_cluster
- - install/kylin_docker
- - install/configuration
- - install/advance_settings
-
-- title: Tutorial
- docs:
- - tutorial/kylin_sample
- - tutorial/web
- - tutorial/create_cube
- - tutorial/cube_build_job
- - tutorial/sql_reference
- - tutorial/project_level_acl
- - tutorial/kylin_client_tool
- - tutorial/setup_systemcube
- - tutorial/use_cube_planner
- - tutorial/use_dashboard
- - tutorial/hybrid
- - tutorial/mysql_metastore
-
-- title: Integration
- docs:
- - tutorial/odbc
- - tutorial/jdbc
- - tutorial/tableau
- - tutorial/tableau_91
- - tutorial/powerbi
- - tutorial/microstrategy
- - tutorial/squirrel
- - tutorial/flink
- - tutorial/spark
- - tutorial/hue
- - tutorial/Qlik
- - tutorial/superset
-
-- title: How To
- docs:
- - howto/howto_use_restapi
- - howto/howto_build_cube_with_restapi
- - howto/howto_backup_metadata
- - howto/howto_cleanup_storage
- - howto/howto_optimize_build_and_query
- - howto/howto_config_spark_pool
- - howto/howto_upgrade
- - howto/howto_use_cli
- - howto/howto_ldap_and_sso
- - howto/howto_use_beeline
- - howto/howto_install_ranger_kylin_plugin
- - howto/howto_enable_zookeeper_acl
-
-- title: Security
- docs:
- - security
\ No newline at end of file
diff --git a/website/_docs/gettingstarted/concepts.md b/website/_docs/gettingstarted/concepts.md
index 05a5fee..9c7b18c 100644
--- a/website/_docs/gettingstarted/concepts.md
+++ b/website/_docs/gettingstarted/concepts.md
@@ -10,55 +10,55 @@ Here are some basic technical concepts used in Apache Kylin, please check them f
For terminology in domain, please refer to: [Terminology](terminology.html)
## CUBE
-* __Table__ - This is the definition of hive tables as source of cubes, which must be synced before building cubes.
-
+* __Table__ - This is definition of hive tables as source of cubes, which must be synced before building cubes.
+
-* __Data Model__ - This describes a [STAR SCHEMA](https://en.wikipedia.org/wiki/Star_schema) data model, which defines fact/lookup tables and filter conditions.
-
+* __Data Model__ - This describes a [STAR SCHEMA](https://en.wikipedia.org/wiki/Star_schema) data model, which defines fact/lookup tables and filter condition.
+
-* __Cube Descriptor__ - This describes the definition and settings for a cube instance, defining which data model to use, what dimensions and measures to have, how to partition into segments and how to handle auto-merge, etc.
-
+* __Cube Descriptor__ - This describes definition and settings for a cube instance, defining which data model to use, what dimensions and measures to have, how to partition to segments and how to handle auto-merge etc.
+
-* __Cube Instance__ - This is the instance of cube built from one cube descriptor, and consists of one or more cube segments according to partition settings.
-
+* __Cube Instance__ - This is instance of cube, built from one cube descriptor, and consist of one or more cube segments according partition settings.
+
-* __Partition__ - User can define a DATE/STRING column as partition column on the cube descriptor to separate one cube into several segments with different date periods.
-
+* __Partition__ - User can define a DATE/STRING column as partition column on cube descriptor, to separate one cube into several segments with different date periods.
+
-* __Cube Segment__ - This is the actual carrier of cube data, and it maps to an HTable in HBase. One building job creates one new segment for the cube instance. Once data changes on specified date period, we can refresh related segments to avoid rebuilding the whole cube.
-
+* __Cube Segment__ - This is actual carrier of cube data, and maps to a HTable in HBase. One building job creates one new segment for the cube instance. Once data change on specified data period, we can refresh related segments to avoid rebuilding whole cube.
+
-* __Aggregation Group__ - Each aggregation group is a subset of dimensions, and cuboid are built with combinations inside. It aims at pruning for optimization.
-
+* __Aggregation Group__ - Each aggregation group is subset of dimensions, and build cuboid with combinations inside. It aims at pruning for optimization.
+
## DIMENSION & MEASURE
-* __Mandatory__ - This dimension type is used for cuboid pruning, if a dimension is specified as “mandatory”, then those combinations without such dimension are pruned.
-* __Hierarchy__ - This dimension type is used for cuboid pruning, if dimensions A,B,C form a “hierarchy” relation, then only combinations with A, AB or ABC shall be remained.
-* __Derived__ - In lookup tables, some dimensions could be generated from their PK, so there are specific mappings between them and the FK from the fact table. So those dimensions are DERIVED, and they don't participate in cuboid generation.
-
+* __Mandotary__ - This dimension type is used for cuboid pruning, if a dimension is specified as “mandatory”, then those combinations without such dimension are pruned.
+* __Hierarchy__ - This dimension type is used for cuboid pruning, if dimension A,B,C forms a “hierarchy” relation, then only combinations with A, AB or ABC shall be remained.
+* __Derived__ - On lookup tables, some dimensions could be generated from its PK, so there's specific mapping between them and FK from fact table. So those dimensions are DERIVED and don't participate in cuboid generation.
+
-* __Count Distinct(HyperLogLog)__ - Immediate COUNT DISTINCT is hard to calculate, an approximate algorithm - [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) is introduced, and it keeps the error rate in a low level.
-* __Count Distinct(Precise)__ - Precise COUNT DISTINCT will be pre-calculated based on RoaringBitmap. Currently, only int and bigint are supported.
-* __Top N__ - For example, with this measure type, user can easily get specified numbers of top sellers/buyers, etc.
-
+* __Count Distinct(HyperLogLog)__ - Immediate COUNT DISTINCT is hard to calculate, a approximate algorithm - [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) is introduced, and keep error rate in a lower level.
+* __Count Distinct(Precise)__ - Precise COUNT DISTINCT will be pre-calculated basing on RoaringBitmap, currently only int or bigint are supported.
+* __Top N__ - For example, with this measure type, user can easily get specified numbers of top sellers/buyers etc.
+
## CUBE ACTIONS
* __BUILD__ - Given an interval of partition column, this action is to build a new cube segment.
-* __REFRESH__ - This action will rebuild the cube segment in some partition periods, which is used in case of source table increasing.
-* __MERGE__ - This action will merge multiple continuous cube segments into a single one. This can be automated with the auto-merge setting in cube descriptor.
-* __PURGE__ - Clear segments under a cube instance. This will only update the metadata, and won't delete the cube data from HBase.
-
+* __REFRESH__ - This action will rebuilt cube segment in some partition period, which is used in case of source table increasing.
+* __MERGE__ - This action will merge multiple continuous cube segments into single one. This can be automated with auto-merge settings in cube descriptor.
+* __PURGE__ - Clear segments under a cube instance. This will only update metadata, and won't delete cube data from HBase.
+
## JOB STATUS
-* __NEW__ - This denotes that one job has been just created.
-* __PENDING__ - This denotes that one job is paused by job scheduler and is waiting for resources.
-* __RUNNING__ - This denotes that one job is running in progress.
-* __FINISHED__ - This denotes that one job is finished successfully.
-* __ERROR__ - This denotes that one job is aborted with errors.
-* __DISCARDED__ - This denotes that one job is cancelled by end users.
-
+* __NEW__ - This denotes one job has been just created.
+* __PENDING__ - This denotes one job is paused by job scheduler and waiting for resources.
+* __RUNNING__ - This denotes one job is running in progress.
+* __FINISHED__ - This denotes one job is successfully finished.
+* __ERROR__ - This denotes one job is aborted with errors.
+* __DISCARDED__ - This denotes one job is cancelled by end users.
+
## JOB ACTION
-* __RESUME__ - Once a job is in ERROR status, this action will try to restore it from the latest successful point.
-* __DISCARD__ - No matter what the status of a job is, users can end it and release resources with the DISCARD action.
-
+* __RESUME__ - Once a job in ERROR status, this action will try to restore it from latest successful point.
+* __DISCARD__ - No matter status of a job is, user can end it and release resources with DISCARD action.
+
diff --git a/website/_docs/gettingstarted/faq.cn.md b/website/_docs/gettingstarted/faq.cn.md
index 5168bef..2b75113 100644
--- a/website/_docs/gettingstarted/faq.cn.md
+++ b/website/_docs/gettingstarted/faq.cn.md
@@ -9,201 +9,10 @@ since: v0.6.x
### 如果在使用 Kylin 中遇到了问题
1. 使用搜索引擎（谷歌/百度）、[Kylin 的邮件列表的存档](http://apache-kylin.74782.x6.nabble.com/), [Kylin 的 JIRA 列表](https://issues.apache.org/jira/projects/KYLIN/issues) 来寻求解决办法。
-2. 浏览 Kylin 官方网页，尤其是[文档](http://kylin.apache.org/docs/) 和 [常见问题](http://kylin.apache.org/docs/gettingstarted/faq.html) 页面。
+2. 浏览 Kylin 官方网页，尤其是[文档](http://kylin.apache.org/docs/) 和 [常见问题](https://cwiki.apache.org/confluence/display/KYLIN/FAQ) 页面。
3. 向社区求助，用户可以在订阅 Apache Kylin 的邮件列表之后，用个人邮箱向 Apache Kylin 邮件列表发送邮件，所有订阅了邮件列表的用户都会看到此邮件，并回复邮件以发表自己的见解。
Apache Kylin 主要有 3 个邮件列表，分别是 dev、user、issues。dev 列表主要讨论 Kylin 的开发及新版本发布，user 列表主要讨论用户使用过程中遇到的问题，issues 主要用于追踪 Kylin 项目管理工具（JIRA）的更新动态，订阅的方法请参考 [Apache Kylin 邮件群组](https://kylin.apache.org/cn/community/) 页面中的订阅方法。
也正因为 Apache Kylin 社区是开源社区，所有用户和 Committer 都是志愿进行贡献的，所有的讨论和求助是没有 SLA（Service Level Agreement）的。为了提高讨论效率、规范提问，建议用户在撰写邮件时详细描述问题的出错情况、重现过程、安装版本和 Hadoop 发行版版本等，并且最好能提供相关的出错日志。另外，因为用户的全球化，建议提问时使用英文撰写邮件内容、至少保证邮件主题使用英文。有一篇关于如何提问的[How To Ask Questions The Smart Way](http://catb.org/~esr/faqs/smart-questions.html) 文章，推荐阅读。
-### Kylin 是大数据的通用 SQL 引擎吗？
-不，Kylin 是一个带有 SQL 接口的 OLAP 引擎。 SQL 查询需要与预定义的 OLAP 模型匹配。
-
-### 什么是使用 Apache Kylin 的典型场景？
-如果用户有一个巨大的表 (如：超过 1 亿行) 与维表进行 JOIN，而且查询需要在仪表盘、交互式报告、BI (商务智能) 中完成，用户并发数量为几十个或者几百个，那么 Kylin 是最好的选择。
-
-### Kylin 支持多大的数据量表？ 性能怎么样？
-Kylin 可以支持 TB 到 PB 级数据集的亚秒级查询。 这已经被 eBay，美团，头条等用户验证过。 以 美团的案例为例（至 2018-08），973 个 Cube，每天 380 万个查询，原始数据 8.9 万亿，总 Cube 大小 971 TB（原始数据更大），50％查询在 <0.5 秒内完成，90％ 查询 <1.2秒。
-
-### Cube 的膨胀率是多大(与原始数据相比)
-Cube 的膨胀率取决于多个因素，例如维度 / 度量的数量，维度的基数，Cuboid 的数量，压缩算法等。用户可以通过多种方式优化 Cube 体积。
-
-### 如何比较 Kylin 与其他 SQL 引擎（如 Hive，Presto，SparkSQL，Impala）
-SQL 引擎以不同的方式回答查询，Kylin 不是它们的替代品，而是它们的查询加速器。很多用户将 Kylin 与其他 SQL 引擎一起使用。对于高频率查询的模式，构建 Cube 可以极大地提高性能并给集群负荷减压。
-
-### 运行 Kylin 需要多少个 Hadoop 节点？
-
-Kylin 可以在 Hadoop 集群上运行，从几个节点到数千个节点，取决于您拥有多少数据。 该架构可水平扩展。
-因为大多数计算都是在 Hadoop（MapReduce / Spark / HBase）中进行的，所以通常只需要在几个节点中安装Kylin。
-
-
-### Cube 可以支持多少个维度？
-
-Cube 的最大物理维度数量 (不包括衍生维度) 是 63，但是不推荐使用大于 30 个维度的 Cube，会引起维度灾难。
-
-
-### 执行 "select \*" 报错
-Cube 中只包含聚合数据，所以用户的所有查询都应该是聚合查询 (包含 "group by")。用户可以使用对所有维度分组（"group by") 的查询来获取尽可能接近的详细结果，但返回结果并不是原始数据。
-为了从某些 BI 工具连接 Kylin，Kylin 会尝试回答 "select \*" 查询，但是请注意结果可能不是预期的。请确保发送到 Kylin 的每条查询都是聚合查询。
-
-
-### 如何从 Cube 中查询原始数据
-
-Cube 不是用于查询原始数据的正确选择；但是如果用户的确存在这个需求，以下是一些解决方案：
-1. 将表中的主键（primiary key，简称 pk) 设置为 Cube 中的维度，然后在查询的时候使用 "group by pk"
-2. 配置 Kylin 查询下压到其他 SQL 引擎如 Hive；但是请注意查询性能可能会有影响。
-
-
-### 什么是超高基（UHC) 维度
-
-UHC 代表 Ultra High Cardinality，即超高基数。基数表示维度不同值的数量。通常，维度的基数从数十到数百万。如果超过百万，我们将其称为超高基维度，例如：用户 ID，电话号码等。
-
-Kylin 支持超高基维度，但是在 Cube 设计中额外注意超高基维度，它们可能会使 Cube 体积非常大、查询变慢。
-
-
-### 如何指定用于回答查询的 Cube
-
-请参考 Wiki 文档：[Specify cube for your query](https://cwiki.apache.org/confluence/display/KYLIN/Specify+cube+for+your+query)
-
-### 存在用于创建项目、模型和 Cube 的 REST API 吗？
-
-存在的，但是他们是代码内部的 API，并且会随着版本进行变化。默认情况下，Kylin 希望用户在网页上创建新的项目、模型和 Cube。
-
-### 如何定义一个雪花模型 (有两张事实表)
-
-在雪花模型中，同样只能指定一张事实表。但是用户可以定义一张维表与另一张维表进行 JOIN 连结。
-
-如果查询中包含两张事实表的 JOIN 连结，用户同样可以将一张事实表定义为维表，并不为这张较大的维表设置维表快照。
-
-### Cube 存放在哪里？可以直接在 HBase 上读取 Cube 数据吗？
-
-Cube 数据存储在 HBase 中，每个 Cube segment 都是一张 HBase 中的表；维度的值将组成行键 (Row Key)；度量的值将按列序列化。为了提高存储效率，维度和度量都被编码为字节。Kylin 将在 HBase 中获取到字节后解码为原始值。
-如果没有 Kylin 的元数据，HBase 表不是可读的。
-
-
-### Cube 设计的最佳实践
-请参考： [Design cube in Apache Kylin](https://www.slideshare.net/YangLi43/design-cube-in-apache-kylin)
-
-### 如何加密 Cube 数据
-
-用户可以在 HBase 端对 Cube 数据进行加密，有关内容请参考：[Transparent Encryption of Data At Rest](https://hbase.apache.org/book.html#hbase.encryption.server)
-
-
-### 如何自动调度 Cube 构建
-
-Kylin 没有内置的调度程度。您可以通过 REST API 从外部调度程度服务中触发 Cube 的定时构建，如 Linux 的命令 `crontab`、Apache Airflow 等。
-
-
-### Kylin 支持 Hadoop 3 和 HBase 2.0 吗？
-
-从 v2.5.0 开始，Kylin 将为 Hadoop 3 和 HBase 2 提供二进制包。
-
-
-### Cube 已经启用了，但是 Insight 页面看不到对应的表
-
-请确认 `$KYLIN_HOME/conf/kylin.properties` 中的配置项 `kylin.server.cluster-servers` 在每个 Kylin 节点上都被正确配置，Kylin 节点通过此配置相互通知刷新缓存，请确保节点间的网络通信健康。
-
-### 如何处理 "java.lang.NoClassDefFoundError" 报错？
-
-Kylin 并不自带这些 Hadoop 的 Jar 包，因为它们应该已经在 Hadoop 节点中存在。所以 Kylin 会尝试通过 `hbase classpath` 和 `hive -e set` 找到它们，并将它们的路径加入 `HBASE_CLASSPATH` 中（Kylin 启动时会运行 `hbase` 脚本，该脚本会读取 `HBASE_CLASSPATH`)。
-由于 Hadoop 的复杂性，可能会存在一些找不到 Jar 包的情况，在这种情况下，请查看并修改 `$KYLIN_HOME/bin/` 目录下的 `find-\*-dependecy.sh` 和 `kylin.sh` 脚本来适应您的环境；或者在某些 Hadoop 的发行版中 (如 AWS EMR 5.0)，`hbase` 脚本不会保留原始的 `HBASE_CLASSPATH` 值，可能会引起 "NoClassDefFoundError" 的报错。为了解决这个问题，请在 `$HBASE_HOME/bin/` 下找到 `hbase` 脚本，并在其中搜索 `HBASE_CLASSPATH`，查看它是否是如下形式：
-```sh
-export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*
-```
-如果是的话，请修改它来保留原始值，如下：
-```sh
-export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
-```
-
-
-### 如何在 Cube 中增加维度和度量？
-
-一旦 Cube 被构建完成，它的结构不能被修改。用户可以通过克隆该 Cube，在新的 Cube 中增加维度和度量，并在新的 Cube 构建完成后，禁用或删除老的 Cube，使得查询能够使用新的 Cube 进行回答。
-
-如果用户能接受新的维度在历史数据中不存在，则可以在老的 Cube 的构建结束时候后开始构建新的 Cube，并基于老的 Cube 和新的 Cube 创建一个混合（Hybrid）模型。
-
-
-### 查询结果和在 Hive 中查询结果不一致
-
-以下是可能的原因：
-1. Hive 中的源数据在导入 Kylin 后发生了改变。
-2. Cube 的时间区间和 Hive 中的不一样。
-3. 另外一个 Cube 回答了查询。
-4. 模型中存在内连接(INNER JOIN)，但是查询时没有包含所有的表的连接关系。
-5. Cube 有一些近似度量，如 HyperLogLog、TopN
-6. 在 Kylin v2.3 及之前版本，Kylin 在从 Hive 中获取数据时可能存在数据丢失，请参考 KYLIN-3388。
-
-### Cube 构建完成后，源数据发生改变怎么办？
-
-用户需要刷新 Cube，如果 Cube 被分区，那么可以刷新某些 Segment。
-
-### 构建时在 “Load HFile to HBase Table” 报错 “bulk load aborted with some files not yet loaded”
-
-根本原因是 Kylin 没有权限执行 HBase CompleteBulkLoad。请检查启动 Kylin 的用户是否有访问 HBase 的权限。
-
-### 执行 `sample.sh` 在 HDFS 上创建 `/tmp/kylin` 文件夹失败
-
-执行 `bash -v $KYLIN_HOME/bin/find-hadoop-conf-dir.sh`，查看报错信息，然后根据报错信息进行排错。
-
-### 使用 Chrome 浏览器时，网页报错显示 “net::ERR_CONTENT_DECODING_FAILED”
-
-修改 `$KYLIN_HOME/tomcat/conf/server.xml`，找到 `compress=on`，修改为 `off`。
-
-
-### 如何配置某个 Cube 在指定的 YARN 队列中构建
-
-用户可以在 Cube 级别进行重写如下参数：
-```properties
-kylin.engine.mr.config-override.mapreduce.job.queuename=YOUR_QUEUE_NAME
-kylin.source.hive.config-override.mapreduce.job.queuename=YOUR_QUEUE_NAME
-kylin.engine.spark-conf.spark.yarn.queue=YOUR_QUEUE_NAME
-```
-
-### 构建 Cube 时报错 “Too high cardinality is not suitable for dictionary”
-
-Kylin 使用字典编码方式来编码/解码维度的值；通常一个维度的基数都是小于百万的，所以字典编码方式非常好用。正如字典需要被持久化、加载进内存，如果一个维度的基数非常高，内存占用会非常大，所以 Kylin 加了一层检查。如果用户看到这类报错，建议先找到哪些维度是超高基维度，并重新对 Cube 进行设计 (如：是否需要对将超高基的列设置为维度)，如果用户必须保留该超高基列作为一个维度，有如下解决方法：
-1. 修改编码方式，如修改为 `fixed_length` 或 `integer`。
-2. 增大 `$KYLIN_HOME/conf/kylin.properties` 中的 `kylin.dictionary.max.cardinality` 的值。
-
-
-### 当某列中的数据都大于 0 时，查询 SUM() 返回负值
-
-如果在 Hive 中将列声明为 integer，则 SQL 引擎 (Calcite) 将使用列的数据类型作为 “SUM()” 的数据类型，而此字段上的聚合值可能超出整数范围；在这种情况下，查询的返回结果可能是负值。
-
-解决方法如下：
-在 Hive 中将该列的数据类型更改为 BIGINT，然后将表同步到 Kylin（对应的 Cube 不需要刷新）。
-
-
-### 为什么需要在构建 Cuboid 之前从事实表中提取不同的列？
-Kylin 使用字典对每列中的值进行编码，这大大减少了 Cube 的存储大小。 而要构建字典，Kylin 需要为每列获取不同的值。
-
-
-### 如何新增用户并修改默认用户的密码
-
-请参考: [How to add new user or change the default password](https://cwiki.apache.org/confluence/display/KYLIN/How+to+add+new+user+or+change+the+default+password) 。
-
-### 构建 Kylin 代码时遇到 NPM 报错 (中国大陆地区用户请特别注意此问题)
-
-用户可以通过以下命令为 NPM 设置代理：
-```sh
-npm config set proxy http://YOUR_PROXY_IP
-```
-请更新您本地的 NPM 仓库以使用国内的 NPM 镜像，例如[淘宝 NPM 镜像]([http://npm.taobao.org](http://npm.taobao.org)
-
-
-### 运行 BuildCubeWithEngineTest 失败 "failed to connect to hbase"
-用户在第一次运行 hbase 客户端时可能会遇到此错误，请检查错误信息以查看是否存在无法访问 "/hadoop/hbase/local/jars" 等文件夹；如果文件夹不存在，请创建它。
-
-### JDBC 驱动程序返回的日期/时间与 REST API 不同
-请参考：[JDBC query result Date column get wrong value](http://apache-kylin.74782.x6.nabble.com/JDBC-query-result-Date-column-get-wrong-value-td5370.html)
-
-
-重启 Kylin 来使得配置生效，如果用户有多个 Kylin 服务器作为一个集群，需要在所有的节点都执行相同操作。
-
-### HDFS 上的工作目录中文件超过了 300G，可以手动删除吗？
-
-HDFS 上的工作目录中的数据包括了中间数据 (将被垃圾清理所清除) 和 Cuboid 数据 (不会被垃圾清理所清除)，Cuboid 数据将为之后的 Segment 合并而保留。所以如果用户确认这些 Segment 在之后不会被合并，可以将 Cuboid 数据移动到其他路径甚至删除。
-
-另外，请留意 HDFS 工作目录下的 "resources" 或 "jdbc-resources" 子目录中会存放一些大的元数据，如字典文件和维表的快照，这些文件不能被删除。
-
-#### 如何对 like 语句中的关键字进行转义？
-"%", "_" 是 "like" 语句中的保留关键字; "%" 可以匹配任意个字符, "_" 匹配单个字符; 如果你想匹配关键字如 "_", 需要使用另一个字符在前面进行转义; 下面是一个使用 "/" 进行转义的例子, 此查询试图匹配 "xiao_":
-"select username from gg_user where username like '%xiao/_%' escape '/'; "
\ No newline at end of file
+为了方便编辑和更新，Kylin 4.x 常见问题收集在Apache kylin Wiki，您可以从中查找是否有自己需要的答案：
+[Apache Kylin4.0 FAQ](https://cwiki.apache.org/confluence/display/KYLIN/FAQ)
\ No newline at end of file
diff --git a/website/_docs/gettingstarted/faq.md b/website/_docs/gettingstarted/faq.md
index 5c5a403..4ecefe8 100644
--- a/website/_docs/gettingstarted/faq.md
+++ b/website/_docs/gettingstarted/faq.md
@@ -8,291 +8,9 @@ since: v0.6.x
**Here are some tips for you when encountering problems with Kylin:**
1. Use search engines (Google / Baidu), Kylin's [Mailing List Archives](http://apache-kylin.74782.x6.nabble.com/), the Kylin Project on the Apache [JIRA](https://issues.apache.org/jira/projects/KYLIN/issues) to seek a solution.
- 2. Browse Kylin's official website, especially the [Docs page](http://kylin.apache.org/docs/) and the [FAQ page](http://kylin.apache.org/docs/gettingstarted/faq.html).
+ 2. Browse Kylin's official website, especially the [Docs page](http://kylin.apache.org/docs/) and the [FAQ page](https://cwiki.apache.org/confluence/display/KYLIN/FAQ).
3. Send an email to Apache Kylin dev or user mailing list: dev@kylin.apache.org, user@kylin.apache.org; before sending, please make sure you have subscribed the mailing list by dropping an email to dev-subscribe@kylin.apache.org or user-subscribe@kylin.apache.org. Your email is supposed to include: the version numbers of Kylin and other components you are using in your env, the log of the error message, the SQL (if you got the query error).
-There is an article about [how to ask a question in a smart way](http://catb.org/~esr/faqs/smart-questions.html).
+There is an article about [How to ask a question in a smart way](http://catb.org/~esr/faqs/smart-questions.html).
-#### Is Kylin a generic SQL engine for big data?
-
- * No, Kylin is an OLAP engine with SQL interface. The SQL queries should be matched with the pre-defined OLAP model.
-
-#### What's a typical scenario to use Apache Kylin?
-
- * Kylin can be the best option if you have a huge table (e.g., >100 million rows), join with lookup tables, while queries need to be finished in the second level (dashboards, interactive reports, business intelligence, etc), and the concurrent users can be dozens or hundreds.
-
-#### How large a data scale can Kylin support? How about the performance?
-
- * Kylin can support second level query performance at TB to PB level dataset. This has been verified by users like eBay, Meituan, Toutiao. Take Meituan's case as an example (till 2018-08), 973 cubes, 3.8 million queries per day, raw data 8.9 trillion, total cube size 971 TB (original data is bigger), 50% of the queries finished in < 0.5 seconds, 90% queries < 1.2 seconds.
-
-#### Who are using Apache Kylin?
-
- * You can find a list in Kylin's [powered by page](/community/poweredby.html). If you want to be added, please email to dev@kylin.apache.org with your use case.
-
-#### What's the expansion rate of Cube (compared with raw data)?
-
- * It depends on a couple of factors, for example, dimension/measure number, dimension cardinality, cuboid number, compression algorithm, etc. You can optimize the cube expansion in many ways to control the size.
-
-#### How to compare Kylin with other SQL engines like Hive, Presto, Spark SQL, Impala?
-
- * They answer a query in different ways. Kylin is not a replacement for them, but a supplement (query accelerator). Many users run Kylin together with other SQL engines. For the high frequent query patterns, building Cubes can greatly improve the performance and also offload cluster workloads. For less queried patterns or ad-hoc queries, other MPP engines are more flexible.
-
-#### How to compare Kylin with Druid?
-
- * Druid is more suitable for real-time analysis. Kylin is more focus on OLAP case. Druid has a good integration with Kafka as real-time streaming; Kylin fetches data from Hive or Kafka in batches. The real-time capability of Kylin is still under development.
-
- * Many internet service providers host both Druid and Kylin, serving different purposes (real-time and historical).
-
- * Some other Kylin's highlights: supports star & snowflake schema; ANSI-SQL support, JDBC/ODBC for BI integrations. Kylin also has a Web GUI with LDAP/SSO user authentication.
-
- * For more information, please do a search or check this [mail thread](https://mail-archives.apache.org/mod_mbox/kylin-dev/201503.mbox/%3CCAKmQrOY0fjZLUU0MGo5aajZ2uLb3T0qJknHQd+Wv1oxd5PKixQ@mail.gmail.com%3E).
-
-#### How to quick start with Kylin?
-
- * To get a quick start, you can run Kylin in a Hadoop sandbox VM or in the cloud, for example, start a small AWS EMR or Azure HDInsight cluster and then install Kylin in one of the node.
-
-#### How many nodes of the Hadoop are needed to run Kylin?
-
- * Kylin can run on a Hadoop cluster from only a couple nodes to thousands of nodes, depends on how much data you have. The architecture is horizontally scalable.
-
- * Because most of the computation is happening in Hadoop (MapReduce/Spark/HBase), usually you just need to install Kylin in a couple of nodes.
-
-#### How many dimensions can be in a cube?
-
- * The max physical dimension number (exclude derived columns in lookup tables) in a cube is 63; If you can normalize some dimensions to lookup tables, with derived dimensions, you can create a cube with more than 100 dimensions.
-
- * But a cube with > 30 physical dimensions is not recommended; You even couldn't save that in Kylin if you don't optimize the aggregation groups. Please search "curse of dimensionality".
-
-#### Why do I got an error when running a "select * " query?
-
- * The cube has only the aggregated data, so all your queries should be aggregated queries ("GROUP BY"). You can use a SQL with all dimensions be grouped to get them as close as the detailed result, but that is not the raw data.
-
- * In order to be connected from some BI tools, Kylin tries to answer "select \*" query but please aware the result might not be expected. Please make sure each query to Kylin is aggregated.
-
-#### How can I query raw data from a cube?
-
- * Cube is not the right option for raw data.
-
-But if you do want, there are some workarounds. 1) Add the primary key as a dimension, then the "group by pk" will return the raw data; 2) Configure Kylin to push down the query to another SQL engine like Hive, but the performance has no assurance.
-
-#### What is the UHC dimension?
-
- * UHC means Ultra High Cardinality. Cardinality means the number of distinct values of a dimension. Usually, a dimension's cardinality is from tens to millions. If above a million, we call it a UHC dimension, for example, user id, cell number, etc.
-
- * Kylin supports UHC dimension, but you need to pay attention to UHC dimensions, especially the encodings and the cuboid combinations. It may cause your Cube to be very large and query to be slow.
-
-#### Can I specify a cube to answer my SQL statements?
-
- * Please refer to this wiki: [Specify cube for your query](https://cwiki.apache.org/confluence/display/KYLIN/Specify+cube+for+your+query)
-
-#### Is there a REST API to create the project/model/cube?
-
- * Yes, but they are private APIs, incline to change over versions (without notification). By design, Kylin expects the user to create a new project/model/cube in Kylin's web GUI.
-
-#### How to define a snowflake model(with two fact tables)?
-
- * In the snowflake model, there is only one fact table also. But you could define lookup table joins with another lookup table.
- * If the query pattern between your two "fact" tables is fixed, just like factA left join with factB. You could define factB as a lookup table and skip the snapshot for this huge lookup table.
-
-#### Where does the cube locate, can I directly read cube from HBase without going through Kylin API?
-
- * Cube is stored in HBase. Each cube segment is an HBase table. The dimension values will be composed as the row key. The measures will be serialized in columns. To improve the storage efficiency, both dimension and measure values will be encoded to bytes. Kylin will decode the bytes to origin values after fetching from HBase. Without Kylin's metadata, the HBase tables are not readable.
-
-#### What's the best practice to design a cube?
-
- * Please check: [https://www.slideshare.net/YangLi43/design-cube-in-apache-kylin](https://www.slideshare.net/YangLi43/design-cube-in-apache-kylin)
-
-#### How to encrypt cube data?
-
- * You can enable encryption at HBase side. Refer to https://hbase.apache.org/book.html#hbase.encryption.server for more details.
-
-#### How to schedule the cube build at a fixed frequency, in an automatic way?
-
- * Kylin doesn't have a built-in scheduler for this. You can trigger that through the Rest API from external scheduler services, like Linux cron job, Apache Airflow, etc.
-
-#### How to export/import cube/project across different Kylin environments?
-
- * Please check: [http://kylin.apache.org/docs/howto/howto_use_cli.html](http://kylin.apache.org/docs/howto/howto_use_cli.html)
-
-#### How to view kylin cube's hbase table without encoding?
-
- * To view the origin data, please use SQL to query Kylin. Kylin will convert the SQL query to HBase access and then decode the data. You can use Rest API, JDBC, ODBC drivers to connect with Kylin.
-
-#### Does Kylin support Hadoop 3 and HBase 2.0?
-
- * From v2.5.0, Kylin will provide a binary package for Hadoop 3 and HBase 2.
-
-#### The Cube is ready, but why the table does not appear in the "Insight" tab?
-
- * Make sure the "kylin.server.cluster-servers" property in `conf/kylin.properties` is configured with EVERY Kylin node, all job and query nodes. Kylin nodes notify each other to flush cache with this configuration. Also, please ensure that the network among them are healthy.
-
-#### What should I do if I encounter a "java.lang.NoClassDefFoundError" error?
-
- * Kylin doesn't ship those Hadoop jars, because they should already exist in the Hadoop node. So Kylin will try to find them and then add to Kylin's classpath. Due to Hadoop's complexity, there might be some case a jar wasn't found. In this case please look at the "bin/find-\*-dependency.sh" and "bin/kylin.sh", modify them to fit your environment.
-
-#### How to query Kylin in Python?
-
- * Please check: [https://github.com/Kyligence/kylinpy](https://github.com/Kyligence/kylinpy)
-
-#### How to add dimension/measure to a cube?
-
- * Once a cube is built, its structure cannot be modified. To add a dimension/measure, you need to clone a new cube, and then add to it.
-
-When the new cube is built, please disable or drop the old one.
-
-If you can accept the absence of new dimensions for historical data, you can build the new cube since the end time of the old cube. And then create a hybrid model over the old and new cube.
-
-#### How to solve the data security problem of Tableau connection client?
-
- * Kylin's ACL control can solve this problem. Different analysts have the authority to work on different projects for Kylin. When you create a Kylin ODBC DSN, you can map different links to different analyst accounts.
-
-#### The query result is not exactly matched with that in Hive, what's the possible reason?
-
- * Possible reasons:
-a) Source data changed in Hive after built into the cube;
-b) Cube's time range is not the same as in Hive;
-c) Another cube answered your query;
-d) The data model has inner joins, but the query doesn't join all tables;
-e) Cube has some approximate measures like HyberLogLog, TopN;
-f) In v2.3 and before, Kylin may have data loss when fetching from Hive, see KYLIN-3388.
-
-#### What to do if the source data changed after being built into the cube?
-
- * You need to refresh the cube. If the cube is partitioned, you can refresh certain segments.
-
-#### What is the possible reason for getting the error ‘bulk load aborted with some files not yet loaded’ in the ‘Load HFile to HBase Table’ step?
-
- * Kylin doesn't have permissions to execute HBase CompleteBulkLoad. Check whether the current user (that run Kylin service) has the permission to access HBase.
-
-#### Why `bin/sample.sh` cannot create the `/tmp/kylin` folder on HDFS?
-
- * Run ./bin/find-hadoop-conf-dir.sh -v, check the error message, then check the environment according to the information reported.
-
-#### In Chrome, web console shows net::ERR_CONTENT_DECODING_FAILED, what should I do?
-
- * Edit $KYLIN_HOME/tomcat/conf/server.xml, find the "compress=on", change it to off.
-
-#### How to configure one cube to be built using a chosen YARN queue?
-
- * Set the YARN queue in Cube’s Configuration Overwrites page, then it will affect only one cube. Here are the three parameters:
-
- {% highlight Groff markup %}
-kylin.engine.mr.config-override.mapreduce.job.queuename=YOUR_QUEUE_NAME
-kylin.source.hive.config-override.mapreduce.job.queuename=YOUR_QUEUE_NAME
-kylin.engine.spark-conf.spark.yarn.queue=YOUR_QUEUE_NAME
- {% endhighlight %}
-
-#### How to add a new JDBC data source dialect?
-
- * It is easy to add a new type of JDBC data source. You can follow such steps:
-
-1) Add the dialect in source-hive/src/main/java/org/apache/kylin/source/jdbc/JdbcDialect.java
-
-2) Implement a new IJdbcMetadata if {database that you want to add}'s metadata fetching is different with others and then register it in JdbcMetadataFactory
-
-3) You may need to customize the SQL for creating/dropping tables in JdbcExplorer for {database that you want to add}.
-
-#### How to ask a question?
-
- * Check Kylin documents first, and doing a Google search can also help. Sometimes the question has been answered, so you don't need ask again. If no matching, please send your question to Apache Kylin user mailing list: user@kylin.apache.org; You need to drop an email to user-subscribe@kylin.apache.org to subscribe if you haven't done so. In the email content, please provide your Kylin and Hadoop version, specific error logs (as much as possible), and also the how to re-produce steps.
-
-#### "bin/find-hive-dependency.sh" can locate hive/hcat jars in local, but Kylin reports error like "java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat" or "java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/session/SessionState"
-
- * Kylin needs many dependent jars (hadoop/hive/hcat/hbase/kafka) on classpath to work, but Kylin doesn't ship them. It will seek these jars from your local machine by running commands like `hbase classpath`, `hive -e set` etc. The founded jars' path will be appended to the environment variable *HBASE_CLASSPATH* (Kylin uses `hbase` shell command to start up, which will read this). But in some Hadoop distribution (like AWS EMR 5.0), the `hbase` shell doesn't keep the origin `HBASE_CLASSP [...]
-
- * To fix this, find the hbase shell script (in hbase/bin folder), and search *HBASE_CLASSPATH*, check whether it overwrites the value like :
-
- {% highlight Groff markup %}
- export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*
- {% endhighlight %}
-
- * If true, change it to keep the origin value like:
-
- {% highlight Groff markup %}
- export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
- {% endhighlight %}
-
-#### Get "java.lang.IllegalArgumentException: Too high cardinality is not suitable for dictionary -- cardinality: 5220674" in "Build Dimension Dictionary" step
-
- * Kylin uses "Dictionary" encoding to encode/decode the dimension values (check [this blog](/blog/2015/08/13/kylin-dictionary/)); Usually a dimension's cardinality is less than millions, so the "Dict" encoding is good to use. As dictionary need to be persisted and loaded into memory, if a dimension's cardinality is very high, the memory footprint will be tremendous, so Kylin add a check on this. If you see this error, please identify the UHC dimension first and then re-evaluate the des [...]
-
-
-#### How to Install Kylin on CDH 5.2 or Hadoop 2.5.x
-
- * Check out discussion: [https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kylin-olap/X0GZfsX1jLc/nzs6xAhNpLkJ](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kylin-olap/X0GZfsX1jLc/nzs6xAhNpLkJ)
-
- {% highlight Groff markup %}
- I was able to deploy Kylin with the following options in POM.
- <hadoop2.version>2.5.0</hadoop2.version>
- <yarn.version>2.5.0</yarn.version>
- <hbase-hadoop2.version>0.98.6-hadoop2</hbase-hadoop2.version>
- <zookeeper.version>3.4.5</zookeeper.version>
- <hive.version>0.13.1</hive.version>
- My Cluster is running on Cloudera Distribution CDH 5.2.0.
- {% endhighlight %}
-
-
-#### SUM(field) returns a negative result while all the numbers in this field are > 0
- * If a column is declared as integer in Hive, the SQL engine (calcite) will use column's type (integer) as the data type for "SUM(field)". While the aggregated value on this field may exceed the scope of integer, the cast will cause a negative value be returned. The workaround is, alter that column's type to BIGINT in hive, and then sync the table schema to Kylin (the cube doesn't need rebuild); Keep in mind that, always declare as BIGINT in hive for an integer column which would be us [...]
-
-#### Why does Kylin need to extract the distinct columns from Fact Table before building cube?
- * Kylin uses dictionary to encode the values in each column, this significantly reduces the cube's storage size. To build the dictionary, Kylin needs to fetch the distinct values for each column.
-
-#### Why does Kylin calculate the HIVE table cardinality?
- * The cardinality of dimensions is an important measure of the cube complexity. The higher the cardinality, the bigger the cube, and thus the longer to build, and the slower to query. Cardinality > 1,000 is worth attention and > 1,000,000 should be avoided at best effort. For optimal cube performance, try to reduce high cardinality by categorize values or derive features.
-
-#### How to add new user or change the default password?
- * Please check the document: [How to add new user or change the default password](https://cwiki.apache.org/confluence/display/KYLIN/How+to+add+new+user+or+change+the+default+password).
-
-#### Using sub-query for un-supported SQL
-
-{% highlight Groff markup %}
-Original SQL:
-select fact.slr_sgmt,
-sum(case when cal.RTL_WEEK_BEG_DT = '2015-09-06' then gmv else 0 end) as W36,
-sum(case when cal.RTL_WEEK_BEG_DT = '2015-08-30' then gmv else 0 end) as W35
-from ih_daily_fact fact
-inner join dw_cal_dt cal on fact.cal_dt = cal.cal_dt
-group by fact.slr_sgmt
-{% endhighlight %}
-
-{% highlight Groff markup %}
-Using sub-query
-select a.slr_sgmt,
-sum(case when a.RTL_WEEK_BEG_DT = '2015-09-06' then gmv else 0 end) as W36,
-sum(case when a.RTL_WEEK_BEG_DT = '2015-08-30' then gmv else 0 end) as W35
-from (
- select fact.slr_sgmt as slr_sgmt,
- cal.RTL_WEEK_BEG_DT as RTL_WEEK_BEG_DT,
- sum(gmv) as gmv36,
- sum(gmv) as gmv35
- from ih_daily_fact fact
- inner join dw_cal_dt cal on fact.cal_dt = cal.cal_dt
- group by fact.slr_sgmt, cal.RTL_WEEK_BEG_DT
-) a
-group by a.slr_sgmt
-{% endhighlight %}
-
-#### Build kylin meet NPM errors (中国大陆地区用户请特别注意此问题)
-
- * Please add proxy for your NPM:
- `npm config set proxy http://YOUR_PROXY_IP`
-
- * Please update your local NPM repository to using any mirror of npmjs.org, like Taobao NPM (请更新您本地的NPM仓库以使用国内的NPM镜像，例如淘宝NPM镜像) :
- [http://npm.taobao.org](http://npm.taobao.org)
-
-#### Failed to run BuildCubeWithEngineTest, saying failed to connect to hbase while hbase is active
- * User may get this error when running hbase client the first time, please check the error trace to see whether there is an error saying couldn't access a folder like "/hadoop/hbase/local/jars"; If that folder doesn't exist, create it.
-
-#### Kylin JDBC driver returns a different Date/time than the REST API, seems it add the timezone to parse the date.
- * Please check the [post in mailing list](http://apache-kylin.74782.x6.nabble.com/JDBC-query-result-Date-column-get-wrong-value-td5370.html)
-
-
-#### What kind of data is left in 'kylin.env.hdfs-working-dir' ? We often execute kylin cleanup storage command, but now our working dir folder is about 300 GB size, can we delete old data manually?
-
- * The data in 'hdfs-working-dir' ('hdfs:///kylin/kylin_metadata/' by default) includes intermediate files (will be GC) and Cuboid data (won't be GC). The Cuboid data is kept for the further segments' merge, as Kylin couldn't merge from HBase. If you're sure those segments won't be merged, you can move them to other paths or even delete.
-
- * Please pay attention to the "resources" or "jdbc-resources" sub-folder under '/kylin/kylin_metadata/', which persists big metadata files like dictionaries and lookup tables' snapshots. They shouldn't be manually moved.
-
-#### How to escape the key word in fuzzy match (like) queries?
-"%", "_" are keywords in the "like" clause; "%" matches any character, and "_" matches a single character; When you want to match the keyword like "_", you need to escape them with another character ahead; Below is a sample with "/" to escape, the query is to match the "xiao_":
-"select username from gg_user where username like '%xiao/_%' escape '/'; "
\ No newline at end of file
+To facilitate editing and updating, kylin 4.X FAQ are collected in the Apache kylin wiki, where you may can find out the answers you need:
+[Apache Kylin4.0 FAQ](https://cwiki.apache.org/confluence/display/KYLIN/FAQ)
\ No newline at end of file
diff --git a/website/_docs/gettingstarted/quickstart.cn.md b/website/_docs/gettingstarted/quickstart.cn.md
index 518c398..4bc7076 100644
--- a/website/_docs/gettingstarted/quickstart.cn.md
+++ b/website/_docs/gettingstarted/quickstart.cn.md
@@ -6,84 +6,16 @@ permalink: /cn/docs/gettingstarted/kylin-quickstart.html
since: v0.6.x
---
-这里是一份从下载安装到体验亚秒级查询的完整流程，分别介绍了有hadoop环境（基于hadoop环境的安装）和没有hadoop环境（从docker镜像安装）两种场景下kylin的安装使用，用户可以根据自己的环境选择其中的任意一种方式。
-你可以按照文章里的步骤对kylin进行初步的了解和体验，掌握kylin的基本使用技能，然后结合自己的业务场景使用kylin来设计模型，加速查询。
+这里是一份从下载安装到体验亚秒级查询的完整流程，你可以按照文章里的步骤对kylin进行初步的了解和体验，掌握kylin的基本使用技能，然后结合自己的业务场景使用kylin来设计模型，加速查询。
+### 基于hadoop环境安装使用kylin
-
-### 一、 从docker镜像安装使用kylin（不需要提前准备hadoop环境）
-为了让用户方便的试用 Kylin，我们提供了 Kylin 的 docker 镜像。该镜像中，Kylin 依赖的各个服务均已正确的安装及部署，包括：
-
-- JDK 1.8
-- Hadoop 2.7.0
-- Hive 1.2.1
-- Hbase 1.1.2 (with Zookeeper)
-- Spark 2.3.1
-- Kafka 1.1.1
-- MySQL 5.1.73
-
-我们已将面向用户的 Kylin 镜像上传至 docker 仓库，用户无需在本地构建镜像，只需要安装docker，就可以体验kylin的一键安装。
-
-#### step1、首先执行以下命令从 docker 仓库 pull 镜像：
-
-```
-docker pull apachekylin/apache-kylin-standalone:3.1.0
-```
-
-此处的镜像包含的是kylin最新Release版本kylin 3.1.0。由于该镜像中包含了所有kylin依赖的大数据组件，所以拉取镜像需要的时间较长，请耐心等待。Pull成功后显示如下：
-
-
-#### step2、执行以下命令来启动容器：
-
-```
-docker run -d \
--m 8G \
--p 7070:7070 \
--p 8088:8088 \
--p 50070:50070 \
--p 8032:8032 \
--p 8042:8042 \
--p 16010:16010 \
-apachekylin/apache-kylin-standalone:3.1.0
-```
-
-容器会很快启动，由于容器内指定端口已经映射到本机端口，可以直接在本机浏览器中打开各个服务的页面，如：
-
-- Kylin 页面：http://127.0.0.1:7070/kylin/
-- Hdfs NameNode 页面：http://127.0.0.1:50070
-- Yarn ResourceManager 页面：http://127.0.0.1:8088
-- HBase 页面：http://127.0.0.1:60010
-
-容器启动时，会自动启动以下服务：
-
-- NameNode, DataNode
-- ResourceManager, NodeManager
-- HBase
-- Kafka
-- Kylin
-
-并自动运行 $KYLIN_HOME/bin/sample.sh及在 Kafka 中创建 kylin_streaming_topic topic 并持续向该 topic 中发送数据。这是为了让用户启动容器后，就能体验以批和流的方式的方式构建 Cube 并进行查询。
-用户可以通过docker exec命令进入容器，容器内相关环境变量如下：
-
-```
-JAVA_HOME=/home/admin/jdk1.8.0_141
-HADOOP_HOME=/home/admin/hadoop-2.7.0
-KAFKA_HOME=/home/admin/kafka_2.11-1.1.1
-SPARK_HOME=/home/admin/spark-2.3.1-bin-hadoop2.6
-HBASE_HOME=/home/admin/hbase-1.1.2
-HIVE_HOME=/home/admin/apache-hive-1.2.1-bin
-KYLIN_HOME=/home/admin/apache-kylin-3.1.0-bin-hbase1x
-```
-
-使用ADMIN/KYLIN的用户名和密码组合登陆Kylin后，用户可以使用sample cube来体验cube的构建和查询，也可以按照下面“基于hadoop环境安装使用kylin”中从step8之后的教程来创建并查询属于自己的model和cube。
-
-### 二、 基于hadoop环境安装使用kylin
对于已经有稳定hadoop环境的用户，可以下载kylin的二进制包将其部署安装在自己的hadoop集群。安装之前请根据以下要求进行环境检查：
- 前置条件：
-Kylin 依赖于 Hadoop 集群处理大量的数据集。您需要准备一个配置好 HDFS，YARN，MapReduce，Hive， HBase，Zookeeper 和其他服务的 Hadoop 集群供 Kylin 运行。
-Kylin 可以在 Hadoop 集群的任意节点上启动。方便起见，您可以在 master 节点上运行 Kylin。但为了更好的稳定性，我们建议您将 Kylin 部署在一个干净的 Hadoop client 节点上，该节点上 Hive，HBase，HDFS 等命令行已安装好且 client 配置（如 core-site.xml，hive-site.xml，hbase-site.xml及其他）也已经合理的配置且其可以自动和其它节点同步。
-运行 Kylin 的 Linux 账户要有访问 Hadoop 集群的权限，包括创建/写入 HDFS 文件夹，Hive 表， HBase 表和提交 MapReduce 任务的权限。
+Kylin 依赖于 Hadoop 集群处理大量的数据集。您需要准备一个配置好 HDFS，YARN，MapReduce，Hive，Zookeeper 和其他服务的 Hadoop 集群供 Kylin 运行。
+Kylin 可以在 Hadoop 集群的任意节点上启动。方便起见，您可以在 master 节点上运行 Kylin。但为了更好的稳定性，我们建议您将 Kylin 部署在一个干净的 Hadoop client 节点上，该节点上 Hive，HDFS 等命令行已安装好且 client 配置（如 core-site.xml，hive-site.xml及其他）也已经合理的配置且其可以自动和其它节点同步。
+运行 Kylin 的 Linux 账户要有访问 Hadoop 集群的权限，包括创建/写入 HDFS 文件夹，Hive 表的权限。
- 硬件要求：
运行 Kylin 的服务器建议最低配置为 4 core CPU，16 GB 内存和 100 GB 磁盘。
@@ -94,20 +26,19 @@ CentOS 6.5+ 或Ubuntu 16.0.4+
- 软件要求：
- Hadoop 2.7+,3.0
- Hive 0.13+,1.2.1+
- - Hbase 1.1+，2.0（从kylin 2.5开始支持）
+ - Spark 2.4.6
- JDK: 1.8+
-建议使用集成的Hadoop环境进行kylin的安装与测试，比如Hortonworks HDP 或Cloudera CDH ，kylin发布前在 Hortonworks HDP 2.2-2.6 and 3.0, Cloudera CDH 5.7-5.11 and 6.0, AWS EMR 5.7-5.10, Azure HDInsight 3.5-3.6上测试通过。
+建议使用集成的Hadoop环境进行kylin的安装与测试，比如Hortonworks HDP 或Cloudera CDH ，kylin发布前在 Hortonworks HDP 2.4, Cloudera CDH 5.7 and 6.0, AWS EMR 5.31 and 6.0, Azure HDInsight 4.0 上测试通过。
当你的环境满足上述前置条件时 ，你可以开始安装使用kylin。
#### step1、下载kylin压缩包
-从[Apache Kylin Download Site](https://kylin.apache.org/download/)下载一个适用于你的Hadoop版本的二进制文件。目前最新Release版本是kylin 3.1.0和kylin 2.6.6，其中3.0版本支持实时摄入数据进行预计算的功能。以CDH 5.的hadoop环境为例，可以使用如下命令行下载kylin 3.1.0：
-
+从[Apache Kylin Download Site](https://kylin.apache.org/download/)下载 kylin4.0 的二进制文件。
```
cd /usr/local/
-wget http://apache.website-solution.net/kylin/apache-kylin-3.1.0/apache-kylin-3.1.0-bin-cdh57.tar.gz
+wget http://apache.website-solution.net/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin.tar.gz
```
#### step2、解压kylin
@@ -115,14 +46,14 @@ wget http://apache.website-solution.net/kylin/apache-kylin-3.1.0/apache-kylin-3.
解压下载得到的kylin压缩包，并配置环境变量KYLIN_HOME指向解压目录：
```
-tar -zxvf apache-kylin-3.1.0-bin-cdh57.tar.gz
-cd apache-kylin-3.1.0-bin-cdh57
+tar -zxvf apache-kylin-4.0.0-bin.tar.gz
+cd apache-kylin-4.0.0-bin-cdh57
export KYLIN_HOME=`pwd`
```
#### step3、下载SPARK
-由于kylin启动时会对SPARK环境进行检查，所以你需要设置SPARK_HOME指向自己的spark安装路径：
+Kylin4.0 使用 Spark 作为查询和构建引擎，所以你需要设置SPARK_HOME指向自己的spark安装路径：
```
export SPARK_HOME=/path/to/spark
@@ -136,7 +67,18 @@ $KYLIN_HOME/bin/download-spark.sh
脚本会将解压好的spark放在$KYLIN_HOME目录下，如果系统中没有设置SPARK_HOME，启动kylin时会自动找到$KYLIN_HOME目录下的spark。
-#### step4、环境检查
+### step4、配置 Mysql 元数据
+
+Kylin 4.0 使用 Mysql 作为元数据存储，需要在kylin.properties做如下配置：
+
+```$xslt
+kylin.metadata.url=kylin_metadata@jdbc,driverClassName=com.mysql.jdbc.Driver,url=jdbc:mysql://localhost:3306/kylin_database,username=,password=
+kylin.env.zookeeper-connect-string=ip:2181
+```
+
+你需要修改其中的 Mysql 用户名和密码，以及存储元数据的database和table。请参考 [配置 Mysql 为 Metastore](/_docs40/tutorial/mysql_metastore.html) 了解 Mysql 作为 Metastore 的详细配置。
+
+#### step5、环境检查
Kylin 运行在 Hadoop 集群上，对各个组件的版本、访问权限及 CLASSPATH 等都有一定的要求，为了避免遇到各种环境问题，您可以执行
@@ -146,7 +88,7 @@ $KYLIN_HOME/bin/check-env.sh
来进行环境检测，如果您的环境存在任何的问题，脚本将打印出详细报错信息。如果没有报错信息，代表您的环境适合 Kylin 运行。
-#### step5、启动kylin
+#### step6、启动kylin
运行如下命令来启动kylin：
@@ -158,18 +100,18 @@ $KYLIN_HOME/bin/kylin.sh start
```
A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
-Check the log at /usr/local/apache-kylin-3.1.0-bin-cdh57/logs/kylin.log
+Check the log at /usr/local/apache-kylin-4.0.0-bin/logs/kylin.log
Web UI is at http://<hostname>:7070/kylin
```
-#### step6、访问kylin
+#### step7、访问kylin
Kylin 启动后您可以通过浏览器 http://<hostname>:port/kylin 进行访问。
其中 <hostname> 为具体的机器名、IP 地址或域名，port为kylin端口，默认为7070。
初始用户名和密码是 ADMIN/KYLIN。
服务器启动后，可以通过查看 $KYLIN_HOME/logs/kylin.log 获得运行时日志。
-#### step7、创建Sample Cube
+#### step8、创建Sample Cube
Kylin提供了一个创建样例Cube的脚本，以供用户快速体验Kylin。
在命令行运行：
@@ -179,39 +121,28 @@ $KYLIN_HOME/bin/sample.sh
```
完成后登陆kylin，点击System->Configuration->Reload Metadata来重载元数据
-元数据重载完成后你可以在左上角的Project中看到一个名为learn_kylin的项目，它包含kylin_sales_cube和kylin_streaming_cube, 它们分别为batch cube和streaming cube，你可以直接对kylin_sales_cube进行构建，构建完成后就可以查询。
-对于kylin_streaming_cube，需要设置KAFKA_HOME指向你的kafka安装目录:
-
-```
-export KAFKA_HOME=/path/to/kafka
-```
-
-然后执行
-
-```
-${KYLIN_HOME}/bin/sample-streaming.sh
-```
-
-该脚本会在 localhost:9092 broker 中创建名为 kylin_streaming_topic 的 Kafka Topic，它也会每秒随机发送 100 条 messages 到 kylin_streaming_topic，然后你可以对kylin_streaming_cube进行构建。
-
-关于sample cube，可以参考[Sample Cube](/cn/docs/tutorial/kylin_sample.html)。
+元数据重载完成后你可以在左上角的Project中看到一个名为learn_kylin的项目，它包含kylin_sales_cube和kylin_streaming_cube, 它们分别为batch cube和streaming cube，不过 kylin4.0 暂时还不支持 streaming cube，你可以直接对kylin_sales_cube进行构建，构建完成后就可以查询。
当然，你也可以根据下面的教程来尝试创建自己的Cube。
-#### step8、创建project
+#### step9、创建project
登陆kylin后，点击左上角的+号来创建Project：
-#### step9、加载Hive表
+#### step10、加载Hive表
点击Model->Data Source->Load Table From Tree，
Kylin会读取到Hive数据源中的表并以树状方式显示出来，你可以选择自己要使用的表，然后点击sync进行将其加载到kylin。
+此外，Kylin4.0 还支持 CSV 格式文件作为数据源，你也可以点击 Model->Data Source->Load CSV File as Table 来加载 CSV 数据源。
+
+本例中仍然使用 Hive 数据源进行讲解与演示。
+
-#### step10、创建模型
+#### step11、创建模型
点击Model->New->New Model：
@@ -235,7 +166,7 @@ Kylin会读取到Hive数据源中的表并以树状方式显示出来，你可
最后点击Save保存模型。
-#### step11、创建Cube
+#### step12、创建Cube
选择Model->New->New Cube
@@ -245,11 +176,11 @@ Kylin会读取到Hive数据源中的表并以树状方式显示出来，你可
-点击Next到下一步，点击+Measure来添加需要预计算的度量。Kylin会默认创建一个Count(1)的度量。Kylin支持SUM、MIN、MAX、COUNT、COUNT_DISTINCT、TOP_N、EXTENDED_COLUMN、PERCENTILE八种度量。请为COUNT_DISTINCT和TOP_N选择合适的返回类型，这关系到Cube的大小。添加完成之后点击ok，该Measure将会显示在Measures列表中
+点击Next到下一步，点击+Measure来添加需要预计算的度量。Kylin会默认创建一个Count(1)的度量。Kylin 4.x支持SUM、MIN、MAX、COUNT、COUNT_DISTINCT、TOP_N、PERCENTILE度量。请为COUNT_DISTINCT和TOP_N选择合适的返回类型，这关系到Cube的大小。添加完成之后点击ok，该Measure将会显示在Measures列表中
-添加完所有Measure后点击Next进行下一步，这一页是关于Cube数据刷新的设置。在这里可以设施自动合并的阈值（Auto Merge Thresholds）、数据保留的最短时间（Retention Threshold）以及第一个Segment的起点时间。
+添加完所有Measure后点击Next进行下一步，这一页是关于Cube数据刷新的设置。在这里可以设置自动合并的阈值（Auto Merge Thresholds）、数据保留的最短时间（Retention Threshold）以及第一个Segment的起点时间。
@@ -261,7 +192,7 @@ Kylin会读取到Hive数据源中的表并以树状方式显示出来，你可
-对于高级设置不是很熟悉时可以先保持默认设置，点击Next跳转到Kylin Properties页面，你可以在这里重写cube级别的kylin配置项，定义覆盖的属性，配置项请参考[配置项](/cn/docs/install/configuration.html)。
+对于高级设置不是很熟悉时可以先保持默认设置，点击Next跳转到Kylin Properties页面，你可以在这里重写cube级别的kylin配置项，定义覆盖的属性，配置项请参考[配置项](https://cwiki.apache.org/confluence/display/KYLIN/Configuration+Kylin+4.X)。
@@ -282,17 +213,17 @@ Cube的构建方式通常有两种：全量构建和增量构建。
-设置好起止时间后，点击Submit提交构建任务。然后你可以在Monitor页面观察构建任务的状态。Kylin会在页面上显示每一个步骤的运行状态、输出日志以及MapReduce任务。可以在${KYLIN_HOME}/logs/kylin.log中查看更详细的日志信息。
+设置好起止时间后，点击Submit提交构建任务。然后你可以在Monitor页面观察构建任务的状态。Kylin会在页面上显示每一个步骤的运行状态、输出日志。可以在${KYLIN_HOME}/logs/kylin.log中查看更详细的日志信息。
-
+
任务构建完成后，Cube状态会变成READY，并且可以看到Segment的信息。
-
+
#### step13、查询Cube
-Cube构建完成后，在Insight页面的Tables列表下面可以看到构建完成的Cube的table，并可以对其进行查询.查询语句击中Cube后会返回存储在Hbase中的预计算结果。
+Cube构建完成后，在Insight页面的Tables列表下面可以看到构建完成的Cube的table，并可以对其进行查询.查询语句击中Cube后会返回存储在HDFS中的预计算结果。
diff --git a/website/_docs/gettingstarted/quickstart.md b/website/_docs/gettingstarted/quickstart.md
index 27fe8dc..33ff52e 100644
--- a/website/_docs/gettingstarted/quickstart.md
+++ b/website/_docs/gettingstarted/quickstart.md
@@ -6,87 +6,21 @@ permalink: /docs/gettingstarted/kylin-quickstart.html
since: v0.6.x
---
-This guide aims to provide novice Kylin users with a complete process guide from download and installation to a sub-second query experience. The guide is divided into two parts, which respectively introduce the installation and use of Kylin in two scenarios – with an installation based on a Hadoop environment and installation from Docker image without a Hadoop environment.
+This guide aims to provide novice Kylin users with a complete process guide from download and installation to a sub-second query experience.
Users can follow these steps to get an initial understanding of how to use Kylin, master the basic skills of Kylin and then use Kylin to design models and speed up queries based on their own business scenarios.
-### 01 Install Kylin From a Docker Image
-
-In order to make it easy for users to try out Kylin, Zhu Weibin of Ant Financial has contributed “Kylin Docker Image” to the community. In this image, various services that Kylin depends on have been installed and deployed, including:
-
-- JDK 1.8
-- Hadoop 2.7.0
-- Hive 1.2.1
-- Hbase 1.1.2 (with Zookeeper)
-- Spark 2.3.1
-- Kafka 1.1.1
-- MySQL 5.1.73
-
-We have uploaded the user facing Kylin image to the Docker repository. Users do not need to build the image locally; they only need to install Docker to experience Kylin’s one-click installation.
-
-#### Step1
-First, execute the following command to pull the image from the Docker repository:
-
-```
-docker pull apachekylin/apache-kylin-standalone:3.1.0
-```
-
-The image here contains the latest version of Kylin: Kylin v3.1.0. This image contains all of the big data components that Kylin depends on, so it takes a long time to pull the image – please be patient. After the pull is successful, it is displayed as follows:
-
-
-
-#### Step2
-Execute the following command to start the container:
-
-```
-docker run -d \
--m 8G \
--p 7070:7070 \
--p 8088:8088 \
--p 50070:50070 \
--p 8032:8032 \
--p 8042:8042 \
--p 16010:16010 \
-apachekylin/apache-kylin-standalone:3.1.0
-```
-
-The container will start shortly. Since the specified port in the container has been mapped to the local port, you can directly open the pages of each service in the local browser, such as:
-- Kylin Page: http://127.0.0.1:7070/kylin/
-- HDFS NameNode Page: http://127.0.0.1:50070
-- Yarn ResourceManager Page: http://127.0.0.1:8088
-- HBase Page: http://127.0.0.1:60010
-
-When the container starts, the following services are automatically started:
-- NameNode, DataNode
-- ResourceManager, NodeManager
-- HBase
-- Kafka
-- Kylin
-
-It will also automatically run $ KYLIN_HOME / bin / sample.sh and create a kylin_streaming_topic in Kafka and continue to send data to that topic to allow users to experience building and querying cubes in batches and streams as soon as the container is launched.
-
-Users can enter the container through the docker exec command. The relevant environment variables in the container are as follows:
-- JAVA_HOME = /home/admin/jdk1.8.0_141
-- HADOOP_HOME = /home/admin/hadoop-2.7.0
-- KAFKA_HOME = /home/admin/kafka_2.11-1.1.1
-- SPARK_HOME = /home/admin/spark-2.3.1-bin-hadoop2.6
-- HBASE_HOME = /home/admin/hbase-1.1.2
-- HIVE_HOME = /home/admin/apache-hive-1.2.1-bin
-- KYLIN_HOME = /home/admin/apache-kylin-3.1.0-bin-hbase1x
-
-After logging in to Kylin with user/password of ADMIN/KYLIN, users can use the sample cube to experience the construction and query of the cube, or they can create and query their own models and cubes by following the tutorial from Step 8 in “Install and Use Kylin Based on a Hadoop Environment” below.
-
-### 02 Install and Use Kylin Based on a Hadoop Environment
+### Install and Use Kylin Based on a Hadoop Environment
Users who already have a stable Hadoop environment can download Kylin’s binary package and deploy it on their Hadoop cluster. Before installation, check the environment according to the following requirements.
#### Environmental Inspection
-(1) Pre-Conditions: Kylin relies on a Hadoop cluster to process large data sets. You need to prepare a Hadoop cluster configured with HDFS, YARN, MapReduce, Hive, HBase, Zookeeper and other services for Kylin to run.
+(1) Pre-Conditions: Kylin relies on a Hadoop cluster to process large data sets. You need to prepare a Hadoop cluster configured with HDFS, YARN, MapReduce, Hive, Zookeeper and other services for Kylin to run.
-Kylin can be started on any node of a Hadoop cluster. For your convenience, you can run Kylin on the master node, but for better stability, we recommend that you deploy Kylin on a clean Hadoop client node. The Hive, HBase, HDFS and other command lines have been installed on the node and the client configuration (such as core-site.xml, hive-site.xml, hbase-site.xml and others) have been properly configured and they can automatically synchronize with other nodes.
+Kylin can be started on any node of a Hadoop cluster. For your convenience, you can run Kylin on the master node, but for better stability, we recommend that you deploy Kylin on a clean Hadoop client node. The Hive, HDFS and other command lines have been installed on the node and the client configuration (such as core-site.xml, hive-site.xml and others) have been properly configured and they can automatically synchronize with other nodes.
-The Linux account running Kylin must have access to the Hadoop cluster, including permissions to create/write HDFS folders, Hive tables, HBase tables and submit MapReduce tasks.
+The Linux account running Kylin must have access to the Hadoop cluster, including permissions to create/write HDFS folders, Hive tables.
@@ -98,34 +32,34 @@ The Linux account running Kylin must have access to the Hadoop cluster, includin
-(4) Software Requirements: Hadoop 2.7+, 3.0-3.1; Hive 0.13+, 1.2.1+; HBase 1.1+, 2.0 (supported since Kylin 2.5); JDK: 1.8+
+(4) Software Requirements: Hadoop 2.7+, 3.0-3.1; Hive 0.13+, 1.2.1+; Spark 2.4.6; JDK: 1.8+
-It is recommended to use an integrated Hadoop environment for Kylin installation and testing, such as Hortonworks HDP or Cloudera CDH. Before Kylin was released, Hortonworks HDP 2.2-2.6 and 3.0, Cloudera CDH 5.7-5.11 and 6.0, AWS EMR 5.7-5.10, and Azure HDInsight 3.5-3.6 passed the test.
+It is recommended to use an integrated Hadoop environment for Kylin installation and testing, such as Hortonworks HDP or Cloudera CDH. Before Kylin was released, Hortonworks HDP 2.4, Cloudera CDH 5.7 and 6.0, AWS EMR 5.31 and 6.0, and Azure HDInsight 4.0 passed the test.
#### Install and Use
When your environment meets the above prerequisites, you can install and start using Kylin.
#### Step1. Download the Kylin Archive
-Download a binary for your version of Hadoop from [Apache Kylin Download Site](https://kylin.apache.org/download/). Currently, the latest versions are Kylin 3.1.0 and Kylin 2.6.6, of which, version 3.0 supports the function of ingesting data in real time for pre-calculation. If your Hadoop environment is CDH 5.7, you can download Kylin 3.1.0 using the following command line:
+Download a kylin4.0 binary package from [Apache Kylin Download Site](https://kylin.apache.org/download/).
```
cd /usr/local/
-wget http://apache.website-solution.net/kylin/apache-kylin-3.1.0/apache-kylin-3.1.0-bin-cdh57.tar.gz
+wget http://apache.website-solution.net/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin.tar.gz
```
#### Step2. Extract Kylin
Extract the downloaded Kylin archive and configure the environment variable KYLIN_HOME to point to the extracted directory:
```
-tar -zxvf apache-kylin-3.1.0-bin-cdh57.tar.gz
-cd apache-kylin-3.1.0-bin-cdh57
+tar -zxvf apache-kylin-4.0.0-bin.tar.gz
+cd apache-kylin-4.0.0-bin
export KYLIN_HOME=`pwd`
```
#### Step3. Download Spark
-Since Kylin checks the Spark environment when it starts, you need to set SPARK_HOME:
+Kylin 4.0 uses spark as query engine and build engine, you need to set SPARK_HOME:
```
export SPARK_HOME=/path/to/spark
@@ -139,12 +73,24 @@ $KYLIN_HOME/bin/download-spark.sh
The script will place the decompressed Spark in the $ KYLIN_HOME directory. If SPARK_HOME is not set in the system, the Spark in the $ KYLIN_HOME directory will be found automatically when Kylin is started.
-#### Step4. Environmental Inspection
+#### Step4. Configure MySQL metastore
+
+Kylin 4.0 uses MySQL as metadata storage, make the following configuration in `kylin.properties`:
+
+```$xslt
+kylin.metadata.url=kylin_metadata@jdbc,driverClassName=com.mysql.jdbc.Driver,url=jdbc:mysql//localhost:3306/kylin_test,username=,password=
+kylin.env.zookeeper-connect-string=ip:2181
+```
+
+You need to change the Mysql user name and password, as well as the database and table where the metadata is stored.
+Please refer to [Configure Mysql as Metastore](/_docs/tutorial/mysql_metastore.html) learn about the detailed configuration of MySQL as a Metastore.
+
+#### Step5. Environmental Inspection
Kylin runs on a Hadoop cluster and has certain requirements for the version, access permissions and CLASSPATH of each component.
To avoid encountering various environmental problems, you can run the $ KYLIN_HOME / bin / check-env.sh script to perform an environment check to see if there are any problems.
The script will print out detailed error messages if any errors are identified. If there is no error message, your environment is suitable for Kylin operation.
-#### Step5. Start Kylin
+#### Step6. Start Kylin
Run
```
@@ -155,17 +101,17 @@ Start script to start Kylin. If the startup is successful, the following will be
```
A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
-Check the log at /usr/local/apache-kylin-3.1.0-bin-cdh57/logs/kylin.log
+Check the log at /usr/local/apache-kylin-4.0.0-bin/logs/kylin.log
Web UI is at http://<hostname>:7070/kylin
```
The default port started by Kylin is 7070. You can use $ KYLIN_HOME/bin/kylin-port-replace-util.sh set number to modify the port. The modified port is 7070 + number.
-#### Step6. Visit Kylin
+#### Step7. Visit Kylin
After Kylin starts, you can access it through your browser: http://<hostname>:port/kylin – where <hostname> is the specific machine name, IP address or domain name, port is the Kylin port and the default is 7070.
The initial username and password are ADMIN/KYLIN. After the server starts, you can get the runtime log by looking at $ KYLIN_HOME/logs/kylin.log.
-#### Step7. Create Sample Cube
+#### Step8. Create Sample Cube
Kylin provides a script to create a sample cube for users to quickly experience Kylin. Run from the command line:
```
@@ -175,29 +121,29 @@ $KYLIN_HOME/bin/sample.sh
After completing, log in to Kylin, click System -> Configuration -> Reload Metadata to reload the metadata.
After the metadata is reloaded, you can see a project named learn_kylin in Project in the upper left corner.
-This contains kylin_sales_cube and kylin_streaming_cube, which are a batch cube and a streaming cube, respectively.
+This contains kylin_sales_cube and kylin_streaming_cube, which are a batch cube and a streaming cube, respectively. However, kylin 4.0 does not support streaming cube yet.
You can build the kylin_sales_cube directly and you can query it after the build is completed.
-For kylin_streaming_cube, you need to set KAFKA_HOME and then execute $ {KYLIN_HOME} /bin/sample-streaming.sh.
-This script will create a Kafka Topic named kylin_streaming_topic in the localhost: 9092 broker and it will also randomly send 100 messages to kylin_streaming_topic, then you can build kylin_streaming_cube.
-
-For sample cube, you can refer to:[Sample Cube](/docs/tutorial/kylin_sample.html)
Of course, you can also try to create your own cube based on the following tutorial.
-#### Step8. Create Project
+#### Step9. Create Project
After logging in to Kylin, click the + in the upper left corner to create a Project.
-#### Step9. Load Hive Table
-Click Model -> the Data Source -> the Load the From the Table Tree.
+#### Step10. Load Hive Table
+Click `Model -> the Data Source -> the Load the From the Table Tree`.
Kylin reads the Hive data source table and displays it in a tree. You can choose the tables you would like to add to models and then click Sync. The selected tables will then be loaded into Kylin.
They then appear in the Tables directory of the data source.
-#### Step10. Create the Model
+In addition, kylin 4.0 also supports CSV file as data source. You can also click `model -> data source -> Load CSV file as table` to load the CSV data source.
+
+In this example, Hive data source is still used for explanation and demonstration.
+
+#### Step11. Create the Model
Click Model -> New -> New Model:
@@ -220,7 +166,7 @@ The next step is to set the time partition column and filter conditions. The tim
Then, click Save to save the model.
-#### Step11. Create Cube
+#### Step12. Create Cube
Model -> New -> New Cube:
@@ -232,7 +178,7 @@ Click Next to add Dimension. The dimensions of the Lookup Table can be set to No
Click Next and click + Measure to add a pre-calculated measure.
-Kylin creates a Count (1) metric by default. Kylin supports eight metrics: SUM, MIN, MAX, COUNT, COUNT_DISTINCT, TOP_N, EXTENDED_COLUMN and PERCENTILE.
+Kylin creates a Count (1) metric by default. Kylin supports metrics: SUM, MIN, MAX, COUNT, COUNT_DISTINCT, TOP_N and PERCENTILE.
Please select the appropriate return type for COUNT_DISTINCT and TOP_N, which is related to the size of the cube.
Click OK after the addition is complete and the measure will be displayed in the Measures list.
@@ -254,7 +200,7 @@ For more dimensional optimization, you can read: [aggregation-group](/blog/2016/
If you are not familiar with Advanced Settings, you can keep the default settings first. Click Next to jump to the Kylin Properties page. Here you can override the cube-level Kylin configuration items and define the properties to be covered.
-For configuration items, please refer to: [configuration](/docs/install/configuration.html).
+For configuration items, please refer to: [configuration](https://cwiki.apache.org/confluence/display/KYLIN/Configuration+Kylin+4.X).
@@ -265,7 +211,7 @@ After that, this cube will appear in your cube list.
-#### Step12. Build Cube
+#### Step13. Build Cube
The cube created in the previous step has definitions but no calculated data. Its status is “DISABLED” and it cannot be queried. If you want the cube to have data, you need to build it. Cubes are usually built in one of two ways: full builds or incremental builds.
@@ -282,15 +228,15 @@ You can then observe the status of the build task on the Monitor page.
Kylin displays the running status of each step on the page, the output log and MapReduce tasks.
You can view more detailed log information in ${KYLIN_HOME}/logs/kylin.log.
-
+
After the job is built, the status of the cube will change to READY and you can see the segment information.
-
+
-#### Step13. Query Cube
+#### Step14. Query Cube
After the cube is built, you can see the table of the built cube and query it under the Tables list on the Insight page.
-After the query hits the cube, it returns the pre-calculated results stored in HBase.
+After the query hits the cube, it returns the pre-calculated results stored in HDFS.
diff --git a/website/_docs/gettingstarted/terminology.md b/website/_docs/gettingstarted/terminology.md
index 2d998b3..ceaec3f 100644
--- a/website/_docs/gettingstarted/terminology.md
+++ b/website/_docs/gettingstarted/terminology.md
@@ -3,7 +3,7 @@ layout: docs
title: "Terminology"
categories: gettingstarted
permalink: /docs/gettingstarted/terminology.html
-since: v0.5.x
+since: v4.0.0
---
diff --git a/website/_docs/howto/howto_backup_metadata.cn.md b/website/_docs/howto/howto_backup_metadata.cn.md
index 16be351..7f92c75 100644
--- a/website/_docs/howto/howto_backup_metadata.cn.md
+++ b/website/_docs/howto/howto_backup_metadata.cn.md
@@ -5,18 +5,18 @@ categories: 帮助
permalink: /cn/docs/howto/howto_backup_metadata.html
---
-Kylin将它全部的元数据（包括cube描述和实例、项目、倒排索引描述和实例、任务、表和字典）组织成层级文件系统的形式。然而，Kylin 使用 HBase 来存储元数据，而不是一个普通的文件系统。如果你查看过Kylin的配置文件（kylin.properties），你会发现这样一行：
+Kylin将它全部的元数据（包括cube描述和实例、项目、倒排索引描述和实例、任务、表和字典）组织成层级文件系统的形式。然而，Kylin 使用 Mysql 来存储元数据，而不是一个普通的文件系统。如果你查看过Kylin的配置文件（kylin.properties），你会发现这样一行：
{% highlight Groff markup %}
-## The metadata store in hbase
-kylin.metadata.url=kylin_metadata@hbase
+## The metadata store in mysql
+kylin.metadata.url=kylin_metadata@jdbc,driverClassName=com.mysql.jdbc.Driver,url=jdbc:mysql://localhost:3306/kylin_database,username=,password=
{% endhighlight %}
-这表明元数据会被保存在一个叫作 “kylin_metadata”的htable 里。你可以在 hbase shell 里 scan 该 htbale 来获取它。
+这表明元数据会被保存在一个叫作 “kylin_metadata” 的 table 里, 并且该元数据 table 位于名为 kylin_database 的 database。
## 元数据路径
-Kylin使用`resource root path + resource name + resource suffix`作为key值(HBase中的rowkey)来存储元数据。你可以参考如下表格使用`./bin/metastore.sh`命令。
+Kylin使用`resource root path + resource name + resource suffix`作为key值来存储元数据。你可以参考如下表格使用`./bin/metastore.sh`命令。
| Resource root path | resource name | resource suffix
| --------------------| :---------------------| :--------------|
@@ -24,20 +24,16 @@ Kylin使用`resource root path + resource name + resource suffix`作为key值(HB
| /cube_desc | /cube name | .json |
| /cube_statistics | /cube name/uuid | .seq |
| /model_desc | /model name | .json |
-| /dict | /DATABASE.TABLE/COLUMN/uuid | .dict |
| /project | /project name | .json |
-| /table_snapshot | /DATABASE.TABLE/uuid | .snapshot |
| /table | /DATABASE.TABLE--project name | .json |
| /table_exd | /DATABASE.TABLE--project name | .json |
| /execute | /job id | |
| /execute_output | /job id-step index | |
-| /kafka | /DATABASE.TABLE | .json |
-| /streaming | /DATABASE.TABLE | .json |
| /user | /user name | |
## 查看元数据
-Kylin以二进制字节的格式将元数据存储在HBase中，如果你想要查看一些元数据，可以运行：
+Kylin 以二进制字节的格式将元数据存储在 Mysql 中，如果你想要查看一些元数据，可以运行：
{% highlight Groff markup %}
./bin/metastore.sh list /path/to/store/metadata
@@ -53,7 +49,7 @@ Kylin以二进制字节的格式将元数据存储在HBase中，如果你想要
## 使用二进制包来备份 metadata
-有时你需要将 Kylin 的 metadata store 从 hbase 备份到磁盘文件系统。在这种情况下，假设你在部署 Kylin 的 hadoop 命令行（或沙盒）里，你可以到KYLIN_HOME并运行：
+有时你需要将 Kylin 的 metadata store 从 Mysql 备份到磁盘文件系统。在这种情况下，假设你在部署 Kylin 的 hadoop 命令行（或沙盒）里，你可以到KYLIN_HOME并运行：
{% highlight Groff markup %}
./bin/metastore.sh backup
@@ -73,7 +69,7 @@ Kylin以二进制字节的格式将元数据存储在HBase中，如果你想要
万一你发现你的元数据被搞得一团糟，想要恢复先前的备份：
-首先，重置 metatdara store（这个会清理 Kylin 在 HBase 的 metadata store的所有信息，请确保先备份）：
+首先，重置 metatdara store（这个会清理 Kylin 在 Mysql 的 metadata store的所有信息，请确保先备份）：
{% highlight Groff markup %}
./bin/metastore.sh reset
@@ -109,23 +105,8 @@ cd $KYLIN_HOME
./bin/metastore.sh restore /path/to/restore_new
{% endhighlight %}
-只有在此文件夹中的文件才会上传到Kylin Metastore。 同样，在恢复完成后，单击 Web UI 上的“Reload Metadata”按钮以刷新缓存。
+只有在此文件夹中的文件才会上传到 Kylin Metastore。 同样，在恢复完成后，单击 Web UI 上的“Reload Metadata”按钮以刷新缓存。
## 在开发环境备份/恢复元数据
在开发调试 Kylin 时，典型的环境是一台装有 IDE 的开发机上和一个后台的沙盒，通常你会写代码并在开发机上运行测试案例，但每次都需要将二进制包放到沙盒里以检查元数据是很麻烦的。这时有一个名为 SandboxMetastoreCLI 工具类可以帮助你在开发机本地下载/上传元数据。
-
-## 从 metadata store 清理无用的资源
-随着运行时间增长，类似字典、表快照的资源变得没有用（cube segment被丢弃或者合并了），但是它们依旧占用空间，你可以运行命令来找到并清除它们：
-
-首先，运行一个检查，这是安全的因为它不会改变任何东西，通过添加 "--jobThreshold 30(默认值，您可以改为任何数字)" 参数，您可以设置要保留的 metadata resource 天数：
-{% highlight Groff markup %}
-./bin/metastore.sh clean --jobThreshold 30
-{% endhighlight %}
-
-将要被删除的资源会被列出来：
-
-接下来，增加 “--delete true” 参数来清理这些资源；在这之前，你应该确保已经备份 metadata store：
-{% highlight Groff markup %}
-./bin/metastore.sh clean --delete true --jobThreshold 30
-{% endhighlight %}
diff --git a/website/_docs/howto/howto_backup_metadata.md b/website/_docs/howto/howto_backup_metadata.md
index 5c8cecf..8a5e418 100644
--- a/website/_docs/howto/howto_backup_metadata.md
+++ b/website/_docs/howto/howto_backup_metadata.md
@@ -5,18 +5,18 @@ categories: howto
permalink: /docs/howto/howto_backup_metadata.html
---
-Kylin organizes all of its metadata (including cube descriptions and instances, projects, inverted index description and instances, jobs, tables and dictionaries) as a hierarchy file system. However, Kylin uses hbase to store it, rather than normal file system. If you check your kylin configuration file(kylin.properties) you will find such a line:
+Kylin organizes all of its metadata (including cube descriptions and instances, projects, inverted index description and instances, jobs, tables and dictionaries) as a hierarchy file system. However, Kylin uses mysql to store it, rather than normal file system. If you check your kylin configuration file(kylin.properties) you will find such a line:
{% highlight Groff markup %}
-## The metadata store in hbase
-kylin.metadata.url=kylin_metadata@hbase
+## The metadata store in mysql
+kylin.metadata.url=kylin_metadata@jdbc,driverClassName=com.mysql.jdbc.Driver,url=jdbc:mysql://localhost:3306/kylin_database,username=,password=
{% endhighlight %}
-This indicates that the metadata will be saved as a htable called `kylin_metadata`. You can scan the htable in hbase shell to check it out.
+This indicates that the metadata will be saved as a table called `kylin_metadata` in mysql database `kylin_database`.
## Metadata directory
-Kylin metastore use `resource root path + resource name + resource suffix` as key (rowkey in hbase) to store metadata. You can refer to the following table to use `./bin/metastore.sh`.
+Kylin metastore use `resource root path + resource name + resource suffix` as key to store metadata. You can refer to the following table to use `./bin/metastore.sh`.
| Resource root path | resource name | resource suffix
| --------------------| :---------------------| :--------------|
@@ -24,20 +24,16 @@ Kylin metastore use `resource root path + resource name + resource suffix` as ke
| /cube_desc | /cube name | .json |
| /cube_statistics | /cube name/uuid | .seq |
| /model_desc | /model name | .json |
-| /dict | /DATABASE.TABLE/COLUMN/uuid | .dict |
| /project | /project name | .json |
-| /table_snapshot | /DATABASE.TABLE/uuid | .snapshot |
| /table | /DATABASE.TABLE--project name | .json |
| /table_exd | /DATABASE.TABLE--project name | .json |
| /execute | /job id | |
| /execute_output | /job id-step index | |
-| /kafka | /DATABASE.TABLE | .json |
-| /streaming | /DATABASE.TABLE | .json |
| /user | /user name | |
## View metadata
-Kylin store metadata in Byte format in HBase. If you want to view some metadata, you can run:
+If you want to view some metadata, you can run:
{% highlight Groff markup %}
./bin/metastore.sh list /path/to/store/metadata
@@ -53,7 +49,7 @@ to view one entity metadata.
## Backup metadata with binary package
-Sometimes you need to backup the Kylin's metadata store from hbase to your disk file system.
+Sometimes you need to backup the Kylin's metadata store from mysql to your disk file system.
In such cases, assuming you're on the hadoop CLI(or sandbox) where you deployed Kylin, you can go to KYLIN_HOME and run :
{% highlight Groff markup %}
@@ -74,7 +70,7 @@ to dump metadata selectively. For example, run `./bin/metastore.sh fetch /cube_d
In case you find your metadata store messed up, and you want to restore to a previous backup:
-Firstly, reset the metadata store (this will clean everything of the Kylin metadata store in hbase, make sure to backup):
+Firstly, reset the metadata store (this will clean everything of the Kylin metadata store in mysql, make sure to backup):
{% highlight Groff markup %}
./bin/metastore.sh reset
@@ -115,18 +111,3 @@ Only the files in the folder will be uploaded to Kylin metastore. Similarly, aft
## Backup/restore metadata in development env
When developing/debugging Kylin, typically you have a dev machine with an IDE, and a backend sandbox. Usually you'll write code and run test cases at dev machine. It would be troublesome if you always have to put a binary package in the sandbox to check the metadata. There is a helper class called SandboxMetastoreCLI to help you download/upload metadata locally at your dev machine. Follow the Usage information and run it in your IDE.
-
-## Cleanup unused resources from metadata store
-As time goes on, some resources like dictionary, table snapshots became useless (as the cube segment be dropped or merged), but they still take space there; You can run command to find and cleanup them from metadata store:
-
-Firstly, run a check, this is safe as it will not change anything, you can set the number of days to keep metadata resource by adding the "--jobThreshold 30(default, you can change to any number)" option:
-{% highlight Groff markup %}
-./bin/metastore.sh clean --jobThreshold 30
-{% endhighlight %}
-
-The resources that will be dropped will be listed;
-
-Next, add the "--delete true" parameter to cleanup those resources; before this, make sure you have made a backup of the metadata store;
-{% highlight Groff markup %}
-./bin/metastore.sh clean --delete true --jobThreshold 30
-{% endhighlight %}
diff --git a/website/_docs/howto/howto_build_cube_with_restapi.cn.md b/website/_docs/howto/howto_build_cube_with_restapi.cn.md
index aeed2e2..0e183f5 100644
--- a/website/_docs/howto/howto_build_cube_with_restapi.cn.md
+++ b/website/_docs/howto/howto_build_cube_with_restapi.cn.md
@@ -52,3 +52,6 @@ Content-Type: application/json;charset=UTF-8
### 5. 如果构建任务出现错误，可以重新开始它
* `PUT http://localhost:7070/kylin/api/jobs/{job_uuid}/resume`
+
+### 6. 调整某个 cube 中的 cuboid list，触发 optimize segment 任务
+* `PUT http://localhost:7070/kylin/api/cubes/{cube_name}/optimize2`
diff --git a/website/_docs/howto/howto_build_cube_with_restapi.md b/website/_docs/howto/howto_build_cube_with_restapi.md
index bb06f8f..13d705f 100644
--- a/website/_docs/howto/howto_build_cube_with_restapi.md
+++ b/website/_docs/howto/howto_build_cube_with_restapi.md
@@ -51,3 +51,6 @@ Content-Type: application/json;charset=UTF-8
### 5. If the job got errors, you can resume it.
* `PUT http://localhost:7070/kylin/api/jobs/{job_uuid}/resume`
+
+### 6. Adjust the cuboid list of a cube and trigger optimize segment job
+* `PUT http://localhost:7070/kylin/api/cubes/{cube_name}/optimize2`
diff --git a/website/_docs/howto/howto_cleanup_storage.cn.md b/website/_docs/howto/howto_cleanup_storage.cn.md
index 2dbd1d6..913dcaa 100644
--- a/website/_docs/howto/howto_cleanup_storage.cn.md
+++ b/website/_docs/howto/howto_cleanup_storage.cn.md
@@ -5,22 +5,4 @@ categories: 帮助
permalink: /cn/docs/howto/howto_cleanup_storage.html
---
-Kylin 在构建 cube 期间会在 HDFS 上生成中间文件；除此之外，当清理/删除/合并 cube 时，一些 HBase 表可能被遗留在 HBase 却以后再也不会被查询；虽然 Kylin 已经开始做自动化的垃圾回收，但不一定能覆盖到所有的情况；你可以定期做离线的存储清理：
-
-步骤：
-1. 检查哪些资源可以清理，这一步不会删除任何东西：
-{% highlight Groff markup %}
-export KYLIN_HOME=/path/to/kylin_home
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false
-{% endhighlight %}
-请将这里的 (version) 替换为你安装的 Kylin jar 版本。
-2. 你可以抽查一两个资源来检查它们是否已经没有被引用了；然后加上“--delete true”选项进行清理。
-{% highlight Groff markup %}
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true
-{% endhighlight %}
-完成后，Hive 里的中间表, HDFS 上的中间文件及 HBase 中的 HTables 都会被移除。
-3. 如果您想要删除所有资源；可添加 "--force true" 选项：
-{% highlight Groff markup %}
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --force true --delete true
-{% endhighlight %}
-完成后，Hive 中所有的中间表, HDFS 上所有的中间文件及 HBase 中的 HTables 都会被移除。
+Wiki : https://cwiki.apache.org/confluence/display/KYLIN/How+to+clean+up+storage+in+Kylin+4
diff --git a/website/_docs/howto/howto_cleanup_storage.md b/website/_docs/howto/howto_cleanup_storage.md
index 043116a..6d6c4df 100644
--- a/website/_docs/howto/howto_cleanup_storage.md
+++ b/website/_docs/howto/howto_cleanup_storage.md
@@ -5,23 +5,4 @@ categories: howto
permalink: /docs/howto/howto_cleanup_storage.html
---
-Kylin will generate intermediate files in HDFS during the cube building; Besides, when purge/drop/merge cubes, some HBase tables may be left in HBase and will no longer be queried; Although Kylin has started to do some
-automated garbage collection, it might not cover all cases; You can do an offline storage cleanup periodically:
-
-Steps:
-1. Check which resources can be cleanup, this will not remove anything:
-{% highlight Groff markup %}
-export KYLIN_HOME=/path/to/kylin_home
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete false
-{% endhighlight %}
-Here please replace (version) with the specific Kylin jar version in your installation;
-2. You can pickup 1 or 2 resources to check whether they're no longer be referred; Then add the "--delete true" option to start the cleanup:
-{% highlight Groff markup %}
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true
-{% endhighlight %}
-On finish, the intermediate Hive tables, HDFS location and HTables should be dropped;
-3. If you want to delete all resources, then add the "--force true" option to start the cleanup:
-{% highlight Groff markup %}
-${KYLIN_HOME}/bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob --force true --delete true
-{% endhighlight %}
-On finish, all the intermediate Hive tables, HDFS location and HTables should be dropped;
+Wiki : https://cwiki.apache.org/confluence/display/KYLIN/How+to+clean+up+storage+in+Kylin+4
diff --git a/website/_docs40/howto/howto_config_spark_pool.md b/website/_docs/howto/howto_config_spark_pool.md
similarity index 67%
rename from website/_docs40/howto/howto_config_spark_pool.md
rename to website/_docs/howto/howto_config_spark_pool.md
index e87abc3..964897a 100644
--- a/website/_docs40/howto/howto_config_spark_pool.md
+++ b/website/_docs/howto/howto_config_spark_pool.md
@@ -1,8 +1,8 @@
---
-layout: docs40
+layout: docs
title: Config different spark Pool for different types of SQL
categories: howto
-permalink: /docs40/howto/howto_config_spark_pool.html
+permalink: /docs/howto/howto_config_spark_pool.html
---
Please check document: [Use different spark pool for different query](https://cwiki.apache.org/confluence/display/KYLIN/Use+different+spark+pool+for+different+query)
\ No newline at end of file
diff --git a/website/_docs40/howto/howto_optimize_build_and_query.cn.md b/website/_docs/howto/howto_optimize_build_and_query.cn.md
similarity index 90%
rename from website/_docs40/howto/howto_optimize_build_and_query.cn.md
rename to website/_docs/howto/howto_optimize_build_and_query.cn.md
index 2e11099..ec13829 100644
--- a/website/_docs40/howto/howto_optimize_build_and_query.cn.md
+++ b/website/_docs/howto/howto_optimize_build_and_query.cn.md
@@ -1,8 +1,8 @@
---
-layout: docs40-cn
+layout: docs-cn
title: 优化构建和查询性能
categories: howto
-permalink: /cn/docs40/howto/howto_optimize_build_and_query.html
+permalink: /cn/docs/howto/howto_optimize_build_and_query.html
since: v4.0.0
---
diff --git a/website/_docs40/howto/howto_optimize_build_and_query.md b/website/_docs/howto/howto_optimize_build_and_query.md
similarity index 90%
rename from website/_docs40/howto/howto_optimize_build_and_query.md
rename to website/_docs/howto/howto_optimize_build_and_query.md
index bc3147f..1f610f0 100644
--- a/website/_docs40/howto/howto_optimize_build_and_query.md
+++ b/website/_docs/howto/howto_optimize_build_and_query.md
@@ -1,8 +1,8 @@
---
-layout: docs40
+layout: docs
title: Optimize Build and Query
categories: Optimize Build and Query
-permalink: /docs40/howto/howto_optimize_build_and_query.html
+permalink: /docs/howto/howto_optimize_build_and_query.html
since: v4.0.0
---
diff --git a/website/_docs/howto/howto_upgrade.md b/website/_docs/howto/howto_upgrade.md
index 35441d6..f8f6909 100644
--- a/website/_docs/howto/howto_upgrade.md
+++ b/website/_docs/howto/howto_upgrade.md
@@ -3,122 +3,8 @@ layout: docs
title: Upgrade From Old Versions
categories: howto
permalink: /docs/howto/howto_upgrade.html
-since: v1.5.1
---
-Running as a Hadoop client, Apache Kylin's metadata and Cube data are persistended in Hadoop (HBase and HDFS), so the upgrade is relatively easy and user does not need worry about data loss. The upgrade can be performed in the following steps:
-
-* Download the new Apache Kylin binary package for your Hadoop version from Kylin download page.
-* Unpack the new version Kylin package to a new folder, e.g, /usr/local/kylin/apache-kylin-2.1.0/ (directly overwrite old instance is not recommended).
-* Merge the old configuration files (`$KYLIN_HOME/conf/*`) into the new ones. It is not recommended to overwrite the new configuration files, although that works in most cases. If you have modified tomcat configuration ($KYLIN_HOME/tomcat/conf/), do the same for it.
-* Stop the current Kylin instance with `bin/kylin.sh stop`
-* Set the `KYLIN_HOME` env variable to the new installation folder. If you have set `KYLIN_HOME` in `~/.bash_profile` or other scripts, remember to update them as well.
-* Start the new Kylin instance with `$KYLIN_HOME/bin/kylin start`. After be started, login Kylin web to check whether your cubes can be loaded correctly.
-* [Upgrade coprocessor](howto_update_coprocessor.html) to ensure the HBase region servers use the latest Kylin coprocessor.
-* Verify your SQL queries can be performed successfully.
-
-Below are versions specific guides:
-
-## Upgrade from 3.0 to 3.1.0
-
-1)`Set Config` on web of kylin v3.1.0 is turned off by default.
-2) [KYLIN-4478](https://issues.apache.org/jira/browse/KYLIN-4478) modified the encryption algorithm. If you enable LDAP in kylin, you need to re-encoded the password of the LDAP server configured in kylin.properties:
-
-```
-cd $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/lib
-java -classpath kylin-server-base-\<versioin\>.jar:kylin-core-common-\<versioin\>.jar:spring-beans-4.3.10.RELEASE.jar:spring-core-4.3.10.RELEASE.jar:commons-codec-1.7.jar org.apache.kylin.rest.security.PasswordPlaceholderConfigurer AES <your_password>
-```
-
-Then update the re-encoded password to `kylin.security.ldap.connection-password=<your_password_encrypted>`.
-
-## Upgrade from 2.4 to 2.5.0
-
-* Kylin 2.5 need Java 8; Please upgrade Java if you're running with Java 7.
-* Kylin metadata is compatible between 2.4 and 2.5. No migration is needed.
-* Spark engine will move more steps from MR to Spark, you may see performance difference for the same cube after the upgrade.
-* Property `kylin.source.jdbc.sqoop-home` need be the location of sqoop installation, not its "bin" subfolder, please modify it if you're using RDBMS as the data source.
-* The Cube planner is enabled by default now; New cubes will be optimized by it on first build. System cube and dashboard still need manual enablement.
-
-## Upgrade from v2.1.0 to v2.2.0
-
-Kylin v2.2.0 cube metadata is compitable with v2.1.0, but you need aware the following changes:
-
-* Cube ACL is removed, use Project Level ACL instead. You need to manually configure Project Permissions to migrate your existing Cube Permissions. Please refer to [Project Level ACL](/docs21/tutorial/project_level_acl.html).
-* Update HBase coprocessor. The HBase tables for existing cubes need be updated to the latest coprocessor. Follow [this guide](/docs21/howto/howto_update_coprocessor.html) to update.
-
-
-## Upgrade from v2.0.0 to v2.1.0
-
-Kylin v2.1.0 cube metadata is compitable with v2.0.0, but you need aware the following changes.
-
-1) In previous version, Kylin uses additional two HBase tables "kylin_metadata_user" and "kylin_metadata_acl" to persistent the user and ACL info. From 2.1, Kylin consolidates all the info into one table: "kylin_metadata". This will make the backup/restore and maintenance more easier. When you start Kylin 2.1.0, it will detect whether need migration; if true, it will print the command to do migration:
-
-```
-ERROR: Legacy ACL metadata detected. Please migrate ACL metadata first. Run command 'bin/kylin.sh org.apache.kylin.tool.AclTableMigrationCLI MIGRATE'.
-```
-
-After the migration finished, you can delete the legacy "kylin_metadata_user" and "kylin_metadata_acl" tables from HBase.
-
-2) From v2.1, Kylin hides the default settings in "conf/kylin.properties"; You only need uncomment or add the customized properties in it.
-
-3) Spark is upgraded from v1.6.3 to v2.1.1, if you customized Spark configurations in kylin.properties, please upgrade them as well by referring to [Spark documentation](https://spark.apache.org/docs/2.1.0/).
-
-4) If you are running Kylin with two clusters (compute/query separated), need copy the big metadata files (which are persisted in HDFS instead of HBase) from the Hadoop cluster to HBase cluster.
-
-```
-hadoop distcp hdfs://compute-cluster:8020/kylin/kylin_metadata/resources hdfs://query-cluster:8020/kylin/kylin_metadata/resources
-```
-
-
-## Upgrade from v1.6.0 to v2.0.0
-
-Kylin v2.0.0 can read v1.6.0 metadata directly. Please follow the common upgrade steps above.
-
-Configuration names in `kylin.properties` have changed since v2.0.0. While the old property names still work, it is recommended to use the new property names as they follow [the naming convention](/development/coding_naming_convention.html) and are easier to understand. There is [a mapping from the old properties to the new properties](https://github.com/apache/kylin/blob/2.0.x/core-common/src/main/resources/kylin-backward-compatibility.properties).
-
-## Upgrade from v1.5.4 to v1.6.0
-
-Kylin v1.5.4 and v1.6.0 are compatible in metadata. Please follow the common upgrade steps above.
-
-## Upgrade from v1.5.3 to v1.5.4
-Kylin v1.5.3 and v1.5.4 are compatible in metadata. Please follow the common upgrade steps above.
-
-## Upgrade from 1.5.2 to v1.5.3
-Kylin v1.5.3 metadata is compatible with v1.5.2, your cubes don't need rebuilt, as usual, some actions need to be performed:
-
-#### 1. Update HBase coprocessor
-The HBase tables for existing cubes need be updated to the latest coprocessor; Follow [this guide](howto_update_coprocessor.html) to update;
-
-#### 2. Update conf/kylin_hive_conf.xml
-From 1.5.3, Kylin doesn't need Hive to merge small files anymore; For users who copy the conf/ from previous version, please remove the "merge" related properties in kylin_hive_conf.xml, including "hive.merge.mapfiles", "hive.merge.mapredfiles", and "hive.merge.size.per.task"; this will save the time on extracting data from Hive.
-
-
-## Upgrade from 1.5.1 to v1.5.2
-Kylin v1.5.2 metadata is compatible with v1.5.1, your cubes don't need upgrade, while some actions need to be performed:
-
-#### 1. Update HBase coprocessor
-The HBase tables for existing cubes need be updated to the latest coprocessor; Follow [this guide](howto_update_coprocessor.html) to update;
-
-#### 2. Update conf/kylin.properties
-In v1.5.2 several properties are deprecated, and several new one are added:
-
-Deprecated:
-
-* kylin.hbase.region.cut.small=5
-* kylin.hbase.region.cut.medium=10
-* kylin.hbase.region.cut.large=50
-
-New:
-
-* kylin.hbase.region.cut=5
-* kylin.hbase.hfile.size.gb=2
-
-These new parameters determines how to split HBase region; To use different size you can overwite these params in Cube level.
-
-When copy from old kylin.properties file, suggest to remove the deprecated ones and add the new ones.
-
-#### 3. Add conf/kylin\_job\_conf\_inmem.xml
-A new job conf file named "kylin\_job\_conf\_inmem.xml" is added in "conf" folder; As Kylin 1.5 introduced the "fast cubing" algorithm, which aims to leverage more memory to do the in-mem aggregation; Kylin will use this new conf file for submitting the in-mem cube build job, which requesting different memory with a normal job; Please update it properly according to your cluster capacity.
-
-Besides, if you have used separate config files for different capacity cubes, for example "kylin\_job\_conf\_small.xml", "kylin\_job\_conf\_medium.xml" and "kylin\_job\_conf\_large.xml", please note that they are deprecated now; Only "kylin\_job\_conf.xml" and "kylin\_job\_conf\_inmem.xml" will be used for submitting cube job; If you have cube level job configurations (like using different Yarn job queue), you can customize at cube level, check [KYLIN-1706](https://issues.apache.org/jira [...]
+Compared with Kylin 3.x and previous versions, Kylin 4.0's storage engine has changed from HBase to Parquet. Therefore, if you need to upgrade from Kylin 3.x and previous versions to kylin4.0, the built cuboid data can't be upgraded, you can only upgrade metadata.
+Please refer to : [How to migrate metadata to Kylin 4](https://cwiki.apache.org/confluence/display/KYLIN/How+to+migrate+metadata+to+Kylin+4)
\ No newline at end of file
diff --git a/website/_docs/howto/howto_use_cli.cn.md b/website/_docs/howto/howto_use_cli.cn.md
index 1fbe5a1..a54f1ff 100644
--- a/website/_docs/howto/howto_use_cli.cn.md
+++ b/website/_docs/howto/howto_use_cli.cn.md
@@ -54,7 +54,7 @@ CubeMetaExtractor.java 用于提取与 cube 相关的信息以达到调试/分
./bin/kylin.sh org.apache.kylin.tool.extractor.CubeMetaExtractor -cube querycube -destDir /root/newconfigdir1
{% endhighlight %}
结果：
-命令执行成功后，您想要抽取的 cube / project / hybrid 将会存在于您指定的 destDir 目录中。
+命令执行成功后，您想要抽取的 cube/project/hybrid 将会存在于您指定的 destDir 目录中。
下面会列出所有支持的参数：
@@ -92,31 +92,20 @@ CubeMetaIngester.java 将提取的 cube 注入到另一个 metadata store 中。
下面会列出所有支持的参数：
-| Parameter | Description |
-| -------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| forceIngest <forceIngest> | Skip the target cube, model and table check and ingest by force. Use in caution because it might break existing cubes! Suggest to backup metadata store first. Default false. |
-| overwriteTables <overwriteTables> | If table meta conflicts, overwrite the one in metadata store with the one in srcPath. Use in caution because it might break existing cubes! Suggest to backup metadata store first. Default false. |
-| createProjectIfNotExists <createProjectIdNotExists>| If the specified project is not exists, kylin will create it. |
-| project <project> | (Required) Specify the target project for the new cubes. |
-| srcPath <srcPath> | (Required) Specify the path to the extracted Cube metadata zip file. |
+| Parameter | Description |
+| --------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| forceIngest <forceIngest> | Skip the target cube, model and table check and ingest by force. Use in caution because it might break existing cubes! Suggest to backup metadata store first. Default false. |
+| overwriteTables <overwriteTables> | If table meta conflicts, overwrite the one in metadata store with the one in srcPath. Use in caution because it might break existing cubes! Suggest to backup metadata store first. Default false. |
+| project <project> | (Required) Specify the target project for the new cubes.
+| srcPath <srcPath> | (Required) Specify the path to the extracted Cube metadata zip file. |
-## CubeMigrationCheckCLI.java
+## CubeMigrationCLI.java
### 作用
-CubeMigrationCheckCLI.java 用于在迁移 Cube 之后检查“KYLIN_HOST”属性是否与 dst 中所有 Cube segment 对应的 HTable 的 MetadataUrlPrefix 一致。CubeMigrationCheckCLI.java 会在 CubeMigrationCLI.java 中被调用，通常不单独使用。
+自 Apache kylin 2.0 以来提供了迁移工具来支持跨不同集群迁移元数据。在 kylin4.0 中，我们对 CubeMigration 工具进行了改进并添加了新功能，增强功能列表如下所示：
+-支持迁移源集群中的所有多维数据集
+-支持在源集群中迁移整个项目
+-支持将元数据从旧版本迁移和升级到Kylin 4
### 如何使用
-{% highlight Groff markup %}
-./bin/kylin.sh org.apache.kylin.tool.CubeMigrationCheckCLI -fix <conf_value> -dstCfgUri <dstCfgUri_value> -cube <cube_name>
-{% endhighlight %}
-例如：
-{% highlight Groff markup %}
-./bin/kylin.sh org.apache.kylin.tool.CubeMigrationCheckCLI -fix true -dstCfgUri kylin-prod:7070 -cube querycube
-{% endhighlight %}
-下面会列出所有支持的参数：
-
-| Parameter | Description |
-| ------------------- | :---------------------------------------------------------------------------- |
-| fix | Fix the inconsistent Cube segments' HOST, default false |
-| dstCfgUri | The KylinConfig of the Cube’s new home |
-| cube | The name of Cube migrated |
+请参考文档：[How to migrate metadata to Kylin4](https://cwiki.apache.org/confluence/display/KYLIN/How+to+migrate+metadata+to+Kylin+4)
\ No newline at end of file
diff --git a/website/_docs/howto/howto_use_cli.md b/website/_docs/howto/howto_use_cli.md
index f99a7d0..972d6f8 100644
--- a/website/_docs/howto/howto_use_cli.md
+++ b/website/_docs/howto/howto_use_cli.md
@@ -48,11 +48,11 @@ CubeMetaExtractor.java is to extract Cube related info for debugging / distribut
### How to use
At least two parameters should be followed.
{% highlight Groff markup %}
-./bin/kylin.sh org.apache.kylin.tool.extractor.CubeMetaExtractor -<conf_name> <conf_value> -destDir <your_dest_dir>
+./bin/kylin.sh org.apache.kylin.tool.CubeMetaExtractor -<conf_name> <conf_value> -destDir <your_dest_dir>
{% endhighlight %}
For example:
{% highlight Groff markup %}
-./bin/kylin.sh org.apache.kylin.tool.extractor.CubeMetaExtractor -cube kylin_sales_cube -destDir /tmp/kylin_sales_cube
+./bin/kylin.sh org.apache.kylin.tool.CubeMetaExtractor -cube kylin_sales_cube -destDir /tmp/kylin_sales_cube
{% endhighlight %}
Result:
After the command is executed, the cube, project or hybrid you want to extract will be dumped in the specified path.
@@ -96,31 +96,20 @@ After the command is successfully executed, the cube you want to ingest will exi
All supported parameters are listed below:
-| Parameter | Description |
-| -------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| forceIngest <forceIngest> | Skip the target Cube, model and table check and ingest by force. Use in caution because it might break existing cubes! Suggest to backup metadata store first. Default false. |
-| overwriteTables <overwriteTables> | If table meta conflicts, overwrite the one in metadata store with the one in srcPath. Use in caution because it might break existing cubes! Suggest to backup metadata store first. Default false. |
-| createProjectIfNotExists <createProjectIdNotExists>| If the specified project is not exists, kylin will create it. |
-| project <project> | (Required) Specify the target project for the new cubes. |
-| srcPath <srcPath> | (Required) Specify the path to the extracted Cube metadata zip file. |
+| Parameter | Description |
+| --------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| forceIngest <forceIngest> | Skip the target Cube, model and table check and ingest by force. Use in caution because it might break existing cubes! Suggest to backup metadata store first. Default false. |
+| overwriteTables <overwriteTables> | If table meta conflicts, overwrite the one in metadata store with the one in srcPath. Use in caution because it might break existing cubes! Suggest to backup metadata store first. Default false. |
+| project <project> | (Required) Specify the target project for the new cubes. |
+| srcPath <srcPath> | (Required) Specify the path to the extracted Cube metadata zip file. |
-## CubeMigrationCheckCLI.java
+## CubeMigrationCLI.java
-### Function
-CubeMigrationCheckCLI.java serves for the purpose of checking the "KYLIN_HOST" property to be consistent with the dst's MetadataUrlPrefix for all of Cube segments' corresponding HTables after migrating a Cube. CubeMigrationCheckCLI.java will be called in CubeMigrationCLI.java and is usually not used separately.
+## Function
+Apache Kylin have provided migration tool to support migrating metadata across different clusters since version 2.0. Recently, we have refined and added new ability to CubeMigration tool, The list of enhanced functions is showed as below:
+- Support migrating all cubes in source cluster
+- Support migrating a whole project in source cluster
+- Support migrating and upgrading metadata from older version to Kylin 4
### How to use
-{% highlight Groff markup %}
-./bin/kylin.sh org.apache.kylin.tool.CubeMigrationCheckCLI -fix <conf_value> -dstCfgUri <dstCfgUri_value> -cube <cube_name>
-{% endhighlight %}
-For example:
-{% highlight Groff markup %}
-./bin/kylin.sh org.apache.kylin.tool.CubeMigrationCheckCLI -fix true -dstCfgUri kylin-prod:7070 -cube querycube
-{% endhighlight %}
-All supported parameters are listed below:
-
-| Parameter | Description |
-| ------------------- | :---------------------------------------------------------------------------- |
-| fix | Fix the inconsistent Cube segments' HOST, default false |
-| dstCfgUri | The KylinConfig of the Cube’s new home |
-| cube | The cube name. |
\ No newline at end of file
+Please check: [How to migrate metadata to Kylin4](https://cwiki.apache.org/confluence/display/KYLIN/How+to+migrate+metadata+to+Kylin+4)
\ No newline at end of file
diff --git a/website/_docs/howto/howto_use_restapi.cn.md b/website/_docs/howto/howto_use_restapi.cn.md
index e4d3851..a49f3cd 100644
--- a/website/_docs/howto/howto_use_restapi.cn.md
+++ b/website/_docs/howto/howto_use_restapi.cn.md
@@ -6,2170 +6,5 @@ permalink: /cn/docs/howto/howto_use_restapi.html
since: v0.7.1
---
-This page lists the major RESTful APIs provided by Kylin.
-* Authentication
- * [Authentication](#authentication)
-* Query
- * [Query](#query)
- * [Prepare query](#prepare-query)
- * [Save query](#save-query)
- * [Remove saved query](#remove-saved-query)
- * [Get saved queries](#get-saved-queries)
- * [Get running queries](#get-running-queries)
- * [Stop query](#stop-query)
- * [List queryable tables](#list-queryable-tables)
-* CUBE
- * [Create cube](#create-cube)
- * [Update cube](#update-cube)
- * [List cubes](#list-cubes)
- * [Get cube](#get-cube)
- * [Get cube descriptor (dimension, measure info, etc)](#get-cube-descriptor)
- * [Get data model (fact and lookup table info)](#get-data-model)
- * [Build cube](#build-cube)
- * [Enable cube](#enable-cube)
- * [Disable cube](#disable-cube)
- * [Purge cube](#purge-cube)
- * [Delete segment](#delete-segment)
- * [Auto-Merge segment](#auto-merge-segment)
- * [Get sql of a cube](#get-sql-of-a-cube)
- * [Get sql of a cube segment](#get-sql-of-a-cube-segment)
- * [Force rebuild lookup table snapshot](#force-rebuild-lookup-table-snapshot)
- * [Clone cube](#clone-cube)
- * [Delete Cube](#delete-cube)
- * [Get hbase info](#get-hbase-info)
- * [Get current cuboid](#get-current-cuboid)
- * [Migrate cube](#migrate-cube)
-* MODEL
- * [Create model](#create-model)
- * [Update model](#update-model)
- * [Get modelDescData](#get-modeldescdata)
- * [Delete model](#delete-model)
- * [Clone model](#clone-model)
-* JOB
- * [Resume job](#resume-job)
- * [Pause job](#pause-job)
- * [Drop job](#drop-job)
- * [Discard job](#discard-job)
- * [Get job status](#get-job-status)
- * [Get job step output](#get-job-step-output)
- * [Get job list](#get-job-list)
- * [Get job status overview](#get-job-status-overview)
- * [Resubmit realtime build job](#resubmit-realtime-build-job)
- * [Rollback job](#rollback-job)
-* Metadata
- * [Get Hive Table](#get-hive-table)
- * [Get Hive Tables](#get-hive-tables)
- * [Load Hive Tables](#load-hive-tables)
- * [Unload Hive Tables](#unload-hive-tables)
- * [Show databases in hive](#show-databases-in-hive)
- * [Show tables in a hive database](#show-tables-in-a-hive-database)
-* Cache
- * [Wipe cache](#wipe-cache)
- * [Announce wipe cache](#announce-wipe-cache)
- * [Hot load kylin config](#hot-load-kylin-config)
-* Streaming
- * [Initiate cube start position](#initiate-cube-start-position)
- * [Build stream cube](#build-stream-cube)
- * [Check segment holes](#check-segment-holes)
- * [Fill segment holes](#fill-segment-holes)
- * [Get streaming configs](#get-streaming-configs)
- * [Get Kafka configs](#get-kafka-configs)
- * [Create streaming schema](#create-streaming-schema)
- * [Update streaming tables](#update-streaming-schema)
-* ACL
- * [Get users can query the table](#get-users-can-query-the-table)
- * [Get users cannot query the table](#get-users-cannot-query-the-table)
- * [Put user to table blacklist](#put-user-to-table-blacklist)
- * [Delete user from table blacklist](#delete-user-from-table-blacklist)
-* Metrics
- * [Get all metrics](#get-all-metrics)
- * [Get specific type of metrics](#get-specific-type-of-metrics)
-
-## Authentication
-`POST /kylin/api/user/authentication`
-
-#### Request Header
-Authorization data encoded by basic auth is needed in the header, such as:
-Authorization:Basic {data}
-You can generate {data} by using below python script
-```
-python -c "import base64; print base64.standard_b64encode('$UserName:$Password')"
-```
-
-#### Response Body
-* userDetails - Defined authorities and status of current user.
-
-#### Response Sample
-
-```sh
-{
- "userDetails": {
- "username": "sample",
- "password": "null",
- "authorities": [
- {
- "authority": "ROLE_ADMIN"
- },
- {
- "authority": "ROLE_ANALYST"
- },
- {
- "authority": "ROLE_MODELER"
- },
- {
- "authority": "ALL_USERS"
- }
- ],
- "disabled": false,
- "defaultPassword": false,
- "locked": false,
- "lockedTime": 0,
- "wrongTime": 0,
- "uuid": "3704ba8c-deb1-ac47-729d-c1039c1bd6ec",
- "last_modified": 1585219480112,
- "version": "3.0.0.20500"
- }
-}
-```
-
-#### Curl Example
-
-```
-curl -c /path/to/cookiefile.txt -X POST -H "Authorization: Basic XXXXXXXXX" -H 'Content-Type: application/json' http://<host>:<port>/kylin/api/user/authentication
-```
-
-If login successfully, the JSESSIONID will be saved into the cookie file; In the subsequent http requests, attach the cookie, for example:
-
-```
-curl -b /path/to/cookiefile.txt -X PUT -H 'Content-Type: application/json' -d '{"startTime":'1423526400000', "endTime":'1423612800000', "buildType":"BUILD"}' http://<host>:<port>/kylin/api/cubes/your_cube/build
-```
-
-Alternatively, you can provide the username/password with option "user" in each curl call; please note this has the risk of password leak in shell history:
-
-
-```
-curl -X PUT --user ADMIN:KYLIN -H "Content-Type: application/json;charset=utf-8" -d '{ "startTime": 820454400000, "endTime": 821318400000, "buildType": "BUILD"}' http://localhost:7070/kylin/api/cubes/kylin_sales/build
-```
-
-***
-
-## Query
-`POST /kylin/api/query`
-
-#### Request Body
-* sql - `required` `string` The text of sql statement.
-* offset - `optional` `int` Query offset. If offset is set in sql, curIndex will be ignored.
-* limit - `optional` `int` Query limit. If limit is set in sql, perPage will be ignored.
-* acceptPartial - `optional` `bool` Whether accept a partial result or not, default be "false". Set to "false" for production use.
-* project - `optional` `string` Project to perform query. Default value is 'DEFAULT'.
-如果你想为你的查询指定某个cube回答，请参考 Wiki 文档：[Specify cube for your query](https://cwiki.apache.org/confluence/display/KYLIN/Specify+cube+for+your+query)
-
-#### Request Sample
-
-```sh
-{
- "sql":"select * from TEST_KYLIN_FACT",
- "offset":0,
- "limit":50000,
- "acceptPartial":false,
- "project":"DEFAULT"
-}
-```
-
-#### Curl Example
-
-```
-curl -X POST -H "Authorization: Basic XXXXXXXXX" -H "Content-Type: application/json" -d '{ "sql":"select count(*) from TEST_KYLIN_FACT", "project":"learn_kylin" }' http://localhost:7070/kylin/api/query
-```
-
-#### Response Body
-* columnMetas - Column metadata information of result set.
-* results - Data set of result.
-* cube - Cube used for this query.
-* affectedRowCount - Count of affected row by this sql statement.
-* isException - Whether this response is an exception.
-* ExceptionMessage - Message content of the exception.
-* Duration - Time cost of this query
-* Partial - Whether the response is a partial result or not. Decided by `acceptPartial` of request.
-
-#### Response Sample
-
-```sh
-{
- "columnMetas":[
- {
- "isNullable":1,
- "displaySize":0,
- "label":"CAL_DT",
- "name":"CAL_DT",
- "schemaName":null,
- "catelogName":null,
- "tableName":null,
- "precision":0,
- "scale":0,
- "columnType":91,
- "columnTypeName":"DATE",
- "readOnly":true,
- "writable":false,
- "caseSensitive":true,
- "searchable":false,
- "currency":false,
- "signed":true,
- "autoIncrement":false,
- "definitelyWritable":false
- },
- {
- "isNullable":1,
- "displaySize":10,
- "label":"LEAF_CATEG_ID",
- "name":"LEAF_CATEG_ID",
- "schemaName":null,
- "catelogName":null,
- "tableName":null,
- "precision":10,
- "scale":0,
- "columnType":4,
- "columnTypeName":"INTEGER",
- "readOnly":true,
- "writable":false,
- "caseSensitive":true,
- "searchable":false,
- "currency":false,
- "signed":true,
- "autoIncrement":false,
- "definitelyWritable":false
- }
- ],
- "results":[
- [
- "2013-08-07",
- "32996",
- "15",
- "15",
- "Auction",
- "10000000",
- "49.048952730908745",
- "49.048952730908745",
- "49.048952730908745",
- "1"
- ],
- [
- "2013-08-07",
- "43398",
- "0",
- "14",
- "ABIN",
- "10000633",
- "85.78317064220418",
- "85.78317064220418",
- "85.78317064220418",
- "1"
- ]
- ],
- "cube":"test_kylin_cube_with_slr_desc",
- "affectedRowCount":0,
- "isException":false,
- "exceptionMessage":null,
- "duration":3451,
- "partial":false
-}
-```
-
-
-## Prepare query
-`POST /kylin/api/query/prestate`
-
-#### Request Body
-* sql - `required` `string` The text of sql statement.
-* offset - `optional` `int` Query offset. If offset is set in sql, curIndex will be ignored.
-* limit - `optional` `int` Query limit. If limit is set in sql, perPage will be ignored.
-* acceptPartial - `optional` `bool` Whether accept a partial result or not, default be "false". Set to "false" for production use.
-* project - `optional` `string` Project to perform query. Default value is 'DEFAULT'.
-
-#### Request Sample
-
-```sh
-{
- "sql":"select * from TEST_KYLIN_FACT",
- "offset":0,
- "limit":50000,
- "acceptPartial":false,
- "project":"DEFAULT"
-}
-```
-
-
-## Save query
-`POST /kylin/api/saved_queries`
-
-#### Request Body
-* sql - `required` `string` The text of sql statement.
-* name - `required` `string` Sql name.
-* project - `required` `string` Project to perform query.
-* description - `optional` `string` Sql description.
-
-#### Request Sample
-
-```sh
-{
- "sql": "select count(*) from kylin_sales",
- "name": "test",
- "project": "learn_kylin"
-}
-```
-
-
-## Remove saved query
-`DELETE /kylin/api/saved_queries/{id}`
-
-#### Request Parameters
-* id - `required` `string` The id of saved query you want to remove
-
-
-## Get saved queries
-`GET /kylin/api/saved_queries`
-
-#### Response Sample
-```
-[
- {
- "name": "test",
- "project": "learn_kylin",
- "sql": "select count(*) from kylin_sales",
- "description": null,
- "id": "-1674470999"
- }
-]
-```
-
-## Get running queries
-`GET /kylin/api/query/runningQueries`
-
-## Stop Query
-`PUT /kylin/api/query/{queryId}/stop`
-
-#### Path Variable
-* queryId - `required` `String` The queryId of you want to stop. You can obtain it by `Get running queries`.
-
-
-## List queryable tables
-`GET /kylin/api/tables_and_columns`
-
-#### Path Variable
-* project - `required` `string` The project to load tables
-
-#### Response Sample
-```sh
-[
- {
- "columns":[
- {
- "table_NAME":"TEST_CAL_DT",
- "table_SCHEM":"EDW",
- "column_NAME":"CAL_DT",
- "data_TYPE":91,
- "nullable":1,
- "column_SIZE":-1,
- "buffer_LENGTH":-1,
- "decimal_DIGITS":0,
- "num_PREC_RADIX":10,
- "column_DEF":null,
- "sql_DATA_TYPE":-1,
- "sql_DATETIME_SUB":-1,
- "char_OCTET_LENGTH":-1,
- "ordinal_POSITION":1,
- "is_NULLABLE":"YES",
- "scope_CATLOG":null,
- "scope_SCHEMA":null,
- "scope_TABLE":null,
- "source_DATA_TYPE":-1,
- "iS_AUTOINCREMENT":null,
- "table_CAT":"defaultCatalog",
- "remarks":null,
- "type_NAME":"DATE"
- },
- {
- "table_NAME":"TEST_CAL_DT",
- "table_SCHEM":"EDW",
- "column_NAME":"WEEK_BEG_DT",
- "data_TYPE":91,
- "nullable":1,
- "column_SIZE":-1,
- "buffer_LENGTH":-1,
- "decimal_DIGITS":0,
- "num_PREC_RADIX":10,
- "column_DEF":null,
- "sql_DATA_TYPE":-1,
- "sql_DATETIME_SUB":-1,
- "char_OCTET_LENGTH":-1,
- "ordinal_POSITION":2,
- "is_NULLABLE":"YES",
- "scope_CATLOG":null,
- "scope_SCHEMA":null,
- "scope_TABLE":null,
- "source_DATA_TYPE":-1,
- "iS_AUTOINCREMENT":null,
- "table_CAT":"defaultCatalog",
- "remarks":null,
- "type_NAME":"DATE"
- }
- ],
- "table_NAME":"TEST_CAL_DT",
- "table_SCHEM":"EDW",
- "ref_GENERATION":null,
- "self_REFERENCING_COL_NAME":null,
- "type_SCHEM":null,
- "table_TYPE":"TABLE",
- "table_CAT":"defaultCatalog",
- "remarks":null,
- "type_CAT":null,
- "type_NAME":null
- }
-]
-```
-
-***
-
-## Create cube
-`POST /kylin/api/cubes`
-
-#### Request Body
-* cubeDescData - `required` `string` cubeDescData to create
-* cubeName - `required` `string` cubeName to create
-* projectName - `required` `string` projectName to which cube belongs
-
-#### Request Sample
-```
-{
-"cubeDescData":"{\"uuid\": \"0ef9b7a8-3929-4dff-b59d-2100aadc8dbf\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"kylin_test_cube\",\"is_draft\": false,\"model_name\": \"kylin_sales_model\",\"description\": \"\",\"null_string\": null,\"dimensions\": [{\"name\": \"TRANS_ID\",\"table\": \"KYLIN_SALES\",\"column\": \"TRANS_ID\",\"derived\": null},{\"name\": \"YEAR_BEG_DT\",\"table\": \"KYLIN_CAL_DT\",\"column\": null,\"derived\": [\"YEAR_BEG_DT\"]},{\"name\": \"MONTH_BEG_DT\ [...]
-"cubeName":"kylin_test_cube",
-"project":"learn_kylin"
-}
-```
-
-#### Response Sample
-```
-{
-"uuid": "7b3faf69-eca8-cc5f-25f9-49b0f0b5d404",
-"cubeName": "kylin_test_cube",
-"cubeDescData":"{\"uuid\": \"0ef9b7a8-3929-4dff-b59d-2100aadc8dbf\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"kylin_test_cube\",\"is_draft\": false,\"model_name\": \"kylin_sales_model\",\"description\": \"\",\"null_string\": null,\"dimensions\": [{\"name\": \"TRANS_ID\",\"table\": \"KYLIN_SALES\",\"column\": \"TRANS_ID\",\"derived\": null},{\"name\": \"YEAR_BEG_DT\",\"table\": \"KYLIN_CAL_DT\",\"column\": null,\"derived\": [\"YEAR_BEG_DT\"]},{\"name\": \"MONTH_BEG_DT\ [...]
-"streamingData": null,
-"kafkaData": null,
-"successful": true,
-"message": null,
-"project": "learn_kylin",
-"streamingCube": null
-}
-```
-
-## Update Cube
-`PUT /kylin/api/cubes`
-
-#### Request Body
-(Same as "Create Cube")
-
-#### Request Sample
-(Same as "Create Cube")
-
-#### Response Sample
-(Same as "Create Cube")
-
-
-## List cubes
-`GET /kylin/api/cubes`
-
-#### Request Parameters
-* offset - `required` `int` Offset used by pagination
-* limit - `required` `int ` Cubes per page.
-* cubeName - `optional` `string` Keyword for cube names. To find cubes whose name contains this keyword.
-* projectName - `optional` `string` Project name.
-
-#### Response Sample
-```sh
-[
- {
- "uuid":"1eaca32a-a33e-4b69-83dd-0bb8b1f8c53b",
- "last_modified":1407831634847,
- "name":"test_kylin_cube_with_slr_empty",
- "owner":null,
- "version":null,
- "descriptor":"test_kylin_cube_with_slr_desc",
- "cost":50,
- "status":"DISABLED",
- "segments":[
- ],
- "create_time":null,
- "source_records_count":0,
- "source_records_size":0,
- "size_kb":0
- }
-]
-```
-
-## Get cube
-`GET /kylin/api/cubes/{cubeName}`
-
-#### Path Variable
-* cubeName - `required` `string` Cube name to find.
-
-## Get cube descriptor
-`GET /kylin/api/cube_desc/{cubeName}`
-Get descriptor for specified cube instance.
-
-#### Path Variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-```sh
-[
- {
- "uuid": "0ef9b7a8-3929-4dff-b59d-2100aadc8dbf",
- "last_modified": 1574402902000,
- "version": "3.0.0.20500",
- "name": "kylin_sales_cube",
- "is_draft": false,
- "model_name": "kylin_sales_model",
- "description": "",
- "null_string": null,
- "dimensions": [
- {
- "name": "TRANS_ID",
- "table": "KYLIN_SALES",
- "column": "TRANS_ID",
- "derived": null
- },
- {
- "name": "YEAR_BEG_DT",
- "table": "KYLIN_CAL_DT",
- "column": null,
- "derived": [
- "YEAR_BEG_DT"
- ]
- },
- {
- "name": "MONTH_BEG_DT",
- "table": "KYLIN_CAL_DT",
- "column": null,
- "derived": [
- "MONTH_BEG_DT"
- ]
- },
- {
- "name": "WEEK_BEG_DT",
- "table": "KYLIN_CAL_DT",
- "column": null,
- "derived": [
- "WEEK_BEG_DT"
- ]
- },
- {
- "name": "USER_DEFINED_FIELD1",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": null,
- "derived": [
- "USER_DEFINED_FIELD1"
- ]
- },
- {
- "name": "USER_DEFINED_FIELD3",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": null,
- "derived": [
- "USER_DEFINED_FIELD3"
- ]
- },
- {
- "name": "META_CATEG_NAME",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": "META_CATEG_NAME",
- "derived": null
- },
- {
- "name": "CATEG_LVL2_NAME",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": "CATEG_LVL2_NAME",
- "derived": null
- },
- {
- "name": "CATEG_LVL3_NAME",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": "CATEG_LVL3_NAME",
- "derived": null
- },
- {
- "name": "LSTG_FORMAT_NAME",
- "table": "KYLIN_SALES",
- "column": "LSTG_FORMAT_NAME",
- "derived": null
- },
- {
- "name": "SELLER_ID",
- "table": "KYLIN_SALES",
- "column": "SELLER_ID",
- "derived": null
- },
- {
- "name": "BUYER_ID",
- "table": "KYLIN_SALES",
- "column": "BUYER_ID",
- "derived": null
- },
- {
- "name": "ACCOUNT_BUYER_LEVEL",
- "table": "BUYER_ACCOUNT",
- "column": "ACCOUNT_BUYER_LEVEL",
- "derived": null
- },
- {
- "name": "ACCOUNT_SELLER_LEVEL",
- "table": "SELLER_ACCOUNT",
- "column": "ACCOUNT_SELLER_LEVEL",
- "derived": null
- },
- {
- "name": "BUYER_COUNTRY",
- "table": "BUYER_ACCOUNT",
- "column": "ACCOUNT_COUNTRY",
- "derived": null
- },
- {
- "name": "SELLER_COUNTRY",
- "table": "SELLER_ACCOUNT",
- "column": "ACCOUNT_COUNTRY",
- "derived": null
- },
- {
- "name": "BUYER_COUNTRY_NAME",
- "table": "BUYER_COUNTRY",
- "column": "NAME",
- "derived": null
- },
- {
- "name": "SELLER_COUNTRY_NAME",
- "table": "SELLER_COUNTRY",
- "column": "NAME",
- "derived": null
- },
- {
- "name": "OPS_USER_ID",
- "table": "KYLIN_SALES",
- "column": "OPS_USER_ID",
- "derived": null
- },
- {
- "name": "OPS_REGION",
- "table": "KYLIN_SALES",
- "column": "OPS_REGION",
- "derived": null
- }
- ],
- "measures": [
- {
- "name": "GMV_SUM",
- "function": {
- "expression": "SUM",
- "parameter": {
- "type": "column",
- "value": "KYLIN_SALES.PRICE"
- },
- "returntype": "decimal(19,4)"
- }
- },
- {
- "name": "BUYER_LEVEL_SUM",
- "function": {
- "expression": "SUM",
- "parameter": {
- "type": "column",
- "value": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL"
- },
- "returntype": "bigint"
- }
- },
- {
- "name": "SELLER_LEVEL_SUM",
- "function": {
- "expression": "SUM",
- "parameter": {
- "type": "column",
- "value": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
- },
- "returntype": "bigint"
- }
- },
- {
- "name": "TRANS_CNT",
- "function": {
- "expression": "COUNT",
- "parameter": {
- "type": "constant",
- "value": "1"
- },
- "returntype": "bigint"
- }
- },
- {
- "name": "SELLER_CNT_HLL",
- "function": {
- "expression": "COUNT_DISTINCT",
- "parameter": {
- "type": "column",
- "value": "KYLIN_SALES.SELLER_ID"
- },
- "returntype": "hllc(10)"
- }
- },
- {
- "name": "TOP_SELLER",
- "function": {
- "expression": "TOP_N",
- "parameter": {
- "type": "column",
- "value": "KYLIN_SALES.PRICE",
- "next_parameter": {
- "type": "column",
- "value": "KYLIN_SALES.SELLER_ID"
- }
- },
- "returntype": "topn(100)",
- "configuration": {
- "topn.encoding.KYLIN_SALES.SELLER_ID": "dict",
- "topn.encoding_version.KYLIN_SALES.SELLER_ID": "1"
- }
- }
- }
- ],
- "rowkey": {
- "rowkey_columns": [
- {
- "column": "KYLIN_SALES.BUYER_ID",
- "encoding": "integer:4",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.SELLER_ID",
- "encoding": "integer:4",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.TRANS_ID",
- "encoding": "integer:4",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.PART_DT",
- "encoding": "date",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.LEAF_CATEG_ID",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "BUYER_COUNTRY.NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "SELLER_COUNTRY.NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.LSTG_FORMAT_NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.LSTG_SITE_ID",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.OPS_USER_ID",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.OPS_REGION",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- }
- ]
- },
- "hbase_mapping": {
- "column_family": [
- {
- "name": "F1",
- "columns": [
- {
- "qualifier": "M",
- "measure_refs": [
- "GMV_SUM",
- "BUYER_LEVEL_SUM",
- "SELLER_LEVEL_SUM",
- "TRANS_CNT"
- ]
- }
- ]
- },
- {
- "name": "F2",
- "columns": [
- {
- "qualifier": "M",
- "measure_refs": [
- "SELLER_CNT_HLL",
- "TOP_SELLER"
- ]
- }
- ]
- }
- ]
- },
- "aggregation_groups": [
- {
- "includes": [
- "KYLIN_SALES.PART_DT",
- "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
- "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
- "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
- "KYLIN_SALES.LEAF_CATEG_ID",
- "KYLIN_SALES.LSTG_FORMAT_NAME",
- "KYLIN_SALES.LSTG_SITE_ID",
- "KYLIN_SALES.OPS_USER_ID",
- "KYLIN_SALES.OPS_REGION",
- "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
- "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
- "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
- "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
- "BUYER_COUNTRY.NAME",
- "SELLER_COUNTRY.NAME"
- ],
- "select_rule": {
- "hierarchy_dims": [
- [
- "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
- "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
- "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
- "KYLIN_SALES.LEAF_CATEG_ID"
- ]
- ],
- "mandatory_dims": [
- "KYLIN_SALES.PART_DT"
- ],
- "joint_dims": [
- [
- "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
- "BUYER_COUNTRY.NAME"
- ],
- [
- "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
- "SELLER_COUNTRY.NAME"
- ],
- [
- "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
- "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
- ],
- [
- "KYLIN_SALES.LSTG_FORMAT_NAME",
- "KYLIN_SALES.LSTG_SITE_ID"
- ],
- [
- "KYLIN_SALES.OPS_USER_ID",
- "KYLIN_SALES.OPS_REGION"
- ]
- ]
- }
- }
- ],
- "signature": null,
- "notify_list": [],
- "status_need_notify": [],
- "partition_date_start": 1325376000000,
- "partition_date_end": 3153600000000,
- "auto_merge_time_ranges": [],
- "volatile_range": 0,
- "retention_range": 0,
- "engine_type": 2,
- "storage_type": 2,
- "override_kylin_properties": {
- "kylin.cube.aggrgroup.is-mandatory-only-valid": "true",
- "kylin.engine.spark.rdd-partition-cut-mb": "500"
- },
- "cuboid_black_list": [],
- "parent_forward": 3,
- "mandatory_dimension_set_list": [],
- "snapshot_table_desc_list": []
- }
-]
-```
-
-## Get data model
-`GET /kylin/api/model/{modelName}`
-
-#### Path Variable
-* modelName - `required` `string` Data model name, by default it should be the same with cube name.
-
-#### Response Sample
-```sh
-{
- "uuid": "ff527b94-f860-44c3-8452-93b17774c647",
- "name": "test_kylin_with_slr_model_desc",
- "lookups": [
- {
- "table": "EDW.TEST_CAL_DT",
- "join": {
- "type": "inner",
- "primary_key": [
- "CAL_DT"
- ],
- "foreign_key": [
- "CAL_DT"
- ]
- }
- },
- {
- "table": "DEFAULT.TEST_CATEGORY_GROUPINGS",
- "join": {
- "type": "inner",
- "primary_key": [
- "LEAF_CATEG_ID",
- "SITE_ID"
- ],
- "foreign_key": [
- "LEAF_CATEG_ID",
- "LSTG_SITE_ID"
- ]
- }
- }
- ],
- "capacity": "MEDIUM",
- "last_modified": 1442372116000,
- "fact_table": "DEFAULT.TEST_KYLIN_FACT",
- "filter_condition": null,
- "partition_desc": {
- "partition_date_column": "DEFAULT.TEST_KYLIN_FACT.CAL_DT",
- "partition_date_start": 0,
- "partition_date_format": "yyyy-MM-dd",
- "partition_type": "APPEND",
- "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
- }
-}
-```
-
-## Build cube
-`PUT /kylin/api/cubes/{cubeName}/build`
-
-#### Path Variable
-* cubeName - `required` `string` Cube name.
-
-#### Request Body
-* startTime - `required` `long` Start timestamp of data to build, e.g. 1388563200000 for 2014-1-1
-* endTime - `required` `long` End timestamp of data to build
-* buildType - `required` `string` Supported build type: 'BUILD', 'MERGE', 'REFRESH'
-
-#### Curl Example
-```
-curl -X PUT -H "Authorization: Basic XXXXXXXXX" -H 'Content-Type: application/json' -d '{"startTime":'1423526400000', "endTime":'1423612800000', "buildType":"BUILD"}' http://<host>:<port>/kylin/api/cubes/{cubeName}/build
-```
-
-#### Response Sample
-```
-{
- "uuid":"c143e0e4-ac5f-434d-acf3-46b0d15e3dc6",
- "last_modified":1407908916705,
- "name":"test_kylin_cube_with_slr_empty - 19700101000000_20140731160000 - BUILD - PDT 2014-08-12 22:48:36",
- "type":"BUILD",
- "duration":0,
- "related_cube":"test_kylin_cube_with_slr_empty",
- "related_segment":"19700101000000_20140731160000",
- "exec_start_time":0,
- "exec_end_time":0,
- "mr_waiting":0,
- "steps":[
- {
- "interruptCmd":null,
- "name":"Create Intermediate Flat Hive Table",
- "sequence_id":0,
- "exec_cmd":"hive -e \"DROP TABLE IF EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6;\nCREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\n(\nCAL_DT date\n,LEAF_CATEG_ID int\n,LSTG_SITE_ID int\n,META_CATEG_NAME string\n,CATEG_LVL2_NAME string\n,CATEG_LVL3_NAME string\n,LSTG_FORMAT_NAME string\n,SLR_SEGMENT_ [...]
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"SHELL_CMD_HADOOP",
- "info":null,
- "run_async":false
- },
- {
- "interruptCmd":null,
- "name":"Extract Fact Table Distinct Columns",
- "sequence_id":1,
- "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns -jobname Kylin_Fact_Distinct_Columns_test_kylin_cube_with_slr_empty_Step_1",
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"JAVA_CMD_HADOOP_FACTDISTINCT",
- "info":null,
- "run_async":true
- },
- {
- "interruptCmd":null,
- "name":"Load HFile to HBase Table",
- "sequence_id":12,
- "exec_cmd":" -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile/ -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -cubename test_kylin_cube_with_slr_empty",
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"JAVA_CMD_HADOOP_NO_MR_BULKLOAD",
- "info":null,
- "run_async":false
- }
- ],
- "job_status":"PENDING",
- "progress":0.0
-}
-```
-
-## Enable Cube
-`PUT /kylin/api/cubes/{cubeName}/enable`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-```sh
-{
- "uuid":"1eaca32a-a33e-4b69-83dd-0bb8b1f8c53b",
- "last_modified":1407909046305,
- "name":"test_kylin_cube_with_slr_ready",
- "owner":null,
- "version":null,
- "descriptor":"test_kylin_cube_with_slr_desc",
- "cost":50,
- "status":"ACTIVE",
- "segments":[
- {
- "name":"19700101000000_20140531160000",
- "storage_location_identifier":"KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_READY-19700101000000_20140531160000_BF043D2D-9A4A-45E9-AA59-5A17D3F34A50",
- "date_range_start":0,
- "date_range_end":1401552000000,
- "status":"READY",
- "size_kb":4758,
- "source_records":6000,
- "source_records_size":620356,
- "last_build_time":1407832663227,
- "last_build_job_id":"2c7a2b63-b052-4a51-8b09-0c24b5792cda",
- "binary_signature":null,
- "dictionaries":{
- "TEST_CATEGORY_GROUPINGS/CATEG_LVL2_NAME":"/dict/TEST_CATEGORY_GROUPINGS/CATEG_LVL2_NAME/16d8185c-ee6b-4f8c-a919-756d9809f937.dict",
- "TEST_KYLIN_FACT/LSTG_SITE_ID":"/dict/TEST_SITES/SITE_ID/0bec6bb3-1b0d-469c-8289-b8c4ca5d5001.dict",
- "TEST_KYLIN_FACT/SLR_SEGMENT_CD":"/dict/TEST_SELLER_TYPE_DIM/SELLER_TYPE_CD/0c5d77ec-316b-47e0-ba9a-0616be890ad6.dict",
- "TEST_KYLIN_FACT/CAL_DT":"/dict/PREDEFINED/date(yyyy-mm-dd)/64ac4f82-f2af-476e-85b9-f0805001014e.dict",
- "TEST_CATEGORY_GROUPINGS/CATEG_LVL3_NAME":"/dict/TEST_CATEGORY_GROUPINGS/CATEG_LVL3_NAME/270fbfb0-281c-4602-8413-2970a7439c47.dict",
- "TEST_KYLIN_FACT/LEAF_CATEG_ID":"/dict/TEST_CATEGORY_GROUPINGS/LEAF_CATEG_ID/2602386c-debb-4968-8d2f-b52b8215e385.dict",
- "TEST_CATEGORY_GROUPINGS/META_CATEG_NAME":"/dict/TEST_CATEGORY_GROUPINGS/META_CATEG_NAME/0410d2c4-4686-40bc-ba14-170042a2de94.dict"
- },
- "snapshots":{
- "TEST_CAL_DT":"/table_snapshot/TEST_CAL_DT.csv/8f7cfc8a-020d-4019-b419-3c6deb0ffaa0.snapshot",
- "TEST_SELLER_TYPE_DIM":"/table_snapshot/TEST_SELLER_TYPE_DIM.csv/c60fd05e-ac94-4016-9255-96521b273b81.snapshot",
- "TEST_CATEGORY_GROUPINGS":"/table_snapshot/TEST_CATEGORY_GROUPINGS.csv/363f4a59-b725-4459-826d-3188bde6a971.snapshot",
- "TEST_SITES":"/table_snapshot/TEST_SITES.csv/78e0aecc-3ec6-4406-b86e-bac4b10ea63b.snapshot"
- }
- }
- ],
- "create_time":null,
- "source_records_count":6000,
- "source_records_size":0,
- "size_kb":4758
-}
-```
-
-## Disable Cube
-`PUT /kylin/api/cubes/{cubeName}/disable`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-(Same as "Enable Cube")
-
-## Purge Cube
-`PUT /kylin/api/cubes/{cubeName}/purge`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-(Same as "Enable Cube")
-
-
-## Delete Segment
-`DELETE /kylin/api/cubes/{cubeName}/segs/{segmentName}`
-
-
-## Auto-merge Segment
-`PUT /kylin/api/cubes/{cubeName}/automerge`
-
-
-## Get sql of a cube
-`GET /kylin/api/cubes/{cubeName}/sql`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-```sh
-{
- "sql": "SELECT\n`KYLIN_SALES`.`TRANS_ID` as `KYLIN_SALES_TRANS_ID`\n,`KYLIN_SALES`.`PART_DT` as `KYLIN_SALES_PART_DT`\n,`KYLIN_CAL_DT`.`YEAR_BEG_DT` as `KYLIN_CAL_DT_YEAR_BEG_DT`\n,`KYLIN_CAL_DT`.`MONTH_BEG_DT` as `KYLIN_CAL_DT_MONTH_BEG_DT`\n,`KYLIN_CAL_DT`.`WEEK_BEG_DT` as `KYLIN_CAL_DT_WEEK_BEG_DT`\n,`KYLIN_SALES`.`LEAF_CATEG_ID` as `KYLIN_SALES_LEAF_CATEG_ID`\n,`KYLIN_SALES`.`LSTG_SITE_ID` as `KYLIN_SALES_LSTG_SITE_ID`\n,`KYLIN_CATEGORY_GROUPINGS`.`USER_DEFINED_FIELD1` as `KYLIN_ [...]
-}
-```
-
-
-## Get sql of a cube segment
-`GET /kylin/api/cubes/{cubeName}/segs/{segmentName}/sql`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-* segmentName - `required` `string` Segment name.
-
-#### Response Sample
-```sh
-{
- "sql": "SELECT\n`KYLIN_SALES`.`TRANS_ID` as `KYLIN_SALES_TRANS_ID`\n,`KYLIN_SALES`.`PART_DT` as `KYLIN_SALES_PART_DT`\n,`KYLIN_CAL_DT`.`YEAR_BEG_DT` as `KYLIN_CAL_DT_YEAR_BEG_DT`\n,`KYLIN_CAL_DT`.`MONTH_BEG_DT` as `KYLIN_CAL_DT_MONTH_BEG_DT`\n,`KYLIN_CAL_DT`.`WEEK_BEG_DT` as `KYLIN_CAL_DT_WEEK_BEG_DT`\n,`KYLIN_SALES`.`LEAF_CATEG_ID` as `KYLIN_SALES_LEAF_CATEG_ID`\n,`KYLIN_SALES`.`LSTG_SITE_ID` as `KYLIN_SALES_LSTG_SITE_ID`\n,`KYLIN_CATEGORY_GROUPINGS`.`USER_DEFINED_FIELD1` as `KYLIN_ [...]
-}
-```
-
-
-## Force rebuild lookup table snapshot
-`PUT /kylin/api/cubes/{cubeName}/refresh_lookup`
-
-
-## Clone cube
-`PUT /kylin/api/cubes/{cubeName}/clone`
-
-
-## Delete cube
-`DELETE /kylin/api/cubes/{cubeName}`
-
-
-## Get hbase info
-`GET /kylin/api/cubes/{cubeName}/hbase`
-
-#### Response Sample
-```sh
-[
- {
- "segmentName": "20120101000000_20120103000000",
- "segmentUUID": null,
- "segmentStatus": "READY",
- "tableName": "KYLIN_E1VT22737D",
- "tableSize": 0,
- "regionCount": 1,
- "dateRangeStart": 1325376000000,
- "dateRangeEnd": 1325548800000,
- "sourceOffsetStart": 0,
- "sourceOffsetEnd": 0,
- "sourceCount": 29
- }
-]
-```
-
-
-## Get current cuboid
-`GET /kylin/api/cubes/{cubeName}/cuboids/current`
-
-
-## Migrate Cube
-`POST /kylin/api/cubes/{cube}/{project}/migrate`
-
-
-***
-
-## Create Model
-`POST /kylin/api/models`
-
-#### Request Body
-* modelDescData - `required` `string` modelDescData to create
-* modelName - `required` `string` modelName to create
-* projectName - `required` `string` projectName to which model belongs
-
-#### Request Sample
-```sh
-{
-"modelDescData": "{\"uuid\": \"0928468a-9fab-4185-9a14-6f2e7c74823f\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"kylin_test_model\",\"owner\": null,\"is_draft\": false,\"description\": \"\",\"fact_table\": \"DEFAULT.KYLIN_SALES\",\"lookups\": [{\"table\": \"DEFAULT.KYLIN_CAL_DT\",\"kind\": \"LOOKUP\",\"alias\": \"KYLIN_CAL_DT\",\"join\": {\"type\": \"inner\",\"primary_key\": [\"KYLIN_CAL_DT.CAL_DT\"],\"foreign_key\": [\"KYLIN_SALES.PART_DT\"]}},{\"table\": \"DEFAULT.KY [...]
-"modelName": "kylin_test_model",
-"project": "learn_kylin"
-}
-```
-
-#### Response Sample
-```sh
-{
-"uuid": "2613d739-14c1-38ac-2e37-f36e46fd9976",
-"modelName": "kylin_test_model",
-"modelDescData": "{\"uuid\": \"0928468a-9fab-4185-9a14-6f2e7c74823f\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"kylin_test_model\",\"owner\": null,\"is_draft\": false,\"description\": \"\",\"fact_table\": \"DEFAULT.KYLIN_SALES\",\"lookups\": [{\"table\": \"DEFAULT.KYLIN_CAL_DT\",\"kind\": \"LOOKUP\",\"alias\": \"KYLIN_CAL_DT\",\"join\": {\"type\": \"inner\",\"primary_key\": [\"KYLIN_CAL_DT.CAL_DT\"],\"foreign_key\": [\"KYLIN_SALES.PART_DT\"]}},{\"table\": \"DEFAULT.KY [...]
-"successful": true,
-"message": null,
-"project": "learn_kylin",
-"ccInCheck": null,
-"seekingExprAdvice": false
-}
-```
-
-
-## Update Model
-`PUT /kylin/api/models`
-
-#### Request Body
-(Same as "Create Model")
-
-#### Request Sample
-(Same as "Create Model")
-
-#### Response Sample
-(Same as "Create Model")
-
-
-## Get ModelDescData
-`GET /kylin/api/models`
-
-#### Request Parameters
-* modelName - `optional` `string` Model name.
-* projectName - `optional` `string` Project Name.
-* limit - `optional` `integer` Offset used by pagination
-* offset - `optional` `integer` Models per page
-
-#### Response Sample
-```sh
-[
- {
- "uuid": "0928468a-9fab-4185-9a14-6f2e7c74823f",
- "last_modified": 1568862496000,
- "version": "3.0.0.20500",
- "name": "kylin_sales_model",
- "owner": null,
- "is_draft": false,
- "description": "",
- "fact_table": "DEFAULT.KYLIN_SALES",
- "lookups": [
- {
- "table": "DEFAULT.KYLIN_CAL_DT",
- "kind": "LOOKUP",
- "alias": "KYLIN_CAL_DT",
- "join": {
- "type": "inner",
- "primary_key": [
- "KYLIN_CAL_DT.CAL_DT"
- ],
- "foreign_key": [
- "KYLIN_SALES.PART_DT"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_CATEGORY_GROUPINGS",
- "kind": "LOOKUP",
- "alias": "KYLIN_CATEGORY_GROUPINGS",
- "join": {
- "type": "inner",
- "primary_key": [
- "KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID",
- "KYLIN_CATEGORY_GROUPINGS.SITE_ID"
- ],
- "foreign_key": [
- "KYLIN_SALES.LEAF_CATEG_ID",
- "KYLIN_SALES.LSTG_SITE_ID"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_ACCOUNT",
- "kind": "LOOKUP",
- "alias": "BUYER_ACCOUNT",
- "join": {
- "type": "inner",
- "primary_key": [
- "BUYER_ACCOUNT.ACCOUNT_ID"
- ],
- "foreign_key": [
- "KYLIN_SALES.BUYER_ID"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_ACCOUNT",
- "kind": "LOOKUP",
- "alias": "SELLER_ACCOUNT",
- "join": {
- "type": "inner",
- "primary_key": [
- "SELLER_ACCOUNT.ACCOUNT_ID"
- ],
- "foreign_key": [
- "KYLIN_SALES.SELLER_ID"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_COUNTRY",
- "kind": "LOOKUP",
- "alias": "BUYER_COUNTRY",
- "join": {
- "type": "inner",
- "primary_key": [
- "BUYER_COUNTRY.COUNTRY"
- ],
- "foreign_key": [
- "BUYER_ACCOUNT.ACCOUNT_COUNTRY"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_COUNTRY",
- "kind": "LOOKUP",
- "alias": "SELLER_COUNTRY",
- "join": {
- "type": "inner",
- "primary_key": [
- "SELLER_COUNTRY.COUNTRY"
- ],
- "foreign_key": [
- "SELLER_ACCOUNT.ACCOUNT_COUNTRY"
- ]
- }
- }
- ],
- "dimensions": [
- {
- "table": "KYLIN_SALES",
- "columns": [
- "TRANS_ID",
- "SELLER_ID",
- "BUYER_ID",
- "PART_DT",
- "LEAF_CATEG_ID",
- "LSTG_FORMAT_NAME",
- "LSTG_SITE_ID",
- "OPS_USER_ID",
- "OPS_REGION"
- ]
- },
- {
- "table": "KYLIN_CAL_DT",
- "columns": [
- "CAL_DT",
- "WEEK_BEG_DT",
- "MONTH_BEG_DT",
- "YEAR_BEG_DT"
- ]
- },
- {
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "columns": [
- "USER_DEFINED_FIELD1",
- "USER_DEFINED_FIELD3",
- "META_CATEG_NAME",
- "CATEG_LVL2_NAME",
- "CATEG_LVL3_NAME",
- "LEAF_CATEG_ID",
- "SITE_ID"
- ]
- },
- {
- "table": "BUYER_ACCOUNT",
- "columns": [
- "ACCOUNT_ID",
- "ACCOUNT_BUYER_LEVEL",
- "ACCOUNT_SELLER_LEVEL",
- "ACCOUNT_COUNTRY",
- "ACCOUNT_CONTACT"
- ]
- },
- {
- "table": "SELLER_ACCOUNT",
- "columns": [
- "ACCOUNT_ID",
- "ACCOUNT_BUYER_LEVEL",
- "ACCOUNT_SELLER_LEVEL",
- "ACCOUNT_COUNTRY",
- "ACCOUNT_CONTACT"
- ]
- },
- {
- "table": "BUYER_COUNTRY",
- "columns": [
- "COUNTRY",
- "NAME"
- ]
- },
- {
- "table": "SELLER_COUNTRY",
- "columns": [
- "COUNTRY",
- "NAME"
- ]
- }
- ],
- "metrics": [
- "KYLIN_SALES.PRICE",
- "KYLIN_SALES.ITEM_COUNT"
- ],
- "filter_condition": "",
- "partition_desc": {
- "partition_date_column": "KYLIN_SALES.PART_DT",
- "partition_time_column": null,
- "partition_date_start": 1325376000000,
- "partition_date_format": "yyyy-MM-dd",
- "partition_time_format": "HH:mm:ss",
- "partition_type": "APPEND",
- "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
- },
- "capacity": "MEDIUM"
- }
-]
-```
-
-## Delete Model
-`DELETE /kylin/api/models/{modelName}`
-
-#### Path variable
-* modelName - `required` `string` Model name.
-
-## Clone Model
-`PUT /kylin/api/models/{modelName}/clone`
-
-***
-
-## Resume Job
-`PUT /kylin/api/jobs/{jobId}/resume`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-#### Response Sample
-```
-{
- "uuid":"c143e0e4-ac5f-434d-acf3-46b0d15e3dc6",
- "last_modified":1407908916705,
- "name":"test_kylin_cube_with_slr_empty - 19700101000000_20140731160000 - BUILD - PDT 2014-08-12 22:48:36",
- "type":"BUILD",
- "duration":0,
- "related_cube":"test_kylin_cube_with_slr_empty",
- "related_segment":"19700101000000_20140731160000",
- "exec_start_time":0,
- "exec_end_time":0,
- "mr_waiting":0,
- "steps":[
- {
- "interruptCmd":null,
- "name":"Create Intermediate Flat Hive Table",
- "sequence_id":0,
- "exec_cmd":"hive -e \"DROP TABLE IF EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6;\nCREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\n(\nCAL_DT date\n,LEAF_CATEG_ID int\n,LSTG_SITE_ID int\n,META_CATEG_NAME string\n,CATEG_LVL2_NAME string\n,CATEG_LVL3_NAME string\n,LSTG_FORMAT_NAME string\n,SLR_SEGMENT_ [...]
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"SHELL_CMD_HADOOP",
- "info":null,
- "run_async":false
- },
- {
- "interruptCmd":null,
- "name":"Extract Fact Table Distinct Columns",
- "sequence_id":1,
- "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns -jobname Kylin_Fact_Distinct_Columns_test_kylin_cube_with_slr_empty_Step_1",
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"JAVA_CMD_HADOOP_FACTDISTINCT",
- "info":null,
- "run_async":true
- },
- {
- "interruptCmd":null,
- "name":"Load HFile to HBase Table",
- "sequence_id":12,
- "exec_cmd":" -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile/ -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -cubename test_kylin_cube_with_slr_empty",
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"JAVA_CMD_HADOOP_NO_MR_BULKLOAD",
- "info":null,
- "run_async":false
- }
- ],
- "job_status":"PENDING",
- "progress":0.0
-}
-```
-## Pause Job
-`PUT /kylin/api/jobs/{jobId}/pause`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-## Discard Job
-`PUT /kylin/api/jobs/{jobId}/cancel`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-## Drop Job
-`DELETE /kylin/api/jobs/{jobId}/drop`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-## Get Job Status
-`GET /kylin/api/jobs/{jobId}`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-#### Response Sample
-(Same as "Resume Job")
-
-## Get job step output
-`GET /kylin/api/jobs/{jobId}/steps/{stepId}/output`
-
-#### Path Variable
-* jobId - `required` `string` Job id.
-* stepId - `required` `string` Step id; the step id is composed by jobId with step sequence id; for example, the jobId is "fb479e54-837f-49a2-b457-651fc50be110", its 3rd step id is "fb479e54-837f-49a2-b457-651fc50be110-3",
-
-#### Response Sample
-```
-{
- "cmd_output":"log string"
-}
-```
-
-## Get job list
-`GET /kylin/api/jobs`
-
-#### Request Variables
-* cubeName - `optional` `string` Cube name.
-* projectName - `required` `string` Project name.
-* status - `optional` `int` Job status, e.g. (NEW: 0, PENDING: 1, RUNNING: 2, STOPPED: 32, FINISHED: 4, ERROR: 8, DISCARDED: 16)
-* offset - `required` `int` Offset used by pagination.
-* limit - `required` `int` Jobs per page.
-* timeFilter - `required` `int`, e.g. (LAST ONE DAY: 0, LAST ONE WEEK: 1, LAST ONE MONTH: 2, LAST ONE YEAR: 3, ALL: 4)
-
-For example, to get the job list in project 'learn_kylin' for cube 'kylin_sales_cube' in lastone week:
-
-```
-GET: /kylin/api/jobs?cubeName=kylin_sales_cube&limit=15&offset=0&projectName=learn_kylin&timeFilter=1
-```
-
-#### Response Sample
-```
-[
- {
- "uuid": "9eb7bccf-4448-4578-9c29-552658b5a2ca",
- "last_modified": 1490957579843,
- "version": "2.0.0",
- "name": "Sample_Cube - 19700101000000_20150101000000 - BUILD - GMT+08:00 2017-03-31 18:36:08",
- "type": "BUILD",
- "duration": 936,
- "related_cube": "Sample_Cube",
- "related_segment": "53a5d7f7-7e06-4ea1-b3ee-b7f30343c723",
- "exec_start_time": 1490956581743,
- "exec_end_time": 1490957518131,
- "mr_waiting": 0,
- "steps": [
- {
- "interruptCmd": null,
- "id": "9eb7bccf-4448-4578-9c29-552658b5a2ca-00",
- "name": "Create Intermediate Flat Hive Table",
- "sequence_id": 0,
- "exec_cmd": null,
- "interrupt_cmd": null,
- "exec_start_time": 1490957508721,
- "exec_end_time": 1490957518102,
- "exec_wait_time": 0,
- "step_status": "DISCARDED",
- "cmd_type": "SHELL_CMD_HADOOP",
- "info": { "endTime": "1490957518102", "startTime": "1490957508721" },
- "run_async": false
- },
- {
- "interruptCmd": null,
- "id": "9eb7bccf-4448-4578-9c29-552658b5a2ca-01",
- "name": "Redistribute Flat Hive Table",
- "sequence_id": 1,
- "exec_cmd": null,
- "interrupt_cmd": null,
- "exec_start_time": 0,
- "exec_end_time": 0,
- "exec_wait_time": 0,
- "step_status": "DISCARDED",
- "cmd_type": "SHELL_CMD_HADOOP",
- "info": {},
- "run_async": false
- }
- ],
- "submitter": "ADMIN",
- "job_status": "FINISHED",
- "progress": 100.0
- }
-]
-```
-
-## Get Job Status Overview
-`GET /kylin/api/jobs/overview`
-
-### Request Variables
-(Same as "Get job list")
-
-#### Response Sample
-```sh
-{
- "DISCARDED": 0,
- "NEW": 0,
- "STOPPED": 0,
- "PENDING": 0,
- "RUNNING": 0,
- "FINISHED": 1,
- "ERROR": 0
-}
-```
-
-## Resubmit realtime build job
-`PUT /kylin/api/jobs/{jobId}/resubmit`
-
-## Rollback job
-`PUT /kylin/api/{jobId}/steps/{stepId}/rollback`
-
-#### Path Parameters
-* jobId - `required` `string` job id you want to rollback
-* stepId - `required` `string` specify rollback step id, e.g.(Create Intermediate Flat Hive:1)
-
-For example, rollback job to Create Intermediate Flat Hive:
-`PUT: kylin/api/jobs/4e84cb5e-a929-89c7-6240-768fa9835d89/steps/1/rollback`
-
-#### Response Sample
-```sh
-{
- "uuid": "4e84cb5e-a929-89c7-6240-768fa9835d89",
- "last_modified": 1590054128311,
- "version": "3.0.0.20500",
- "name": "BUILD CUBE - kylin_sales_cube - 20120102000000_20120103000000 - CST 2020-05-21 17:38:59",
- "projectName": "learn_kylin",
- "type": "BUILD",
- "duration": 187,
- "related_cube": "kylin_sales_cube",
- "display_cube_name": "kylin_sales_cube",
- "related_segment": "fca98f62-cb3f-8b53-5bf1-94a85334560b",
- "exec_start_time": 1590053963522,
- "exec_end_time": 0,
- "exec_interrupt_time": 0,
- "mr_waiting": 39,
- "steps": [
- {
- "interruptCmd": null,
- "id": "4e84cb5e-a929-89c7-6240-768fa9835d89-00",
- "name": "Create Intermediate Flat Hive Table",
- "sequence_id": 0,
- "exec_cmd": null,
- "interrupt_cmd": null,
- "exec_start_time": 0,
- "exec_end_time": 0,
- "exec_wait_time": 0,
- "step_status": "PENDING",
- "cmd_type": "SHELL_CMD_HADOOP",
- "info": {},
- "run_async": false
- },
- ...
- ],
- "submitter": "ADMIN",
- "job_status": "RUNNING",
- "build_instance": "20984@host",
- "progress": 0.0
-}
-```
-
-***
-
-## Get Hive Table
-`GET /kylin/api/tables/{project}/{tableName}`
-
-#### Path Parameters
-* project - `required` `string` project name
-* tableName - `required` `string` table name to find.
-
-#### Response Sample
-```sh
-{
- uuid: "69cc92c0-fc42-4bb9-893f-bd1141c91dbe",
- name: "SAMPLE_07",
- columns: [{
- id: "1",
- name: "CODE",
- datatype: "string"
- }, {
- id: "2",
- name: "DESCRIPTION",
- datatype: "string"
- }, {
- id: "3",
- name: "TOTAL_EMP",
- datatype: "int"
- }, {
- id: "4",
- name: "SALARY",
- datatype: "int"
- }],
- database: "DEFAULT",
- last_modified: 1419330476755
-}
-```
-
-## Get Hive Tables
-`GET /kylin/api/tables`
-
-#### Request Parameters
-* project- `required` `string` will list all tables in the project.
-* ext- `optional` `boolean` set true to get extend info of table.
-
-#### Response Sample
-```sh
-[
- {
- uuid: "53856c96-fe4d-459e-a9dc-c339b1bc3310",
- name: "SAMPLE_08",
- columns: [{
- id: "1",
- name: "CODE",
- datatype: "string"
- }, {
- id: "2",
- name: "DESCRIPTION",
- datatype: "string"
- }, {
- id: "3",
- name: "TOTAL_EMP",
- datatype: "int"
- }, {
- id: "4",
- name: "SALARY",
- datatype: "int"
- }],
- database: "DEFAULT",
- cardinality: {},
- last_modified: 0,
- exd: {
- minFileSize: "46069",
- totalNumberFiles: "1",
- location: "hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/sample_08",
- lastAccessTime: "1398176495945",
- lastUpdateTime: "1398176495981",
- columns: "struct columns { string code, string description, i32 total_emp, i32 salary}",
- partitionColumns: "",
- EXD_STATUS: "true",
- maxFileSize: "46069",
- inputformat: "org.apache.hadoop.mapred.TextInputFormat",
- partitioned: "false",
- tableName: "sample_08",
- owner: "hue",
- totalFileSize: "46069",
- outputformat: "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
- }
- }
-]
-```
-
-## Load Hive Tables
-`POST /kylin/api/tables/{tables}/{project}`
-
-#### Request Parameters
-* tables - `required` `string` table names you want to load from hive, separated with comma.
-* project - `required` `String` the project which the tables will be loaded into.
-
-#### Request Body
-* calculate - `optional` `boolean`
-
-#### Response Sample
-```
-{
- "result.loaded": ["DEFAULT.SAMPLE_07"],
- "result.unloaded": ["sapmle_08"]
-}
-```
-
-## Unload Hive Tables
-`DELETE /kylin/api/tables/{tables}/{project}`
-
-#### Path Parameters
-* tables - `required` `string` table names you want to unload, separated with comma.
-* project - `required` `String` the project which the tables belong to.
-
-#### Response Sample
-```sh
-{
- "result.unload.success": [
- "kylin_sales"
- ],
- "result.unload.fail": []
-}
-```
-
-## Show databases in hive
-`GET /kylin/api/tables/hive`
-
-## Show tables in a hive database
-`GET /kylin/api/tables/hive/{database}`
-
-***
-
-## Wipe cache
-`PUT /kylin/api/cache/{type}/{name}/{action}`
-
-#### Path variable
-* type - `required` `string` 'METADATA' or 'CUBE'
-* name - `required` `string` Cache key, e.g the cube name.
-* action - `required` `string` 'create', 'update' or 'drop'
-
-
-## Announce wipe cache
-`PUT /kylin/api/cache/announce/{type}/{name}/{action}`
-
-#### Path variable
-(Same as "Wipe cache")
-
-## Hot load kylin config
-`POST /kylin/api/cache/announce/config`
-
-
-***
-
-## Initiate cube start position
-Set the stream cube's start position to the current latest offsets; This can avoid building from the earlist position of Kafka topic (if you have set a long retension time);
-
-`PUT /kylin/api/cubes/{cubeName}/init_start_offsets`
-
-#### Path variable
-* cubeName - `required` `string` Cube name
-
-#### Response Sample
-```sh
-{
- "result": "success",
- "offsets": "{0=246059529, 1=253547684, 2=253023895, 3=172996803, 4=165503476, 5=173513896, 6=19200473, 7=26691891, 8=26699895, 9=26694021, 10=19204164, 11=26694597}"
-}
-```
-
-## Build stream cube
-`PUT /kylin/api/cubes/{cubeName}/build2`
-
-This API is specific for stream cube's building;
-
-#### Path variable
-* cubeName - `required` `string` Cube name
-
-#### Request Body
-
-* sourceOffsetStart - `required` `long` The start offset, 0 represents from previous position;
-* sourceOffsetEnd - `required` `long` The end offset, 9223372036854775807 represents to the end position of current stream data
-* buildType - `required` Build type, "BUILD", "MERGE" or "REFRESH"
-
-#### Request Sample
-
-```sh
-{
- "sourceOffsetStart": 0,
- "sourceOffsetEnd": 9223372036854775807,
- "buildType": "BUILD"
-}
-```
-
-#### Response Sample
-```sh
-{
- "uuid": "3afd6e75-f921-41e1-8c68-cb60bc72a601",
- "last_modified": 1480402541240,
- "version": "1.6.0",
- "name": "embedded_cube_clone - 1409830324_1409849348 - BUILD - PST 2016-11-28 22:55:41",
- "type": "BUILD",
- "duration": 0,
- "related_cube": "embedded_cube_clone",
- "related_segment": "42ebcdea-cbe9-4905-84db-31cb25f11515",
- "exec_start_time": 0,
- "exec_end_time": 0,
- "mr_waiting": 0,
- ...
-}
-```
-
-## Check segment holes
-`GET /kylin/api/cubes/{cubeName}/holes`
-
-#### Path variable
-* cubeName - `required` `string` Cube name
-
-## Fill segment holes
-`PUT /kylin/api/cubes/{cubeName}/holes`
-
-#### Path variable
-* cubeName - `required` `string` Cube name
-
-## Get streaming configs
-`GET /kylin/api/streaming/getConfig`
-
-#### Response sample
-```sh
-[
- {
- "uuid": "8b2b9dfe-777c-4d39-bf89-8472ec929193",
- "last_modified": 1587528491000,
- "version": "3.0.0.20500",
- "name": "DEFAULT.KYLIN_STREAMING_TABLE",
- "type": "kafka"
- }
-]
-```
-
-## Get kafka configs
-`GET /kylin/api/streaming/getKfkConfig`
-
-#### Response sample
-```sh
-[
- {
- "uuid": "8b2b9dfe-777c-4d39-bf89-8472ec919193",
- "last_modified": 1587528491000,
- "version": "3.0.0.20500",
- "name": "DEFAULT.KYLIN_STREAMING_TABLE",
- "clusters": [
- {
- "uuid": null,
- "last_modified": 0,
- "version": "3.0.0.20500",
- "brokers": [
- {
- "id": 0,
- "host": "localhost",
- "port": 9092
- }
- ]
- }
- ],
- "topic": "kylin_streaming_topic",
- "timeout": 60000,
- "parserName": "org.apache.kylin.source.kafka.TimedJsonStreamParser",
- "timestampField": "order_time",
- "margin": 0,
- "splitRows": 1000000,
- "parserProperties": null
- }
-]
-```
-
-## Create streaming schema
-`POST /kylin/api/streaming`
-
-#### Request body
-* project - `required` `string` Project which you want create streaming schema to.
-* tableData - `required` `string` Streaming table desc.
-* streamingConfig - `required` `string` Streaming config.
-* kafkaConfig - `required` `string` Kafka config.
-
-#### Request sample
-```
-{
- "project":"test",
- "tableData":"{\"uuid\": \"e286e39e-41d7-44c2-8fa2-41b365123987\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"KYLIN_TEST_STREAMING_TABLE\",\"columns\": [{\"id\": \"1\",\"name\": \"AMOUNT\",\"datatype\": \"decimal(19,4)\"},{\"id\": 3,\"name\": \"ORDER_TIME\",\"datatype\": \"timestamp\",\"index\": \"T\"}],\"source_type\": 1,\"table_type\": null,\"database\": \"DEFAULT\"}",
- "streamingConfig":"{\"uuid\": \"8b2b9dfe-777c-4d39-bf89-8472ec929193\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"DEFAULT.KYLIN_TEST_STREAMING_TABLE\",\"type\": \"kafka\"}",
- "kafkaConfig":"{\"uuid\": \"8b2b9dfe-777c-4d39-bf89-8472ec919193\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"DEFAULT.KYLIN_STREAMING_TABLE\",\"clusters\": [{\"uuid\": null,\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"brokers\": [{\"id\": 0,\"host\": \"localhost\",\"port\": 9092}]}],\"topic\": \"kylin_streaming_topic\",\"timeout\": 60000,\"parserName\": \"org.apache.kylin.source.kafka.TimedJsonStreamParser\",\"timestampField\": \"order_time\",\"margin\": 0,\ [...]
-}
-```
-
-#### Response sample
-```sh
-{
- "project": "test",
- "tableData": "{\"uuid\": \"e286e39e-41d7-44c2-8fa2-41b365123987\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"KYLIN_TEST_STREAMING_TABLE\",\"columns\": [{\"id\": \"1\",\"name\": \"AMOUNT\",\"datatype\": \"decimal(19,4)\"},{\"id\": 3,\"name\": \"ORDER_TIME\",\"datatype\": \"timestamp\",\"index\": \"T\"}],\"source_type\": 1,\"table_type\": null,\"database\": \"DEFAULT\"}",
- "streamingConfig": "{\"uuid\": \"8b2b9dfe-777c-4d39-bf89-8472ec929193\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"DEFAULT.KYLIN_TEST_STREAMING_TABLE\",\"type\": \"kafka\"}",
- "kafkaConfig": "{\"uuid\": \"8b2b9dfe-777c-4d39-bf89-8472ec919193\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"DEFAULT.KYLIN_STREAMING_TABLE\",\"clusters\": [{\"uuid\": null,\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"brokers\": [{\"id\": 0,\"host\": \"localhost\",\"port\": 9092}]}],\"topic\": \"kylin_streaming_topic\",\"timeout\": 60000,\"parserName\": \"org.apache.kylin.source.kafka.TimedJsonStreamParser\",\"timestampField\": \"order_time\",\"margin\": 0, [...]
- "successful": true,
- "message": null
-}
-```
-
-## Update streaming schema
-`PUT /kylin/api/streaming`
-
-#### Request body
-(Same as "Create streaming schema)
-
-#### Request sample
-(Same as "Create streaming schema)
-
-#### Response sample
-(Same as "Create streaming schema)
-
-
-***
-
-## Get users can query the table
-`GET /kylin/api/acl/table/{project}/{type}/{table}`
-
-#### Path variable
-* project - `required` `string` projectName to which table belongs
-* type - `required` `string` user or group
-* table - `required` `string` table name
-
-## Get users cannot query the table
-`GET /kylin/api/acl/table/{project}/{type}/black/{table}`
-
-#### Path variable
-* project - `required` `string` projectName to which table belongs
-* type - `required` `string` user or group
-* table - `required` `string` table name
-
-## Put user to table blacklist
-`DELETE /kylin/api/table/{project}/{type}/{table}/{name}`
-
-#### Path variable
-* project - `required` `string` projectName to which table belongs
-* type - `required` `string` user or group
-* table - `required` `string` table name
-* name - `required` `string` user name or group name you want to put to table blacklist
-
-## Delete user from table blacklist
-`POST /kylin/api/table/{project}/{type}/{table}/{name}`
-
-#### Path variable
-* project - `required` `string` projectName to which table belongs
-* type - `required` `string` user or group
-* table - `required` `string` table name
-* name - `required` `string` user name or group name you want to delete from table blacklist
-
-***
-
-## Get all metrics
-`GET /kylin/api/jmetrics/`
-
-#### Response sample
-```
-{
- "version": "3.0.0",
- "gauges": {},
- "counters": {},
- "histograms": {},
- "meters": {},
- "timers": {}
-}
-```
-
-## Get specific type of metrics
-`GET /kylin/api/jmetrics/{type}`
-
-#### Path variable
-* type - `required` `string` Specific type of metrics you want, e.g meters
-
-## Use RESTful API in Javascript
-
-Keypoints of call Kylin RESTful API in web page are:
-
-1. Add basic access authorization info in http headers.
-
-2. Use proper request type and data synax.
-
-Kylin security is based on basic access authorization, if you want to use API in your javascript, you need to add authorization info in http headers; for example:
-
-```
-$.ajaxSetup({
- headers: { 'Authorization': "Basic eWFu**********X***ZA==", 'Content-Type': 'application/json;charset=utf-8' } // use your own authorization code here
- });
- var request = $.ajax({
- url: "http://hostname/kylin/api/query",
- type: "POST",
- data: '{"sql":"select count(*) from SUMMARY;","offset":0,"limit":50000,"acceptPartial":true,"project":"test"}',
- dataType: "json"
- });
- request.done(function( msg ) {
- alert(msg);
- });
- request.fail(function( jqXHR, textStatus ) {
- alert( "Request failed: " + textStatus );
- });
-
-```
-
-To generate your authorization code, download and import "jquery.base64.js" from [https://github.com/yckart/jquery.base64.js](https://github.com/yckart/jquery.base64.js)).
-
-```
-var authorizationCode = $.base64('encode', 'NT_USERNAME' + ":" + 'NT_PASSWORD');
-
-$.ajaxSetup({
- headers: {
- 'Authorization': "Basic " + authorizationCode,
- 'Content-Type': 'application/json;charset=utf-8'
- }
-});
-```
+All Rest API of Kylin 4.0 lists in Apache Kylin Wiki:
+[Apache Kylin4.0 RESTful API](https://cwiki.apache.org/confluence/display/KYLIN/REST+API)
\ No newline at end of file
diff --git a/website/_docs/howto/howto_use_restapi.md b/website/_docs/howto/howto_use_restapi.md
index 4ed24ee..8d3951f 100644
--- a/website/_docs/howto/howto_use_restapi.md
+++ b/website/_docs/howto/howto_use_restapi.md
@@ -6,2170 +6,5 @@ permalink: /docs/howto/howto_use_restapi.html
since: v0.7.1
---
-This page lists the major RESTful APIs provided by Kylin.
-* Authentication
- * [Authentication](#authentication)
-* Query
- * [Query](#query)
- * [Prepare query](#prepare-query)
- * [Save query](#save-query)
- * [Remove saved query](#remove-saved-query)
- * [Get saved queries](#get-saved-queries)
- * [Get running queries](#get-running-queries)
- * [Stop query](#stop-query)
- * [List queryable tables](#list-queryable-tables)
-* CUBE
- * [Create cube](#create-cube)
- * [Update cube](#update-cube)
- * [List cubes](#list-cubes)
- * [Get cube](#get-cube)
- * [Get cube descriptor (dimension, measure info, etc)](#get-cube-descriptor)
- * [Get data model (fact and lookup table info)](#get-data-model)
- * [Build cube](#build-cube)
- * [Enable cube](#enable-cube)
- * [Disable cube](#disable-cube)
- * [Purge cube](#purge-cube)
- * [Delete segment](#delete-segment)
- * [Auto-Merge segment](#auto-merge-segment)
- * [Get sql of a cube](#get-sql-of-a-cube)
- * [Get sql of a cube segment](#get-sql-of-a-cube-segment)
- * [Force rebuild lookup table snapshot](#force-rebuild-lookup-table-snapshot)
- * [Clone cube](#clone-cube)
- * [Delete Cube](#delete-cube)
- * [Get hbase info](#get-hbase-info)
- * [Get current cuboid](#get-current-cuboid)
- * [Migrate cube](#migrate-cube)
-* MODEL
- * [Create model](#create-model)
- * [Update model](#update-model)
- * [Get modelDescData](#get-modeldescdata)
- * [Delete model](#delete-model)
- * [Clone model](#clone-model)
-* JOB
- * [Resume job](#resume-job)
- * [Pause job](#pause-job)
- * [Drop job](#drop-job)
- * [Discard job](#discard-job)
- * [Get job status](#get-job-status)
- * [Get job step output](#get-job-step-output)
- * [Get job list](#get-job-list)
- * [Get job status overview](#get-job-status-overview)
- * [Resubmit realtime build job](#resubmit-realtime-build-job)
- * [Rollback job](#rollback-job)
-* Metadata
- * [Get Hive Table](#get-hive-table)
- * [Get Hive Tables](#get-hive-tables)
- * [Load Hive Tables](#load-hive-tables)
- * [Unload Hive Tables](#unload-hive-tables)
- * [Show databases in hive](#show-databases-in-hive)
- * [Show tables in a hive database](#show-tables-in-a-hive-database)
-* Cache
- * [Wipe cache](#wipe-cache)
- * [Announce wipe cache](#announce-wipe-cache)
- * [Hot load kylin config](#hot-load-kylin-config)
-* Streaming
- * [Initiate cube start position](#initiate-cube-start-position)
- * [Build stream cube](#build-stream-cube)
- * [Check segment holes](#check-segment-holes)
- * [Fill segment holes](#fill-segment-holes)
- * [Get streaming configs](#get-streaming-configs)
- * [Get Kafka configs](#get-kafka-configs)
- * [Create streaming schema](#create-streaming-schema)
- * [Update streaming tables](#update-streaming-schema)
-* ACL
- * [Get users can query the table](#get-users-can-query-the-table)
- * [Get users cannot query the table](#get-users-cannot-query-the-table)
- * [Put user to table blacklist](#put-user-to-table-blacklist)
- * [Delete user from table blacklist](#delete-user-from-table-blacklist)
-* Metrics
- * [Get all metrics](#get-all-metrics)
- * [Get specific type of metrics](#get-specific-type-of-metrics)
-
-## Authentication
-`POST /kylin/api/user/authentication`
-
-#### Request Header
-Authorization data encoded by basic auth is needed in the header, such as:
-Authorization:Basic {data}
-You can generate {data} by using below python script
-```
-python -c "import base64; print base64.standard_b64encode('$UserName:$Password')"
-```
-
-#### Response Body
-* userDetails - Defined authorities and status of current user.
-
-#### Response Sample
-
-```sh
-{
- "userDetails": {
- "username": "sample",
- "password": "null",
- "authorities": [
- {
- "authority": "ROLE_ADMIN"
- },
- {
- "authority": "ROLE_ANALYST"
- },
- {
- "authority": "ROLE_MODELER"
- },
- {
- "authority": "ALL_USERS"
- }
- ],
- "disabled": false,
- "defaultPassword": false,
- "locked": false,
- "lockedTime": 0,
- "wrongTime": 0,
- "uuid": "3704ba8c-deb1-ac47-729d-c1039c1bd6ec",
- "last_modified": 1585219480112,
- "version": "3.0.0.20500"
- }
-}
-```
-
-#### Curl Example
-
-```
-curl -c /path/to/cookiefile.txt -X POST -H "Authorization: Basic XXXXXXXXX" -H 'Content-Type: application/json' http://<host>:<port>/kylin/api/user/authentication
-```
-
-If login successfully, the JSESSIONID will be saved into the cookie file; In the subsequent http requests, attach the cookie, for example:
-
-```
-curl -b /path/to/cookiefile.txt -X PUT -H 'Content-Type: application/json' -d '{"startTime":'1423526400000', "endTime":'1423612800000', "buildType":"BUILD"}' http://<host>:<port>/kylin/api/cubes/your_cube/build
-```
-
-Alternatively, you can provide the username/password with option "user" in each curl call; please note this has the risk of password leak in shell history:
-
-
-```
-curl -X PUT --user ADMIN:KYLIN -H "Content-Type: application/json;charset=utf-8" -d '{ "startTime": 820454400000, "endTime": 821318400000, "buildType": "BUILD"}' http://localhost:7070/kylin/api/cubes/kylin_sales/build
-```
-
-***
-
-## Query
-`POST /kylin/api/query`
-
-#### Request Body
-* sql - `required` `string` The text of sql statement.
-* offset - `optional` `int` Query offset. If offset is set in sql, curIndex will be ignored.
-* limit - `optional` `int` Query limit. If limit is set in sql, perPage will be ignored.
-* acceptPartial - `optional` `bool` Whether accept a partial result or not, default be "false". Set to "false" for production use.
-* project - `optional` `string` Project to perform query. Default value is 'DEFAULT'.
-If you want to specify cube for your query, please refer to this wiki: [Specify cube for your query](https://cwiki.apache.org/confluence/display/KYLIN/Specify+cube+for+your+query)
-
-#### Request Sample
-
-```sh
-{
- "sql":"select * from TEST_KYLIN_FACT",
- "offset":0,
- "limit":50000,
- "acceptPartial":false,
- "project":"DEFAULT"
-}
-```
-
-#### Curl Example
-
-```
-curl -X POST -H "Authorization: Basic XXXXXXXXX" -H "Content-Type: application/json" -d '{ "sql":"select count(*) from TEST_KYLIN_FACT", "project":"learn_kylin" }' http://localhost:7070/kylin/api/query
-```
-
-#### Response Body
-* columnMetas - Column metadata information of result set.
-* results - Data set of result.
-* cube - Cube used for this query.
-* affectedRowCount - Count of affected row by this sql statement.
-* isException - Whether this response is an exception.
-* ExceptionMessage - Message content of the exception.
-* Duration - Time cost of this query
-* Partial - Whether the response is a partial result or not. Decided by `acceptPartial` of request.
-
-#### Response Sample
-
-```sh
-{
- "columnMetas":[
- {
- "isNullable":1,
- "displaySize":0,
- "label":"CAL_DT",
- "name":"CAL_DT",
- "schemaName":null,
- "catelogName":null,
- "tableName":null,
- "precision":0,
- "scale":0,
- "columnType":91,
- "columnTypeName":"DATE",
- "readOnly":true,
- "writable":false,
- "caseSensitive":true,
- "searchable":false,
- "currency":false,
- "signed":true,
- "autoIncrement":false,
- "definitelyWritable":false
- },
- {
- "isNullable":1,
- "displaySize":10,
- "label":"LEAF_CATEG_ID",
- "name":"LEAF_CATEG_ID",
- "schemaName":null,
- "catelogName":null,
- "tableName":null,
- "precision":10,
- "scale":0,
- "columnType":4,
- "columnTypeName":"INTEGER",
- "readOnly":true,
- "writable":false,
- "caseSensitive":true,
- "searchable":false,
- "currency":false,
- "signed":true,
- "autoIncrement":false,
- "definitelyWritable":false
- }
- ],
- "results":[
- [
- "2013-08-07",
- "32996",
- "15",
- "15",
- "Auction",
- "10000000",
- "49.048952730908745",
- "49.048952730908745",
- "49.048952730908745",
- "1"
- ],
- [
- "2013-08-07",
- "43398",
- "0",
- "14",
- "ABIN",
- "10000633",
- "85.78317064220418",
- "85.78317064220418",
- "85.78317064220418",
- "1"
- ]
- ],
- "cube":"test_kylin_cube_with_slr_desc",
- "affectedRowCount":0,
- "isException":false,
- "exceptionMessage":null,
- "duration":3451,
- "partial":false
-}
-```
-
-
-## Prepare query
-`POST /kylin/api/query/prestate`
-
-#### Request Body
-* sql - `required` `string` The text of sql statement.
-* offset - `optional` `int` Query offset. If offset is set in sql, curIndex will be ignored.
-* limit - `optional` `int` Query limit. If limit is set in sql, perPage will be ignored.
-* acceptPartial - `optional` `bool` Whether accept a partial result or not, default be "false". Set to "false" for production use.
-* project - `optional` `string` Project to perform query. Default value is 'DEFAULT'.
-
-#### Request Sample
-
-```sh
-{
- "sql":"select * from TEST_KYLIN_FACT",
- "offset":0,
- "limit":50000,
- "acceptPartial":false,
- "project":"DEFAULT"
-}
-```
-
-
-## Save query
-`POST /kylin/api/saved_queries`
-
-#### Request Body
-* sql - `required` `string` The text of sql statement.
-* name - `required` `string` Sql name.
-* project - `required` `string` Project to perform query.
-* description - `optional` `string` Sql description.
-
-#### Request Sample
-
-```sh
-{
- "sql": "select count(*) from kylin_sales",
- "name": "test",
- "project": "learn_kylin"
-}
-```
-
-
-## Remove saved query
-`DELETE /kylin/api/saved_queries/{id}`
-
-#### Request Parameters
-* id - `required` `string` The id of saved query you want to remove
-
-
-## Get saved queries
-`GET /kylin/api/saved_queries`
-
-#### Response Sample
-```
-[
- {
- "name": "test",
- "project": "learn_kylin",
- "sql": "select count(*) from kylin_sales",
- "description": null,
- "id": "-1674470999"
- }
-]
-```
-
-## Get running queries
-`GET /kylin/api/query/runningQueries`
-
-## Stop Query
-`PUT /kylin/api/query/{queryId}/stop`
-
-#### Path Variable
-* queryId - `required` `String` The queryId of you want to stop. You can obtain it by `Get running queries`.
-
-
-## List queryable tables
-`GET /kylin/api/tables_and_columns`
-
-#### Path Variable
-* project - `required` `string` The project to load tables
-
-#### Response Sample
-```sh
-[
- {
- "columns":[
- {
- "table_NAME":"TEST_CAL_DT",
- "table_SCHEM":"EDW",
- "column_NAME":"CAL_DT",
- "data_TYPE":91,
- "nullable":1,
- "column_SIZE":-1,
- "buffer_LENGTH":-1,
- "decimal_DIGITS":0,
- "num_PREC_RADIX":10,
- "column_DEF":null,
- "sql_DATA_TYPE":-1,
- "sql_DATETIME_SUB":-1,
- "char_OCTET_LENGTH":-1,
- "ordinal_POSITION":1,
- "is_NULLABLE":"YES",
- "scope_CATLOG":null,
- "scope_SCHEMA":null,
- "scope_TABLE":null,
- "source_DATA_TYPE":-1,
- "iS_AUTOINCREMENT":null,
- "table_CAT":"defaultCatalog",
- "remarks":null,
- "type_NAME":"DATE"
- },
- {
- "table_NAME":"TEST_CAL_DT",
- "table_SCHEM":"EDW",
- "column_NAME":"WEEK_BEG_DT",
- "data_TYPE":91,
- "nullable":1,
- "column_SIZE":-1,
- "buffer_LENGTH":-1,
- "decimal_DIGITS":0,
- "num_PREC_RADIX":10,
- "column_DEF":null,
- "sql_DATA_TYPE":-1,
- "sql_DATETIME_SUB":-1,
- "char_OCTET_LENGTH":-1,
- "ordinal_POSITION":2,
- "is_NULLABLE":"YES",
- "scope_CATLOG":null,
- "scope_SCHEMA":null,
- "scope_TABLE":null,
- "source_DATA_TYPE":-1,
- "iS_AUTOINCREMENT":null,
- "table_CAT":"defaultCatalog",
- "remarks":null,
- "type_NAME":"DATE"
- }
- ],
- "table_NAME":"TEST_CAL_DT",
- "table_SCHEM":"EDW",
- "ref_GENERATION":null,
- "self_REFERENCING_COL_NAME":null,
- "type_SCHEM":null,
- "table_TYPE":"TABLE",
- "table_CAT":"defaultCatalog",
- "remarks":null,
- "type_CAT":null,
- "type_NAME":null
- }
-]
-```
-
-***
-
-## Create cube
-`POST /kylin/api/cubes`
-
-#### Request Body
-* cubeDescData - `required` `string` cubeDescData to create
-* cubeName - `required` `string` cubeName to create
-* projectName - `required` `string` projectName to which cube belongs
-
-#### Request Sample
-```
-{
-"cubeDescData":"{\"uuid\": \"0ef9b7a8-3929-4dff-b59d-2100aadc8dbf\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"kylin_test_cube\",\"is_draft\": false,\"model_name\": \"kylin_sales_model\",\"description\": \"\",\"null_string\": null,\"dimensions\": [{\"name\": \"TRANS_ID\",\"table\": \"KYLIN_SALES\",\"column\": \"TRANS_ID\",\"derived\": null},{\"name\": \"YEAR_BEG_DT\",\"table\": \"KYLIN_CAL_DT\",\"column\": null,\"derived\": [\"YEAR_BEG_DT\"]},{\"name\": \"MONTH_BEG_DT\ [...]
-"cubeName":"kylin_test_cube",
-"project":"learn_kylin"
-}
-```
-
-#### Response Sample
-```
-{
-"uuid": "7b3faf69-eca8-cc5f-25f9-49b0f0b5d404",
-"cubeName": "kylin_test_cube",
-"cubeDescData":"{\"uuid\": \"0ef9b7a8-3929-4dff-b59d-2100aadc8dbf\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"kylin_test_cube\",\"is_draft\": false,\"model_name\": \"kylin_sales_model\",\"description\": \"\",\"null_string\": null,\"dimensions\": [{\"name\": \"TRANS_ID\",\"table\": \"KYLIN_SALES\",\"column\": \"TRANS_ID\",\"derived\": null},{\"name\": \"YEAR_BEG_DT\",\"table\": \"KYLIN_CAL_DT\",\"column\": null,\"derived\": [\"YEAR_BEG_DT\"]},{\"name\": \"MONTH_BEG_DT\ [...]
-"streamingData": null,
-"kafkaData": null,
-"successful": true,
-"message": null,
-"project": "learn_kylin",
-"streamingCube": null
-}
-```
-
-## Update Cube
-`PUT /kylin/api/cubes`
-
-#### Request Body
-(Same as "Create Cube")
-
-#### Request Sample
-(Same as "Create Cube")
-
-#### Response Sample
-(Same as "Create Cube")
-
-
-## List cubes
-`GET /kylin/api/cubes`
-
-#### Request Parameters
-* offset - `required` `int` Offset used by pagination
-* limit - `required` `int ` Cubes per page.
-* cubeName - `optional` `string` Keyword for cube names. To find cubes whose name contains this keyword.
-* projectName - `optional` `string` Project name.
-
-#### Response Sample
-```sh
-[
- {
- "uuid":"1eaca32a-a33e-4b69-83dd-0bb8b1f8c53b",
- "last_modified":1407831634847,
- "name":"test_kylin_cube_with_slr_empty",
- "owner":null,
- "version":null,
- "descriptor":"test_kylin_cube_with_slr_desc",
- "cost":50,
- "status":"DISABLED",
- "segments":[
- ],
- "create_time":null,
- "source_records_count":0,
- "source_records_size":0,
- "size_kb":0
- }
-]
-```
-
-## Get cube
-`GET /kylin/api/cubes/{cubeName}`
-
-#### Path Variable
-* cubeName - `required` `string` Cube name to find.
-
-## Get cube descriptor
-`GET /kylin/api/cube_desc/{cubeName}`
-Get descriptor for specified cube instance.
-
-#### Path Variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-```sh
-[
- {
- "uuid": "0ef9b7a8-3929-4dff-b59d-2100aadc8dbf",
- "last_modified": 1574402902000,
- "version": "3.0.0.20500",
- "name": "kylin_sales_cube",
- "is_draft": false,
- "model_name": "kylin_sales_model",
- "description": "",
- "null_string": null,
- "dimensions": [
- {
- "name": "TRANS_ID",
- "table": "KYLIN_SALES",
- "column": "TRANS_ID",
- "derived": null
- },
- {
- "name": "YEAR_BEG_DT",
- "table": "KYLIN_CAL_DT",
- "column": null,
- "derived": [
- "YEAR_BEG_DT"
- ]
- },
- {
- "name": "MONTH_BEG_DT",
- "table": "KYLIN_CAL_DT",
- "column": null,
- "derived": [
- "MONTH_BEG_DT"
- ]
- },
- {
- "name": "WEEK_BEG_DT",
- "table": "KYLIN_CAL_DT",
- "column": null,
- "derived": [
- "WEEK_BEG_DT"
- ]
- },
- {
- "name": "USER_DEFINED_FIELD1",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": null,
- "derived": [
- "USER_DEFINED_FIELD1"
- ]
- },
- {
- "name": "USER_DEFINED_FIELD3",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": null,
- "derived": [
- "USER_DEFINED_FIELD3"
- ]
- },
- {
- "name": "META_CATEG_NAME",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": "META_CATEG_NAME",
- "derived": null
- },
- {
- "name": "CATEG_LVL2_NAME",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": "CATEG_LVL2_NAME",
- "derived": null
- },
- {
- "name": "CATEG_LVL3_NAME",
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "column": "CATEG_LVL3_NAME",
- "derived": null
- },
- {
- "name": "LSTG_FORMAT_NAME",
- "table": "KYLIN_SALES",
- "column": "LSTG_FORMAT_NAME",
- "derived": null
- },
- {
- "name": "SELLER_ID",
- "table": "KYLIN_SALES",
- "column": "SELLER_ID",
- "derived": null
- },
- {
- "name": "BUYER_ID",
- "table": "KYLIN_SALES",
- "column": "BUYER_ID",
- "derived": null
- },
- {
- "name": "ACCOUNT_BUYER_LEVEL",
- "table": "BUYER_ACCOUNT",
- "column": "ACCOUNT_BUYER_LEVEL",
- "derived": null
- },
- {
- "name": "ACCOUNT_SELLER_LEVEL",
- "table": "SELLER_ACCOUNT",
- "column": "ACCOUNT_SELLER_LEVEL",
- "derived": null
- },
- {
- "name": "BUYER_COUNTRY",
- "table": "BUYER_ACCOUNT",
- "column": "ACCOUNT_COUNTRY",
- "derived": null
- },
- {
- "name": "SELLER_COUNTRY",
- "table": "SELLER_ACCOUNT",
- "column": "ACCOUNT_COUNTRY",
- "derived": null
- },
- {
- "name": "BUYER_COUNTRY_NAME",
- "table": "BUYER_COUNTRY",
- "column": "NAME",
- "derived": null
- },
- {
- "name": "SELLER_COUNTRY_NAME",
- "table": "SELLER_COUNTRY",
- "column": "NAME",
- "derived": null
- },
- {
- "name": "OPS_USER_ID",
- "table": "KYLIN_SALES",
- "column": "OPS_USER_ID",
- "derived": null
- },
- {
- "name": "OPS_REGION",
- "table": "KYLIN_SALES",
- "column": "OPS_REGION",
- "derived": null
- }
- ],
- "measures": [
- {
- "name": "GMV_SUM",
- "function": {
- "expression": "SUM",
- "parameter": {
- "type": "column",
- "value": "KYLIN_SALES.PRICE"
- },
- "returntype": "decimal(19,4)"
- }
- },
- {
- "name": "BUYER_LEVEL_SUM",
- "function": {
- "expression": "SUM",
- "parameter": {
- "type": "column",
- "value": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL"
- },
- "returntype": "bigint"
- }
- },
- {
- "name": "SELLER_LEVEL_SUM",
- "function": {
- "expression": "SUM",
- "parameter": {
- "type": "column",
- "value": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
- },
- "returntype": "bigint"
- }
- },
- {
- "name": "TRANS_CNT",
- "function": {
- "expression": "COUNT",
- "parameter": {
- "type": "constant",
- "value": "1"
- },
- "returntype": "bigint"
- }
- },
- {
- "name": "SELLER_CNT_HLL",
- "function": {
- "expression": "COUNT_DISTINCT",
- "parameter": {
- "type": "column",
- "value": "KYLIN_SALES.SELLER_ID"
- },
- "returntype": "hllc(10)"
- }
- },
- {
- "name": "TOP_SELLER",
- "function": {
- "expression": "TOP_N",
- "parameter": {
- "type": "column",
- "value": "KYLIN_SALES.PRICE",
- "next_parameter": {
- "type": "column",
- "value": "KYLIN_SALES.SELLER_ID"
- }
- },
- "returntype": "topn(100)",
- "configuration": {
- "topn.encoding.KYLIN_SALES.SELLER_ID": "dict",
- "topn.encoding_version.KYLIN_SALES.SELLER_ID": "1"
- }
- }
- }
- ],
- "rowkey": {
- "rowkey_columns": [
- {
- "column": "KYLIN_SALES.BUYER_ID",
- "encoding": "integer:4",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.SELLER_ID",
- "encoding": "integer:4",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.TRANS_ID",
- "encoding": "integer:4",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.PART_DT",
- "encoding": "date",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.LEAF_CATEG_ID",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "BUYER_COUNTRY.NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "SELLER_COUNTRY.NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.LSTG_FORMAT_NAME",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.LSTG_SITE_ID",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.OPS_USER_ID",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- },
- {
- "column": "KYLIN_SALES.OPS_REGION",
- "encoding": "dict",
- "encoding_version": 1,
- "isShardBy": false
- }
- ]
- },
- "hbase_mapping": {
- "column_family": [
- {
- "name": "F1",
- "columns": [
- {
- "qualifier": "M",
- "measure_refs": [
- "GMV_SUM",
- "BUYER_LEVEL_SUM",
- "SELLER_LEVEL_SUM",
- "TRANS_CNT"
- ]
- }
- ]
- },
- {
- "name": "F2",
- "columns": [
- {
- "qualifier": "M",
- "measure_refs": [
- "SELLER_CNT_HLL",
- "TOP_SELLER"
- ]
- }
- ]
- }
- ]
- },
- "aggregation_groups": [
- {
- "includes": [
- "KYLIN_SALES.PART_DT",
- "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
- "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
- "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
- "KYLIN_SALES.LEAF_CATEG_ID",
- "KYLIN_SALES.LSTG_FORMAT_NAME",
- "KYLIN_SALES.LSTG_SITE_ID",
- "KYLIN_SALES.OPS_USER_ID",
- "KYLIN_SALES.OPS_REGION",
- "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
- "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
- "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
- "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
- "BUYER_COUNTRY.NAME",
- "SELLER_COUNTRY.NAME"
- ],
- "select_rule": {
- "hierarchy_dims": [
- [
- "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
- "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
- "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
- "KYLIN_SALES.LEAF_CATEG_ID"
- ]
- ],
- "mandatory_dims": [
- "KYLIN_SALES.PART_DT"
- ],
- "joint_dims": [
- [
- "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
- "BUYER_COUNTRY.NAME"
- ],
- [
- "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
- "SELLER_COUNTRY.NAME"
- ],
- [
- "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
- "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
- ],
- [
- "KYLIN_SALES.LSTG_FORMAT_NAME",
- "KYLIN_SALES.LSTG_SITE_ID"
- ],
- [
- "KYLIN_SALES.OPS_USER_ID",
- "KYLIN_SALES.OPS_REGION"
- ]
- ]
- }
- }
- ],
- "signature": null,
- "notify_list": [],
- "status_need_notify": [],
- "partition_date_start": 1325376000000,
- "partition_date_end": 3153600000000,
- "auto_merge_time_ranges": [],
- "volatile_range": 0,
- "retention_range": 0,
- "engine_type": 2,
- "storage_type": 2,
- "override_kylin_properties": {
- "kylin.cube.aggrgroup.is-mandatory-only-valid": "true",
- "kylin.engine.spark.rdd-partition-cut-mb": "500"
- },
- "cuboid_black_list": [],
- "parent_forward": 3,
- "mandatory_dimension_set_list": [],
- "snapshot_table_desc_list": []
- }
-]
-```
-
-## Get data model
-`GET /kylin/api/model/{modelName}`
-
-#### Path Variable
-* modelName - `required` `string` Data model name, by default it should be the same with cube name.
-
-#### Response Sample
-```sh
-{
- "uuid": "ff527b94-f860-44c3-8452-93b17774c647",
- "name": "test_kylin_with_slr_model_desc",
- "lookups": [
- {
- "table": "EDW.TEST_CAL_DT",
- "join": {
- "type": "inner",
- "primary_key": [
- "CAL_DT"
- ],
- "foreign_key": [
- "CAL_DT"
- ]
- }
- },
- {
- "table": "DEFAULT.TEST_CATEGORY_GROUPINGS",
- "join": {
- "type": "inner",
- "primary_key": [
- "LEAF_CATEG_ID",
- "SITE_ID"
- ],
- "foreign_key": [
- "LEAF_CATEG_ID",
- "LSTG_SITE_ID"
- ]
- }
- }
- ],
- "capacity": "MEDIUM",
- "last_modified": 1442372116000,
- "fact_table": "DEFAULT.TEST_KYLIN_FACT",
- "filter_condition": null,
- "partition_desc": {
- "partition_date_column": "DEFAULT.TEST_KYLIN_FACT.CAL_DT",
- "partition_date_start": 0,
- "partition_date_format": "yyyy-MM-dd",
- "partition_type": "APPEND",
- "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
- }
-}
-```
-
-## Build cube
-`PUT /kylin/api/cubes/{cubeName}/build`
-
-#### Path Variable
-* cubeName - `required` `string` Cube name.
-
-#### Request Body
-* startTime - `required` `long` Start timestamp of data to build, e.g. 1388563200000 for 2014-1-1
-* endTime - `required` `long` End timestamp of data to build
-* buildType - `required` `string` Supported build type: 'BUILD', 'MERGE', 'REFRESH'
-
-#### Curl Example
-```
-curl -X PUT -H "Authorization: Basic XXXXXXXXX" -H 'Content-Type: application/json' -d '{"startTime":'1423526400000', "endTime":'1423612800000', "buildType":"BUILD"}' http://<host>:<port>/kylin/api/cubes/{cubeName}/build
-```
-
-#### Response Sample
-```
-{
- "uuid":"c143e0e4-ac5f-434d-acf3-46b0d15e3dc6",
- "last_modified":1407908916705,
- "name":"test_kylin_cube_with_slr_empty - 19700101000000_20140731160000 - BUILD - PDT 2014-08-12 22:48:36",
- "type":"BUILD",
- "duration":0,
- "related_cube":"test_kylin_cube_with_slr_empty",
- "related_segment":"19700101000000_20140731160000",
- "exec_start_time":0,
- "exec_end_time":0,
- "mr_waiting":0,
- "steps":[
- {
- "interruptCmd":null,
- "name":"Create Intermediate Flat Hive Table",
- "sequence_id":0,
- "exec_cmd":"hive -e \"DROP TABLE IF EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6;\nCREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\n(\nCAL_DT date\n,LEAF_CATEG_ID int\n,LSTG_SITE_ID int\n,META_CATEG_NAME string\n,CATEG_LVL2_NAME string\n,CATEG_LVL3_NAME string\n,LSTG_FORMAT_NAME string\n,SLR_SEGMENT_ [...]
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"SHELL_CMD_HADOOP",
- "info":null,
- "run_async":false
- },
- {
- "interruptCmd":null,
- "name":"Extract Fact Table Distinct Columns",
- "sequence_id":1,
- "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns -jobname Kylin_Fact_Distinct_Columns_test_kylin_cube_with_slr_empty_Step_1",
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"JAVA_CMD_HADOOP_FACTDISTINCT",
- "info":null,
- "run_async":true
- },
- {
- "interruptCmd":null,
- "name":"Load HFile to HBase Table",
- "sequence_id":12,
- "exec_cmd":" -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile/ -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -cubename test_kylin_cube_with_slr_empty",
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"JAVA_CMD_HADOOP_NO_MR_BULKLOAD",
- "info":null,
- "run_async":false
- }
- ],
- "job_status":"PENDING",
- "progress":0.0
-}
-```
-
-## Enable Cube
-`PUT /kylin/api/cubes/{cubeName}/enable`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-```sh
-{
- "uuid":"1eaca32a-a33e-4b69-83dd-0bb8b1f8c53b",
- "last_modified":1407909046305,
- "name":"test_kylin_cube_with_slr_ready",
- "owner":null,
- "version":null,
- "descriptor":"test_kylin_cube_with_slr_desc",
- "cost":50,
- "status":"ACTIVE",
- "segments":[
- {
- "name":"19700101000000_20140531160000",
- "storage_location_identifier":"KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_READY-19700101000000_20140531160000_BF043D2D-9A4A-45E9-AA59-5A17D3F34A50",
- "date_range_start":0,
- "date_range_end":1401552000000,
- "status":"READY",
- "size_kb":4758,
- "source_records":6000,
- "source_records_size":620356,
- "last_build_time":1407832663227,
- "last_build_job_id":"2c7a2b63-b052-4a51-8b09-0c24b5792cda",
- "binary_signature":null,
- "dictionaries":{
- "TEST_CATEGORY_GROUPINGS/CATEG_LVL2_NAME":"/dict/TEST_CATEGORY_GROUPINGS/CATEG_LVL2_NAME/16d8185c-ee6b-4f8c-a919-756d9809f937.dict",
- "TEST_KYLIN_FACT/LSTG_SITE_ID":"/dict/TEST_SITES/SITE_ID/0bec6bb3-1b0d-469c-8289-b8c4ca5d5001.dict",
- "TEST_KYLIN_FACT/SLR_SEGMENT_CD":"/dict/TEST_SELLER_TYPE_DIM/SELLER_TYPE_CD/0c5d77ec-316b-47e0-ba9a-0616be890ad6.dict",
- "TEST_KYLIN_FACT/CAL_DT":"/dict/PREDEFINED/date(yyyy-mm-dd)/64ac4f82-f2af-476e-85b9-f0805001014e.dict",
- "TEST_CATEGORY_GROUPINGS/CATEG_LVL3_NAME":"/dict/TEST_CATEGORY_GROUPINGS/CATEG_LVL3_NAME/270fbfb0-281c-4602-8413-2970a7439c47.dict",
- "TEST_KYLIN_FACT/LEAF_CATEG_ID":"/dict/TEST_CATEGORY_GROUPINGS/LEAF_CATEG_ID/2602386c-debb-4968-8d2f-b52b8215e385.dict",
- "TEST_CATEGORY_GROUPINGS/META_CATEG_NAME":"/dict/TEST_CATEGORY_GROUPINGS/META_CATEG_NAME/0410d2c4-4686-40bc-ba14-170042a2de94.dict"
- },
- "snapshots":{
- "TEST_CAL_DT":"/table_snapshot/TEST_CAL_DT.csv/8f7cfc8a-020d-4019-b419-3c6deb0ffaa0.snapshot",
- "TEST_SELLER_TYPE_DIM":"/table_snapshot/TEST_SELLER_TYPE_DIM.csv/c60fd05e-ac94-4016-9255-96521b273b81.snapshot",
- "TEST_CATEGORY_GROUPINGS":"/table_snapshot/TEST_CATEGORY_GROUPINGS.csv/363f4a59-b725-4459-826d-3188bde6a971.snapshot",
- "TEST_SITES":"/table_snapshot/TEST_SITES.csv/78e0aecc-3ec6-4406-b86e-bac4b10ea63b.snapshot"
- }
- }
- ],
- "create_time":null,
- "source_records_count":6000,
- "source_records_size":0,
- "size_kb":4758
-}
-```
-
-## Disable Cube
-`PUT /kylin/api/cubes/{cubeName}/disable`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-(Same as "Enable Cube")
-
-## Purge Cube
-`PUT /kylin/api/cubes/{cubeName}/purge`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-(Same as "Enable Cube")
-
-
-## Delete Segment
-`DELETE /kylin/api/cubes/{cubeName}/segs/{segmentName}`
-
-
-## Auto-merge Segment
-`PUT /kylin/api/cubes/{cubeName}/automerge`
-
-
-## Get sql of a cube
-`GET /kylin/api/cubes/{cubeName}/sql`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-
-#### Response Sample
-```sh
-{
- "sql": "SELECT\n`KYLIN_SALES`.`TRANS_ID` as `KYLIN_SALES_TRANS_ID`\n,`KYLIN_SALES`.`PART_DT` as `KYLIN_SALES_PART_DT`\n,`KYLIN_CAL_DT`.`YEAR_BEG_DT` as `KYLIN_CAL_DT_YEAR_BEG_DT`\n,`KYLIN_CAL_DT`.`MONTH_BEG_DT` as `KYLIN_CAL_DT_MONTH_BEG_DT`\n,`KYLIN_CAL_DT`.`WEEK_BEG_DT` as `KYLIN_CAL_DT_WEEK_BEG_DT`\n,`KYLIN_SALES`.`LEAF_CATEG_ID` as `KYLIN_SALES_LEAF_CATEG_ID`\n,`KYLIN_SALES`.`LSTG_SITE_ID` as `KYLIN_SALES_LSTG_SITE_ID`\n,`KYLIN_CATEGORY_GROUPINGS`.`USER_DEFINED_FIELD1` as `KYLIN_ [...]
-}
-```
-
-
-## Get sql of a cube segment
-`GET /kylin/api/cubes/{cubeName}/segs/{segmentName}/sql`
-
-#### Path variable
-* cubeName - `required` `string` Cube name.
-* segmentName - `required` `string` Segment name.
-
-#### Response Sample
-```sh
-{
- "sql": "SELECT\n`KYLIN_SALES`.`TRANS_ID` as `KYLIN_SALES_TRANS_ID`\n,`KYLIN_SALES`.`PART_DT` as `KYLIN_SALES_PART_DT`\n,`KYLIN_CAL_DT`.`YEAR_BEG_DT` as `KYLIN_CAL_DT_YEAR_BEG_DT`\n,`KYLIN_CAL_DT`.`MONTH_BEG_DT` as `KYLIN_CAL_DT_MONTH_BEG_DT`\n,`KYLIN_CAL_DT`.`WEEK_BEG_DT` as `KYLIN_CAL_DT_WEEK_BEG_DT`\n,`KYLIN_SALES`.`LEAF_CATEG_ID` as `KYLIN_SALES_LEAF_CATEG_ID`\n,`KYLIN_SALES`.`LSTG_SITE_ID` as `KYLIN_SALES_LSTG_SITE_ID`\n,`KYLIN_CATEGORY_GROUPINGS`.`USER_DEFINED_FIELD1` as `KYLIN_ [...]
-}
-```
-
-
-## Force rebuild lookup table snapshot
-`PUT /kylin/api/cubes/{cubeName}/refresh_lookup`
-
-
-## Clone cube
-`PUT /kylin/api/cubes/{cubeName}/clone`
-
-
-## Delete cube
-`DELETE /kylin/api/cubes/{cubeName}`
-
-
-## Get hbase info
-`GET /kylin/api/cubes/{cubeName}/hbase`
-
-#### Response Sample
-```sh
-[
- {
- "segmentName": "20120101000000_20120103000000",
- "segmentUUID": null,
- "segmentStatus": "READY",
- "tableName": "KYLIN_E1VT22737D",
- "tableSize": 0,
- "regionCount": 1,
- "dateRangeStart": 1325376000000,
- "dateRangeEnd": 1325548800000,
- "sourceOffsetStart": 0,
- "sourceOffsetEnd": 0,
- "sourceCount": 29
- }
-]
-```
-
-
-## Get current cuboid
-`GET /kylin/api/cubes/{cubeName}/cuboids/current`
-
-
-## Migrate Cube
-`POST /kylin/api/cubes/{cube}/{project}/migrate`
-
-
-***
-
-## Create Model
-`POST /kylin/api/models`
-
-#### Request Body
-* modelDescData - `required` `string` modelDescData to create
-* modelName - `required` `string` modelName to create
-* projectName - `required` `string` projectName to which model belongs
-
-#### Request Sample
-```sh
-{
-"modelDescData": "{\"uuid\": \"0928468a-9fab-4185-9a14-6f2e7c74823f\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"kylin_test_model\",\"owner\": null,\"is_draft\": false,\"description\": \"\",\"fact_table\": \"DEFAULT.KYLIN_SALES\",\"lookups\": [{\"table\": \"DEFAULT.KYLIN_CAL_DT\",\"kind\": \"LOOKUP\",\"alias\": \"KYLIN_CAL_DT\",\"join\": {\"type\": \"inner\",\"primary_key\": [\"KYLIN_CAL_DT.CAL_DT\"],\"foreign_key\": [\"KYLIN_SALES.PART_DT\"]}},{\"table\": \"DEFAULT.KY [...]
-"modelName": "kylin_test_model",
-"project": "learn_kylin"
-}
-```
-
-#### Response Sample
-```sh
-{
-"uuid": "2613d739-14c1-38ac-2e37-f36e46fd9976",
-"modelName": "kylin_test_model",
-"modelDescData": "{\"uuid\": \"0928468a-9fab-4185-9a14-6f2e7c74823f\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"kylin_test_model\",\"owner\": null,\"is_draft\": false,\"description\": \"\",\"fact_table\": \"DEFAULT.KYLIN_SALES\",\"lookups\": [{\"table\": \"DEFAULT.KYLIN_CAL_DT\",\"kind\": \"LOOKUP\",\"alias\": \"KYLIN_CAL_DT\",\"join\": {\"type\": \"inner\",\"primary_key\": [\"KYLIN_CAL_DT.CAL_DT\"],\"foreign_key\": [\"KYLIN_SALES.PART_DT\"]}},{\"table\": \"DEFAULT.KY [...]
-"successful": true,
-"message": null,
-"project": "learn_kylin",
-"ccInCheck": null,
-"seekingExprAdvice": false
-}
-```
-
-
-## Update Model
-`PUT /kylin/api/models`
-
-#### Request Body
-(Same as "Create Model")
-
-#### Request Sample
-(Same as "Create Model")
-
-#### Response Sample
-(Same as "Create Model")
-
-
-## Get ModelDescData
-`GET /kylin/api/models`
-
-#### Request Parameters
-* modelName - `optional` `string` Model name.
-* projectName - `optional` `string` Project Name.
-* limit - `optional` `integer` Offset used by pagination
-* offset - `optional` `integer` Models per page
-
-#### Response Sample
-```sh
-[
- {
- "uuid": "0928468a-9fab-4185-9a14-6f2e7c74823f",
- "last_modified": 1568862496000,
- "version": "3.0.0.20500",
- "name": "kylin_sales_model",
- "owner": null,
- "is_draft": false,
- "description": "",
- "fact_table": "DEFAULT.KYLIN_SALES",
- "lookups": [
- {
- "table": "DEFAULT.KYLIN_CAL_DT",
- "kind": "LOOKUP",
- "alias": "KYLIN_CAL_DT",
- "join": {
- "type": "inner",
- "primary_key": [
- "KYLIN_CAL_DT.CAL_DT"
- ],
- "foreign_key": [
- "KYLIN_SALES.PART_DT"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_CATEGORY_GROUPINGS",
- "kind": "LOOKUP",
- "alias": "KYLIN_CATEGORY_GROUPINGS",
- "join": {
- "type": "inner",
- "primary_key": [
- "KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID",
- "KYLIN_CATEGORY_GROUPINGS.SITE_ID"
- ],
- "foreign_key": [
- "KYLIN_SALES.LEAF_CATEG_ID",
- "KYLIN_SALES.LSTG_SITE_ID"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_ACCOUNT",
- "kind": "LOOKUP",
- "alias": "BUYER_ACCOUNT",
- "join": {
- "type": "inner",
- "primary_key": [
- "BUYER_ACCOUNT.ACCOUNT_ID"
- ],
- "foreign_key": [
- "KYLIN_SALES.BUYER_ID"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_ACCOUNT",
- "kind": "LOOKUP",
- "alias": "SELLER_ACCOUNT",
- "join": {
- "type": "inner",
- "primary_key": [
- "SELLER_ACCOUNT.ACCOUNT_ID"
- ],
- "foreign_key": [
- "KYLIN_SALES.SELLER_ID"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_COUNTRY",
- "kind": "LOOKUP",
- "alias": "BUYER_COUNTRY",
- "join": {
- "type": "inner",
- "primary_key": [
- "BUYER_COUNTRY.COUNTRY"
- ],
- "foreign_key": [
- "BUYER_ACCOUNT.ACCOUNT_COUNTRY"
- ]
- }
- },
- {
- "table": "DEFAULT.KYLIN_COUNTRY",
- "kind": "LOOKUP",
- "alias": "SELLER_COUNTRY",
- "join": {
- "type": "inner",
- "primary_key": [
- "SELLER_COUNTRY.COUNTRY"
- ],
- "foreign_key": [
- "SELLER_ACCOUNT.ACCOUNT_COUNTRY"
- ]
- }
- }
- ],
- "dimensions": [
- {
- "table": "KYLIN_SALES",
- "columns": [
- "TRANS_ID",
- "SELLER_ID",
- "BUYER_ID",
- "PART_DT",
- "LEAF_CATEG_ID",
- "LSTG_FORMAT_NAME",
- "LSTG_SITE_ID",
- "OPS_USER_ID",
- "OPS_REGION"
- ]
- },
- {
- "table": "KYLIN_CAL_DT",
- "columns": [
- "CAL_DT",
- "WEEK_BEG_DT",
- "MONTH_BEG_DT",
- "YEAR_BEG_DT"
- ]
- },
- {
- "table": "KYLIN_CATEGORY_GROUPINGS",
- "columns": [
- "USER_DEFINED_FIELD1",
- "USER_DEFINED_FIELD3",
- "META_CATEG_NAME",
- "CATEG_LVL2_NAME",
- "CATEG_LVL3_NAME",
- "LEAF_CATEG_ID",
- "SITE_ID"
- ]
- },
- {
- "table": "BUYER_ACCOUNT",
- "columns": [
- "ACCOUNT_ID",
- "ACCOUNT_BUYER_LEVEL",
- "ACCOUNT_SELLER_LEVEL",
- "ACCOUNT_COUNTRY",
- "ACCOUNT_CONTACT"
- ]
- },
- {
- "table": "SELLER_ACCOUNT",
- "columns": [
- "ACCOUNT_ID",
- "ACCOUNT_BUYER_LEVEL",
- "ACCOUNT_SELLER_LEVEL",
- "ACCOUNT_COUNTRY",
- "ACCOUNT_CONTACT"
- ]
- },
- {
- "table": "BUYER_COUNTRY",
- "columns": [
- "COUNTRY",
- "NAME"
- ]
- },
- {
- "table": "SELLER_COUNTRY",
- "columns": [
- "COUNTRY",
- "NAME"
- ]
- }
- ],
- "metrics": [
- "KYLIN_SALES.PRICE",
- "KYLIN_SALES.ITEM_COUNT"
- ],
- "filter_condition": "",
- "partition_desc": {
- "partition_date_column": "KYLIN_SALES.PART_DT",
- "partition_time_column": null,
- "partition_date_start": 1325376000000,
- "partition_date_format": "yyyy-MM-dd",
- "partition_time_format": "HH:mm:ss",
- "partition_type": "APPEND",
- "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
- },
- "capacity": "MEDIUM"
- }
-]
-```
-
-## Delete Model
-`DELETE /kylin/api/models/{modelName}`
-
-#### Path variable
-* modelName - `required` `string` Model name.
-
-## Clone Model
-`PUT /kylin/api/models/{modelName}/clone`
-
-***
-
-## Resume Job
-`PUT /kylin/api/jobs/{jobId}/resume`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-#### Response Sample
-```
-{
- "uuid":"c143e0e4-ac5f-434d-acf3-46b0d15e3dc6",
- "last_modified":1407908916705,
- "name":"test_kylin_cube_with_slr_empty - 19700101000000_20140731160000 - BUILD - PDT 2014-08-12 22:48:36",
- "type":"BUILD",
- "duration":0,
- "related_cube":"test_kylin_cube_with_slr_empty",
- "related_segment":"19700101000000_20140731160000",
- "exec_start_time":0,
- "exec_end_time":0,
- "mr_waiting":0,
- "steps":[
- {
- "interruptCmd":null,
- "name":"Create Intermediate Flat Hive Table",
- "sequence_id":0,
- "exec_cmd":"hive -e \"DROP TABLE IF EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6;\nCREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\n(\nCAL_DT date\n,LEAF_CATEG_ID int\n,LSTG_SITE_ID int\n,META_CATEG_NAME string\n,CATEG_LVL2_NAME string\n,CATEG_LVL3_NAME string\n,LSTG_FORMAT_NAME string\n,SLR_SEGMENT_ [...]
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"SHELL_CMD_HADOOP",
- "info":null,
- "run_async":false
- },
- {
- "interruptCmd":null,
- "name":"Extract Fact Table Distinct Columns",
- "sequence_id":1,
- "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns -jobname Kylin_Fact_Distinct_Columns_test_kylin_cube_with_slr_empty_Step_1",
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"JAVA_CMD_HADOOP_FACTDISTINCT",
- "info":null,
- "run_async":true
- },
- {
- "interruptCmd":null,
- "name":"Load HFile to HBase Table",
- "sequence_id":12,
- "exec_cmd":" -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile/ -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -cubename test_kylin_cube_with_slr_empty",
- "interrupt_cmd":null,
- "exec_start_time":0,
- "exec_end_time":0,
- "exec_wait_time":0,
- "step_status":"PENDING",
- "cmd_type":"JAVA_CMD_HADOOP_NO_MR_BULKLOAD",
- "info":null,
- "run_async":false
- }
- ],
- "job_status":"PENDING",
- "progress":0.0
-}
-```
-## Pause Job
-`PUT /kylin/api/jobs/{jobId}/pause`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-## Discard Job
-`PUT /kylin/api/jobs/{jobId}/cancel`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-## Drop Job
-`DELETE /kylin/api/jobs/{jobId}/drop`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-## Get Job Status
-`GET /kylin/api/jobs/{jobId}`
-
-#### Path variable
-* jobId - `required` `string` Job id.
-
-#### Response Sample
-(Same as "Resume Job")
-
-## Get job step output
-`GET /kylin/api/jobs/{jobId}/steps/{stepId}/output`
-
-#### Path Variable
-* jobId - `required` `string` Job id.
-* stepId - `required` `string` Step id; the step id is composed by jobId with step sequence id; for example, the jobId is "fb479e54-837f-49a2-b457-651fc50be110", its 3rd step id is "fb479e54-837f-49a2-b457-651fc50be110-3",
-
-#### Response Sample
-```
-{
- "cmd_output":"log string"
-}
-```
-
-## Get job list
-`GET /kylin/api/jobs`
-
-#### Request Variables
-* cubeName - `optional` `string` Cube name.
-* projectName - `required` `string` Project name.
-* status - `optional` `int` Job status, e.g. (NEW: 0, PENDING: 1, RUNNING: 2, STOPPED: 32, FINISHED: 4, ERROR: 8, DISCARDED: 16)
-* offset - `required` `int` Offset used by pagination.
-* limit - `required` `int` Jobs per page.
-* timeFilter - `required` `int`, e.g. (LAST ONE DAY: 0, LAST ONE WEEK: 1, LAST ONE MONTH: 2, LAST ONE YEAR: 3, ALL: 4)
-
-For example, to get the job list in project 'learn_kylin' for cube 'kylin_sales_cube' in lastone week:
-
-```
-GET: /kylin/api/jobs?cubeName=kylin_sales_cube&limit=15&offset=0&projectName=learn_kylin&timeFilter=1
-```
-
-#### Response Sample
-```
-[
- {
- "uuid": "9eb7bccf-4448-4578-9c29-552658b5a2ca",
- "last_modified": 1490957579843,
- "version": "2.0.0",
- "name": "Sample_Cube - 19700101000000_20150101000000 - BUILD - GMT+08:00 2017-03-31 18:36:08",
- "type": "BUILD",
- "duration": 936,
- "related_cube": "Sample_Cube",
- "related_segment": "53a5d7f7-7e06-4ea1-b3ee-b7f30343c723",
- "exec_start_time": 1490956581743,
- "exec_end_time": 1490957518131,
- "mr_waiting": 0,
- "steps": [
- {
- "interruptCmd": null,
- "id": "9eb7bccf-4448-4578-9c29-552658b5a2ca-00",
- "name": "Create Intermediate Flat Hive Table",
- "sequence_id": 0,
- "exec_cmd": null,
- "interrupt_cmd": null,
- "exec_start_time": 1490957508721,
- "exec_end_time": 1490957518102,
- "exec_wait_time": 0,
- "step_status": "DISCARDED",
- "cmd_type": "SHELL_CMD_HADOOP",
- "info": { "endTime": "1490957518102", "startTime": "1490957508721" },
- "run_async": false
- },
- {
- "interruptCmd": null,
- "id": "9eb7bccf-4448-4578-9c29-552658b5a2ca-01",
- "name": "Redistribute Flat Hive Table",
- "sequence_id": 1,
- "exec_cmd": null,
- "interrupt_cmd": null,
- "exec_start_time": 0,
- "exec_end_time": 0,
- "exec_wait_time": 0,
- "step_status": "DISCARDED",
- "cmd_type": "SHELL_CMD_HADOOP",
- "info": {},
- "run_async": false
- }
- ],
- "submitter": "ADMIN",
- "job_status": "FINISHED",
- "progress": 100.0
- }
-]
-```
-
-## Get Job Status Overview
-`GET /kylin/api/jobs/overview`
-
-### Request Variables
-(Same as "Get job list")
-
-#### Response Sample
-```sh
-{
- "DISCARDED": 0,
- "NEW": 0,
- "STOPPED": 0,
- "PENDING": 0,
- "RUNNING": 0,
- "FINISHED": 1,
- "ERROR": 0
-}
-```
-
-## Resubmit realtime build job
-`PUT /kylin/api/jobs/{jobId}/resubmit`
-
-## Rollback job
-`PUT /kylin/api/{jobId}/steps/{stepId}/rollback`
-
-#### Path Parameters
-* jobId - `required` `string` job id you want to rollback
-* stepId - `required` `string` specify rollback step id, e.g.(Create Intermediate Flat Hive:1)
-
-For example, rollback job to Create Intermediate Flat Hive:
-`PUT: kylin/api/jobs/4e84cb5e-a929-89c7-6240-768fa9835d89/steps/1/rollback`
-
-#### Response Sample
-```sh
-{
- "uuid": "4e84cb5e-a929-89c7-6240-768fa9835d89",
- "last_modified": 1590054128311,
- "version": "3.0.0.20500",
- "name": "BUILD CUBE - kylin_sales_cube - 20120102000000_20120103000000 - CST 2020-05-21 17:38:59",
- "projectName": "learn_kylin",
- "type": "BUILD",
- "duration": 187,
- "related_cube": "kylin_sales_cube",
- "display_cube_name": "kylin_sales_cube",
- "related_segment": "fca98f62-cb3f-8b53-5bf1-94a85334560b",
- "exec_start_time": 1590053963522,
- "exec_end_time": 0,
- "exec_interrupt_time": 0,
- "mr_waiting": 39,
- "steps": [
- {
- "interruptCmd": null,
- "id": "4e84cb5e-a929-89c7-6240-768fa9835d89-00",
- "name": "Create Intermediate Flat Hive Table",
- "sequence_id": 0,
- "exec_cmd": null,
- "interrupt_cmd": null,
- "exec_start_time": 0,
- "exec_end_time": 0,
- "exec_wait_time": 0,
- "step_status": "PENDING",
- "cmd_type": "SHELL_CMD_HADOOP",
- "info": {},
- "run_async": false
- },
- ...
- ],
- "submitter": "ADMIN",
- "job_status": "RUNNING",
- "build_instance": "20984@host",
- "progress": 0.0
-}
-```
-
-***
-
-## Get Hive Table
-`GET /kylin/api/tables/{project}/{tableName}`
-
-#### Path Parameters
-* project - `required` `string` project name
-* tableName - `required` `string` table name to find.
-
-#### Response Sample
-```sh
-{
- uuid: "69cc92c0-fc42-4bb9-893f-bd1141c91dbe",
- name: "SAMPLE_07",
- columns: [{
- id: "1",
- name: "CODE",
- datatype: "string"
- }, {
- id: "2",
- name: "DESCRIPTION",
- datatype: "string"
- }, {
- id: "3",
- name: "TOTAL_EMP",
- datatype: "int"
- }, {
- id: "4",
- name: "SALARY",
- datatype: "int"
- }],
- database: "DEFAULT",
- last_modified: 1419330476755
-}
-```
-
-## Get Hive Tables
-`GET /kylin/api/tables`
-
-#### Request Parameters
-* project- `required` `string` will list all tables in the project.
-* ext- `optional` `boolean` set true to get extend info of table.
-
-#### Response Sample
-```sh
-[
- {
- uuid: "53856c96-fe4d-459e-a9dc-c339b1bc3310",
- name: "SAMPLE_08",
- columns: [{
- id: "1",
- name: "CODE",
- datatype: "string"
- }, {
- id: "2",
- name: "DESCRIPTION",
- datatype: "string"
- }, {
- id: "3",
- name: "TOTAL_EMP",
- datatype: "int"
- }, {
- id: "4",
- name: "SALARY",
- datatype: "int"
- }],
- database: "DEFAULT",
- cardinality: {},
- last_modified: 0,
- exd: {
- minFileSize: "46069",
- totalNumberFiles: "1",
- location: "hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/sample_08",
- lastAccessTime: "1398176495945",
- lastUpdateTime: "1398176495981",
- columns: "struct columns { string code, string description, i32 total_emp, i32 salary}",
- partitionColumns: "",
- EXD_STATUS: "true",
- maxFileSize: "46069",
- inputformat: "org.apache.hadoop.mapred.TextInputFormat",
- partitioned: "false",
- tableName: "sample_08",
- owner: "hue",
- totalFileSize: "46069",
- outputformat: "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
- }
- }
-]
-```
-
-## Load Hive Tables
-`POST /kylin/api/tables/{tables}/{project}`
-
-#### Request Parameters
-* tables - `required` `string` table names you want to load from hive, separated with comma.
-* project - `required` `String` the project which the tables will be loaded into.
-
-#### Request Body
-* calculate - `optional` `boolean`
-
-#### Response Sample
-```
-{
- "result.loaded": ["DEFAULT.SAMPLE_07"],
- "result.unloaded": ["sapmle_08"]
-}
-```
-
-## Unload Hive Tables
-`DELETE /kylin/api/tables/{tables}/{project}`
-
-#### Path Parameters
-* tables - `required` `string` table names you want to unload, separated with comma.
-* project - `required` `String` the project which the tables belong to.
-
-#### Response Sample
-```sh
-{
- "result.unload.success": [
- "kylin_sales"
- ],
- "result.unload.fail": []
-}
-```
-
-## Show databases in hive
-`GET /kylin/api/tables/hive`
-
-## Show tables in a hive database
-`GET /kylin/api/tables/hive/{database}`
-
-***
-
-## Wipe cache
-`PUT /kylin/api/cache/{type}/{name}/{action}`
-
-#### Path variable
-* type - `required` `string` 'METADATA' or 'CUBE'
-* name - `required` `string` Cache key, e.g the cube name.
-* action - `required` `string` 'create', 'update' or 'drop'
-
-
-## Announce wipe cache
-`PUT /kylin/api/cache/announce/{type}/{name}/{action}`
-
-#### Path variable
-(Same as "Wipe cache")
-
-## Hot load kylin config
-`POST /kylin/api/cache/announce/config`
-
-
-***
-
-## Initiate cube start position
-Set the stream cube's start position to the current latest offsets; This can avoid building from the earlist position of Kafka topic (if you have set a long retension time);
-
-`PUT /kylin/api/cubes/{cubeName}/init_start_offsets`
-
-#### Path variable
-* cubeName - `required` `string` Cube name
-
-#### Response Sample
-```sh
-{
- "result": "success",
- "offsets": "{0=246059529, 1=253547684, 2=253023895, 3=172996803, 4=165503476, 5=173513896, 6=19200473, 7=26691891, 8=26699895, 9=26694021, 10=19204164, 11=26694597}"
-}
-```
-
-## Build stream cube
-`PUT /kylin/api/cubes/{cubeName}/build2`
-
-This API is specific for stream cube's building;
-
-#### Path variable
-* cubeName - `required` `string` Cube name
-
-#### Request Body
-
-* sourceOffsetStart - `required` `long` The start offset, 0 represents from previous position;
-* sourceOffsetEnd - `required` `long` The end offset, 9223372036854775807 represents to the end position of current stream data
-* buildType - `required` Build type, "BUILD", "MERGE" or "REFRESH"
-
-#### Request Sample
-
-```sh
-{
- "sourceOffsetStart": 0,
- "sourceOffsetEnd": 9223372036854775807,
- "buildType": "BUILD"
-}
-```
-
-#### Response Sample
-```sh
-{
- "uuid": "3afd6e75-f921-41e1-8c68-cb60bc72a601",
- "last_modified": 1480402541240,
- "version": "1.6.0",
- "name": "embedded_cube_clone - 1409830324_1409849348 - BUILD - PST 2016-11-28 22:55:41",
- "type": "BUILD",
- "duration": 0,
- "related_cube": "embedded_cube_clone",
- "related_segment": "42ebcdea-cbe9-4905-84db-31cb25f11515",
- "exec_start_time": 0,
- "exec_end_time": 0,
- "mr_waiting": 0,
- ...
-}
-```
-
-## Check segment holes
-`GET /kylin/api/cubes/{cubeName}/holes`
-
-#### Path variable
-* cubeName - `required` `string` Cube name
-
-## Fill segment holes
-`PUT /kylin/api/cubes/{cubeName}/holes`
-
-#### Path variable
-* cubeName - `required` `string` Cube name
-
-## Get streaming configs
-`GET /kylin/api/streaming/getConfig`
-
-#### Response sample
-```sh
-[
- {
- "uuid": "8b2b9dfe-777c-4d39-bf89-8472ec929193",
- "last_modified": 1587528491000,
- "version": "3.0.0.20500",
- "name": "DEFAULT.KYLIN_STREAMING_TABLE",
- "type": "kafka"
- }
-]
-```
-
-## Get kafka configs
-`GET /kylin/api/streaming/getKfkConfig`
-
-#### Response sample
-```sh
-[
- {
- "uuid": "8b2b9dfe-777c-4d39-bf89-8472ec919193",
- "last_modified": 1587528491000,
- "version": "3.0.0.20500",
- "name": "DEFAULT.KYLIN_STREAMING_TABLE",
- "clusters": [
- {
- "uuid": null,
- "last_modified": 0,
- "version": "3.0.0.20500",
- "brokers": [
- {
- "id": 0,
- "host": "localhost",
- "port": 9092
- }
- ]
- }
- ],
- "topic": "kylin_streaming_topic",
- "timeout": 60000,
- "parserName": "org.apache.kylin.source.kafka.TimedJsonStreamParser",
- "timestampField": "order_time",
- "margin": 0,
- "splitRows": 1000000,
- "parserProperties": null
- }
-]
-```
-
-## Create streaming schema
-`POST /kylin/api/streaming`
-
-#### Request body
-* project - `required` `string` Project which you want create streaming schema to.
-* tableData - `required` `string` Streaming table desc.
-* streamingConfig - `required` `string` Streaming config.
-* kafkaConfig - `required` `string` Kafka config.
-
-#### Request sample
-```
-{
- "project":"test",
- "tableData":"{\"uuid\": \"e286e39e-41d7-44c2-8fa2-41b365123987\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"KYLIN_TEST_STREAMING_TABLE\",\"columns\": [{\"id\": \"1\",\"name\": \"AMOUNT\",\"datatype\": \"decimal(19,4)\"},{\"id\": 3,\"name\": \"ORDER_TIME\",\"datatype\": \"timestamp\",\"index\": \"T\"}],\"source_type\": 1,\"table_type\": null,\"database\": \"DEFAULT\"}",
- "streamingConfig":"{\"uuid\": \"8b2b9dfe-777c-4d39-bf89-8472ec929193\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"DEFAULT.KYLIN_TEST_STREAMING_TABLE\",\"type\": \"kafka\"}",
- "kafkaConfig":"{\"uuid\": \"8b2b9dfe-777c-4d39-bf89-8472ec919193\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"DEFAULT.KYLIN_STREAMING_TABLE\",\"clusters\": [{\"uuid\": null,\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"brokers\": [{\"id\": 0,\"host\": \"localhost\",\"port\": 9092}]}],\"topic\": \"kylin_streaming_topic\",\"timeout\": 60000,\"parserName\": \"org.apache.kylin.source.kafka.TimedJsonStreamParser\",\"timestampField\": \"order_time\",\"margin\": 0,\ [...]
-}
-```
-
-#### Response sample
-```sh
-{
- "project": "test",
- "tableData": "{\"uuid\": \"e286e39e-41d7-44c2-8fa2-41b365123987\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"KYLIN_TEST_STREAMING_TABLE\",\"columns\": [{\"id\": \"1\",\"name\": \"AMOUNT\",\"datatype\": \"decimal(19,4)\"},{\"id\": 3,\"name\": \"ORDER_TIME\",\"datatype\": \"timestamp\",\"index\": \"T\"}],\"source_type\": 1,\"table_type\": null,\"database\": \"DEFAULT\"}",
- "streamingConfig": "{\"uuid\": \"8b2b9dfe-777c-4d39-bf89-8472ec929193\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"DEFAULT.KYLIN_TEST_STREAMING_TABLE\",\"type\": \"kafka\"}",
- "kafkaConfig": "{\"uuid\": \"8b2b9dfe-777c-4d39-bf89-8472ec919193\",\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"name\": \"DEFAULT.KYLIN_STREAMING_TABLE\",\"clusters\": [{\"uuid\": null,\"last_modified\": 0,\"version\": \"3.0.0.20500\",\"brokers\": [{\"id\": 0,\"host\": \"localhost\",\"port\": 9092}]}],\"topic\": \"kylin_streaming_topic\",\"timeout\": 60000,\"parserName\": \"org.apache.kylin.source.kafka.TimedJsonStreamParser\",\"timestampField\": \"order_time\",\"margin\": 0, [...]
- "successful": true,
- "message": null
-}
-```
-
-## Update streaming schema
-`PUT /kylin/api/streaming`
-
-#### Request body
-(Same as "Create streaming schema)
-
-#### Request sample
-(Same as "Create streaming schema)
-
-#### Response sample
-(Same as "Create streaming schema)
-
-
-***
-
-## Get users can query the table
-`GET /kylin/api/acl/table/{project}/{type}/{table}`
-
-#### Path variable
-* project - `required` `string` projectName to which table belongs
-* type - `required` `string` user or group
-* table - `required` `string` table name
-
-## Get users cannot query the table
-`GET /kylin/api/acl/table/{project}/{type}/black/{table}`
-
-#### Path variable
-* project - `required` `string` projectName to which table belongs
-* type - `required` `string` user or group
-* table - `required` `string` table name
-
-## Put user to table blacklist
-`DELETE /kylin/api/table/{project}/{type}/{table}/{name}`
-
-#### Path variable
-* project - `required` `string` projectName to which table belongs
-* type - `required` `string` user or group
-* table - `required` `string` table name
-* name - `required` `string` user name or group name you want to put to table blacklist
-
-## Delete user from table blacklist
-`POST /kylin/api/table/{project}/{type}/{table}/{name}`
-
-#### Path variable
-* project - `required` `string` projectName to which table belongs
-* type - `required` `string` user or group
-* table - `required` `string` table name
-* name - `required` `string` user name or group name you want to delete from table blacklist
-
-***
-
-## Get all metrics
-`GET /kylin/api/jmetrics/`
-
-#### Response sample
-```
-{
- "version": "3.0.0",
- "gauges": {},
- "counters": {},
- "histograms": {},
- "meters": {},
- "timers": {}
-}
-```
-
-## Get specific type of metrics
-`GET /kylin/api/jmetrics/{type}`
-
-#### Path variable
-* type - `required` `string` Specific type of metrics you want, e.g meters
-
-## Use RESTful API in Javascript
-
-Keypoints of call Kylin RESTful API in web page are:
-
-1. Add basic access authorization info in http headers.
-
-2. Use proper request type and data synax.
-
-Kylin security is based on basic access authorization, if you want to use API in your javascript, you need to add authorization info in http headers; for example:
-
-```
-$.ajaxSetup({
- headers: { 'Authorization': "Basic eWFu**********X***ZA==", 'Content-Type': 'application/json;charset=utf-8' } // use your own authorization code here
- });
- var request = $.ajax({
- url: "http://hostname/kylin/api/query",
- type: "POST",
- data: '{"sql":"select count(*) from SUMMARY;","offset":0,"limit":50000,"acceptPartial":true,"project":"test"}',
- dataType: "json"
- });
- request.done(function( msg ) {
- alert(msg);
- });
- request.fail(function( jqXHR, textStatus ) {
- alert( "Request failed: " + textStatus );
- });
-
-```
-
-To generate your authorization code, download and import "jquery.base64.js" from [https://github.com/yckart/jquery.base64.js](https://github.com/yckart/jquery.base64.js)).
-
-```
-var authorizationCode = $.base64('encode', 'NT_USERNAME' + ":" + 'NT_PASSWORD');
-
-$.ajaxSetup({
- headers: {
- 'Authorization': "Basic " + authorizationCode,
- 'Content-Type': 'application/json;charset=utf-8'
- }
-});
-```
+Kylin 4.0 支持的 REST API 及其用法在 Apache Kylin Wiki 页面列出：
+[Apache Kylin4.0 RESTful API](https://cwiki.apache.org/confluence/display/KYLIN/REST+API)
\ No newline at end of file
diff --git a/website/_docs/index.cn.md b/website/_docs/index.cn.md
index ce6c2c7..02ac835 100644
--- a/website/_docs/index.cn.md
+++ b/website/_docs/index.cn.md
@@ -1,6 +1,6 @@
---
layout: docs-cn
-title: 概述
+title: Apache Kylin4 概述
categories: docs
permalink: /cn/docs/index.html
---
@@ -11,62 +11,135 @@ permalink: /cn/docs/index.html
Apache Kylin™是一个开源的、分布式的分析型数据仓库，提供 Hadoop 之上的 SQL 查询接口及多维分析（OLAP）能力以支持超大规模数据，最初由eBay Inc.开发并贡献至开源社区。
-查看其它版本文档:
-* [v4.0 document](/cn/docs40)
-* [v3.0 document](/cn/docs30/)
+查看其它版本文档:
+* [v3.1 document](/cn/docs31/)
* [v2.4 document](/cn/docs24/)
* [归档](/archive/)
-如果您对 Kylin v4.0.0-alpha 感兴趣，请通过我们的 [v4.0 Wiki](https://cwiki.apache.org/confluence/display/KYLIN/User+Manual+4.X) 来了解 。
+Apache Kylin4.0 是 Apache Kylin3.x 之后一次重大的版本更新，它采用了全新的 Spark 构建引擎和 Parquet 作为存储，同时使用 Spark 作为查询引擎。
+
+Apache Kylin4.0 的第一个版本 kylin4.0.0-alpha 于 2020 年 7 月份发布，此后相继发布 kylin4.0.0-beta 以及正式版本。
+
+为了方便用户对 Kylin4.x 有更全面更深层的了解，本篇文档会着重从 Kylin4.x 与之前版本有何异同的角度对 Kylin4.x 做全面概述。文章分为以下几个部分：
+
+- 为什么选择 Parquet 替换 HBase
+- 预计算结果在 Kylin4.0 中如何存储
+- Kylin 4.0 的构建引擎
+- Kylin 4.0 的查询引擎
+- Kylin4.0 与 Kylin3.1 功能对比
+- Kylin 4.0 性能表现
+- Kylin 4.0 查询和构建调优
+- Kylin 4.0 用户案例
+
+## 为什么选择 Parquet 替换 HBase
+在 3.x 以及之前的版本中，kylin 一直使用 HBase 作为存储引擎来保存 cube 构建后产生的预计算结果。HBase 作为 HDFS 之上面向列族的数据库，查询表现已经算是比较优秀，但是它仍然存在以下几个缺点：
+1. HBase 不是真正的列式存储；
+2. HBase 没有二级索引，Rowkey 是它唯一的索引；
+3. HBase 没有对存储的数据进行编码，kylin 必须自己进行对数据编码的过程；
+4. HBase 不适合云上部署和自动伸缩；
+5. HBase 不同版本之间的 API 版本不同，存在兼容性问题（比如，0.98，1.0，1.1，2.0）；
+6. HBase 存在不同的供应商版本，他们之间有兼容性问题。
+
+针对以上问题，社区提出了对使用 Apache Parquet + Spark 来代替 HBase 的提议，理由如下：
+1. Parquet 是一种开源并且已经成熟稳定的列式存储格式；
+2. Parquet 对云更加友好，可以兼容各种文件系统，包括 HDFS、S3、Azure Blob store、Ali OSS 等；
+3. Parquet 可以很好地与 Hadoop、Hive、Spark、Impala 等集成；
+4. Parquet 支持自定义索引。
+
+## 预计算结果在 Kylin4.0 中如何存储
+在 Kylin4.x 中，预计算结果以 Parquet 格式存储在文件系统中，文件存储结构对于 I/O 优化很重要，提前对存储目录结构进行设计，就能够在查询时通过目录或者文件名过滤数据文件，避免不必要的扫描。
+Kylin4 对 cube 进行构建得到的预计算结果的 Parquet 文件在文件系统中存储的目录结构如下：
+- cube_name
+ - SegmentA
+ - Cuboid-111
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - Cuboid-222
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - SegmentB
+ - Cuboid-111
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - Cuboid-222
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+
+可以看出，与 HBase 相比，采用 Parquet 存储可以很方便地增删 cuboid 而不影响其他数据。利用这种特点，Kylin4 中实现了支持用户手动增删 cuboid 的功能，请参考：[How to update cuboid list for a cube](https://cwiki.apache.org/confluence/display/KYLIN/How+to+update+cuboid+list+for+a+cube)
+
+## Kylin 4.0 的构建引擎
+在 Kylin4 中，Spark Engine 是唯一的构建引擎，与之前版本中的构建引擎相比，存在如下特点：
+
+1、Kylin4 的构建简化了很多步骤。比如在 Cube Build Job 中， kylin4 只需要资源探测和 cubing 两个步骤，就可以完成构建；
+2、由于 Parquet 会对存储的数据进行编码，所以在 kylin4 中不再需要维度字典和对维度列编码的过程；
+3、Kylin4 对全局字典做了全新的实现，更多细节请参考：[Kylin4 全局字典](https://cwiki.apache.org/confluence/display/KYLIN/Global+Dictionary+on+Spark+CN) ；
+4、Kylin4 会根据集群资源、构建任务情况等对 Spark 进行自动调参；
+5、Kylin4 提高了构建速度。
+
+用户可以通过 `kylin.build.spark-conf` 开头的配置项手动修改构建相关的 Spark 配置，经过用户手动修改的 Spark 配置项不会再参与自动调参。
+
+## Kylin 4.0 的查询引擎
+Kylin4 的查询引擎 `Sparder(SparderContext)` 是由 spark application 后端实现的新型分布式查询引擎，相比于原来的查询引擎，Sparder 的优势体现在以下几点：
+- 分布式的查询引擎，有效避免单点故障；
+- 与构建所使用的计算引擎统一为 Spark；
+- 对于复杂查询的性能有很大提高；
+- 可以从 Spark 的新功能及其生态中获益。
+
+在 Kylin4 中，Sparder 是作为一个 long-running 的 spark application 存在的。 Sparder 会根据 `kylin.query.spark-conf` 开头的配置项中配置的 Spark 参数来获取 Yarn 资源，如果配置的资源参数过大，可能会影响构建任务甚至无法成功启动 Sparder，如果 Sparder 没有成功启动，则所有查询任务都会失败，用户可以在 kylin WebUI 的 System 页面中检查 Sparder 状态。
+
+默认情况下，用于查询的 spark 参数会设置的比较小，在生产环境中，大家可以适当把这些参数调大一些，以提升查询性能。
+`kylin.query.auto-sparder-context` 参数用于控制是否在启动 kylin 的同时启动 Sparder，默认值为 false，即默认情况下会在执行第一条 SQL 的时候才启动 Sparder，由于这个原因，执行第一条 SQL 的时候的会花费较长时间。
+如果你不希望第一条 SQL 的查询速度低于预期，可以设置 `kylin.query.auto-sparder-context` 为 `true`，此时 Sparder 会随 Kylin 一起启动。
+
+## Kylin 4.0 与 Kylin 3.1 功能对比
+
+| Feature | Kylin 3.1.0 | Kylin 4.0 |
+| --------------------- | :------------------------------------------- | :----------------------------------------------|
+| Storage | HBase | Parquet |
+| BuildEngine | MapReduce/Spark/Flink | New Spark Engine |
+| Metastore | HBase(Default)/Mysql | Mysql(Default) |
+| DataSource | Kafka/Hive/JDBC | Hive/CSV |
+| Global Dictionary | Two implementation | New implementation |
+| Cube Optimization Tool | Cube Planner | Cube Planner phase1 and Optimize cube manually |
+| Self-monitoring | System cube and Dashboard | System cube and Dashboard |
+| PushDown Engine | Hive/JDBC | Spark SQL |
+| Hadoop platform | HDP2/HDP3/CDH5/CDH6/EMR5 | HDP2/CDH5/CDH6/EMR5/EMR6/HDI |
+| Deployment mode | Single node/Cluster/Read and write separation| Single node/Cluster/Read and write separation |
+
+## Kylin 4.0 性能表现
+为了测试 Kylin4.0 的性能，我们分别在 SSB 数据集和 TPC-H 数据集上做了 benchmark，与 Kylin3.1.0 进行对比。测试环境为 4 个节点的 CDH 集群，所使用的 yarn 队列分配了 400G 内存和 128 cpu cores。
+性能测试对比结果如下：
+- Comparison of build duration and result size（SSB）
+
+
+
+测试结果可以体现以下两点：
+- kylin4 的构建速度与 kylin3.1.0 的 Spark Engine 相比有明显提升；
+- Kylin4 构建后得到的预计算结果 Parquet 文件大小与 HBase 相比有明显减小；
+
+- Comparison of query response(SSB and TPC-H)
+
+
+
+从查询结果对比中可以看出，对于***简单查询***，kylin3 与 Kylin4 不相上下，kylin4 略有不足；而对于***复杂查询***，kylin4 则体现出了明显的优势，查询速度比 kylin3 快很多。
+并且，Kylin4 中的***简单查询***的性能还存在很大的优化空间。在有赞使用 Kylin4 的实践中，对于***简单查询***的性能可以优化到 1 秒以内。
+
+## 如何升级
+请参考文档：[How to migrate metadata to Kylin4](https://cwiki.apache.org/confluence/display/KYLIN/How+to+migrate+metadata+to+Kylin+4)
+
+## Kylin 4.0 查询和构建调优
+对于 Kylin4 的调优，请参考：[How to improve cube building and query performance](/docs/howto/howto_optimize_build_and_query.html)
+
+## Kylin 4.0 用户案例
+[Why did Youzan choose Kylin4](/blog/2021/06/17/Why-did-Youzan-choose-Kylin4)
+
+参考链接：
+[Kylin Improvement Proposal 1: Parquet Storage](https://cwiki.apache.org/confluence/display/KYLIN/KIP-1%3A+Parquet+storage)
-安装
-------------
-1. [安装指南](install/index.html)
-2. [Kylin 配置](install/configuration.html)
-3. [集群模式部署](install/kylin_cluster.html)
-4. [高级配置](install/advance_settings.html)
-5. [用 Docker 运行 Kylin](install/kylin_docker.html)
-6. [在 AWS EMR 上安装 Kylin](install/kylin_aws_emr.html)
-
-教程
-------------
-1. [样例 Cube 快速入门](tutorial/kylin_sample.html)
-2. [Web 界面](tutorial/web.html)
-3. [Cube 创建](tutorial/create_cube.html)
-4. [Cube 构建和 Job 监控](tutorial/cube_build_job.html)
-5. [Cube 迁移](tutorial/cube_migration.html)
-6. [SQL 快速参考](tutorial/sql_reference.html)
-7. [用 Kafka 流构建 Cube](tutorial/cube_streaming.html)
-8. [用 Spark 构建 Cube](tutorial/cube_spark.html)
-9. [优化 Cube 构建](tutorial/cube_build_performance.html)
-10. [查询下压](tutorial/query_pushdown.html)
-11. [建立 System Cube](tutorial/setup_systemcube.html)
-12. [使用 Cube Planner](tutorial/use_cube_planner.html)
-13. [使用 Dashboard](tutorial/use_dashboard.html)
-14. [建立 JDBC 数据源](tutorial/setup_jdbc_datasource.html)
-
-
-工具集成
-------------
-1. [ODBC 驱动](tutorial/odbc.html)
-2. [JDBC 驱动](howto/howto_jdbc.html)
-3. [RESTful API 列表](howto/howto_use_restapi.html)
-4. [用 API 构建 Cube](howto/howto_build_cube_with_restapi.html)
-5. [MS Excel 及 PowerBI 教程](tutorial/powerbi.html)
-6. [Tableau 8](tutorial/tableau.html)
-7. [Tableau 9](tutorial/tableau_91.html)
-8. [SQuirreL](tutorial/squirrel.html)
-9. [Qlik Sense 集成](tutorial/Qlik.html)
-10. [Apache Superset](tutorial/superset.html)
-11. [Redash](/blog/2018/05/08/redash-kylin-plugin-strikingly/)
-12. [Davinci](/cn_blog/2019/11/29/Davinci-Kylin-Insight/)
-
-
-帮助
-------------
-1. [备份元数据](howto/howto_backup_metadata.html)
-2. [清理存储](howto/howto_cleanup_storage.html)
-3. [版本升级](howto/howto_upgrade.html)
diff --git a/website/_docs/index.md b/website/_docs/index.md
index 403c187..d67c250 100644
--- a/website/_docs/index.md
+++ b/website/_docs/index.md
@@ -1,6 +1,6 @@
---
layout: docs
-title: Overview
+title: Overview of Apache Kylin4.x
categories: docs
permalink: /docs/index.html
---
@@ -11,65 +11,139 @@ Welcome to Apache Kylin™: Analytical Data Warehouse for Big Data
Apache Kylin™ is an open source Distributed Analytical Data Warehouse for big data; It was designed to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop supporting extremely large datasets.
-This is the document for the latest released version (v3.1.0). Document of other versions:
-* [v4.0 document](/docs40)
+This is the document for Apache Kylin4.0. Document of other versions:
* [v3.0 document](/docs30)
* [v2.4 document](/docs24)
* [Archived](/archive/)
-If you are interested in Kylin v4.0.0-alpha, feel free to check detail at [v4.0 Wiki](https://cwiki.apache.org/confluence/display/KYLIN/User+Manual+4.X) .
+Apache kylin 4.0 is a major version after Apache kylin 3.x. Kylin4 uses a new spark build engine and parquet as storage, and uses spark as query engine.
-Installation & Setup
-------------
-1. [Installation Guide](install/index.html)
-2. [Configurations](install/configuration.html)
-3. [Deploy in cluster mode](install/kylin_cluster.html)
-4. [Advanced settings](install/advance_settings.html)
-5. [Run Kylin with Docker](install/kylin_docker.html)
-6. [Install Kylin on AWS EMR](install/kylin_aws_emr.html)
-
-Tutorial
-------------
-1. [Quick Start with Sample Cube](tutorial/kylin_sample.html)
-2. [Web Interface](tutorial/web.html)
-3. [Cube Wizard](tutorial/create_cube.html)
-4. [Cube Build and Job Monitoring](tutorial/cube_build_job.html)
-5. [Cube Migration](tutorial/cube_migration.html)
-6. [SQL reference](tutorial/sql_reference.html)
-7. [Build Cube with Streaming Data](tutorial/cube_streaming.html)
-8. [Build Cube with Spark Engine](tutorial/cube_spark.html)
-9. [Cube Build Tuning](tutorial/cube_build_performance.html)
-10. [Enable Query Pushdown](tutorial/query_pushdown.html)
-11. [Setup System Cube](tutorial/setup_systemcube.html)
-12. [Optimize with Cube Planner](tutorial/use_cube_planner.html)
-13. [Use System Dashboard](tutorial/use_dashboard.html)
-14. [Setup JDBC Data Source](tutorial/setup_jdbc_datasource.html)
-
-
-Connectivity and APIs
-------------
-1. [ODBC driver](tutorial/odbc.html)
-2. [JDBC driver](howto/howto_jdbc.html)
-3. [RESTful API list](howto/howto_use_restapi.html)
-4. [Build cube with RESTful API](howto/howto_build_cube_with_restapi.html)
-5. [Connect from MS Excel and PowerBI](tutorial/powerbi.html)
-6. [Connect from Tableau 8](tutorial/tableau.html)
-7. [Connect from Tableau 9](tutorial/tableau_91.html)
-8. [Connect from MicroStrategy](tutorial/microstrategy.html)
-9. [Connect from SQuirreL](tutorial/squirrel.html)
-10. [Connect from Apache Flink](tutorial/flink.html)
-11. [Connect from Apache Spark](tutorial/spark.html)
-12. [Connect from Hue](tutorial/hue.html)
-13. [Connect from Qlik Sense](tutorial/Qlik.html)
-14. [Connect from Apache Superset](tutorial/superset.html)
-15. [Connect from Redash](/blog/2018/05/08/redash-kylin-plugin-strikingly/)
-
-
-Operations
-------------
-1. [Backup/restore Kylin metadata](howto/howto_backup_metadata.html)
-2. [Cleanup storage](howto/howto_cleanup_storage.html)
-3. [Upgrade from old version](howto/howto_upgrade.html)
+Kylin 4.0.0-alpha, the first version of Apache kylin 4.0, was released in July 2020, and then kylin 4.0.0-beta and official version were released.
+
+In order to facilitate users to have a more comprehensive and deeper understanding of kylin4.x, this document will focus on a comprehensive overview of kylin4.x from the perspective of the similarities and differences between kylin4.x and previous versions.
+
+The article includes the following parts:
+
+- Why replace HBase with Parquet
+- How to store pre calculation results in kylin 4.0
+- Build engine of Kylin 4.0
+- Query engine of kylin 4.0
+- Feature comparison between kylin 4.0 and kylin 3.1
+- Kylin 4.0 performance
+- Kylin 4.0 query and build tuning
+- Kylin 4.0 use case
+
+## Why replace HBase with Parquet
+In versions 3.x and before, Kylin has been using HBase as a storage engine to save the precomputing results generated after cube builds. HBase, as the database of HDFS, has been excellent in query performance, but it still has the following disadvantages:
+
+1. HBase is not real columnar storage;
+2. HBase has no secondary index; Rowkey is the only index;
+3. HBase has no encoding, Kylin has to do the encoding by itself;
+4. HBase does not fit for cloud deployment and auto-scaling;
+5. HBase has different API versions and has compatible issues (e.g, 0.98, 1.0, 1.1, 2.0);
+6. HBase has different vendor releases and has compatible issues (e.g, Cloudera's is not compatible with others);
+
+In view of the above problems, Kylin community proposed to replace HBase with Apache parquet + spark, for the following reasons:
+
+1. parquet is an open source and mature and stable column storage format;
+2. Parquet is more cloud-friendly, can work with most FS including HDFS, S3, Azure Blob store, Ali OSS, etc;
+3. parquet can be well integrated with Hadoop, hive, spark, impala, etc;
+4. parquet supports custom index.
+
+
+## How to store pre calculation results in kylin 4.0
+In kylin4.x, the pre calculation results are stored in the file system in parquet format. The file storage structure is very important for I/O optimization. If the storage directory structure is designed in advance, the data files can be filtered through the directory or file name during query to avoid unnecessary file scan.
+
+The directory structure of parquet file stored in the file system is as follows:
+- cube_name
+ - SegmentA
+ - Cuboid-111
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - Cuboid-222
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - SegmentB
+ - Cuboid-111
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - Cuboid-222
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+
+It can be seen that, using parquet storage can add and delete cuboid easily without affecting other data. With this feature, kylin4 realizes the feature of supporting users to add and delete cuboid manually. Please refer to: [how to update cuboid list for a cube]（ https://cwiki.apache.org/confluence/display/KYLIN/How+to+update+cuboid+list+for+a+cube )
+
+## Build engine of Kylin 4.0
+In kylin4, spark engine is the only build engine. Compared with the build engine in previous versions, it has the following characteristics:
+
+1. Building kylin4 simplifies many steps. For example, in cube build job, kylin4 only needs two steps: resource detection and cubing;
+2. Since parquet encodes the stored data, the process of dimension dictionary and dimension column encoding is no longer needed in kylin4;
+3. Kylin4 implements a new global dictionary. For more details, please refer to [kylin4 global dictionary](https://cwiki.apache.org/confluence/display/KYLIN/Global+Dictionary+on+Spark);
+4. Kylin4 will automatically adjust parameters of spark according to cluster resources and build job;
+5. Kylin4 can improve the build performance.
+
+Users can manually modify the build the relevant spark configuration through the configuration item beginning with `kylin.build.spark-conf`. The manually modified spark configuration item will no longer participate in automatic parameter adjustment.
+
+## Query engine of kylin 4.0
+
+`Sparder (spardercontext)`, the query engine of Kylin4, is a new distributed query engine implemented by the back end of spark application. Compared with the original query engine, sparder has the following advantages:
+1. Distributed query engine，avoid single-point-of-failure;
+2. Unified calculation engine for building and querying;
+3. There is a substantial increase in complex query performance;
+4. Can benefit from spark new features and ecology.
+
+In kylin4, sparder exists as a long-running spark application. Sparder will obtain the horn resource according to the spark parameter configured in the configuration item beginning with `kylin.query.spark-conf`. If the configured resource parameter is too large, the build engine may be affected, and even sparder cannot be started successfully. If sparder is not started successfully, all query tasks will fail, Users can check the sparder status in the system page of kylin webui.
+By default, the spark parameter used for query will be set smaller. In the production environment, you can increase these parameters appropriately to improve query performance.
+`Kylin.query.auto-sparder-context` parameter is used to control whether to start sparder when kylin is started. The default value is `false`, that is, sparder will be started only when the first SQL is executed by default. For this reason, it will take a long time to execute the first SQL.
+If you don't want the query speed of the first SQL to be lower than expected, you can set `kylin.query.auto-sparder-context` to `true`, and sparder will start with kylin.
+
+## Feature comparison between kylin 4.0 and kylin 3.1
+
+| Feature | Kylin 3.1.0 | Kylin 4.0 |
+| --------------------- | :------------------------------------------- | :----------------------------------------------|
+| Storage | HBase | Parquet |
+| BuildEngine | MapReduce/Spark/Flink | New Spark Engine |
+| Metastore | HBase(Default)/Mysql | Mysql(Default) |
+| DataSource | Kafka/Hive/JDBC | Hive/CSV |
+| Global Dictionary | Two implementation | New implementation |
+| Cube Optimization Tool | Cube Planner | Cube Planner phase1 and Optimize cube manually |
+| Self-monitoring | System cube and Dashboard | System cube and Dashboard |
+| PushDown Engine | Hive/JDBC | Spark SQL |
+| Hadoop platform | HDP2/HDP3/CDH5/CDH6/EMR5 | HDP2/CDH5/CDH6/EMR5/EMR6/HDI |
+| Deployment mode | Single node/Cluster/Read and write separation| Single node/Cluster/Read and write separation |
+
+## Kylin 4.0 performance
+In order to test the performance of kylin 4.0, we benchmark SSB dataset and TPC-H dataset respectively, and compare with kylin 3.1.0. The test environment is a 4-node CDH cluster, and the yarn queue is used to allocate 400G memory and 128 CPU cores.
+The results of performance test are as follows:
+
+- Comparison of build duration and result size（SSB）
+
+
+
+
+The test results can reflect the following two points:
+- The build speed of kylin4 is significantly higher than that of kylin3.1.0 spark engine;
+- Compared with HBase, the parquet file size of kylin4 is significantly reduced;
+
+- Comparison of query response(SSB and TPC-H)
+
+
+
+From the comparison of query results, it can be seen that kylin3 and kylin4 are the same for ***simple query***, kylin4 is slightly insufficient; However, kylin4 has obvious advantages over kylin3 for ***complex query***.
+Moreover, there is still a lot of room to optimize the performance of ***simple query*** in kylin4. In the practice of Youzan using kylin4, the performance of ***simple query*** can be optimized to less than 1 second.
+
+## How to upgrade
+Please check: [How to migrate metadata to Kylin4](https://cwiki.apache.org/confluence/display/KYLIN/How+to+migrate+metadata+to+Kylin+4)
+## Kylin 4.0 query and build tuning
+For kylin4 tuning, please refer to: [How to improve cube building and query performance](/docs/howto/howto_optimize_build_and_query.html)
+## Kylin 4.0 use case
+[Why did Youzan choose Kylin4](/blog/2021/06/17/Why-did-Youzan-choose-Kylin4)
+Reference link:
+[Kylin Improvement Proposal 1: Parquet Storage](https://cwiki.apache.org/confluence/display/KYLIN/KIP-1%3A+Parquet+storage)
diff --git a/website/_docs/install/advance_settings.cn.md b/website/_docs/install/advance_settings.cn.md
index 3305d8e..62a3524 100644
--- a/website/_docs/install/advance_settings.cn.md
+++ b/website/_docs/install/advance_settings.cn.md
@@ -10,67 +10,10 @@ permalink: /cn/docs/install/advance_settings.html
-两个示例：
-
- * `kylin.cube.algorithm`：定义了 job engine 选择的 Cubing 算法；默认值为 "auto"，意味着 engine 会通过采集数据动态的选择一个算法 ("layer" or "inmem")。如果您很了解 Kylin 和 您的数据/集群，您可以直接设置您喜欢的算法。
-
- * `kylin.storage.hbase.region-cut-gb`：定义了创建 HBase 表时一个 region 的大小。默认一个 region "5" (GB)。对于小的或中等大小的 cube 来说它的值可能太大了，所以您可以设置更小的值来获得更多的 regions，可获得更好的查询性能。
-
-## 在 Cube 级别重写默认的 Hadoop job conf 值
-`conf/kylin_job_conf.xml` 和 `conf/kylin_job_conf_inmem.xml` 管理 Hadoop jobs 的默认配置。如果您想通过 cube 自定义配置，您可以通过和上面相似的方式获得，但是需要加一个前缀 `kylin.engine.mr.config-override.`；当提交 jobs 这些配置会被解析并应用。下面是两个示例:
-
- * 希望 job 从 Yarn 获得更多 memory，您可以这样定义：`kylin.engine.mr.config-override.mapreduce.map.java.opts=-Xmx7g` 和 `kylin.engine.mr.config-override.mapreduce.map.memory.mb=8192`
- * 希望 cube's job 使用不同的 Yarn resource queue，您可以这样定义：`kylin.engine.mr.config-override.mapreduce.job.queuename=myQueue` ("myQueue" 是一个举例，可更换成您的 queue 名字)
-
-## 在 Cube 级别重写默认的 Hive job conf 值
-
-`conf/kylin_hive_conf.xml` 管理运行时 Hive job 的默认配置 (例如创建 flat hive table)。如果您想通过 cube 自定义配置，您可以通过和上面相似的方式获得，但需要另一个前缀 `kylin.source.hive.config-override.`；当运行 "hive -e" 或 "beeline" 命令，这些配置会被解析并应用。请看下面示例:
-
- * 希望 hive 使用不同的 Yarn resource queue，您可以这样定义：`kylin.source.hive.config-override.mapreduce.job.queuename=myQueue` ("myQueue" 是一个举例，可更换成您的 queue 名字)
-
## 在 Cube 级别重写默认的 Spark conf 值
Spark 的配置是在 `conf/kylin.properties` 中管理，前缀为 `kylin.engine.spark-conf.`。例如，如果您想要使用 job queue "myQueue" 运行 Spark，设置 "kylin.engine.spark-conf.spark.yarn.queue=myQueue" 会让 Spark 在提交应用时获取 "spark.yarn.queue=myQueue"。参数可以在 Cube 级别进行配置，将会覆盖 `conf/kylin.properties` 中的默认值。
-## 支持压缩
-
-默认情况，Kylin 不支持压缩，在产品环境这不是一个推荐的设置，但对于新的 Kylin 用户是个权衡。一个合适的算法将会减少存储负载。不支持的算法会阻碍 Kylin job build。Kylin 可以使用三种类型的压缩，HBase 表压缩，Hive 输出压缩 和 MR jobs 输出压缩。
-
-* HBase 表压缩
-压缩设置通过 `kylin.hbase.default.compression.codec` 定义在 `kyiln.properties` 中，默认值为 *none*。有效的值包括 *none*，*snappy*，*lzo*，*gzip* 和 *lz4*。在变换压缩算法前，请确保您的 Hbase 集群支持所选算法。尤其是 snappy，lzo 和 lz4，不是所有的 Hadoop 分布式都会包含。
-
-* Hive 输出压缩
-压缩设置定义在 `kylin_hive_conf.xml`。默认设置为 empty 其利用了 Hive 的默认配置。如果您重写配置，请在 `kylin_hive_conf.xml` 中添加 (或替换) 下列属性。以 snappy 压缩为例:
-{% highlight Groff markup %}
- <property>
- <name>mapreduce.map.output.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
- </property>
- <property>
- <name>mapreduce.output.fileoutputformat.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
- </property>
-{% endhighlight %}
-
-* MR jobs 输出压缩
-压缩设置定义在 `kylin_job_conf.xml` 和 `kylin_job_conf_inmem.xml`中。默认设置为 empty 其利用了 MR 的默认配置。如果您重写配置，请在 `kylin_job_conf.xml` 和 `kylin_job_conf_inmem.xml` 中添加 (或替换) 下列属性。以 snappy 压缩为例:
-{% highlight Groff markup %}
- <property>
- <name>mapreduce.map.output.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
- </property>
- <property>
- <name>mapreduce.output.fileoutputformat.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
- </property>
-{% endhighlight %}
-
-压缩设置只有在重启 Kylin 服务器实例后才会生效。
-
## 分配更多内存给 Kylin 实例
打开 `bin/setenv.sh`，这里有两个 `KYLIN_JVM_SETTINGS` 环境变量的样例设置；默认设置较小 (最大为 4GB)，您可以注释它然后取消下一行的注释来给其分配 16GB:
@@ -153,38 +96,3 @@ java -classpath kylin-server-base-\<version\>.jar:kylin-core-common-\<version\>.
{% endhighlight %}
*启动 Kylin
-
-
-## 使用 SparkSql 创建 Hive 中间表
-
-**注意：当第二次连接 thriftserver 进行构建时将会出现问题，详细信息请查看 [https://issues.apache.org/jira/browse/SPARK-21067](https://issues.apache.org/jira/browse/SPARK-21067)**
-
-Kylin 能够使用 SparkSql 创建 Hive 中间表；使其生效之前：
-
-- 确保以下参数存在于 hive-site.xml：
-
-{% highlight Groff markup %}
-
-<property>
- <name>hive.security.authorization.sqlstd.confwhitelist</name>
- <value>mapred.*|hive.*|mapreduce.*|spark.*</value>
-</property>
-
-<property>
- <name>hive.security.authorization.sqlstd.confwhitelist.append</name>
- <value>mapred.*|hive.*|mapreduce.*|spark.*</value>
-</property>
-
-{% endhighlight %}
-- 将 `hive.execution.engine` 改为 mr（可选），如果您想要使用 tez，请确保 tez 相关依赖已导入
-- 将 hive-site.xml 拷贝到 $SPARK_HOME/conf
-- 确保设置了 HADOOP_CONF_DIR 环境变量
-- 使用 `sbin/start-thriftserver.sh --master spark://sparkmasterip:sparkmasterport` 命令启动 thriftserver，通常端口为 7077
-- 编辑 `conf/kylin.properties`，设置如下参数：
-{% highlight Groff markup %}
-kylin.source.hive.enable-sparksql-for-table-ops=true
-kylin.source.hive.sparksql-beeline-shell=/path/to/spark-client/bin/beeline
-kylin.source.hive.sparksql-beeline-params=-n root -u 'jdbc:hive2://thriftserverip:thriftserverport'
-{% endhighlight %}
-
-重启 Kylin 令其生效。通过将 `kylin.source.hive.enable-sparksql-for-table-ops` 设置为 `false` 来令其失效
diff --git a/website/_docs/install/advance_settings.md b/website/_docs/install/advance_settings.md
index d3edc76..56d69b7 100644
--- a/website/_docs/install/advance_settings.md
+++ b/website/_docs/install/advance_settings.md
@@ -10,67 +10,10 @@ In `conf/kylin.properties` there are many parameters, which control/impact on Ky
-Here take two example:
-
- * `kylin.cube.algorithm`: it defines the Cubing algorithm that the job engine will select; Its default value is "auto", means the engine will dynamically pick an algorithm ("layer" or "inmem") by sampling the data. If you knows Kylin and your data/cluster well, you can set your preferred algorithm directly.
-
- * `kylin.storage.hbase.region-cut-gb`: it defines how big a region is when creating the HBase table. The default value is "5" (GB) per region. It might be too big for a small or medium cube, so you can give it a smaller value to get more regions created, then can gain better query performance.
-
-## Overwrite default Hadoop job conf at Cube level
-The `conf/kylin_job_conf.xml` and `conf/kylin_job_conf_inmem.xml` manage the default configurations for Hadoop jobs. If you have the need to customize the configs by cube, you can achieve that with the similar way as above, but need adding a prefix `kylin.engine.mr.config-override.`; These configs will be parsed out and then applied when submitting jobs. See two examples below:
-
- * If want a cube's job getting more memory from Yarn, you can define: `kylin.engine.mr.config-override.mapreduce.map.java.opts=-Xmx7g` and `kylin.engine.mr.config-override.mapreduce.map.memory.mb=8192`
- * If want a cube's job going to a different Yarn resource queue, you can define: `kylin.engine.mr.config-override.mapreduce.job.queuename=myQueue` ("myQueue" is just a sample, change to your queue name)
-
-## Overwrite default Hive job conf at Cube level
-
-The `conf/kylin_hive_conf.xml` manages the default configurations when running Hive job (like creating intermediate flat hive table). If you have the need to customize the configs by cube, you can achieve that with the similar way as above, but need using another prefix `kylin.source.hive.config-override.`; These configs will be parsed out and then applied when running "hive -e" or "beeline" commands. See example below:
-
- * If want hive goes to a different Yarn resource queue, you can define: `kylin.source.hive.config-override.mapreduce.job.queuename=myQueue` ("myQueue" is just a sample, change to your queue name)
-
## Overwrite default Spark conf at Cube level
The configurations for Spark are managed in `conf/kylin.properties` with prefix `kylin.engine.spark-conf.`. For example, if you want to use job queue "myQueue" to run Spark, setting "kylin.engine.spark-conf.spark.yarn.queue=myQueue" will let Spark get "spark.yarn.queue=myQueue" feeded when submitting applications. The parameters can be configured at Cube level, which will override the default values in `conf/kylin.properties`.
-## Enable compression
-
-By default, Kylin does not enable compression, this is not the recommend settings for production environment, but a tradeoff for new Kylin users. A suitable compression algorithm will reduce the storage overhead. But unsupported algorithm will break the Kylin job build also. There are three kinds of compression used in Kylin, HBase table compression, Hive output compression and MR jobs output compression.
-
-* HBase table compression
-The compression settings define in `kyiln.properties` by `kylin.hbase.default.compression.codec`, default value is *none*. The valid value includes *none*, *snappy*, *lzo*, *gzip* and *lz4*. Before changing the compression algorithm, please make sure the selected algorithm is supported on your HBase cluster. Especially for snappy, lzo and lz4, not all Hadoop distributions include these.
-
-* Hive output compression
-The compression settings define in `kylin_hive_conf.xml`. The default setting is empty which leverages the Hive default configuration. If you want to override the settings, please add (or replace) the following properties into `kylin_hive_conf.xml`. Take the snappy compression for example:
-{% highlight Groff markup %}
- <property>
- <name>mapreduce.map.output.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
- </property>
- <property>
- <name>mapreduce.output.fileoutputformat.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
- </property>
-{% endhighlight %}
-
-* MR jobs output compression
-The compression settings define in `kylin_job_conf.xml` and `kylin_job_conf_inmem.xml`. The default setting is empty which leverages the MR default configuration. If you want to override the settings, please add (or replace) the following properties into `kylin_job_conf.xml` and `kylin_job_conf_inmem.xml`. Take the snappy compression for example:
-{% highlight Groff markup %}
- <property>
- <name>mapreduce.map.output.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
- </property>
- <property>
- <name>mapreduce.output.fileoutputformat.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
- </property>
-{% endhighlight %}
-
-Compression settings only take effect after restarting Kylin server instance.
-
## Allocate more memory to Kylin instance
Open `bin/setenv.sh`, which has two sample settings for `KYLIN_JVM_SETTINGS` environment variable; The default setting is small (4GB at max.), you can comment it and then un-comment the next line to allocate 16GB:
@@ -149,41 +92,4 @@ cd $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/lib
java -classpath kylin-server-base-\<version\>.jar:kylin-core-common-\<version\>.jar:spring-beans-4.3.10.RELEASE.jar:spring-core-4.3.10.RELEASE.jar:commons-codec-1.7.jar org.apache.kylin.rest.security.PasswordPlaceholderConfigurer AES <your_password>
{% endhighlight %}
-* Start Kylin
-
-
-## Use SparkSql to create intermediate flat Hive table
-
-**Note: There will be an issue when connecting thriftserver again, detail info, please check [https://issues.apache.org/jira/browse/SPARK-21067](https://issues.apache.org/jira/browse/SPARK-21067)**
-
-Kylin can use SparkSql to create intermediate flat Hive table; Before enable this:
-
-- Make sure the following parameters exist in hive-site.xml:
-
-{% highlight Groff markup %}
-
-<property>
- <name>hive.security.authorization.sqlstd.confwhitelist</name>
- <value>mapred.*|hive.*|mapreduce.*|spark.*</value>
-</property>
-
-<property>
- <name>hive.security.authorization.sqlstd.confwhitelist.append</name>
- <value>mapred.*|hive.*|mapreduce.*|spark.*</value>
-</property>
-
-{% endhighlight %}
-- Change `hive.execution.engine` to mr (Optional), if you want to use tez, please make sure the dependency of tez has been added
-- Copy the hive-site.xml to $SPARK_HOME/conf
-- Make sure the environmental variable HADOOP_CONF_DIR has been set
-- Use `sbin/start-thriftserver.sh --master spark://sparkmasterip:sparkmasterport` to start the thriftserver, usually port is 7077
-- Edit `conf/kylin.properties`, set the following parameters:
-{% highlight Groff markup %}
-kylin.source.hive.enable-sparksql-for-table-ops=true
-kylin.source.hive.sparksql-beeline-shell=/path/to/spark-client/bin/beeline
-kylin.source.hive.sparksql-beeline-params=-n root -u 'jdbc:hive2://thriftserverip:thriftserverport'
-{% endhighlight %}
-
-Restart Kylin server to take effective. To disable, set `kylin.source.hive.enable-sparksql-for-table-ops` back to `false`.
-
-
+* Start Kylin
\ No newline at end of file
diff --git a/website/_docs/install/configuration.cn.md b/website/_docs/install/configuration.cn.md
index 6da26d6..630363f 100644
--- a/website/_docs/install/configuration.cn.md
+++ b/website/_docs/install/configuration.cn.md
@@ -6,835 +6,5 @@ permalink: /cn/docs/install/configuration.html
---
-
-- [配置文件及参数重写](#kylin-config)
- - [Kylin 配置文件](#kylin-config)
- - [配置重写](#config-override)
- - [项目级别配置重写](#project-config-override)
- - [Cube 级别配置重写](#cube-config-override)
- - [重写 MapReduce 参数](#mr-config-override)
- - [重写 Hive 参数](#hive-config-override)
- - [重写 Spark 参数](#spark-config-override)
-- [部署配置](#kylin-deploy)
- - [部署 Kylin](#deploy-config)
- - [分配更多内存给 Kylin 实例](#kylin-jvm-settings)
- - [任务引擎高可用](#job-engine-ha)
- - [任务引擎安全模式](#job-engine-safemode)
- - [读写分离配置](#rw-deploy)
- - [RESTful Webservice](#rest-config)
-- [Metastore 配置](#kylin_metastore)
- - [元数据相关](#metadata)
- - [基于 MySQL 的 Metastore](#mysql-metastore)
-- [构建配置](#kylin-build)
- - [Hive 客户端 & SparkSQL](#hive-client-and-sparksql)
- - [配置 JDBC 数据源](#jdbc-datasource)
- - [数据类型精度](#precision-config)
- - [Cube 设计](#cube-config)
- - [Cube 大小估计](#cube-estimate)
- - [Cube 构建算法](#cube-algorithm)
- - [自动合并](#auto-merge)
- - [维表快照](#snapshot)
- - [Cube 构建](#cube-build)
- - [字典相关](#dict-config)
- - [超高基维度的处理](#uhc-config)
- - [Spark 构建引擎](#spark-cubing)
- - [通过 Livy 提交 Spark 任务](#livy-submit-spark-job)
- - [Spark 资源动态分配](#dynamic-allocation)
- - [任务相关](#job-config)
- - [启用邮件通知](#email-notification)
- - [启用 Cube Planner](#cube-planner)
- - [HBase 存储](#hbase-config)
- - [备用 Hbase 存储](#secondary-hbase)
- - [任务输出](#job-output)
- - [启用压缩](#compress-config)
- - [实时 OLAP](#realtime-olap)
-- [清理存储配置](#storage-clean-up-configuration)
- - [存储清理相关](#storage-clean-up-config)
-- [查询配置](#kylin-query)
- - [查询相关](#query-config)
- - [模糊查询](#fuzzy)
- - [查询缓存](#cache-config)
- - [查询限制](#query-limit)
- - [坏查询](#bad-query)
- - [查询下压](#query-pushdown)
- - [查询改写](#convert-sql)
- - [收集查询指标到 JMX](#jmx-metrics)
- - [收集查询指标到 dropwizard](#dropwizard-metrics)
-- [安全配置](#kylin-security)
- - [用户名和密码](#user-name-and-password)
- - [集成 LDAP 实现单点登录](#ldap-sso)
- - [集成 Apache Ranger](#ranger)
- - [启用 ZooKeeper ACL](#zookeeper-acl)
-- [启用 Memcached 做分布式查询缓存](#distributed-cache)
-
-
-### 配置文件及参数重写 {#kylin-config}
-
-本小节介绍 Kylin 的配置文件和如何进行配置重写。
-
-
-
-### Kylin 配置文件 {#kylin-config-file}
-
-Kylin 会自动从环境中读取 Hadoop 配置（`core-site.xml`），Hive 配置（`hive-site.xml`）和 HBase 配置（`hbase-site.xml`），另外，Kylin 的配置文件在 `$KYLIN_HOME/conf/` 目录下，如下：
-
-- `kylin_hive_conf.xml`：该文件包含了 Hive 任务的配置项。
-- `kylin_job_conf.xml` & `kylin_job_conf_inmem.xml`：该文件包含了 MapReduce 任务的配置项。当执行 **In-mem Cubing** 任务时，需要在 `kylin_job_conf_inmem.xml` 中为 mapper 申请更多的内存
-- `kylin-kafka-consumer.xml`：该文件包含了 Kafka 任务的配置项。
-- `kylin-server-log4j.properties`：该文件包含了 Kylin 服务器的日志配置项。
-- `kylin-tools-log4j.properties`：该文件包含了 Kylin 命令行的日志配置项。
-- `setenv.sh` ：该文件是用于设置环境变量的 shell 脚本，可以通过 `KYLIN_JVM_SETTINGS` 调整 Kylin JVM 栈的大小，且可以设置 `KAFKA_HOME` 等其他环境变量。
-- `kylin.properties`：该文件是 Kylin 使用的全局配置文件。
-
-
-
-### 配置重写 {#config-override}
-
-`$KYLIN_HOME/conf/` 中有部分配置项可以在 Web UI 界面进行重写，配置重写分为**项目级别配置重写**和 **Cube 级别配置重写**。配置重写的优先级关系为：Cube 级别配置重写 > 项目级别配置重写 > 全局配置文件。
-
-
-
-### 项目级别配置重写 {#project-config-override}
-
-在 Web UI 界面点击 **Manage Project** ，选中某个项目，点击 **Edit** -> **Project Config** -> **+ Property**，进行项目级别的配置重写，如下图所示：
-
-
-
-
-### Cube 级别配置重写 {#cube-config-override}
-
-在设计 Cube （**Cube Designer**）的 **Configuration Overwrites** 步骤可以添加配置项，进行 Cube 级别的配置重写，如下图所示：
-
-
-以下参数可以在 Cube 级别重写：
-
-- `kylin.cube.size-estimate*`
-- `kylin.cube.algorithm*`
-- `kylin.cube.aggrgroup*`
-- `kylin.metadata.dimension-encoding-max-length`
-- `kylin.cube.max-building-segments`
-- `kylin.cube.is-automerge-enabled`
-- `kylin.job.allow-empty-segment`
-- `kylin.job.sampling-percentage`
-- `kylin.source.hive.redistribute-flat-table`
-- `kylin.engine.spark*`
-- `kylin.query.skip-empty-segments`
-
-
-
-### MapReduce 任务配置重写 {#mr-config-override}
-
-Kylin 支持在项目和 Cube 级别重写 `kylin_job_conf.xml` 和 `kylin_job_conf_inmem.xml` 中参数，以键值对的性质，按照如下格式替换：
-`kylin.engine.mr.config-override.<key> = <value>`
- * 如果用户希望任务从 Yarn 获得更多内存，可以这样设置：
- `kylin.engine.mr.config-override.mapreduce.map.java.opts=-Xmx7g` 和 `kylin.engine.mr.config-override.mapreduce.map.memory.mb=8192`
- * 如果用户希望 Cube 的构建任务使用不同的 YARN resource queue，可以设置：
- `kylin.engine.mr.config-override.mapreduce.job.queuename={queueName}`
-
-
-### Hive 任务配置重写 {#hive-config-override}
-
-Kylin 支持在项目和 Cube 级别重写 `kylin_hive_conf.xml` 中参数，以键值对的性质，按照如下格式替换：
-`kylin.source.hive.config-override.<key> = <value>`
-如果用户希望 Hive 使用不同的 YARN resource queue，可以设置：
-`kylin.source.hive.config-override.mapreduce.job.queuename={queueName}`
-
-
-
-### Spark 任务配置重写 {#spark-config-override}
-
-Kylin 支持在项目和 Cube 级别重写 `kylin.properties` 中的 Spark 参数，以键值对的性质，按照如下格式替换：
-`kylin.engine.spark-conf.<key> = <value>`
-如果用户希望 Spark 使用不同的 YARN resource queue，可以设置：
-`kylin.engine.spark-conf.spark.yarn.queue={queueName}`
-
-
-
-### 部署配置 {#kylin-deploy}
-
-本小节介绍部署 Kylin 相关的配置。
-
-
-
-### 部署 Kylin {#deploy-config}
-
-- `kylin.env.hdfs-working-dir`：指定 Kylin 服务所用的 HDFS 路径，默认值为 `/kylin`，请确保启动 Kylin 实例的用户有读写该目录的权限
-- `kylin.env`：指定 Kylin 部署的用途，参数值可选 `DEV`，`QA`， `PROD`，默认值为 `DEV`，在 DEV 模式下一些开发者功能将被启用
-- `kylin.env.zookeeper-base-path`：指定 Kylin 服务所用的 ZooKeeper 路径，默认值为 `/kylin`
-- `kylin.env.zookeeper-connect-string`：指定 ZooKeeper 连接字符串，如果为空，使用 HBase 的 ZooKeeper
-- `kylin.env.hadoop-conf-dir`：指定 Hadoop 配置文件目录，如果不指定的话，获取环境中的 `HADOOP_CONF_DIR`
-- `kylin.server.mode`：指定 Kylin 实例的运行模式，参数值可选 `all`， `job`， `query`，默认值为 `all`，job 模式代表该服务仅用于任务调度，不用于查询；query 模式代表该服务仅用于查询，不用于构建任务的调度；all 模式代表该服务同时用于任务调度和 SQL 查询。
-- `kylin.server.cluster-name`：指定集群名称
-
-
-
-### 分配更多内存给 Kylin 实例 {#kylin-jvm-settings}
-
-在 `$KYLIN_HOME/conf/setenv.sh` 中存在对 `KYLIN_JVM_SETTINGS` 的两种示例配置。
-默认配置使用的内存较少，用户可以根据自己的实际情况，注释掉默认配置并取消另一配置前的注释符号以启用另一配置，从而为 Kylin 实例分配更多的内存资源，该项配置的默认值如下：
-
-```shell
-export KYLIN_JVM_SETTINGS="-Xms1024M -Xmx4096M -Xss1024K -XX`MaxPermSize=512M -verbose`gc -XX`+PrintGCDetails -XX`+PrintGCDateStamps -Xloggc`$KYLIN_HOME/logs/kylin.gc.$$ -XX`+UseGCLogFileRotation -XX`NumberOfGCLogFiles=10 -XX`GCLogFileSize=64M"
-# export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX`MaxPermSize=512m -XX`NewSize=3g -XX`MaxNewSize=3g -XX`SurvivorRatio=4 -XX`+CMSClassUnloadingEnabled -XX`+CMSParallelRemarkEnabled -XX`+UseConcMarkSweepGC -XX`+CMSIncrementalMode -XX`CMSInitiatingOccupancyFraction=70 -XX`+UseCMSInitiatingOccupancyOnly -XX`+DisableExplicitGC -XX`+HeapDumpOnOutOfMemoryError -verbose`gc -XX`+PrintGCDetails -XX`+PrintGCDateStamps -Xloggc`$KYLIN_HOME/logs/kylin.gc.$$ -XX`+UseGCLogFileRotation -XX`NumberOfGCLogFi [...]
-```
-
-
-
-### 任务引擎高可用 {#job-engine-ha}
-
-- `kylin.job.scheduler.default=2`：启用分布式任务调度器
-- `kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperJobLock`：开启分布式任务锁
-
-> 提示：更多信息请参考 [集群模式部署](/cn/docs/install/kylin_cluster.html) 中的**任务引擎高可用**部分。
-
-
-### 任务引擎安全模式 {#job-engine-safemode}
-
-安全模式仅在默认调度器中生效
-
-- `kylin.job.scheduler.safemode=TRUE`: 启用安全模式，新提交的任务不会被执行。
-- `kylin.job.scheduler.safemode.runable-projects=project1,project2`: 安全模式下仍然可以执行的项目列表，支持设置多个。
-
-### 读写分离配置 {#rw-deploy}
-
-- `kylin.storage.hbase.cluster-fs`：指明 HBase 集群的 HDFS 文件系统
-- `kylin.storage.hbase.cluster-hdfs-config-file`：指向 HBase 集群的 HDFS 配置文件
-
-> 提示：更多信息请参考 [Deploy Apache Kylin with Standalone HBase Cluster](http://kylin.apache.org/blog/2016/06/10/standalone-hbase-cluster/)
-
-
-
-### RESTful Webservice {#rest-config}
-
-- `kylin.web.timezone`：指定 Kylin 的 REST 服务所使用的时区，默认值为 GMT+8
-- `kylin.web.cross-domain-enabled`：是否支持跨域访问，默认值为 TRUE
-- `kylin.web.export-allow-admin`：是否支持管理员用户导出信息，默认值为 TRUE
-- `kylin.web.export-allow-other`：是否支持其他用户导出信息，默认值为 TRUE
-- `kylin.web.dashboard-enabled`：是否启用 Dashboard，默认值为 FALSE
-
-
-
-### Metastore 配置 {#kylin_metastore}
-
-本小节介绍 Kylin Metastore 相关的配置。
-
-
-
-### 元数据相关 {#metadata}
-
-- `kylin.metadata.url`：指定元数据库路径，默认值为 kylin_metadata@hbase
-- `kylin.metadata.sync-retries`：指定元数据同步重试次数，默认值为 3
-- `kylin.metadata.sync-error-handler`：默认值为 `DefaultSyncErrorHandler`
-- `kylin.metadata.check-copy-on-write`：清除元数据缓存，默认值为 `FALSE`
-- `kylin.metadata.hbase-client-scanner-timeout-period`：表示 HBase 客户端发起一次 scan 操作的 RPC 调用至得到响应之间总的超时时间，默认值为 10000(ms)
-- `kylin.metadata.hbase-rpc-timeout`：指定 HBase 执行 RPC 操作的超时时间，默认值为 5000(ms)
-- `kylin.metadata.hbase-client-retries-number`：指定 HBase 重试次数，默认值为 1（次）
-- `kylin.metadata.resource-store-provider.jdbc`：指定 JDBC 使用的类，默认值为 `org.apache.kylin.common.persistence.JDBCResourceStore`
-
-
-
-### 基于 MySQL 的 Metastore {#mysql-metastore}
-
-- `kylin.metadata.url`：指定元数据路径
-- `kylin.metadata.jdbc.dialect`：指定 JDBC 方言
-- `kylin.metadata.jdbc.json-always-small-cell`：默认值为 TRUE
-- `kylin.metadata.jdbc.small-cell-meta-size-warning-threshold`：默认值为 100(MB)
-- `kylin.metadata.jdbc.small-cell-meta-size-error-threshold`：默认值为 1(GB)
-- `kylin.metadata.jdbc.max-cell-size`：默认值为 1(MB)
-- `kylin.metadata.resource-store-provider.jdbc`：指定 JDBC 使用的类，默认值为 org.apache.kylin.common.persistence.JDBCResourceStore
-
-> 提示：更多信息请参考[基于 MySQL 的 Metastore 配置](/cn/docs/tutorial/mysql_metastore.html)
-
-
-
-### 构建配置 {#kylin-build}
-
-本小节介绍 Kylin 数据建模及构建相关的配置。
-
-
-
-### Hive 客户端 & SparkSQL {#hive-client-and-sparksql}
-
-- `kylin.source.hive.client`：指定 Hive 命令行类型，参数值可选 cli 或 beeline，默认值为 cli
-- `kylin.source.hive.beeline-shell`：指定 Beeline shell 的绝对路径，默认值为 beeline
-- `kylin.source.hive.beeline-params`：当使用 Beeline 做为 Hive 的 Client 工具时，需要配置此参数，以提供更多信息给 Beeline
-- `kylin.source.hive.enable-sparksql-for-table-ops`：默认值为 FALSE，当使用 SparkSQL 时需要设置为 TRUE
-- `kylin.source.hive.sparksql-beeline-shell`：当使用 SparkSQL Beeline 做为 Hive 的 Client 工具时，需要配置此参数为 /path/to/spark-client/bin/beeline
-- `kylin.source.hive.sparksql-beeline-params`：当使用 SparkSQL Beeline 做为 Hive 的 Client 工具时，需要配置此参数
-
-
-
-
-### 配置 JDBC 数据源 {#jdbc-datasource}
-
-- `kylin.source.default`：JDBC 使用的数据源种类
-- `kylin.source.jdbc.connection-url`：JDBC 连接字符串
-- `kylin.source.jdbc.driver`：JDBC 驱动类名
-- `kylin.source.jdbc.dialect`：JDBC方言，默认值为 default
-- `kylin.source.jdbc.user`：JDBC 连接用户名
-- `kylin.source.jdbc.pass`：JDBC 连接密码
-- `kylin.source.jdbc.sqoop-home`：Sqoop 安装路径
-- `kylin.source.jdbc.sqoop-mapper-num`：指定应该分为多少个切片，Sqoop 将为每一个切片运行一个 mapper，默认值为 4
-- `kylin.source.jdbc.field-delimiter`：指定字段分隔符， 默认值为 \
-
-> 提示：更多信息请参考[建立 JDBC 数据源](/cn/docs/tutorial/setup_jdbc_datasource.html)。
-
-
-
-
-### 数据类型精度 {#precision-config}
-
-- `kylin.source.hive.default-varchar-precision`：指定 varchar 字段的最大长度，默认值为256
-- `kylin.source.hive.default-char-precision`：指定 char 字段的最大长度，默认值为 255
-- `kylin.source.hive.default-decimal-precision`：指定 decimal 字段的精度，默认值为 19
-- `kylin.source.hive.default-decimal-scale`：指定 decimal 字段的范围，默认值为 4
-
-
-
-### Cube 设计 {#cube-config}
-
-- `kylin.cube.ignore-signature-inconsistency`：Cube desc 中的 signature 信息能保证 Cube 不被更改为损坏状态，默认值为 FALSE
-- `kylin.cube.aggrgroup.max-combination`：指定一个 Cube 的聚合组 Cuboid 上限，默认值为 32768
-- `kylin.cube.aggrgroup.is-mandatory-only-valid`：是否允许 Cube 只包含 Base Cuboid，默认值为 FALSE，当使用 Spark Cubing 时需设置为 TRUE
-- `kylin.cube.rowkey.max-size`：指定可以设置为 Rowkeys 的最大列数，默认值为 63，且最大不能超过 63
-- `kylin.cube.allow-appear-in-multiple-projects`：是否允许一个 Cube 出现在多个项目中
-- `kylin.cube.gtscanrequest-serialization-level`：默认值为 1
-- `kylin.metadata.dimension-encoding-max-length`：指定维度作为 Rowkeys 时使用 fix_length 编码时的最大长度，默认值为 256
-- `kylin.web.hide-measures`: 隐藏一些可能不需要的度量，默认值是RAW
-
-
-
-
-### Cube 大小估计 {#cube-estimate}
-
-Kylin 和 HBase 都在写入磁盘时使用压缩，因此，Kylin 将在其原来的大小上乘以比率来估计 Cube 大小。
-
-- `kylin.cube.size-estimate-ratio`：普通的 Cube，默认值为 0.25
-- `kylin.cube.size-estimate-memhungry-ratio`：已废弃，默认值为 0.05
-- `kylin.cube.size-estimate-countdistinct-ratio`：包含精确去重度量的 Cube 大小估计，默认值为 0.5
-- `kylin.cube.size-estimate-topn-ratio`：包含 TopN 度量的 Cube 大小估计，默认值为 0.5
-
-
-
-
-### Cube 构建算法 {#cube-algorithm}
-
-- `kylin.cube.algorithm`：指定 Cube 构建的算法，参数值可选 `auto`，`layer` 和 `inmem`， 默认值为 auto，即 Kylin 会通过采集数据动态地选择一个算法 (layer or inmem)，如果用户很了解 Kylin 和自身的数据、集群，可以直接设置喜欢的算法
-- `kylin.cube.algorithm.layer-or-inmem-threshold`：默认值为 7
-- `kylin.cube.algorithm.inmem-split-limit`：默认值为 500
-- `kylin.cube.algorithm.inmem-concurrent-threads`：默认值为 1
-- `kylin.job.sampling-percentage`：指定数据采样百分比，默认值为 100
-
-
-
-### 自动合并 {#auto-merge}
-
-- `kylin.cube.is-automerge-enabled`：是否启用自动合并，默认值为 TRUE，将该参数设置为 FALSE 时，自动合并功能会被关闭，即使 Cube 设置中开启了自动合并、设置了自动合并阈值，也不会触发合并任务。
-
-
-
-### 维表快照 {#snapshot}
-
-- `kylin.snapshot.max-mb`：允许维表的快照大小的上限，默认值为 300(M)
-- `kylin.snapshot.max-cache-entry`：缓存中最多可以存储的 snapshot 数量，默认值为 500
-- `kylin.snapshot.ext.shard-mb`：设置存储维表快照的 HBase 分片大小，默认值为 500(M)
-- `kylin.snapshot.ext.local.cache.path`：本地缓存路径，默认值为 lookup_cache
-- `kylin.snapshot.ext.local.cache.max-size-gb`：本地维表快照缓存大小，默认值为 200(M)
-
-
-
-### Cube 构建 {#cube-build}
-
-- `kylin.storage.default`：指定默认的构建引擎，默认值为 2，即 HBase
-- `kylin.source.hive.keep-flat-table`：是否在构建完成后保留 Hive 中间表，默认值为 FALSE
-- `kylin.source.hive.database-for-flat-table`：指定存放 Hive 中间表的 Hive 数据库名字，默认值为 default，请确保启动 Kylin 实例的用户有操作该数据库的权限
-- `kylin.source.hive.flat-table-storage-format`：指定 Hive 中间表的存储格式，默认值为 SEQUENCEFILE
-- `kylin.source.hive.flat-table-field-delimiter`：指定 Hive 中间表的分隔符，默认值为 \u001F
-- `kylin.source.hive.intermediate-table-prefix`：指定 Hive 中间表的表名前缀，默认值为 kylin\_intermediate\_
-- `kylin.source.hive.redistribute-flat-table`：是否重分配 Hive 平表，默认值为 TRUE
-- `kylin.source.hive.redistribute-column-count`：重分配列的数量，默认值为 3
-- `kylin.source.hive.table-dir-create-first`：默认值为 FALSE
-- `kylin.storage.partition.aggr-spill-enabled`：默认值为 TRUE
-- `kylin.engine.mr.lib-dir`：指定 MapReduce 任务所使用的 jar 包的路径
-- `kylin.engine.mr.reduce-input-mb`：MapReduce 任务启动前会依据输入预估 Reducer 接收数据的总量，再除以该参数得出 Reducer 的数目，默认值为 500（MB）
-- `kylin.engine.mr.reduce-count-ratio`：用于估算 Reducer 数目，默认值为 1.0
-- `kylin.engine.mr.min-reducer-number`：MapReduce 任务中 Reducer 数目的最小值，默认值为 1
-- `kylin.engine.mr.max-reducer-number`：MapReduce 任务中 Reducer 数目的最大值，默认值为 500
-- `kylin.engine.mr.mapper-input-rows`：每个 Mapper 可以处理的行数，默认值为 1000000，如果将这个值调小，会起更多的 Mapper
-- `kylin.engine.mr.max-cuboid-stats-calculator-number`：用于计算 Cube 统计数据的线程数量，默认值为 1
-- `kylin.engine.mr.build-dict-in-reducer`：是否在构建任务 **Extract Fact Table Distinct Columns** 的 Reduce 阶段构建字典，默认值为 TRUE
-- `kylin.engine.mr.yarn-check-interval-seconds`：构建引擎间隔多久检查 Hadoop 任务的状态，默认值为 10（s）
-- `kylin.engine.mr.use-local-classpath`: 是否使用本地 mapreduce 应用的 classpath。默认值为 TRUE
-
-
-
-### 字典相关 {#dict-config}
-
-- `kylin.dictionary.use-forest-trie`：默认值为 TRUE
-- `kylin.dictionary.forest-trie-max-mb`：默认值为 500
-- `kylin.dictionary.max-cache-entry`：默认值为 3000
-- `kylin.dictionary.growing-enabled`：默认值为 FALSE
-- `kylin.dictionary.append-entry-size`：默认值为 10000000
-- `kylin.dictionary.append-max-versions`：默认值为 3
-- `kylin.dictionary.append-version-ttl`：默认值为 259200000
-- `kylin.dictionary.resuable`：是否重用字典，默认值为 FALSE
-- `kylin.dictionary.shrunken-from-global-enabled`：是否缩小全局字典，默认值为 TRUE
-
-
-
-### 超高基维度的处理 {#uhc-config}
-
-Cube 构建默认在 **Extract Fact Table Distinct Column** 这一步为每一列分配一个 Reducer，对于超高基维度，可以通过以下参数增加 Reducer 个数
-
-- `kylin.engine.mr.build-uhc-dict-in-additional-step`：默认值为 FALSE，设置为 TRUE
-- `kylin.engine.mr.uhc-reducer-count`：默认值为 1，可以设置为 5，即为每个超高基的列分配 5 个 Reducer。
-
-
-
-### Spark 构建引擎 {#spark-cubing}
-
-- `kylin.engine.spark-conf.spark.master`：指定 Spark 运行模式，默认值为 yarn
-- `kylin.engine.spark-conf.spark.submit.deployMode`：指定 Spark on YARN 的部署模式，默认值为 cluster
-- `kylin.engine.spark-conf.spark.yarn.queue`：指定 Spark 资源队列，默认值为 default
-- `kylin.engine.spark-conf.spark.driver.memory`：指定 Spark Driver 内存大小，默认值为 2G
-- `kylin.engine.spark-conf.spark.executor.memory`：指定 Spark Executor 内存大小，默认值为 4G
-- `kylin.engine.spark-conf.spark.yarn.executor.memoryOverhead`：指定 Spark Executor 堆外内存大小，默认值为 1024(MB)
-- `kylin.engine.spark-conf.spark.executor.cores`：指定单个 Spark Executor可用核心数，默认值为 1
-- `kylin.engine.spark-conf.spark.network.timeout`：指定 Spark 网络超时时间，600
-- `kylin.engine.spark-conf.spark.executor.instances`：指定一个 Application 拥有的 Spark Executor 数量，默认值为 1
-- `kylin.engine.spark-conf.spark.eventLog.enabled`：是否记录 Spark 时间，默认值为 TRUE
-- `kylin.engine.spark-conf.spark.hadoop.dfs.replication`：HDFS 的副本数，默认值为 2
-- `kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress`：是否压缩输出，默认值为 TRUE
-- `kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress.codec`：输出所用压缩，默认值为 org.apache.hadoop.io.compress.DefaultCodec
-- `kylin.engine.spark.rdd-partition-cut-mb`：Kylin 用该参数的大小来分割 partition，默认值为 10(MB)，可以在 Cube 级别重写这个参数，调整至更大，来减少分区数
-- `kylin.engine.spark.min-partition`：最小分区数，默认值为 1
-- `kylin.engine.spark.max-partition`：最大分区数，默认值为 5000
-- `kylin.engine.spark.storage-level`：RDD 分区数据缓存级别，默认值为 MEMORY_AND_DISK_SER
-- `kylin.engine.spark-conf-mergedict.spark.executor.memory`：为合并字典申请更多的内存，默认值为 6G
-- `kylin.engine.spark-conf-mergedict.spark.memory.fraction`：给系统预留的内存百分比，默认值为 0.2
-
-> 提示：更多信息请参考 [用 Spark 构建 Cube](/cn/docs/tutorial/cube_spark.html)。
-
-
-
-### 通过 Livy 提交 Spark 任务 {#livy-submit-spark-job}
-
-- `kylin.engine.livy-conf.livy-enabled`：是否开启 Livy 进行 Spark 任务的提交。默认值为 *FALSE*
-- `kylin.engine.livy-conf.livy-url`：指定了 Livy 的 URL。例如 *http://127.0.0.1:8998*
-- `kylin.engine.livy-conf.livy-key.*`：指定了 Livy 的 name-key 配置。例如 *kylin.engine.livy-conf.livy-key.name=kylin-livy-1*
-- `kylin.engine.livy-conf.livy-arr.*`：指定了 Livy 数组类型的配置。以逗号分隔。例如 *kylin.engine.livy-conf.livy-arr.jars=hdfs://your_self_path/hbase-common-1.4.8.jar,hdfs://your_self_path/hbase-server-1.4.8.jar,hdfs://your_self_path/hbase-client-1.4.8.jar*
-- `kylin.engine.livy-conf.livy-map.*`：指定了 Spark 配置。例如 *kylin.engine.livy-conf.livy-map.spark.executor.instances=10*
-
-> 提示：更多信息请参考 [Apache Livy Rest API](http://livy.incubator.apache.org/docs/latest/rest-api.html)。
-
-
-### Spark 资源动态分配 {#dynamic-allocation}
-
-- `kylin.engine.spark-conf.spark.shuffle.service.enabled`：是否开启 shuffle service
-- `kylin.engine.spark-conf.spark.dynamicAllocation.enabled`：是否启用 Spark 资源动态分配
-- `kylin.engine.spark-conf.spark.dynamicAllocation.initialExecutors`：如果所有的 Executor 都移除了，重新请求启动时初始 Executor 数量
-- `kylin.engine.spark-conf.spark.dynamicAllocation.minExecutors`：最少保留的 Executor 数量
-- `kylin.engine.spark-conf.spark.dynamicAllocation.maxExecutors`：最多申请的 Executor 数量
-- `kylin.engine.spark-conf.spark.dynamicAllocation.executorIdleTimeout`：Executor 空闲时间超过设置的值后，除非有缓存数据，不然会被移除，默认值为 60(s)
-
-> 提示：更多信息请参考 [Dynamic Resource Allocation](http://spark.apache.org/docs/1.6.2/job-scheduling.html#dynamic-resource-allocation)。
-
-
-
-### 任务相关 {#job-config}
-
-- `kylin.job.log-dir`：默认值为 /tmp/kylin/logs
-- `kylin.job.allow-empty-segment`：是否容忍数据源为空，默认值为 TRUE
-- `kylin.job.max-concurrent-jobs`：最大构建并发数，默认值为 10
-- `kylin.job.retry`：构建任务失败后的重试次数，默认值为 0
-- `kylin.job.retry-interval`: 每次重试的间隔毫秒数。默认值为 30000
-- `kylin.job.scheduler.priority-considered`：是否考虑任务优先级，默认值为 FALSE
-- `kylin.job.scheduler.priority-bar-fetch-from-queue`：指定从优先级队列中获取任务的时间间隔，默认值为 20(s)
-- `kylin.job.scheduler.poll-interval-second`：从队列中获取任务的时间间隔，默认值为 30(s)
-- `kylin.job.error-record-threshold`：指定任务抛出错误信息的阈值，默认值为 0
-- `kylin.job.cube-auto-ready-enabled`：是否在构建完成后自动启用 Cube，默认值为 TRUE
-- `kylin.cube.max-building-segments`：指定对同一个 Cube 的最大构建数量，默认值为 10
-
-
-
-### 启用邮件通知 {#email-notification}
-
-- `kylin.job.notification-enabled`：是否在任务成功或者失败时进行邮件通知，默认值为 FALSE
-- `kylin.job.notification-mail-enable-starttls`：# 是否启用 starttls，默认值为 FALSE
-- `kylin.job.notification-mail-host`：指定邮件的 SMTP 服务器地址
-- `kylin.job.notification-mail-port`：指定邮件的 SMTP 服务器端口，默认值为 25
-- `kylin.job.notification-mail-username`：指定邮件的登录用户名
-- `kylin.job.notification-mail-password`：指定邮件的用户名密码
-- `kylin.job.notification-mail-sender`：指定邮件的发送邮箱地址
-- `kylin.job.notification-admin-emails`：指定邮件通知的管理员邮箱
-
-
-
-### 启用 Cube Planner {#cube-planner}
-
-- `kylin.cube.cubeplanner.enabled`：默认值为 TRUE
-- `kylin.server.query-metrics2-enabled`：默认值为 TRUE
-- `kylin.metrics.reporter-query-enabled`：默认值为 TRUE
-- `kylin.metrics.reporter-job-enabled`：默认值为 TRUE
-- `kylin.metrics.monitor-enabled`：默认值为 TRUE
-- `kylin.cube.cubeplanner.enabled`：是否启用 Cube Planner，默认值为 TRUE
-- `kylin.cube.cubeplanner.enabled-for-existing-cube`：是否对已有的 Cube 启用 Cube Planner，默认值为 TRUE
-- `kylin.cube.cubeplanner.algorithm-threshold-greedy`：默认值为 8
-- `kylin.cube.cubeplanner.expansion-threshold`：默认值为 15.0
-- `kylin.cube.cubeplanner.recommend-cache-max-size`：默认值为 200
-- `kylin.cube.cubeplanner.query-uncertainty-ratio`：默认值为 0.1
-- `kylin.cube.cubeplanner.bpus-min-benefit-ratio`：默认值为 0.01
-- `kylin.cube.cubeplanner.algorithm-threshold-genetic`：默认值为 23
-
-
-> 提示：更多信息请参考 [使用 Cube Planner](/cn/docs/tutorial/use_cube_planner.html)。
-
-
-
-### HBase 存储 {#hbase-config}
-
-- `kylin.storage.hbase.table-name-prefix`：默认值为 KYLIN\_
-- `kylin.storage.hbase.namespace`：指定 HBase 存储默认的 namespace，默认值为 default
-- `kylin.storage.hbase.coprocessor-local-jar`：指向 HBase 协处理器有关 jar 包
-- `kylin.storage.hbase.coprocessor-mem-gb`：设置 HBase 协处理器内存大小，默认值为 3.0(GB)
-- `kylin.storage.hbase.run-local-coprocessor`：是否运行本地 HBase 协处理器，默认值为 FALSE
-- `kylin.storage.hbase.coprocessor-timeout-seconds`：设置超时时间，默认值为 0
-- `kylin.storage.hbase.region-cut-gb`：单个 Region 的大小，默认值为 5.0
-- `kylin.storage.hbase.min-region-count`：指定最小 Region 个数，默认值为 1
-- `kylin.storage.hbase.max-region-count`：指定最大 Region 个数，默认值为 500
-- `kylin.storage.hbase.hfile-size-gb`：指定 HFile 大小，默认值为 2.0(GB)
-- `kylin.storage.hbase.max-scan-result-bytes`：指定扫描返回结果的最大值，默认值为 5242880(byte)，即 5(MB)
-- `kylin.storage.hbase.compression-codec`：是否压缩，默认值为 none，即不开启压缩
-- `kylin.storage.hbase.rowkey-encoding`：指定 Rowkey 的编码方式，默认值为 FAST_DIFF
-- `kylin.storage.hbase.block-size-bytes`：默认值为 1048576
-- `kylin.storage.hbase.small-family-block-size-bytes`：指定 Block 大小，默认值为 65536(byte)，即 64(KB)
-- `kylin.storage.hbase.owner-tag`：指定 Kylin 平台的所属人，默认值为 whoami@kylin.apache.org
-- `kylin.storage.hbase.endpoint-compress-result`：是否返回压缩结果，默认值为 TRUE
-- `kylin.storage.hbase.max-hconnection-threads`：指定连接线程数量的最大值，默认值为 2048
-- `kylin.storage.hbase.core-hconnection-threads`：指定核心连接线程的数量，默认值为 2048
-- `kylin.storage.hbase.hconnection-threads-alive-seconds`：指定线程存活时间，默认值为 60
-- `kylin.storage.hbase.replication-scope`：指定集群复制范围，默认值为 0
-- `kylin.storage.hbase.scan-cache-rows`：指定扫描缓存行数，默认值为 1024
-
-
-
-### 备用 Hbase 存储 {#secondary-hbase}
-
-Kylin支持用户配置备用Hbase，这样在集群迁移时，Kylin仍然可以从旧集群中查询到构建好的Cube数据。
-
-- `kylin.secondary.storage.url`: 指定备用Hbase的集群地址以及metadata url，zookeeper信息. 例如 *kylin.secondary.storage.url=hostname:kylin_metadata@hbase,hbase.zookeeper.quorum=hostname:11000,zookeeper.znode.parent=/hbase/*.如果还有其他的配置项，可以以<key> = <value>的形式添加在后面.
-
-
-> 提示：更多信息请参考 [KYLIN-4175](https://issues.apache.org/jira/browse/KYLIN-4175)。
-
-
-
-### 任务输出 {#job-output}
-
-为了避免job output的内容太多，用户可以设置output的最大长度。
-
-- `kylin.job.execute-output.max-size`: Job output的最大长度. 默认值为10484760.
-- `kylin.engine.spark.output.max-size`: Spark job output的最大长度. 默认值为10484760.
-
-
-
-### 启用压缩 {#compress-config}
-
-Kylin 在默认状态下不会启用压缩，不支持的压缩算法会阻碍 Kylin 的构建任务，但是一个合适的压缩算法可以减少存储开销和网络开销，提高整体系统运行效率。
-Kylin 可以使用三种类型的压缩，分别是 HBase 表压缩，Hive 输出压缩 和 MapReduce 任务输出压缩。
-
-> **注意**：压缩设置只有在重启 Kylin 实例后才会生效。
-
-* HBase 表压缩
-
-该项压缩通过 `kyiln.properties` 中的 `kylin.storage.hbase.compression-codec` 进行配置，参数值可选 `none`，`snappy`， `lzo`， `gzip`， `lz4`），默认值为 none，即不压缩数据。
-
-> **注意**：在修改压缩算法前，请确保用户的 HBase 集群支持所选压缩算法。
-
-
-* Hive 输出压缩
-
-该项压缩通过 `kylin_hive_conf.xml` 进行配置，默认配置为空，即直接使用了 Hive 的默认配置。如果想重写配置，请在 `kylin_hive_conf.xml` 中添加 (或替换) 下列属性。以 SNAPPY 压缩为例：
-
-```xml
-<property>
- <name>mapreduce.map.output.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
-</property>
-<property>
- <name>mapreduce.output.fileoutputformat.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
-</property>
-```
-
-* MapReduce 任务输出压缩
-
-该项压缩通过 `kylin_job_conf.xml` 和 `kylin_job_conf_inmem.xml` 进行配置。默认值为空，即使用 MapReduce 的默认配置。如果想重写配置，请在 `kylin_job_conf.xml` 和 `kylin_job_conf_inmem.xml` 中添加 (或替换) 下列属性。以 SNAPPY 压缩为例：
-
-```xml
-<property>
- <name>mapreduce.map.output.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
-</property>
-<property>
- <name>mapreduce.output.fileoutputformat.compress.codec</name>
- <value>org.apache.hadoop.io.compress.SnappyCodec</value>
- <description></description>
-</property>
-```
-
-
-
-### 实时 OLAP {#realtime-olap}
-
-#### 全局设置
-
-- `kylin.stream.job.dfs.block.size`: 指定了流式构建 Cuboid 任务所需 HDFS 块的大小。默认值为 *16M*。
-- `kylin.stream.index.path`: 指定了存储segment cache file的本地路径(包括本地fragment file和checkpoint file)。支持相对路径和绝对路径，默认值是 *stream_index*，也就是写到`$KYLIN_HOME/stream_index`，如果数据量很大的话将会占用大量磁盘空间，您也可以根据您的需求写成绝对路径以将数据放到数据盘。
-- `kylin.stream.node`: 指定了 receiver/coordinator的地址。格式应该为`hostname:port`或者`port`。如果设置成`port`，Kylin将会自动补全hostname；如果不设置该属性，将会使用默认的端口(Coordinator:7070，Receiver:9090)。当进程启动时，会将自身注册到Metadata。
-- `kylin.stream.metadata.store.type`: 指定了Realtime集群信息的元数据存储。默认值是 *zk*。
-- `kylin.stream.receiver.use-threads-per-query`: 指定了每个查询使用的线程资源数量。默认值是*8*。
-
-#### Cube 级别设置
-
-- `kylin.stream.index.maxrows`: 指定了缓存在堆内的聚合后的事件最大行数。默认值是*50000*。这个参数会影响Fragment File的数量，可以根据需求适当调高。
-- `kylin.stream.cube-num-of-consumer-tasks`: 指定了一个topic的全部消息的摄入将由哪多少Replica Set来负责。如果您的消息速率较大，需要适当提升这个数值。默认值是*3*。
-- `kylin.stream.segment.retention.policy`: 当Segment状态变为*IMMUTABLE*，该配置指定了Receiver如何处理本地Segment Cache。可选值包含`purge`和`fullBuild`。设置为`purge`后，Receiver会等待一定时间后删除本地数据；设置为`fullBuild`后，数据会上传到HDFS并等待构建。默认值是*fullBuild*。
-- `kylin.stream.build.additional.cuboids`: 默认情况下Receiver只构建base cuboid来回答查询，可以在Receiver端是否构建额外的cuboid，如果你希望优化某些查询的响应时间。具体哪些额外的Cuboid需要被构建由高级配置页面的强制Cuboid指定。
-- `kylin.stream.cube.window`: 指定了Streaming Segment的长度。默认值是*3600*。详情参阅[deep-dive-real-time-olap](http://kylin.apache.org/blog/2019/07/01/deep-dive-real-time-olap/)。
-- `kylin.stream.cube.duration`: 指定了Streaming Segment会等待迟到的消息多久，默认值是 *7200*(秒)。 详情参阅[deep-dive-real-time-olap](http://kylin.apache.org/blog/2019/07/01/deep-dive-real-time-olap/)。
-- `kylin.stream.cube.duration.max`: 指定了Streaming Segment保持Active的最长时间。默认值是 *43200*。详情参阅[deep-dive-real-time-olap](http://kylin.apache.org/blog/2019/07/01/deep-dive-real-time-olap/)。
-- `kylin.stream.checkpoint.file.max.num`: 指定了Receiver为每一个Cube保留的checkpoint文件数量。默认值是 *5*。
-- `kylin.stream.index.checkpoint.intervals`: 指定了Receiver进行checkpoint的间隔。默认值是 *300*。
-- `kylin.stream.immutable.segments.max.num`: 指定了在Receiver端，一个Cube最多可以保持多少个*IMMUTABLE*segment，因为Receiver端的性能和Fragment File的数量呈负相关。默认值是 *100*。
-- `kylin.stream.consume.offsets.latest`:指定了Receiver从什么位置开始消费，设置成*true*则从最新的offset开始消费，false则从最老的位置消费。默认值是 *true*。
-
-#### 高级设置
-
-- `kylin.stream.assigner`: 值是一个类的名字，这个类应该是`org.apache.kylin.stream.coordinator.assign.Assigner`的实现类，用于指定如何将Kafka Topic 下的各个Partition分配给各个Replica Set。默认值是 *DefaultAssigner*，其策略会努力将工作负载分配给负责partition数量少的Replica Set，以使得各个Replica Set工作负载相对均衡。
-- `kylin.stream.coordinator.client.timeout.millsecond`: 指定和Coordinator HTTP连接的Timeout，默认值是 *5000*。
-- `kylin.stream.receiver.client.timeout.millsecond`:指定和Receiver HTTP连接的Timeout，默认值是 *5000*。
-- `kylin.stream.receiver.http.max.threads`: 指定了Receiver端的Http连接最大线程数。默认值为 *200*。
-- `kylin.stream.receiver.http.min.threads`: 指定了Receiver端的Http连接最小线程数。默认值为 *10*。
-- `kylin.stream.receiver.query-core-threads`: 指定了Receiver用于scan的线程数量，默认值是*50*。
-- `kylin.stream.receiver.query-max-threads`: 指定了Receiver用于scan的线程最大数量，默认值是*200*。
-- `kylin.stream.segment-max-fragments`: Receiver端每次MemoryStore大小达到阈值(`kylin.stream.index.maxrows`)，会落盘形成一个Fragment File，Receiver会尝试尽可能合并这些Fragment File来减少数据冗余。这个配置项会指定触发merge的阈值，默认值是*50*。
-- `kylin.stream.segment-min-fragments`: Receiver端的每次merge后不会使文件数量少于这个阈值，默认值是 *15*。
-- `kylin.stream.max-fragment-size-mb`: 合并后，每个Fragment File的大小不会超过该值，默认值是 *300*。
-- `kylin.stream.fragments-auto-merge-enable`: 是否开启后台自动合并Fragment File。默认值是 *true*。
-- `kylin.stream.metrics.option`: 指定是否开启Receiver端的metrics信息收集, 可选值是 csv/console/jmx。
-- `kylin.stream.event.timezone`: 指定从Event Time衍生出来的时间衍生列如`HOUR_START`/`DAY_START`使用哪种时区，默认是UTC时间。
-- `kylin.stream.auto-resubmit-after-discard-enabled`: 当用户 discard了某一个 Realtime的构建任务，是否自动重新提交新任务。
-
-> 提示：入门教程 请参考 [Real-time OLAP](/docs/tutorial/realtime_olap.html)。
-
-
-
-### 存储清理配置 {#storage-clean-up-configuration}
-
-本小节介绍 Kylin 存储清理有关的配置。
-
-
-
-### 存储清理相关 {#storage-clean-up-config}
-
-- `kylin.storage.clean-after-delete-operation`: 是否清理 HBase 和 HDFS 中的 segment 数据。默认值为 FALSE。
-
-
-
-### 查询配置 {#kylin-query}
-
-本小节介绍 Kylin 查询有关的配置。
-
-
-
-### 查询相关 {#query-config}
-
-- `kylin.query.skip-empty-segments`：查询是否跳过数据量为 0 的 segment，默认值为 TRUE
-- `kylin.query.large-query-threshold`：指定最大返回行数，默认值为 1000000
-- `kylin.query.security-enabled`：是否在查询时检查 ACL，默认值为 TRUE
-- `kylin.query.security.table-acl-enabled`：是否在查询时检查对应表的 ACL，默认值为 TRUE
-- `kylin.query.calcite.extras-props.conformance`：是否严格解析，默认值为 LENIENT
-- `kylin.query.calcite.extras-props.caseSensitive`：是否大小写敏感，默认值为 TRUE
-- `kylin.query.calcite.extras-props.unquotedCasing`：是否需要将查询语句进行大小写转换，参数值可选 `UNCHANGED`， `TO_UPPER`， `TO_LOWER` ，默认值为 `TO_UPPER`，即全部大写
-- `kylin.query.calcite.extras-props.quoting`：是否添加引号，参数值可选 `DOUBLE_QUOTE`， `BACK_TICK`，`BRACKET`，默认值为 `DOUBLE_QUOTE`
-- `kylin.query.statement-cache-max-num`：缓存的 PreparedStatement 的最大条数，默认值为 50000
-- `kylin.query.statement-cache-max-num-per-key`：每个键缓存的 PreparedStatement 的最大条数，默认值为 50
-- `kylin.query.enable-dict-enumerator`：是否启用字典枚举器，默认值为 FALSE
-- `kylin.query.enable-dynamic-column`：是否启用动态列，默认值为 FALSE，设置为 TRUE 后可以查询一列中不包含 NULL 的行数
-
-
-
-### 模糊查询 {#fuzzy}
-
-- `kylin.storage.hbase.max-fuzzykey-scan`：设置扫描的模糊键的阈值，超过该参数值便不再扫描模糊键，默认值为 200
-- `kylin.storage.hbase.max-fuzzykey-scan-split`：分割大模糊键集来减少每次扫描中模糊键的数量，默认值为 1
-- `kylin.storage.hbase.max-visit-scanrange`：默认值为 1000000
-
-
-
-### 查询缓存 {#cache-config}
-
-- `kylin.query.cache-enabled`：是否启用缓存，默认值为 TRUE
-- `kylin.query.cache-threshold-duration`：查询延时超过阈值则保存进缓存，默认值为 2000(ms)
-- `kylin.query.cache-threshold-scan-count`：查询所扫描的数据行数超过阈值则保存进缓存，默认值为 10240(rows)
-- `kylin.query.cache-threshold-scan-bytes`：查询所扫描的数据字节数超过阈值则保存进缓存，默认值为 1048576(byte)
-
-
-
-### 查询限制 {#query-limit}
-
-- `kylin.query.timeout-seconds`：设置查询超时时间，默认值为 0，即没有限制，如果设置的值小于 60，会被强制替换成 60 秒
-- `kylin.query.timeout-seconds-coefficient`：设置查询超时秒数的系数，默认值为 0.5
-- `kylin.query.max-scan-bytes`：设置查询扫描字节的上限，默认值为 0，即没有限制
-- `kylin.storage.partition.max-scan-bytes`：设置查询扫描的最大字节数，默认值为 3221225472(bytes)，即 3GB
-- `kylin.query.max-return-rows`：指定查询返回行数的上限，默认值为 5000000
-
-
-
-### 坏查询 {#bad-query}
-
-`kylin.query.timeout-seconds` 的值为大于 60 或为 0，`kylin.query.timeout-seconds-coefficient` 其最大值为 double 的上限。这两个参数的乘积为坏查询检查的间隔时间，如果为 0，那么会设为 60 秒，最长秒数是 int 的最大值。
-
-- `kylin.query.badquery-stacktrace-depth`：设置堆栈追踪的深度，默认值为 10
-- `kylin.query.badquery-history-number`：设置要展示的历史坏查询的数量，默认为 50
-- `kylin.query.badquery-alerting-seconds`：默认为 90，如果运行时间大于这个值，那么首先就会打出该查询的日志信息，包括（时长、项目、线程、用户、查询 id）。至于是否保存最近的查询，取决于另一个参数。然后记录 Stack 日志信息，记录的深度由另一个参数指定，方便后续问题分析
-- `kylin.query.badquery-persistent-enabled`：默认为 true，会保存最近的一些坏查询，而且不可在 Cube 级别进行覆盖
-
-
-
-
-### 查询下压 {#query-pushdown}
-
-- `kylin.query.pushdown.runner-class-name=org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl`：如果需要启用查询下压，需要移除这句配置的注释
-- `kylin.query.pushdown.jdbc.url`：JDBC 的 URL
-- `kylin.query.pushdown.jdbc.driver`：JDBC 的 driver 类名，默认值为 `org.apache.hive.jdbc.HiveDriver`
-- `kylin.query.pushdown.jdbc.username`：JDBC 对应数据库的用户名，默认值为 `hive`
-- `kylin.query.pushdown.jdbc.password`：JDBC 对应数据库的密码
-- `kylin.query.pushdown.jdbc.pool-max-total`：JDBC 连接池的最大连接数，默认值为8
-- `kylin.query.pushdown.jdbc.pool-max-idle`：JDBC 连接池的最大等待连接数，默认值为8
-- `kylin.query.pushdown.jdbc.pool-min-idle`：默认值为 0
-- `kylin.query.pushdown.update-enabled`：指定是否在查询下压中开启 update，默认值为 FALSE
-- `kylin.query.pushdown.cache-enabled`：是否开启下压查询的缓存来提高相同查询语句的查询效率，默认值为 FALSE
-
-> 提示：更多信息请参考[查询下压](/cn/docs/tutorial/query_pushdown.html)
-
-
-
-### 查询改写 {#convert-sql}
-
-- `kylin.query.force-limit`：该参数通过为 select * 语句强制添加 LIMIT 分句，达到缩短数据返回时间的目的，该参数默认值为 -1，将该参数值设置为正整数，如 1000，该值会被应用到 LIMIT 分句，查询语句最终会被转化成 select * from fact_table limit 1000
-- `kylin.storage.limit-push-down-enabled`: 默认值为 *TRUE*，设置为 *FALSE* 意味着关闭存储层的 limit-pushdown
-- `kylin.query.flat-filter-max-children`：指定打平 filter 时 filter 的最大值。默认值为 500000
-
-
-
-### 收集查询指标到 JMX {#jmx-metrics}
-
-- `kylin.server.query-metrics-enabled`：默认值为 FALSE，设为 TRUE 来将查询指标收集到 JMX
-
-> 提示：更多信息请参考 [JMX](https://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html)
-
-
-
-### 收集查询指标到 dropwizard {#dropwizard-metrics}
-
-- `kylin.server.query-metrics2-enabled`：默认值为 FALSE，设为 TRUE 来将查询指标收集到 dropwizard
-
-> 提示：更多信息请参考 [dropwizard](https://metrics.dropwizard.io/4.0.0/)
-
-
-
-### 安全配置 {#kylin-security}
-
-本小节介绍 Kylin 安全有关的配置。
-
-
-### 用户名和密码 {#user-name-and-password}
-请参考文档：[How to add new user or change the default password](https://cwiki.apache.org/confluence/display/KYLIN/How+to+add+new+user+or+change+the+default+password)
-
-
-### 集成 LDAP 实现单点登录 {#ldap-sso}
-
-- `kylin.security.profile`：安全认证的方式，参数值可选 `ldap`，`testing`，`saml`。集成 LDAP 实现单点登录时应设置为 `ldap`
-- `kylin.security.ldap.connection-server`：LDAP 服务器，如 ldap://ldap_server:389
-- `kylin.security.ldap.connection-username`：LDAP 用户名
-- `kylin.security.ldap.connection-password`：LDAP 密码
-- `kylin.security.ldap.user-search-base`：定义同步到 Kylin 的用户的范围
-- `kylin.security.ldap.user-search-pattern`：定义登录验证匹配的用户名
-- `kylin.security.ldap.user-group-search-base`：定义同步到 Kylin 的用户组的范围
-- `kylin.security.ldap.user-group-search-filter`：定义同步到 Kylin 的用户的类型
-- `kylin.security.ldap.service-search-base`：当需要服务账户可以访问 Kylin 时需要定义
-- `kylin.security.ldap.service-search-pattern`：当需要服务账户可以访问 Kylin 时需要定义
-- `kylin.security.ldap.service-group-search-base`：当需要服务账户可以访问 Kylin 时需要定义
-- `kylin.security.acl.admin-role`：将一个 LDAP 群组映射成管理员角色（组名大小写敏感）
-- `kylin.server.auth-user-cache.expire-seconds`：LDAP 用户信息缓存时间，默认值为 300(s)
-- `kylin.server.auth-user-cache.max-entries`：LDAP 用户数目最大缓存，默认值为 100
-
-
-
-### 集成 Apache Ranger {#ranger}
-
-- `kylin.server.external-acl-provider=org.apache.ranger.authorization.kylin.authorizer.RangerKylinAuthorizer`
-
-> 提示：更多信息请参考[Ranger 的安装文档之如何集成 Kylin 插件](https://cwiki.apache.org/confluence/display/RANGER/Kylin+Plugin)
-
-
-
-### 启用 ZooKeeper ACL {#zookeeper-acl}
-
-- `kylin.env.zookeeper-acl-enabled`：启用 ZooKeeper ACL 以阻止未经授权的用户访问 Znode 或降低由此导致的不良操作的风险，默认值为 `FALSE`
-- `kylin.env.zookeeper.zk-auth`：使用 用户名：密码 作为 ACL 标识，默认值为 `digest:ADMIN:KYLIN`
-- `kylin.env.zookeeper.zk-acl`：使用单个 ID 作为 ACL 标识，默认值为 `world:anyone:rwcda`，`anyone` 表示任何人
-
-### 使用 Memcached 作为 Kylin 查询缓存 {#distributed-cache}
-
-从 v2.6.0，Kylin 可以使用 Memcached 作为查询缓存，一起引入的还有一系列缓存增强 ([KYLIN-2895](https://issues.apache.org/jira/browse/KYLIN-2895))。想要启用该功能，您需要执行以下步骤：
-
-1. 在一个或多个节点上安装 Memcached (最新稳定版 v1.5.12); 如果资源够的话，可以在每个安装 Kylin 的节点上安装 Memcached。
-
-2. 按照如下所示方式修改 $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes 目录下的 applicationContext.xml 的内容：
-
-注释如下代码：
-{% highlight Groff markup %}
-<bean id="ehcache"
- class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean"
- p:configLocation="classpath:ehcache-test.xml" p:shared="true"/>
-
-<bean id="cacheManager" class="org.springframework.cache.ehcache.EhCacheCacheManager"
- p:cacheManager-ref="ehcache"/>
-{% endhighlight %}
-取消如下代码的注释：
-{% highlight Groff markup %}
-<bean id="ehcache" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean"
- p:configLocation="classpath:ehcache-test.xml" p:shared="true"/>
-
-<bean id="remoteCacheManager" class="org.apache.kylin.cache.cachemanager.MemcachedCacheManager" />
-<bean id="localCacheManager" class="org.apache.kylin.cache.cachemanager.InstrumentedEhCacheCacheManager"
- p:cacheManager-ref="ehcache"/>
-<bean id="cacheManager" class="org.apache.kylin.cache.cachemanager.RemoteLocalFailOverCacheManager" />
-
-<bean id="memcachedCacheConfig" class="org.apache.kylin.cache.memcached.MemcachedCacheConfig">
- <property name="timeout" value="500" />
- <property name="hosts" value="${kylin.cache.memcached.hosts}" />
-</bean>
-{% endhighlight %}
-applicationContext.xml 中 `${kylin.cache.memcached.hosts}` 的值就是在 conf/kylin.properties 中指定的 `kylin.cache.memcached.hosts` 的值。
-
-3.在 `conf/kylin.properties` 中添加如下参数：
-{% highlight Groff markup %}
-kylin.query.cache-enabled=true
-kylin.query.lazy-query-enabled=true
-kylin.query.cache-signature-enabled=true
-kylin.query.segment-cache-enabled=true
-kylin.cache.memcached.hosts=memcached1:11211,memcached2:11211,memcached3:11211
-{% endhighlight %}
-
-- `kylin.query.cache-enabled` 是否开启查询缓存的总开关，默认值为 `true`。
-- `kylin.query.lazy-query-enabled` 是否为短时间内重复发送的查询，等待并重用前次查询的结果，默认为 `false`。
-- `kylin.query.cache-signature-enabled` 是否为缓存进行签名检查，依据签名变化来决定缓存的有效性。缓存的签名由项目中的 cube / hybrid 的状态以及它们的最后构建时间等来动态计算（在缓存被记录时）,默认为 `false`，高度推荐设置为 `true`。
-- `kylin.query.segment-cache-enabled` 是否在 segment 级别缓存从 存储引擎(HBase)返回的数据，默认为 `false`；设置为 `true`，且启用 Memcached 分布式缓存开启的时候，此功能才会生效。可为频繁构建的 cube （如 streaming cube）提升缓存命中率，从而提升性能。
-- `kylin.cache.memcached.hosts` 指明了 memcached 的机器名和端口。
\ No newline at end of file
+Kylin 4.x 的所有配置项定义及用法在 Apache Kylin Wiki 列出：
+[Apache Kylin4.0 Configuration](https://cwiki.apache.org/confluence/display/KYLIN/Kylin+4.X+Configuration)
diff --git a/website/_docs/install/configuration.md b/website/_docs/install/configuration.md
index f4aaf0a..1204ed0 100644
--- a/website/_docs/install/configuration.md
+++ b/website/_docs/install/configuration.md
@@ -5,823 +5,5 @@ categories: install
permalink: /docs/install/configuration.html
---
-
-- [Configuration Files and Overriding](#kylin-config)
- - [Kylin Configuration Files](#kylin-config)
- - [Configuration Overriding](#config-override)
- - [Project-level Configuration Overriding](#project-config-override)
- - [Cube-level Configuration Overriding](#cube-config-override)
- - [MapReduce Configuration Overriding](#mr-config-override)
- - [Hive Configuration Overriding](#hive-config-override)
- - [Spark Configuration Overriding](#spark-config-override)
-- [Deployment configuration](#kylin-deploy)
- - [Deploy Kylin](#deploy-config)
- - [Allocate More Memory for Kylin](#kylin-jvm-settings)
- - [Job Engine HA](#job-engine-ha)
- - [Job Engine Safemode](#job-engine-safemode)
- - [Read/Write Separation](#rw-deploy)
- - [RESTful Webservice](#rest-config)
-- [Metastore Configuration](#kylin_metastore)
- - [Metadata-related](#metadata)
- - [MySQL Metastore Configuration](#mysql-metastore)
-- [Modeling Configuration](#kylin-build)
- - [Hive Client and SparkSQL](#hive-client-and-sparksql)
- - [JDBC Datasource Configuration](#jdbc-datasource)
- - [Data Type Precision](#precision-config)
- - [Cube Design](#cube-config)
- - [Cube Size Estimation](#cube-estimate)
- - [Cube Algorithm](#cube-algorithm)
- - [Auto Merge Segments](#auto-merge)
- - [Lookup Table Snapshot](#snapshot)
- - [Build Cube](#cube-build)
- - [Dictionary-related](#dict-config)
- - [Deal with Ultra-High-Cardinality Columns](#uhc-config)
- - [Spark as Build Engine](#spark-cubing)
- - [Submit Spark jobs via Livy](#livy-submit-spark-job)
- - [Spark Dynamic Allocation](#dynamic-allocation)
- - [Job-related](#job-config)
- - [Enable Email Notification](#email-notification)
- - [Enable Cube Planner](#cube-planner)
- - [HBase Storage](#hbase-config)
- - [Secondary Hbase Storage](#secondary-hbase)
- - [Job Output](#job-output)
- - [Enable Compression](#compress-config)
- - [Real-time OLAP](#realtime-olap)
-- [Storage Clean up Configuration](#storage-clean-up-configuration)
- - [Storage-clean-up-related](#storage-clean-up-config)
-- [Query Configuration](#kylin-query)
- - [Query-related](#query-config)
- - [Fuzzy Query](#fuzzy)
- - [Query Cache](#cache-config)
- - [Query Limits](#query-limit)
- - [Bad Query](#bad-query)
- - [Query Pushdown](#query-pushdown)
- - [Query rewriting](#convert-sql)
- - [Collect Query Metrics to JMX](#jmx-metrics)
- - [Collect Query Metrics to dropwizard](#dropwizard-metrics)
-- [Security Configuration](#kylin-security)
- - [User name and password](#user-name-and-password)
- - [Integrated LDAP for SSO](#ldap-sso)
- - [Integrate with Apache Ranger](#ranger)
- - [Enable ZooKeeper ACL](#zookeeper-acl)
-- [Distributed query cache with Memcached](#distributed-cache)
-
-
-### Configuration Files and Overriding {#kylin-config}
-
-This section introduces Kylin's configuration files and how to perform Configuration Overriding.
-
-
-
-### Kylin Configuration Files {#kylin-config-file}
-
-Kylin will automatically read the Hadoop configuration (`core-site.xml`), Hive configuration (`hive-site.xml`) and HBase configuration (`hbase-site.xml`) from the environment, in addition, Kylin's configuration files are in the `$KYLIN_HOME/conf/` directory.
-Kylin's configuration file is as follows:
-
-- `kylin_hive_conf.xml`: This file contains the configuration for the Hive job.
-- `kylin_job_conf.xml` & `kylin_job_conf_inmem.xml`: This file contains configuration for the MapReduce job. When performing the *In-mem Cubing* job, user need to request more memory for the mapper in `kylin_job_conf_inmem.xml`
-- `kylin-kafka-consumer.xml`: This file contains the configuration for the Kafka job.
-- `kylin-server-log4j.properties`: This file contains the log configuration for the Kylin server.
-- `kylin-tools-log4j.properties`: This file contains the log configuration for the Kylin command line.
-- `setenv.sh` : This file is a shell script for setting environment variables. Users can adjust the size of the Kylin JVM stack with `KYLIN_JVM_SETTINGS` and set other environment variables such as `KAFKA_HOME`.
-- `kylin.properties`: This file contains Kylin global configuration.
-
-
-
-### Configuration Overriding {#config-override}
-
-Some configuration files in `$KYLIN_HOME/conf/` can be overridden in the Web UI. Configuration Overriding has two scope: *Project level* and *Cube level*. The priority order can be stated as: Cube level configurations > Project level configurations > configuration files.
-
-
-
-### Project-level Configuration Overriding {#project-config-override}
-
-Click *Manage Project* in the web UI interface, select a project, click *Edit* -> *Project Config* -> *+ Property* to add configuration properties which could override property values in configuration files, as the figure below shown,
-
-
-
-
-### Cube-level Configuration Overriding {#cube-config-override}
-
-In the *Configuration overrides* step of *Cube Designer*, user could rewrite property values to override those in project level and configuration files, as the figure below shown,
-
-
-The following configurations can be override in the Cube-level,
-
-- `kylin.cube.size-estimate*`
-- `kylin.cube.algorithm*`
-- `kylin.cube.aggrgroup*`
-- `kylin.metadata.dimension-encoding-max-length`
-- `kylin.cube.max-building-segments`
-- `kylin.cube.is-automerge-enabled`
-- `kylin.job.allow-empty-segment`
-- `kylin.job.sampling-percentage`
-- `kylin.source.hive.redistribute-flat-table`
-- `kylin.engine.spark*`
-- `kylin.query.skip-empty-segments`
-
-
-
-### MapReduce Configuration Overriding {#mr-config-override}
-
-Kylin supports overriding configuration properties in `kylin_job_conf.xml` and `kylin_job_conf_inmem.xml` at the project and cube level, in the form of key-value pairs, in the following format:
-`kylin.engine.mr.config-override.<key> = <value>`
-* If user wants getting more memory from YARN for jobs, user can set: `kylin.engine.mr.config-override.mapreduce.map.java.opts=-Xmx7g` and `kylin.engine.mr.config-override.mapreduce.map.memory.mb=8192`
-* If user wants the cube's build job to use a different YARN resource queue, user can set:
-`kylin.engine.mr.config-override.mapreduce.job.queuename={queueName}`
-
-
-
-### Hive Configuration Overriding {#hive-config-override}
-
-Kylin supports overriding configuration properties in `kylin_hive_conf.xml` at the project and cube level, in the form of key-value pairs, in the following format:
-`kylin.source.hive.config-override.<key> = <value>`
-If user wants Hive to use a different YARN resource queue, user can set:
-`kylin.source.hive.config-override.mapreduce.job.queuename={queueName}`
-
-
-
-### Spark Configuration Overriding {#spark-config-override}
-
-Kylin supports overriding configuration properties in `kylin.properties` at the project and cube level, in the form of key-value pairs, in the following format:
-`kylin.engine.spark-conf.<key> = <value>`
-If user wants Spark to use a different YARN resource queue, user can set:
-`kylin.engine.spark-conf.spark.yarn.queue={queueName}`
-
-
-
-### Deployment configuration {#kylin-deploy}
-
-This section introduces Kylin Deployment related configuration.
-
-
-
-### Deploy Kylin {#deploy-config}
-
-- `kylin.env.hdfs-working-dir`: specifies the HDFS path used by Kylin service. The default value is `/kylin`. Make sure that the user who starts the Kylin instance has permission to read and write to this directory.
-- `kylin.env`: specifies the purpose of the Kylin deployment. Optional values include `DEV`, `QA` and `PROD`. The default value is *DEV*. Some developer functions will be enabled in *DEV* mode.
-- `kylin.env.zookeeper-base-path`: specifies the ZooKeeper path used by the Kylin service. The default value is `/kylin`
-- `kylin.env.zookeeper-connect-string`: specifies the ZooKeeper connection string. If it is empty, use HBase's ZooKeeper
-- `kylin.env.hadoop-conf-dir`: specifies the Hadoop configuration file directory. If not specified, get `HADOOP_CONF_DIR` in the environment.
-- `kylin.server.mode`: Optional values include `all`, `job` and `query`, among them *all* is the default one. *job* mode means the instance schedules Cube job only; *query* mode means the instance serves SQL queries only; *all* mode means the instance handles both of them.
-- `kylin.server.cluster-name`: specifies the cluster name
-
-
-
-### Allocate More Memory for Kylin {#kylin-jvm-settings}
-
-There are two sample settings for `KYLIN_JVM_SETTINGS` are given in `$KYLIN_HOME/conf/setenv.sh`.
-The default setting use relatively less memory. You can comment it and then uncomment the next line to allocate more memory for Kyligence Enterprise. The default configuration is:
-
-```shell
-Export KYLIN_JVM_SETTINGS="-Xms1024M -Xmx4096M -Xss1024K -XX`MaxPermSize=512M -verbose`gc -XX`+PrintGCDetails -XX`+PrintGCDateStamps -Xloggc`$KYLIN_HOME/logs/kylin.gc.$$ -XX`+UseGCLogFileRotation - XX`NumberOfGCLogFiles=10 -XX`GCLogFileSize=64M"
-# export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX`MaxPermSize=512m -XX`NewSize=3g -XX`MaxNewSize=3g -XX`SurvivorRatio=4 -XX`+CMSClassUnloadingEnabled -XX`+CMSParallelRemarkEnabled -XX`+UseConcMarkSweepGC -XX `+CMSIncrementalMode -XX`CMSInitiatingOccupancyFraction=70 -XX`+UseCMSInitiatingOccupancyOnly -XX`+DisableExplicitGC -XX`+HeapDumpOnOutOfMemoryError -verbose`gc -XX`+PrintGCDetails -XX`+PrintGCDateStamps -Xloggc`$KYLIN_HOME/logs/kylin.gc. $$ -XX`+UseGCLogFileRotation -XX`NumberOfGCLog [...]
-```
-
-
-
-### Job Engine HA {#job-engine-ha}
-
-- `kylin.job.scheduler.default=2`: to enable the distributed job scheduler.
-- `kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperJobLock`: to enable distributed job lock
-
-> Note: For more information, please refer to the **Enable Job Engine HA** section in [Deploy in Cluster Mode](/docs/install/kylin_cluster.html)
-
-
-### Job Engine Safemode {#job-engine-safemode}
-
-Safemode can be only used in default schedule.
-
-- `kylin.job.scheduler.safemode=TRUE`: to enable job scheduler safemode. In safemode, Newly submitted job will not be executed
-- `kylin.job.scheduler.safemode.runable-projects=project1,project2`: provide list of projects as exceptional case in safemode.
-
-
-### Read/Write Separation {#rw-deploy}
-
-- `kylin.storage.hbase.cluster-fs`: specifies the HDFS file system of the HBase cluster
-- `kylin.storage.hbase.cluster-hdfs-config-file`: specifies HDFS configuration file pointing to the HBase cluster
-
-> Note: For more information, please refer to [Deploy Apache Kylin with Standalone HBase Cluster](http://kylin.apache.org/blog/2016/06/10/standalone-hbase-cluster/)
-
-
-
-### RESTful Webservice {#rest-config}
-
-- `kylin.web.timezone`: specifies the time zone used by Kylin's REST service. The default value is GMT+8.
-- `kylin.web.cross-domain-enabled`: whether cross-domain access is supported. The default value is TRUE
-- `kylin.web.export-allow-admin`: whether to support administrator user export information. The default value is TRUE
-- `kylin.web.export-allow-other`: whether to support other users to export information. The default value is TRUE
-- `kylin.web.dashboard-enabled`: whether to enable Dashboard. The default value is FALSE
-
-
-
-### Metastore Configuration {#kylin_metastore}
-
-This section introduces Kylin Metastore related configuration.
-
-
-
-### Metadata-related {#metadata}
-
-- `kylin.metadata.url`: specifies the Metadata path. The default value is *kylin_metadata@hbase*
-- `kylin.metadata.dimension-encoding-max-length`: specifies the maximum length when the dimension is used as Rowkeys with fix_length encoding. The default value is 256.
-- `kylin.metadata.sync-retries`: specifies the number of Metadata sync retries. The default value is 3.
-- `kylin.metadata.sync-error-handler`: The default value is *DefaultSyncErrorHandler*
-- `kylin.metadata.check-copy-on-write`: whether clear metadata cache, default value is *FALSE*
-- `kylin.metadata.hbase-client-scanner-timeout-period`: specifies the total timeout between the RPC call initiated by the HBase client. The default value is 10000 (ms).
-- `kylin.metadata.hbase-rpc-timeout`: specifies the timeout for HBase to perform RPC operations. The default value is 5000 (ms).
-- `kylin.metadata.hbase-client-retries-number`: specifies the number of HBase retries. The default value is 1 (times).
-- `kylin.metadata.resource-store-provider.jdbc`: specifies the class used by JDBC. The default value is *org.apache.kylin.common.persistence.JDBCResourceStore*
-
-
-
-### MySQL Metastore Configuration {#mysql-metastore}
-
-
-- `kylin.metadata.url`: specifies the metadata path
-- `kylin.metadata.jdbc.dialect`: specifies JDBC dialect
-- `kylin.metadata.jdbc.json-always-small-cell`: The default value is TRUE
-- `kylin.metadata.jdbc.small-cell-meta-size-warning-threshold`: The default value is 100 (MB)
-- `kylin.metadata.jdbc.small-cell-meta-size-error-threshold`: The default value is 1 (GB)
-- `kylin.metadata.jdbc.max-cell-size`: The default value is 1 (MB)
-- `kylin.metadata.resource-store-provider.jdbc`: specifies the class used by JDBC. The default value is org.apache.kylin.common.persistence.JDBCResourceStore
-
-> Note: For more information, please refer to [MySQL-based Metastore Configuration](/docs/tutorial/mysql_metastore.html)
-
-
-
-### Modeling Configuration {#kylin-build}
-
-This section introduces Kylin data modeling and build related configuration.
-
-
-
-### Hive Client and SparkSQL {#hive-client-and-sparksql}
-
-- `kylin.source.hive.client`: specifies the Hive command line type. Optional values include *cli* or *beeline*. The default value is *cli*.
-- `kylin.source.hive.beeline-shell`: specifies the absolute path of the Beeline shell. The default is beeline
-- `kylin.source.hive.beeline-params`: when using Beeline as the Client tool for Hive, user need to configure this parameter to provide more information to Beeline
-- `kylin.source.hive.enable-sparksql-for-table-ops`: the default value is *FALSE*, which needs to be set to *TRUE* when using SparkSQL
-- `kylin.source.hive.sparksql-beeline-shell`: when using SparkSQL Beeline as the client tool for Hive, user need to configure this parameter as /path/to/spark-client/bin/beeline
-- `kylin.source.hive.sparksql-beeline-params`: when using SparkSQL Beeline as the Client tool for Hive,user need to configure this parameter to provide more information to SparkSQL
-
-
-
-
-### JDBC Datasource Configuration {#jdbc-datasource}
-
-- `kylin.source.default`: specifies the type of data source used by JDBC
-- `kylin.source.jdbc.connection-url`: specifies JDBC connection string
-- `kylin.source.jdbc.driver`: specifies JDBC driver class name
-- `kylin.source.jdbc.dialect`: specifies JDBC dialect. The default value is default
-- `kylin.source.jdbc.user`: specifies JDBC connection username
-- `kylin.source.jdbc.pass`: specifies JDBC connection password
-- `kylin.source.jdbc.sqoop-home`: specifies Sqoop installation path
-- `kylin.source.jdbc.sqoop-mapper-num`: specifies how many slices should be split. Sqoop will run a mapper for each slice. The default value is 4.
-- `kylin.source.jdbc.field-delimiter`: specifies the field separator. The default value is \
-
-> Note: For more information, please refer to [Building a JDBC Data Source](/docs/tutorial/setup_jdbc_datasource.html).
-
-
-
-
-### Data Type Precision {#precision-config}
-
-- `kylin.source.hive.default-varchar-precision`: specifies the maximum length of the *varchar* field. The default value is 256.
-- `kylin.source.hive.default-char-precision`: specifies the maximum length of the *char* field. The default value is 255.
-- `kylin.source.hive.default-decimal-precision`: specifies the precision of the *decimal* field. The default value is 19
-- `kylin.source.hive.default-decimal-scale`: specifies the scale of the *decimal* field. The default value is 4.
-
-
-
-### Cube Design {#cube-config}
-
-- `kylin.cube.ignore-signature-inconsistency`:The signature in Cube desc ensures that the cube is not changed to a corrupt state. The default value is *FALSE*
-- `kylin.cube.aggrgroup.max-combination`: specifies the max combination number of aggregation groups. The default value is 32768.
-- `kylin.cube.aggrgroup.is-mandatory-only-valid`: whether to allow Cube contains only Base Cuboid. The default value is *FALSE*, set to *TRUE* when using Spark Cubing
-- `kylin.cube.rowkey.max-size`: specifies the maximum number of columns that can be set to Rowkeys. The default value is 63, and it can not be more than 63.
-- `kylin.cube.allow-appear-in-multiple-projects`: whether to allow a cube to appear in multiple projects
-- `kylin.cube.gtscanrequest-serialization-level`: the default value is 1
-- `kylin.web.hide-measures`: hides some measures that may not be needed, the default value is RAW.
-
-
-
-### Cube Size Estimation {#cube-estimate}
-
-Both Kylin and HBase use compression when writing to disk, so Kylin will multiply its original size by the ratio to estimate the size of the cube.
-
-- `kylin.cube.size-estimate-ratio`: normal cube, default value is 0.25
-- `kylin.cube.size-estimate-memhungry-ratio`: Deprecated, default is 0.05
-- `kylin.cube.size-estimate-countdistinct-ratio`: Cube Size Estimation with count distinct h= metric, default value is 0.5
-- `kylin.cube.size-estimate-topn-ratio`: Cube Size Estimation with TopN metric, default value is 0.5
-
-
-### Cube Algorithm {#cube-algorithm}
-
-- `kylin.cube.algorithm`: specifies the algorithm of the Build Cube. Optional values include `auto`, `layer` and `inmem`. The default value is `auto`, that is, Kylin will dynamically select an algorithm by collecting data ( Layer or inmem), if user knows Kylin, user data and cluster condition well, user can directly set the algorithm.
-- `kylin.cube.algorithm.layer-or-inmem-threshold`: the default value is 7
-- `kylin.cube.algorithm.inmem-split-limit`: the default value is 500
-- `kylin.cube.algorithm.inmem-concurrent-threads`: the default value is 1
-- `kylin.job.sampling-percentage`: specifies the data sampling percentage. The default value is 100.
-
-
-
-### Auto Merge Segments {#auto-merge}
-
-- `kylin.cube.is-automerge-enabled`: whether to enable auto-merge. The default value is *TRUE*. When this parameter is set to *FALSE*, the auto-merge function will be turned off, even if it is enabled in Cube Design.
-
-
-
-### Lookup Table Snapshot {#snapshot}
-
-- `kylin.snapshot.max-mb`: specifies the max size of the snapshot. The default value is 300(M)
-- `kylin.snapshot.max-cache-entry`: The maximum number of snapshots that can be stored in the cache. The default value is 500.
-- `kylin.snapshot.ext.shard-mb`: specifies the size of HBase shard. The default value is 500(M).
-- `kylin.snapshot.ext.local.cache.path`: specifies local cache path, default value is lookup_cache
-- `kylin.snapshot.ext.local.cache.max-size-gb`: specifies local snapshot cache size, default is 200(M)
-
-
-
-### Build Cube {#cube-build}
-
-- `kylin.storage.default`: specifies the default build engine. The default value is 2, which means HBase.
-- `kylin.source.hive.keep-flat-table`: whether to keep the Hive intermediate table after the build job is complete. The default value is *FALSE*
-- `kylin.source.hive.database-for-flat-table`: specifies the name of the Hive database that stores the Hive intermediate table. The default is *default*. Make sure that the user who started the Kylin instance has permission to operate the database.
-- `kylin.source.hive.flat-table-storage-format`: specifies the storage format of the Hive intermediate table. The default value is *SEQUENCEFILE*
-- `kylin.source.hive.flat-table-field-delimiter`: specifies the delimiter of the Hive intermediate table. The default value is *\u001F*
-- `kylin.source.hive.intermediate-table-prefix`: specifies the table name prefix of the Hive intermediate table. The default value is *kylin\_intermediate\_*
-- `kylin.source.hive.redistribute-flat-table`: whether to redistribute the Hive flat table. The default value is *TRUE*
-- `kylin.source.hive.redistribute-column-count`: number of redistributed columns. The default value is *3*
-- `kylin.source.hive.table-dir-create-first`: the default value is *FALSE*
-- `kylin.storage.partition.aggr-spill-enabled`: the default value is *TRUE*
-- `kylin.engine.mr.lib-dir`: specifies the path to the jar package used by the MapReduce job
-- `kylin.engine.mr.reduce-input-mb`: used to estimate the number of Reducers. The default value is 500(MB).
-- `kylin.engine.mr.reduce-count-ratio`: used to estimate the number of Reducers. The default value is 1.0
-- `kylin.engine.mr.min-reducer-number`: specifies the minimum number of Reducers in the MapReduce job. The default is 1
-- `kylin.engine.mr.max-reducer-number`: specifies the maximum number of Reducers in the MapReduce job. The default is 500.
-- `kylin.engine.mr.mapper-input-rows`: specifies the number of rows that each Mapper can handle. The default value is 1000000. If user change this value, it will start more Mapper.
-- `kylin.engine.mr.max-cuboid-stats-calculator-number`: specifies the number of threads used to calculate Cube statistics. The default value is 1
-- `kylin.engine.mr.build-dict-in-reducer`: whether to build the dictionary in the Reduce phase of the build job *Extract Fact Table Distinct Columns*. The default value is `TRUE`
-- `kylin.engine.mr.yarn-check-interval-seconds`: How often the build engine is checked for the status of the Hadoop job. The default value is 10(s)
-- `kylin.engine.mr.use-local-classpath`: whether to use local mapreduce application classpath. The default value is TRUE
-
-
-
-### Dictionary-related {#dict-config}
-
-- `kylin.dictionary.use-forest-trie`: The default value is TRUE
-- `kylin.dictionary.forest-trie-max-mb`: The default value is 500
-- `kylin.dictionary.max-cache-entry`: The default value is 3000
-- `kylin.dictionary.growing-enabled`: The default value is FALSE
-- `kylin.dictionary.append-entry-size`: The default value is 10000000
-- `kylin.dictionary.append-max-versions`: The default value is 3
-- `kylin.dictionary.append-version-ttl`: The default value is 259200000
-- `kylin.dictionary.resuable`: whether to reuse the dictionary. The default value is FALSE
-- `kylin.dictionary.shrunken-from-global-enabled`: whether to reduce the size of global dictionary. The default value is *TRUE*
-
-
-
-### Deal with Ultra-High-Cardinality Columns {#uhc-config}
-
-- `kylin.engine.mr.build-uhc-dict-in-additional-step`: the default value is *FALSE*, set to *TRUE*
-- `kylin.engine.mr.uhc-reducer-count`: the default value is 1, which can be set to 5 to allocate 5 Reducers for each super-high base column.
-
-
-
-### Spark as Build Engine {#spark-cubing}
-
-- `kylin.engine.spark-conf.spark.master`: specifies the Spark operation mode. The default value is *yarn*
-- `kylin.engine.spark-conf.spark.submit.deployMode`: specifies the deployment mode of Spark on YARN. The default value is *cluster*
-- `kylin.engine.spark-conf.spark.yarn.queue`: specifies the Spark resource queue. The default value is *default*
-- `kylin.engine.spark-conf.spark.driver.memory`: specifies the Spark Driver memory The default value is 2G.
-- `kylin.engine.spark-conf.spark.executor.memory`: specifies the Spark Executor memory. The default value is 4G.
-- `kylin.engine.spark-conf.spark.yarn.executor.memoryOverhead`: specifies the size of the Spark Executor heap memory. The default value is 1024(MB).
-- `kylin.engine.spark-conf.spark.executor.cores`: specifies the number of cores available for a single Spark Executor. The default value is 1
-- `kylin.engine.spark-conf.spark.network.timeout`: specifies the Spark network timeout period, 600
-- `kylin.engine.spark-conf.spark.executor.instances`: specifies the number of Spark Executors owned by an Application. The default value is 1
-- `kylin.engine.spark-conf.spark.eventLog.enabled`: whether to record the Spark event. The default value is *TRUE*
-- `kylin.engine.spark-conf.spark.hadoop.dfs.replication`: replication number of HDFS, default is 2
-- `kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress`: whether to compress the output. The default value is *TRUE*
-- `kylin.engine.spark-conf.spark.hadoop.mapreduce.output.fileoutputformat.compress.codec`: specifies Output compression, default is *org.apache.hadoop.io.compress.DefaultCodec*
-- `kylin.engine.spark.rdd-partition-cut-mb`: Kylin uses the size of this parameter to split the partition. The default value is 10 (MB)
-- `kylin.engine.spark.min-partition`: specifies the minimum number of partitions. The default value is 1
-- `kylin.engine.spark.max-partition`: specifies maximum number of partitions, default is 5000
-- `kylin.engine.spark.storage-level`: specifies RDD partition data cache level, default value is *MEMORY_AND_DISK_SER*
-- `kylin.engine.spark-conf-mergedict.spark.executor.memory`: whether to request more memory for merging dictionary.The default value is 6G.
-- `kylin.engine.spark-conf-mergedict.spark.memory.fraction`: specifies the percentage of memory reserved for the system. The default value is 0.2
-
-> Note: For more information, please refer to [Building Cubes with Spark](/docs/tutorial/cube_spark.html).
-
-
-
-### Submit Spark jobs via Livy {#livy-submit-spark-job}
-
-- `kylin.engine.livy-conf.livy-enabled`: whether to enable Livy as submit Spark job service. The default value is *FALSE*
-- `kylin.engine.livy-conf.livy-url`: specifies the URL of Livy. Such as *http://127.0.0.1:8998*
-- `kylin.engine.livy-conf.livy-key.*`: specifies the name-key configuration of Livy. Such as *kylin.engine.livy-conf.livy-key.name=kylin-livy-1*
-- `kylin.engine.livy-conf.livy-arr.*`: specifies the array type configuration of Livy. Separated by commas. Such as *kylin.engine.livy-conf.livy-arr.jars=hdfs://your_self_path/hbase-common-1.4.8.jar,hdfs://your_self_path/hbase-server-1.4.8.jar,hdfs://your_self_path/hbase-client-1.4.8.jar*
-- `kylin.engine.livy-conf.livy-map.*`: specifies the Spark configuration properties. Such as *kylin.engine.livy-conf.livy-map.spark.executor.instances=10*
-
-> Note: For more information, please refer to [Apache Livy Rest API](http://livy.incubator.apache.org/docs/latest/rest-api.html).
-
-
-### Spark Dynamic Allocation {#dynamic-allocation}
-
-- `kylin.engine.spark-conf.spark.shuffle.service.enabled`: whether to enable shuffle service
-- `kylin.engine.spark-conf.spark.dynamicAllocation.enabled`: whether to enable Spark Dynamic Allocation
-- `kylin.engine.spark-conf.spark.dynamicAllocation.initialExecutors`: specifies the initial number of Executors
-- `kylin.engine.spark-conf.spark.dynamicAllocation.minExecutors`: specifies the minimum number of Executors retained
-- `kylin.engine.spark-conf.spark.dynamicAllocation.maxExecutors`: specifies the maximum number of Executors applied for
-- `kylin.engine.spark-conf.spark.dynamicAllocation.executorIdleTimeout`: specifies the threshold of Executor being removed after being idle. The default value is 60(s)
-
-> Note: For more information, please refer to the official documentation: [Dynamic Resource Allocation](http://spark.apache.org/docs/1.6.2/job-scheduling.html#dynamic-resource-allocation).
-
-
-
-### Job-related {#job-config}
-
-- `kylin.job.log-dir`: the default value is */tmp/kylin/logs*
-- `kylin.job.allow-empty-segment`: whether tolerant data source is empty. The default value is *TRUE*
-- `kylin.job.max-concurrent-jobs`: specifies maximum build concurrency, default is 10
-- `kylin.job.retry`: specifies retry times after the job is failed. The default value is 0
-- `kylin.job.retry-interval`: specifies retry interval in milliseconds. The default value is 30000
-- `kylin.job.scheduler.priority-considered`: whether to consider the job priority. The default value is FALSE
-- `kylin.job.scheduler.priority-bar-fetch-from-queue`: specifies the time interval for getting jobs from the priority queue. The default value is 20(s)
-- `kylin.job.scheduler.poll-interval-second`: The time interval for getting the job from the queue. The default value is 30(s)
-- `kylin.job.error-record-threshold`: specifies the threshold for the job to throw an error message. The default value is 0
-- `kylin.job.cube-auto-ready-enabled`: whether to enable Cube automatically after the build is complete. The default value is *TRUE*
-- `kylin.cube.max-building-segments`: specifies the maximum number of building job for the one Cube. The default value is 10
-
-
-
-### Enable Email Notification {#email-notification}
-
-- `kylin.job.notification-enabled`: whether to notify the email when the job succeeds or fails. The default value is *FALSE*
-- `kylin.job.notification-mail-enable-starttls`:# whether to enable starttls. The default value is *FALSE*
-- `kylin.job.notification-mail-host`: specifies the SMTP server address of the mail
-- `kylin.job.notification-mail-port`: specifies the SMTP server port of the mail. The default value is 25
-- `kylin.job.notification-mail-username`: specifies the login user name of the mail
-- `kylin.job.notification-mail-password`: specifies the username and password of the email
-- `kylin.job.notification-mail-sender`: specifies the email address of the email
-- `kylin.job.notification-admin-emails`: specifies the administrator's mailbox for email notifications
-
-
-
-### Enable Cube Planner {#cube-planner}
-
-- `kylin.cube.cubeplanner.enabled`: the default value is *TRUE*
-- `kylin.server.query-metrics2-enabled`: the default value is *TRUE*
-- `kylin.metrics.reporter-query-enabled`: the default value is *TRUE*
-- `kylin.metrics.reporter-job-enabled`: the default value is *TRUE*
-- `kylin.metrics.monitor-enabled`: the default value is *TRUE*
-- `kylin.cube.cubeplanner.enabled`: whether to enable Cube Planner, The default value is *TRUE*
-- `kylin.cube.cubeplanner.enabled-for-existing-cube`: whether to enable Cube Planner for the existing Cube. The default value is *TRUE*
-- `kylin.cube.cubeplanner.algorithm-threshold-greedy`: the default value is 8
-- `kylin.cube.cubeplanner.expansion-threshold`: the default value is 15.0
-- `kylin.cube.cubeplanner.recommend-cache-max-size`: the default value is 200
-- `kylin.cube.cubeplanner.query-uncertainty-ratio`: the default value is 0.1
-- `kylin.cube.cubeplanner.bpus-min-benefit-ratio`: the default value is 0.01
-- `kylin.cube.cubeplanner.algorithm-threshold-genetic`: the default value is 23
-
-> Note: For more information, please refer to [Using Cube Planner](/docs/tutorial/use_cube_planner.html).
-
-
-
-### HBase Storage {#hbase-config}
-
-- `kylin.storage.hbase.table-name-prefix`: specifies the prefix of HTable. The default value is *KYLIN\_*
-- `kylin.storage.hbase.namespace`: specifies the default namespace of HBase Storage. The default value is *default*
-- `kylin.storage.hbase.coprocessor-local-jar`: specifies jar package related to HBase coprocessor
-- `kylin.storage.hbase.coprocessor-mem-gb`: specifies the HBase coprocessor memory. The default value is 3.0(GB).
-- `kylin.storage.hbase.run-local-coprocessor`: whether to run the local HBase coprocessor. The default value is *FALSE*
-- `kylin.storage.hbase.coprocessor-timeout-seconds`: specifies the timeout period. The default value is 0
-- `kylin.storage.hbase.region-cut-gb`: specifies the size of a single Region, default is 5.0
-- `kylin.storage.hbase.min-region-count`: specifies the minimum number of regions. The default value is 1
-- `kylin.storage.hbase.max-region-count`: specifies the maximum number of Regions. The default value is 500
-- `kylin.storage.hbase.hfile-size-gb`: specifies the HFile size. The default value is 2.0(GB)
-- `kylin.storage.hbase.max-scan-result-bytes`: specifies the maximum value of the scan return. The default value is 5242880 (byte), which is 5 (MB).
-- `kylin.storage.hbase.compression-codec`: whether it is compressed. The default value is *none*, that is, compression is not enabled
-- `kylin.storage.hbase.rowkey-encoding`: specifies the encoding method of Rowkey. The default value is *FAST_DIFF*
-- `kylin.storage.hbase.block-size-bytes`: the default value is 1048576
-- `kylin.storage.hbase.small-family-block-size-bytes`: specifies the block size. The default value is 65536 (byte), which is 64 (KB).
-- `kylin.storage.hbase.owner-tag`: specifies the owner of the Kylin platform. The default value is whoami@kylin.apache.org
-- `kylin.storage.hbase.endpoint-compress-result`: whether to return the compression result. The default value is TRUE
-- `kylin.storage.hbase.max-hconnection-threads`: specifies the maximum number of connection threads. The default value is 2048.
-- `kylin.storage.hbase.core-hconnection-threads`: specifies the number of core connection threads. The default value is 2048.
-- `kylin.storage.hbase.hconnection-threads-alive-seconds`: specifies the thread lifetime. The default value is 60.
-- `kylin.storage.hbase.replication-scope`: specifies the cluster replication range. The default value is 0
-- `kylin.storage.hbase.scan-cache-rows`: specifies the number of scan cache lines. The default value is 1024.
-
-
-
-### Secondary Hbase Storage {#secondary-hbase}
-
-Kylin support secondary hbase storage and made the kylin cluster can query cube data from the old hbase cluster during the cluster migration.
-
-- `kylin.secondary.storage.url`: specifies the secondary hbase cluster and metadata path. Such as *kylin.secondary.storage.url=hostname:kylin_metadata@hbase,hbase.zookeeper.quorum=hostname:11000,zookeeper.znode.parent=/hbase/*. if there are other parameters, they can be added in the form of <key> = <value>.
-
-> Note: For more information, please refer to [KYLIN-4175](https://issues.apache.org/jira/browse/KYLIN-4175).
-
-
-
-### Job Output {#job-output}
-
-In order to avoid job output content is too large, user could set the max length of output.
-
-- `kylin.job.execute-output.max-size`: The max length of job output. The default value is 10484760.
-- `kylin.engine.spark.output.max-size`: The max length of spark job output. The default value is 10484760.
-
-
-
-### Enable Compression {#compress-config}
-
-Kylin does not enable Enable Compression by default. Unsupported compression algorithms can hinder Kylin's build jobs, but a suitable compression algorithm can reduce storage overhead and network overhead and improve overall system operation efficiency.
-Kylin can use three types of compression, HBase table compression, Hive output compression, and MapReduce job output compression.
-
-> *Note*: The compression settings will not take effect until the Kylin instance is restarted.
-
-* HBase table compression
-
-This compression is configured by `kylin.storage.hbase.compression-codec` in `kyiln.properties`. Optional values include `none`, `snappy`, `lzo`, `gzip` and `lz4`. The default value is none, which means no data is compressed.
-
-> *Note*: Before modifying the compression algorithm, make sure userr HBase cluster supports the selected compression algorithm.
-
-
-* Hive output compression
-
-This compression is configured by `kylin_hive_conf.xml`. The default configuration is empty, which means that the default configuration of Hive is used directly. If user want to override the configuration, add (or replace) the following properties in `kylin_hive_conf.xml`. Take SNAPPY compression as an example:
-
-```xml
-<property>
-<name>mapreduce.map.output.compress.codec</name>
-<value>org.apache.hadoop.io.compress.SnappyCodec</value>
-<description></description>
-</property>
-<property>
-<name>mapreduce.output.fileoutputformat.compress.codec</name>
-<value>org.apache.hadoop.io.compress.SnappyCodec</value>
-<description></description>
-</property>
-```
-
-* MapReduce job output compression
-
-This compression is configured via `kylin_job_conf.xml` and `kylin_job_conf_inmem.xml`. The default is empty, which uses the default configuration of MapReduce. If user want to override the configuration, add (or replace) the following properties in `kylin_job_conf.xml` and `kylin_job_conf_inmem.xml`. Take SNAPPY compression as an example:
-
-```xml
-<property>
-<name>mapreduce.map.output.compress.codec</name>
-<value>org.apache.hadoop.io.compress.SnappyCodec</value>
-<description></description>
-</property>
-<property>
-<name>mapreduce.output.fileoutputformat.compress.codec</name>
-<value>org.apache.hadoop.io.compress.SnappyCodec</value>
-<description></description>
-</property>
-```
-
-
-
-### Real-time OLAP {#realtime-olap}
-#### Global level config
-
-- `kylin.stream.job.dfs.block.size`: specifies the HDFS block size of the streaming Base Cuboid job using. The default value is *16M*.
-- `kylin.stream.index.path`: specifies the local path to store segment cache files(including fragment and checkpoint files). The default value is *stream_index*.
-- `kylin.stream.node`: specifies the node of coordinator/receiver. Value should be `hostname:port` or `port`. If set to `port`, Kylin will complete hostname automatically. When Kylin process started, it will register it into metadata. The default value is *null*.
-- `kylin.stream.metadata.store.type`: specifies the position of metadata store. The default value is *zk*. This entry is trivial because it has only one option.
-- `kylin.stream.receiver.use-threads-per-query`: specifies the threads number that each query use. The default value is *8*.
-
-#### Cube level config
-
-- `kylin.stream.index.maxrows`: specifies the maximum number of the aggregated event keep in JVM heap. The default value is *50000*. Try to advance it if you have enough heap size.
-- `kylin.stream.cube-num-of-consumer-tasks`: specifies the number of replica sets that share the whole topic partition. It affects how many partitions will be assigned to different replica sets. The default value is *3*.
-- `kylin.stream.segment.retention.policy`: specifies the strategy to process local segment cache when segment become *IMMUTABLE*. Optional values include `purge` and `fullBuild`. `purge` means when the segment become *IMMUTABLE*, it will be deleted. `fullBuild` means when the segment become *IMMUTABLE*, it will be uploaded to HDFS. The default value is *fullBuild*.
-- `kylin.stream.build.additional.cuboids`: whether to build additional Cuboids. The additional Cuboids mean the aggregation of Mandatory Dimensions that chosen in *Cube Advanced Setting* page. The default value is *false*. Only build Base Cuboid by default. Try to enable it if you care the QPS and most query pattern can be foresaw.
-- `kylin.stream.cube.window`: specifies the length of duration of each segment, value in seconds. The default value is *3600*. Please check detail at[deep-dive-real-time-olap](http://kylin.apache.org/blog/2019/07/01/deep-dive-real-time-olap/).
-- `kylin.stream.cube.duration`: specifies the wait time that a segment's status changes from active to IMMUTABLE, value in seconds. The default value is *7200*. Please check detail at[deep-dive-real-time-olap](http://kylin.apache.org/blog/2019/07/01/deep-dive-real-time-olap/).
-- `kylin.stream.cube.duration.max`: specifies the maximum duration that segment can keep active, value in seconds. The default value is *43200*. Please check detail at[deep-dive-real-time-olap](http://kylin.apache.org/blog/2019/07/01/deep-dive-real-time-olap/).
-- `kylin.stream.checkpoint.file.max.num`: specifies the maximum number of checkpoint file for each cube. The default value is *5*.
-- `kylin.stream.index.checkpoint.intervals`: specifies the time interval between setting two checkpoints. The default value is *300*.
-- `kylin.stream.immutable.segments.max.num`: specifies the maximum number of the IMMUTABLE segment in each Cube of the current streaming receiver, if exceed, consumption of current topic will be paused. The default value is *100*.
-- `kylin.stream.consume.offsets.latest`: whether to consume from the latest offset or the earliest offset. The default value is *true*.
-
-#### Advanced config
-
-- `kylin.stream.assigner`: specifies the implementation class which used to assign the topic partition to different replica sets. The class should be the implementation class of `org.apache.kylin.stream.coordinator.assign.Assigner`. The default value is *DefaultAssigner*.
-- `kylin.stream.coordinator.client.timeout.millsecond`: specifies the connection timeout of the coordinator client. The default value is *5000*.
-- `kylin.stream.receiver.client.timeout.millsecond`: specifies the connection timeout of the receiver client. The default value is *5000*.
-- `kylin.stream.receiver.http.max.threads`: specifies the maximum connection threads of the receiver. The default value is *200*.
-- `kylin.stream.receiver.http.min.threads`: specifies the minimum connection threads of the receiver. The default value is *10*.
-- `kylin.stream.receiver.query-core-threads`: specifies the number of query threads be used for the current streaming receiver. The default value is *50*.
-- `kylin.stream.receiver.query-max-threads`: specifies the maximum number of query threads be used for the current streaming receiver. The default value is *200*.
-- `kylin.stream.segment-max-fragments`: specifies the maximum number of fragments that each segment keep. The default value is *50*.
-- `kylin.stream.segment-min-fragments`: specifies the minimum number of fragments that each segment keep. The default value is *15*.
-- `kylin.stream.max-fragment-size-mb`: specifies the maximum size of each fragment. The default value is *300*.
-- `kylin.stream.fragments-auto-merge-enable`: whether to enable fragments auto merge in streaming receiver side. The default value is *true*.
-- `kylin.stream.metrics.option`: specifies how to report metrics in streaming receiver side, option value are csv/console/jmx.
-- `kylin.stream.event.timezone`: specifies which timezone should derived time column like `HOUR_START`/`DAY_START` used.
-- `kylin.stream.auto-resubmit-after-discard-enabled`: whether to resubmit new building job automatically when finding previous job be discarded by user.
-
-> Note: For step by step tutorial, please refer to the [Real-time OLAP](/docs/tutorial/realtime_olap.html).
-
-### Storage Clean up Configuration {#storage-clean-up-configuration}
-
-This section introduces Kylin storage clean up related configuration.
-
-
-
-### Storage-clean-up-related {#storage-clean-up-config}
-
-- `kylin.storage.clean-after-delete-operation`: whether to clean segment data in HBase and HDFS. The default value is FALSE.
-
-
-
-### Query Configuration {#kylin-query}
-
-This section introduces Kylin query related configuration.
-
-
-
-### Query-related {#query-config}
-
-- `kylin.query.skip-empty-segments`: whether to skip empty segments when querying. The default value is *TRUE*
-- `kylin.query.large-query-threshold`: specifies the maximum number of rows returned. The default value is 1000000.
-- `kylin.query.security-enabled`: whether to check the ACL when querying. The default value is *TRUE*
-- `kylin.query.security.table-acl-enabled`: whether to check the ACL of the corresponding table when querying. The default value is *TRUE*
-- `kylin.query.calcite.extras-props.conformance`: whether to strictly parsed. The default value is *LENIENT*
-- `kylin.query.calcite.extras-props.caseSensitive`: whether is case sensitive. The default value is *TRUE*
-- `kylin.query.calcite.extras-props.unquotedCasing`: optional values include `UNCHANGED`, `TO_UPPER` and `TO_LOWER`. The default value is *TO_UPPER*, that is, all uppercase
-- `kylin.query.calcite.extras-props.quoting`: whether to add quotes, Optional values include `DOUBLE_QUOTE`, `BACK_TICK` and `BRACKET`. The default value is *DOUBLE_QUOTE*
-- `kylin.query.statement-cache-max-num`: specifies the maximum number of cached PreparedStatements. The default value is 50000
-- `kylin.query.statement-cache-max-num-per-key`: specifies the maximum number of PreparedStatements per key cache. The default value is 50.
-- `kylin.query.enable-dict-enumerator`: whether to enable the dictionary enumerator. The default value is *FALSE*
-- `kylin.query.enable-dynamic-column`: whether to enable dynamic columns. The default value is *FALSE*, set to *TRUE* to query the number of rows in a column that do not contain NULL
-
-
-
-### Fuzzy Query {#fuzzy}
-
-- `kylin.storage.hbase.max-fuzzykey-scan`: specifies the threshold for the scanned fuzzy key. If the value is exceeded, the fuzzy key will not be scanned. The default value is 200.
-- `kylin.storage.hbase.max-fuzzykey-scan-split`: split the large fuzzy key set to reduce the number of fuzzy keys per scan. The default value is 1
-- `kylin.storage.hbase.max-visit-scanrange`: the default value is 1000000
-
-
-
-### Query Cache {#cache-config}
-
-- `kylin.query.cache-enabled`: whether to enable caching. The default value is TRUE
-- `kylin.query.cache-threshold-duration`: the query duration exceeding the threshold is saved in the cache. The default value is 2000 (ms).
-- `kylin.query.cache-threshold-scan-count`: the row count scanned in the query exceeding the threshold is saved in the cache. The default value is 10240 (rows).
-- `kylin.query.cache-threshold-scan-bytes`: the bytes scanned in the query exceeding the threshold is saved in the cache. The default value is 1048576 (byte).
-
-
-
-### Query Limits {#query-limit}
-
-- `kylin.query.timeout-seconds`: specifies the query timeout in seconds. The default value is 0, that is, no timeout limit on query. If the value is less than 60, it will set to 60 seconds.
-- `kylin.query.timeout-seconds-coefficient`: specifies the coefficient of the query timeout seconds. The default value is 0.5.
-- `kylin.query.max-scan-bytes`: specifies the maximum bytes scanned by the query. The default value is 0, that is, there is no limit.
-- `kylin.storage.partition.max-scan-bytes`: specifies the maximum number of bytes for the query scan. The default value is 3221225472 (bytes), which is 3GB.
-- `kylin.query.max-return-rows`: specifies the maximum number of rows returned by the query. The default value is 5000000.
-
-
-
-### Bad Query {#bad-query}
-
-The value of `kylin.query.timeout-seconds` is greater than 60 or equals 0, the max value of `kylin.query.timeout-seconds-coefficient` is the upper limit of double. The result of multiplying two properties is the interval time of detecting bad query, if it equals 0, it will be set to 60 seconds, the max value of it is the upper limit of int.
-
-- `kylin.query.badquery-stacktrace-depth`: specifies the depth of stack trace. The default value is 10.
-- `kylin.query.badquery-history-number`: specifies the showing number of bad query history. The default value is 50.
-- `kylin.query.badquery-alerting-seconds`: The default value is 90, if the time of running is greater than the value of this property, it will print the log of query firstly, including (duration, project, thread, user, query id). Whether to save the recent query, it depends on another property. Secondly, record the stack log, the depth of log depend on another property, so as to the analysis later
-- `kylin.query.badquery-persistent-enabled`: The default value is true, it will save the recent bad query, and cannot override in Cube-level
-
-
-### Query Pushdown {#query-pushdown}
-
-- `kylin.query.pushdown.runner-class-name=org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl`: whether to enable query pushdown
-- `kylin.query.pushdown.jdbc.url`: specifies JDBC URL
-- `kylin.query.pushdown.jdbc.driver`: specifies JDBC driver class name. The default value is *org.apache.hive.jdbc.HiveDriver*
-- `kylin.query.pushdown.jdbc.username`: specifies the Username of the JDBC database. The default value is *hive*
-- `kylin.query.pushdown.jdbc.password`: specifies JDBC password for the database. The default value is
-- `kylin.query.pushdown.jdbc.pool-max-total`: specifies the maximum number of connections to the JDBC connection pool. The default value is 8.
-- `kylin.query.pushdown.jdbc.pool-max-idle`: specifies the maximum number of idle connections for the JDBC connection pool. The default value is 8.
-- `kylin.query.pushdown.jdbc.pool-min-idle`: the default value is 0
-- `kylin.query.pushdown.update-enabled`: specifies whether to enable update in Query Pushdown. The default value is *FALSE*
-- `kylin.query.pushdown.cache-enabled`: whether to enable the cache of the pushdown query to improve the query efficiency of the same query. The default value is *FALSE*
-
-> Note: For more information, please refer to [Query Pushdown](/docs/tutorial/query_pushdown.html)
-
-
-
-### Query rewriting {#convert-sql}
-
-- `kylin.query.force-limit`: this parameter achieves the purpose of shortening the query duration by forcing a LIMIT clause for the select * statement. The default value is *-1*, and the parameter value is set to a positive integer, such as 1000, the value will be applied to the LIMIT clause, and the query will eventually be converted to select * from fact_table limit 1000
-- `kylin.storage.limit-push-down-enabled`: the default value is *TRUE*, set to *FALSE* to close the limit-pushdown of storage layer
-- `kylin.query.flat-filter-max-children`: specifies the maximum number of filters when flatting filter. The default value is 500000
-
-
-
-### Collect Query Metrics to JMX {#jmx-metrics}
-
-- `kylin.server.query-metrics-enabled`: the default value is *FALSE*, set to *TRUE* to collect query metrics to JMX
-
-> Note: For more information, please refer to [JMX](https://www.oracle.com/technetwork/java/javase/tech/javamanagement-140525.html)
-
-
-
-### Collect Query Metrics to dropwizard {#dropwizard-metrics}
-
-- `kylin.server.query-metrics2-enabled`: the default value is *FALSE*, set to *TRUE* to collect query metrics into dropwizard
-
-> Note: For more information, please refer to [dropwizard](https://metrics.dropwizard.io/4.0.0/)
-
-
-
-### Security Configuration {#kylin-security}
-
-This section introduces Kylin security-related configuration.
-
-
-
-### User name and password {#user-name-and-password}
-
-Please check the document: [How to add new user or change the default password](https://cwiki.apache.org/confluence/display/KYLIN/How+to+add+new+user+or+change+the+default+password)
-
-### Integrated LDAP for SSO {#ldap-sso}
-
-- `kylin.security.profile`: specifies the way of security authentication, optional values include `ldap`, `testing`, `saml`, it should be set to `ldap` when integrating LDAP for SSO
-- `kylin.security.ldap.connection-server`: specifies LDAP server, such as *ldap://ldap_server:389*
-- `kylin.security.ldap.connection-username`: specifies LDAP username
-- `kylin.security.ldap.connection-password`: specifies LDAP password
-- `kylin.security.ldap.user-search-base`: specifies the scope of users synced to Kylin
-- `kylin.security.ldap.user-search-pattern`: specifies the username for the login verification match
-- `kylin.security.ldap.user-group-search-base`: specifies the scope of the user group synchronized to Kylin
-- `kylin.security.ldap.user-group-search-filter`: specifies the type of user synced to Kylin
-- `kylin.security.ldap.service-search-base`: need to be specifies when a service account is required to access Kylin
-- `kylin.security.ldap.service-search-pattern`: need to be specifies when a service account is required to access Kylin
-- `kylin.security.ldap.service-group-search-base`: need to be specifies when a service account is required to access Kylin
-- `kylin.security.acl.admin-role`: map an LDAP group to an admin role (group name case sensitive)
-- `kylin.server.auth-user-cache.expire-seconds`: specifies LDAP user information cache time, default is 300(s)
-- `kylin.server.auth-user-cache.max-entries`: specifies maximum number of LDAP users, default is 100
-
-### Integrate with Apache Ranger {#ranger}
-
-- `kylin.server.external-acl-provider=org.apache.ranger.authorization.kylin.authorizer.RangerKylinAuthorizer`
-
-> Note: For more information, please refer to [How to integrate the Kylin plugin in the installation documentation for Ranger](https://cwiki.apache.org/confluence/display/RANGER/Kylin+Plugin)
-
-
-
-### Enable ZooKeeper ACL {#zookeeper-acl}
-
-- `kylin.env.zookeeper-acl-enabled`: Enable ZooKeeper ACL to prevent unauthorized users from accessing the Znode or reducing the risk of bad operations resulting from this. The default value is *FALSE*
-- `kylin.env.zookeeper.zk-auth`: use username: password as the ACL identifier. The default value is *digest:ADMIN:KYLIN*
-- `kylin.env.zookeeper.zk-acl`: Use a single ID as the ACL identifier. The default value is *world:anyone:rwcda*, *anyone* for anyone
-
-### Distributed query cache with Memcached {#distributed-cache}
-
-From v2.6.0, Kylin can use Memcached as the distributed cache, and also there are some improvements on the cache policy ([KYLIN-2895](https://issues.apache.org/jira/browse/KYLIN-2895)). To enable these new features, you need to do the following steps:
-
-1. Install Memcached (latest v1.5.12) on 1 or multiple nodes; You can install it on all Kylin nodes if resource is enough;
-
-2. Modify the applicationContext.xml under $KYLIN_HOME/tomcat/webapps/kylin/WEB-INF/classes directory, comment the following code:
-{% highlight Groff markup %}
-<bean id="ehcache"
- class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean"
- p:configLocation="classpath:ehcache-test.xml" p:shared="true"/>
-
-<bean id="cacheManager" class="org.springframework.cache.ehcache.EhCacheCacheManager"
- p:cacheManager-ref="ehcache"/>
-{% endhighlight %}
-Uncomment the following code:
-{% highlight Groff markup %}
-<bean id="ehcache" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean"
- p:configLocation="classpath:ehcache-test.xml" p:shared="true"/>
-
-<bean id="remoteCacheManager" class="org.apache.kylin.cache.cachemanager.MemcachedCacheManager" />
-<bean id="localCacheManager" class="org.apache.kylin.cache.cachemanager.InstrumentedEhCacheCacheManager"
- p:cacheManager-ref="ehcache"/>
-<bean id="cacheManager" class="org.apache.kylin.cache.cachemanager.RemoteLocalFailOverCacheManager" />
-
-<bean id="memcachedCacheConfig" class="org.apache.kylin.cache.memcached.MemcachedCacheConfig">
- <property name="timeout" value="500" />
- <property name="hosts" value="${kylin.cache.memcached.hosts}" />
-</bean>
-{% endhighlight %}
-The value of `${kylin.cache.memcached.hosts}` in applicationContext.xml is from the value of `kylin.cache.memcached.hosts` in conf/kylin.properties.
-
-3.Add the following parameters to `conf/kylin.properties`:
-{% highlight Groff markup %}
-kylin.query.cache-enabled=true
-kylin.query.lazy-query-enabled=true
-kylin.query.cache-signature-enabled=true
-kylin.query.segment-cache-enabled=true
-kylin.cache.memcached.hosts=memcached1:11211,memcached2:11211,memcached3:11211
-{% endhighlight %}
-
-- `kylin.query.cache-enabled` controls the on-off of query cache, its default value is `true`.
-- `kylin.query.lazy-query-enabled` : whether to lazily answer the queries that be sent repeatedly in a short time (hold it until the previous query be returned, and then reuse the result); The default value is `false`.
-- `kylin.query.cache-signature-enabled` : whether to use the signature of a query to determine the cache's validity. The signature is calculated by the cube/hybrid list of the project, their last build time and other information (at the moment when cache is persisted); It's default value is `false`, highly recommend to set it to `true`.
-- `kylin.query.segment-cache-enabled` : whether to cache the segment level returned data (from HBase storage) into Memcached. This feature is mainly for the cube that built very frequently (e.g, streaming cube, whose last build time always changed a couple minutes, the whole SQL statement level cache is very likely be cleaned; in this case, the by-segment cache can reduce the I/O). This only works when Memcached configured, the default value is `false`.
-- `kylin.cache.memcached.hosts`: a list of memcached node and port, connected with comma.
\ No newline at end of file
+All configuration of Kylin 4.x lists in Apache Kylin Wiki:
+[Apache Kylin4.0 Configuration](https://cwiki.apache.org/confluence/display/KYLIN/Kylin+4.X+Configuration)
\ No newline at end of file
diff --git a/website/_docs40/install/deploy_without_hadoop.cn.md b/website/_docs/install/deploy_without_hadoop.cn.md
similarity index 99%
rename from website/_docs40/install/deploy_without_hadoop.cn.md
rename to website/_docs/install/deploy_without_hadoop.cn.md
index c7f1252..4253ae3 100644
--- a/website/_docs40/install/deploy_without_hadoop.cn.md
+++ b/website/_docs/install/deploy_without_hadoop.cn.md
@@ -1,8 +1,8 @@
---
-layout: docs40-cn
+layout: docs-cn
title: "无 Hadoop 环境部署 Kylin4"
categories: install
-permalink: /cn/docs40/install/deploy_without_hadoop.html
+permalink: /cn/docs/install/deploy_without_hadoop.html
---
相比于 Kylin 3.x，Kylin 4.0 实现了全新 spark 构建引擎和 parquet 存储，使 kylin 不依赖 hadoop 环境部署成为可能。与在 AWS EMR 之上部署 Kylin 3.x 相比，直接在 AWS EC2 实例上部署 Kylin 4.0 存在以下优势：
diff --git a/website/_docs40/install/deploy_without_hadoop.md b/website/_docs/install/deploy_without_hadoop.md
similarity index 99%
rename from website/_docs40/install/deploy_without_hadoop.md
rename to website/_docs/install/deploy_without_hadoop.md
index 779176d..1196e4f 100644
--- a/website/_docs40/install/deploy_without_hadoop.md
+++ b/website/_docs/install/deploy_without_hadoop.md
@@ -1,8 +1,8 @@
---
-layout: docs40
+layout: docs
title: "Deploy kylin on AWS EC2 without hadoop"
categories: install
-permalink: /docs40/install/deploy_without_hadoop.html
+permalink: /docs/install/deploy_without_hadoop.html
---
Compared with Kylin 3.x, Kylin 4.0 implements a new Spark build engine and parquet storage, making it possible for Kylin to deploy without Hadoop environment. Compared with deploying Kylin 3.x on AWS EMR, deploying kylin4 directly on AWS EC2 instances has the following advantages:
diff --git a/website/_docs/install/index.cn.md b/website/_docs/install/index.cn.md
index 0117d5a..cd2adb9 100644
--- a/website/_docs/install/index.cn.md
+++ b/website/_docs/install/index.cn.md
@@ -7,111 +7,100 @@ permalink: /cn/docs/install/index.html
### 软件要求
-* Hadoop: 2.7+, 3.1+ (since v2.5)
+* Hadoop: cdh5.x, cdh6.x, hdp2.x, EMR5.x, EMR6.x, HDI4.x
* Hive: 0.13 - 1.2.1+
-* HBase: 1.1+, 2.0 (since v2.5)
-* Spark (可选) 2.3.0+
-* Kafka (可选) 1.0.0+ (since v2.5)
-* JDK: 1.8+ (since v2.5)
+* Spark: 2.4.7
+* Mysql: 5.1.17及以上
+* JDK: 1.8+
* OS: Linux only, CentOS 6.5+ or Ubuntu 16.0.4+
-在 Hortonworks HDP 2.2-2.6 and 3.0, Cloudera CDH 5.7-5.11 and 6.0, AWS EMR 5.7-5.10, Azure HDInsight 3.5-3.6 上测试通过。
+在 Hortonworks HDP2.4, Cloudera CDH 5.7 and 6.3.2, AWS EMR 5.31 and 6.0, Azure HDInsight 4.0 上测试通过。
我们建议您使用集成的 sandbox 来试用 Kylin 或进行开发，比如 [HDP sandbox](http://hortonworks.com/products/hortonworks-sandbox/)，且要保证其有至少 10 GB 内存。在配置沙箱时，我们推荐您使用 Bridged Adapter 模型替代 NAT 模型。
-
### 硬件要求
运行 Kylin 的服务器的最低配置为 4 core CPU，16 GB 内存和 100 GB 磁盘。 对于高负载的场景，建议使用 24 core CPU，64 GB 内存或更高的配置。
### Hadoop 环境
-Kylin 依赖于 Hadoop 集群处理大量的数据集。您需要准备一个配置好 HDFS，YARN，MapReduce，Hive， HBase，Zookeeper 和其他服务的 Hadoop 集群供 Kylin 运行。
-Kylin 可以在 Hadoop 集群的任意节点上启动。方便起见，您可以在 master 节点上运行 Kylin。但为了更好的稳定性，我们建议您将 Kylin 部署在一个干净的 Hadoop client 节点上，该节点上 Hive，HBase，HDFS 等命令行已安装好且 client 配置（如 `core-site.xml`，`hive-site.xml`，`hbase-site.xml`及其他）也已经合理的配置且其可以自动和其它节点同步。
-
-运行 Kylin 的 Linux 账户要有访问 Hadoop 集群的权限，包括创建/写入 HDFS 文件夹，Hive 表， HBase 表和提交 MapReduce 任务的权限。
+Kylin 依赖于 Hadoop 集群处理大量的数据集。您需要准备一个配置好 HDFS, YARN, Hive, Zookeeper, Spark以及你可能需要的其他服务的 Hadoop 集群供 Kylin 运行。
+Kylin 可以在 Hadoop 集群的任意节点上启动。方便起见，您可以在 master 节点上运行 Kylin。但为了更好的稳定性，我们建议您将 Kylin 部署在一个干净的 Hadoop client 节点上，该节点上 Hive，HDFS 等命令行已安装好且 client 配置（如 `core-site.xml`，`hive-site.xml`及其他）也已经合理的配置且其可以自动和其它节点同步。
+运行 Kylin 的 Linux 账户要有访问 Hadoop 集群的权限，包括创建/写入 HDFS 文件夹，Hive 表的权限。
### Kylin 安装
-- 从 [Apache Kylin下载网站](https://kylin.apache.org/download/) 下载一个适用于您 Hadoop 版本的二进制文件。例如，适用于 HBase 1.x 的 Kylin 2.5.0 可通过如下命令行下载得到：
+- 从 [Apache Kylin下载网站](https://kylin.apache.org/download/) 下载一个 Apache Kylin 4.0 的二进制文件。可通过如下命令行下载得到：
```shell
cd /usr/local/
-wget http://mirror.bit.edu.cn/apache/kylin/apache-kylin-2.5.0/apache-kylin-2.5.0-bin-hbase1x.tar.gz
+wget http://mirror.bit.edu.cn/apache/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin.tar.gz
```
- 解压 tar 包，配置环境变量 `$KYLIN_HOME` 指向 Kylin 文件夹。
```shell
-tar -zxvf apache-kylin-2.5.0-bin-hbase1x.tar.gz
-cd apache-kylin-2.5.0-bin-hbase1x
+tar -zxvf apache-kylin-4.0.0-bin.tar.gz
+cd apache-kylin-4.0.0-bin
export KYLIN_HOME=`pwd`
```
-- 从 v2.6.1 开始， Kylin 不再包含 Spark 二进制包; 您需要另外下载 Spark，然后设置 `SPARK_HOME` 系统变量到 Spark 安装目录：
+- 使用脚本下载spark:
```shell
-export SPARK_HOME=/path/to/spark
+$KYLIN_HOME/bin/download-spark.sh
```
-或者使用脚本下载:
+或者配置 SPARK_HOME 指向环境中的 spark2.4.7 的路径。
+
+- 配置 Mysql 元数据
+
+Kylin 4.0 使用 Mysql 作为元数据存储，需要在 kylin.properties 中做如下配置：
```shell
-$KYLIN_HOME/bin/download-spark.sh
+kylin.metadata.url=kylin_metadata@jdbc,driverClassName=com.mysql.jdbc.Driver,url=jdbc:mysql://localhost:3306/kylin_test,username=,password=
+kylin.env.zookeeper-connect-string=ip
```
+你需要修改其中的 Mysql 用户名和密码，以及存储元数据的database和table。并将 mysql jdbc connector 放在 `$KYLIN_HOME/ext` 目录下，没有该目录时请自行创建。
+请参考 [配置 Mysql 为 Metastore](/_docs40/tutorial/mysql_metastore.html) 了解 Mysql 作为 Metastore 的详细配置。
+
### Kylin tarball 目录
* `bin`: shell 脚本，用于启动／停止 Kylin，备份／恢复 Kylin 元数据，以及一些检查端口、获取 Hive/HBase 依赖的方法等；
* `conf`: Hadoop 任务的 XML 配置文件，这些文件的作用可参考[配置页面](/docs/install/configuration.html)
-* `lib`: 供外面应用使用的 jar 文件，例如 Hadoop 任务 jar, JDBC 驱动, HBase coprocessor 等.
+* `lib`: 供外面应用使用的 jar 文件，例如 Hadoop 任务 jar.
* `meta_backups`: 执行 `bin/metastore.sh backup` 后的默认的备份目录;
* `sample_cube` 用于创建样例 Cube 和表的文件。
-* `spark`: 自带的 spark。
+* `spark`: 使用kylin脚本下载得到的 spark。
* `tomcat`: 自带的 tomcat，用于启动 Kylin 服务。
* `tool`: 用于执行一些命令行的jar文件。
+### 为某些环境执行额外步骤
+对于 CDH6.x, EMR5.x, EMR6.x 的 hadoop 环境，在启动 kylin 之前需要执行一些额外的步骤。
+CDH6.x 环境请查看文档: https://cwiki.apache.org/confluence/display/KYLIN/Deploy+Kylin+4+on+CDH+6
+EMR 环境请查看文档: https://cwiki.apache.org/confluence/display/KYLIN/Deploy+Kylin+4+on+AWS+EMR
### 检查运行环境
Kylin 运行在 Hadoop 集群上，对各个组件的版本、访问权限及 CLASSPATH 等都有一定的要求，为了避免遇到各种环境问题，您可以运行 `$KYLIN_HOME/bin/check-env.sh` 脚本来进行环境检测，如果您的环境存在任何的问题，脚本将打印出详细报错信息。如果没有报错信息，代表您的环境适合 Kylin 运行。
-
### 启动 Kylin
运行 `$KYLIN_HOME/bin/kylin.sh start` 脚本来启动 Kylin，界面输出如下：
```
Retrieving hadoop conf dir...
-KYLIN_HOME is set to /usr/local/apache-kylin-2.5.0-bin-hbase1x
+KYLIN_HOME is set to /usr/local/apache-kylin-4.0.0-bin
......
A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
-Check the log at /usr/local/apache-kylin-2.5.0-bin-hbase1x/logs/kylin.log
+Check the log at /usr/local/apache-kylin-4.0.0-bin/logs/kylin.log
Web UI is at http://<hostname>:7070/kylin
```
-**注**：如果在启动 kylin 时遇到如下报错(Hadoop3 环境下可能出现)：
-
-```
-Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
- at org.apache.hadoop.conf.Configuration.set(Configuration.java:1358)
- at org.apache.hadoop.conf.Configuration.set(Configuration.java:1339)
- at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:77)
- at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:63)
- at org.apache.kylin.storage.hbase.HBaseConnection.newHBaseConfiguration(HBaseConnection.java:170)
- at org.apache.kylin.storage.hbase.HBaseConnection.get(HBaseConnection.java:259)
- at org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:96)
- at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:119)
- at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:89)
- ... 8 more
-```
-
-可以尝试如下方法解决：
-
-下载[guava-28.0-jre.jar](https://repo1.maven.org/maven2/com/google/guava/guava/28.0-jre/guava-28.0-jre.jar), 将其放到`$KYLIN_HOME/tool/` 和 `$KYLIN_HOME/tomcat/lib/`目录下，然后重启kylin `bin/kylin.sh restart`。
### 使用 Kylin
@@ -127,7 +116,7 @@ Kylin 启动后您可以通过浏览器 `http://<hostname>:7070/kylin` 进行访
```
Retrieving hadoop conf dir...
-KYLIN_HOME is set to /usr/local/apache-kylin-2.5.0-bin-hbase1x
+KYLIN_HOME is set to /usr/local/apache-kylin-4.0.0-bin
Stopping Kylin: 25964
Stopping in progress. Will check after 2 secs again...
Kylin with pid 25964 has been stopped.
@@ -136,11 +125,9 @@ Kylin with pid 25964 has been stopped.
您可以运行 `ps -ef | grep kylin` 来查看 Kylin 进程是否已停止。
### HDFS 目录结构
-Kylin 会在 HDFS 上生成文件，根目录是 "/kylin/", 然后会使用 Kylin 集群的元数据表名作为第二层目录名，默认为 "kylin_metadata" (可以在`conf/kylin.properties`中定制).
-
-通常, `/kylin/kylin_metadata` 目录下会有这么几种子目录：`cardinality`, `coprocessor`, `kylin-job_id`, `resources`, `jdbc-resources`.
-1. `cardinality`: Kylin 加载 Hive 表时，会启动一个 MR 任务来计算各个列的基数，输出结果会暂存在此目录。此目录可以安全清除。
-2. `coprocessor`: Kylin 用于存放 HBase coprocessor jar 的目录；请勿删除。
-3. `kylin-job_id`: Cube 计算过程的数据存储目录，请勿删除。 如需要清理，请遵循 [storage cleanup guide](/docs/howto/howto_cleanup_storage.html).
-4. `resources`: Kylin 默认会将元数据存放在 HBase，但对于太大的文件（如字典或快照），会转存到 HDFS 的该目录下，请勿删除。如需要清理，请遵循 [cleanup resources from metadata](/docs/howto/howto_backup_metadata.html)
-5. `jdbc-resources`：性质同上，只在使用 MySQL 做元数据存储时候出现。
+Kylin 会在 HDFS 上生成文件，默认根目录是 "/kylin/", 然后会使用 Kylin 集群的元数据表名作为第二层目录名，默认为 "kylin_metadata" (可以在`conf/kylin.properties`中定制).
+
+通常, `/kylin/kylin_metadata` 目录下按照不同的 project 存放数据，比如 learn_kylin 项目的数据目录为 `/kylin/kylin_metadata/learn_kylin`, 该目录下通常包括以下子目录：
+1.`job_tmp`: 存放执行任务过程中生成的临时文件。
+2.`parquet`: 存放各个 cube 的 cuboid 文件。
+3.`table_snapshot`: 存放维度表快照。
diff --git a/website/_docs/install/index.md b/website/_docs/install/index.md
index a11bf2f..42a4ffa 100644
--- a/website/_docs/install/index.md
+++ b/website/_docs/install/index.md
@@ -7,15 +7,14 @@ permalink: /docs/install/index.html
### Software Requirements
-* Hadoop: 2.7+, 3.1+ (since v2.5)
+* Hadoop: cdh5.x, cdh6.x, hdp2.x, EMR5.x, EMR6.x, HDI4.x
* Hive: 0.13 - 1.2.1+
-* HBase: 1.1+, 2.0 (since v2.5)
-* Spark (optional) 2.3.0+
-* Kafka (optional) 1.0.0+ (since v2.5)
-* JDK: 1.8+ (since v2.5)
+* Spark: 2.4.7
+* Mysql: 5.1.17及以上
+* JDK: 1.8+
* OS: Linux only, CentOS 6.5+ or Ubuntu 16.0.4+
-Tests passed on Hortonworks HDP 2.2-2.6 and 3.0, Cloudera CDH 5.7-5.11 and 6.0, AWS EMR 5.7-5.10, Azure HDInsight 3.5-3.6.
+Tests passed on Hortonworks HDP2.4, Cloudera CDH 5.7 and 6.3.2, AWS EMR 5.31 and 6.0, Azure HDInsight 4.0.
We recommend you to try out Kylin or develop it using the integrated sandbox, such as [HDP sandbox](http://hortonworks.com/products/hortonworks-sandbox/), and make sure it has at least 10 GB of memory. When configuring a sandbox, we recommend that you use the Bridged Adapter model instead of the NAT model.
@@ -29,52 +28,65 @@ The minimum configuration of a server running Kylin is 4 core CPU, 16 GB RAM and
### Hadoop Environment
-Kylin relies on Hadoop clusters to handle large data sets. You need to prepare a Hadoop cluster with HDFS, YARN, MapReduce, Hive, HBase, Zookeeper and other services for Kylin to run.
-Kylin can be launched on any node in a Hadoop cluster. For convenience, you can run Kylin on the master node. For better stability, it is recommended to deploy Kylin on a clean Hadoop client node with Hive, HBase, HDFS and other command lines installed and client configuration (such as `core-site.xml`, `hive-site.xml`, `hbase-site.xml` and others) are also reasonably configured and can be automatically synchronized with other nodes.
+Kylin relies on Hadoop clusters to handle large data sets. You need to prepare a Hadoop cluster with HDFS, YARN, Hive, Zookeeper and other services for Kylin to run.
+Kylin can be launched on any node in a Hadoop cluster. For convenience, you can run Kylin on the master node. For better stability, it is recommended to deploy Kylin on a clean Hadoop client node with Hive, HDFS and other command lines installed and client configuration (such as `core-site.xml`, `hive-site.xml`and others) are also reasonably configured and can be automatically synchronized with other nodes.
-Linux accounts running Kylin must have access to the Hadoop cluster, including the permission to create/write HDFS folders, Hive tables, HBase tables, and submit MapReduce tasks.
+Linux accounts running Kylin must have access to the Hadoop cluster, including the permission to create/write HDFS folders, Hive tables.
### Kylin Installation
-- Download a binary package for your Hadoop version from the [Apache Kylin Download Site](https://kylin.apache.org/download/). For example, Kylin 2.5.0 for HBase 1.x can be downloaded from the following command line:
+- Download a Apache kylin 4.0.0 binary package from the [Apache Kylin Download Site](https://kylin.apache.org/download/). For example, the following command line can be used:
```shell
cd /usr/local/
-wget http://mirror.bit.edu.cn/apache/kylin/apache-kylin-2.5.0/apache-kylin-2.5.0-bin-hbase1x.tar.gz
+wget http://mirror.bit.edu.cn/apache/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin.tar.gz
```
- Unzip the tarball and configure the environment variable `$KYLIN_HOME` to the Kylin folder.
```shell
-tar -zxvf apache-kylin-2.5.0-bin-hbase1x.tar.gz
-cd apache-kylin-2.5.0-bin-hbase1x
+tar -zxvf apache-kylin-4.0.0-bin.tar.gz
+cd apache-kylin-4.0.0-bin
export KYLIN_HOME=`pwd`
```
-- From v2.6.1, Kylin will not ship Spark binary anymore; You need to install Spark seperately, and then point `SPARK_HOME` system environment variable to it:
+- Run the script to download spark:
```shell
-export SPARK_HOME=/path/to/spark
+$KYLIN_HOME/bin/download-spark.sh
```
-or run the script to download it:
+Or configure SPARK_HOME points to the path of spark2.4.7 in the environment.
+
+- Configure MySQL metastore
+
+Kylin 4.0 uses MySQL as metadata storage, make the following configuration in `kylin.properties`:
```shell
-$KYLIN_HOME/bin/download-spark.sh
+kylin.metadata.url=kylin_metadata@jdbc,driverClassName=com.mysql.jdbc.Driver,url=jdbc:mysql//localhost:3306/kylin_test,username=,password=
+kylin.env.zookeeper-connect-string=ip:2181
```
+You need to change the Mysql user name and password, as well as the database and table where the metadata is stored. And put mysql jdbc connector into `$KYLIN_HOME/ext/`, if there is no such directory, please create it.
+Please refer to [配置 Mysql 为 Metastore](/_docs40/tutorial/mysql_metastore.html) learn about the detailed configuration of MySQL as a Metastore.
+
### Kylin tarball structure
* `bin`: shell scripts to start/stop Kylin service, backup/restore metadata, as well as some utility scripts.
* `conf`: XML configuration files. The function of these xml files can be found in [configuration page](/docs/install/configuration.html)
* `lib`: Kylin jar files for external use, like the Hadoop job jar, JDBC driver, HBase coprocessor jar, etc.
* `meta_backups`: default backup folder when run "bin/metastore.sh backup";
* `sample_cube`: files to create the sample cube and its tables.
-* `spark`: the default spark binary that built with Kylin.
+* `spark`: Spark by $KYLIN_HOME/bin/download.sh download.
* `tomcat` the tomcat web server that run Kylin application.
* `tool`: the jar file for running utility CLI.
+### Perform additional steps for some environments
+For Hadoop environment of CDH6.X, EMR5.X, EMR6.X, you need to perform some additional steps before starting kylin.
+For CDH6.X environment, please check the document: [Deploy kylin4.0 on CDH6](https://cwiki.apache.org/confluence/display/KYLIN/Deploy+Kylin+4+on+CDH+6)
+For EMR environment, please check the document: [Deploy kylin4.0 on EMR](https://cwiki.apache.org/confluence/display/KYLIN/Deploy+Kylin+4+on+AWS+EMR)
+
### Checking the operating environment
Kylin runs on a Hadoop cluster and has certain requirements for the version, access rights, and CLASSPATH of each component. To avoid various environmental problems, you can run the script, `$KYLIN_HOME/bin/check-env.sh` to have a test on your environment, if there are any problems with your environment, the script will print a detailed error message. If there is no error message, it means that your environment is suitable for Kylin to run.
@@ -86,31 +98,13 @@ Run the script, `$KYLIN_HOME/bin/kylin.sh start` , to start Kylin. The interface
```
Retrieving hadoop conf dir...
-KYLIN_HOME is set to /usr/local/apache-kylin-2.5.0-bin-hbase1x
+KYLIN_HOME is set to /usr/local/apache-kylin-4.0.0-bin
......
A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
-Check the log at /usr/local/apache-kylin-2.5.0-bin-hbase1x/logs/kylin.log
+Check the log at /usr/local/apache-kylin-4.0.0-bin/logs/kylin.log
Web UI is at http://<hostname>:7070/kylin
```
-**Note**: If you encounter the following error when starting kylin (possible in Hadoop3 environment)：
-
-```
-Caused by: java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
- at org.apache.hadoop.conf.Configuration.set(Configuration.java:1358)
- at org.apache.hadoop.conf.Configuration.set(Configuration.java:1339)
- at org.apache.kylin.common.util.HadoopUtil.healSickConfig(HadoopUtil.java:77)
- at org.apache.kylin.common.util.HadoopUtil.getCurrentConfiguration(HadoopUtil.java:63)
- at org.apache.kylin.storage.hbase.HBaseConnection.newHBaseConfiguration(HBaseConnection.java:170)
- at org.apache.kylin.storage.hbase.HBaseConnection.get(HBaseConnection.java:259)
- at org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:96)
- at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:119)
- at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:89)
- ... 8 more
-```
-
-You can try download [guava-28.0-jre.jar](https://repo1.maven.org/maven2/com/google/guava/guava/28.0-jre/guava-28.0-jre.jar), put it into `$KYLIN_HOME/tool/` and `$KYLIN_HOME/tomcat/lib/` and restart kylin `bin/kylin.sh restart`.
-
### Using Kylin
Once Kylin is launched, you can access it via the browser `http://<hostname>:7070/kylin` with
@@ -125,7 +119,7 @@ Run the `$KYLIN_HOME/bin/kylin.sh stop` script to stop Kylin. The console output
```
Retrieving hadoop conf dir...
-KYLIN_HOME is set to /usr/local/apache-kylin-2.5.0-bin-hbase1x
+KYLIN_HOME is set to /usr/local/apache-kylin-4.0.0-bin
Stopping Kylin: 25964
Stopping in progress. Will check after 2 secs again...
Kylin with pid 25964 has been stopped.
@@ -135,11 +129,9 @@ You can run `ps -ef | grep kylin` to see if the Kylin process has stopped.
### HDFS folder structure
-Kylin will generate files on HDFS. The root folder is "/kylin/", but will have the second level folder for each Kylin cluster, named with the metadata table name, by default it is "kylin_metadata" (can be customized in `conf/kylin.properties`).
-
-Usually, there are at least these four kind of directories under `/kylin/kylin_metadata`: `cardinality`, `coprocessor`, `kylin-job_id`, `resources`.
-1. `cardinality`: the output folder of the cardinality calculation job when Kylin loads a Hive table. It can be cleaned when there is no job running;
-2. `coprocessor`: the folder that Kylin puts HBase coprocessor jar file. Please do not delete it.
-3. `kylin-job_id`: the cubing job's output folder. Please keep them; if need a cleanup, follow the [storage cleanup guide](/docs/howto/howto_cleanup_storage.html).
-4. `resources`: the metadata entries that too big to persisted in HBase (e.g, a dictionary or table snapshot); Please do not delete it; if need a cleanup, follow the [cleanup resources from metadata](/docs/howto/howto_backup_metadata.html)
-5. `jdbc-resources`: similar as `resources`, only appeared when using MySQL as the metadata storage。
+Kylin will generate files on HDFS. The default root directory is "kylin/", and then the metadata table name of kylin cluster will be used as the second layer directory name, and the default is "kylin_metadata"(can be customized in `conf/kylin.properties`)
+
+Generally, `/kylin/kylin_metadata` directory stores data according to different projects, such as data directory of "learn_kylin" project is `/kylin/kylin_metadata/learn_kylin`, which usually includes the following subdirectories:
+1.`job_tmp`: store temporary files generated during the execution of tasks.
+2.`parquet`: the cuboid file of each cube.
+3.`table_snapshot`: stores the dimension table snapshot.
diff --git a/website/_docs/install/kylin_cluster.cn.md b/website/_docs/install/kylin_cluster.cn.md
index 7e8eb8e..3e5a1e7 100644
--- a/website/_docs/install/kylin_cluster.cn.md
+++ b/website/_docs/install/kylin_cluster.cn.md
@@ -5,16 +5,16 @@ categories: install
permalink: /cn/docs/install/kylin_cluster.html
---
-Kylin 实例是无状态的服务，运行时的状态信息存储在 HBase metastore 中。 出于负载均衡的考虑，您可以启用多个共享一个 metastore 的 Kylin 实例，使得各个节点分担查询压力且互为备份，从而提高服务的可用性。下图描绘了 Kylin 集群模式部署的一个典型场景：
-
+Kylin 实例是无状态的服务，运行时的状态信息存储在 Mysql metastore 中。 出于负载均衡的考虑，您可以启用多个共享一个 metastore 的 Kylin 实例，使得各个节点分担查询压力且互为备份，从而提高服务的可用性。下图描绘了 Kylin 集群模式部署的一个典型场景：
+
### Kylin 集群模式部署
-如果您需要将多个 Kylin 节点组成集群，请确保他们使用同一个 Hadoop 集群、HBase 集群。然后在每个节点的配置文件 `$KYLIN_HOME/conf/kylin.properties` 中执行下述操作：
+如果您需要将多个 Kylin 节点组成集群，请确保他们使用同一个 Hadoop 集群。然后在每个节点的配置文件 `$KYLIN_HOME/conf/kylin.properties` 中执行下述操作：
-1. 配置相同的 `kylin.metadata.url` 值，即配置所有的 Kylin 节点使用同一个 HBase metastore。
+1. 配置相同的 `kylin.metadata.url` 值，即配置所有的 Kylin 节点使用同一个 Mysql metastore。
2. 配置 Kylin 节点列表 `kylin.server.cluster-servers`，包括所有节点（包括当前节点），当事件变化时，接收变化的节点需要通知其他所有节点（包括当前节点）。
3. 配置 Kylin 节点的运行模式 `kylin.server.mode`，参数值可选 `all`, `job`, `query` 中的一个，默认值为 `all`。
`job` 模式代表该服务仅用于任务调度，不用于查询；`query` 模式代表该服务仅用于查询，不用于构建任务的调度；`all` 模式代表该服务同时用于任务调度和 SQL 查询。
@@ -33,33 +33,23 @@ Kylin 实例是无状态的服务，运行时的状态信息存储在 HBase meta
```properties
kylin.job.scheduler.default=2
-kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperJobLock
+kylin.job.lock=org.apache.kylin.job.lock.zookeeper.ZookeeperJobLock
```
然后将所有任务和查询节点的地址注册到 `kylin.server.cluster-servers`。
-### 配置`CuratorScheduler`进行任务调度
+### 配置`CuratorScheculer`进行任务调度
-从 v3.0.0-alpha 开始，kylin引入基于Curator的主从模式多任务引擎调度器，用户可以修改如下配置来启用CuratorScheduler：
+从 v3.0.0-alpha 开始，kylin引入基于Curator的主从模式多任务引擎调度器，用户可以修改如下配置来启用CuratorScheculer：
```properties
kylin.job.scheduler.default=100
kylin.server.self-discovery-enabled=true
```
-更多关于kylin任务调度器的细节可以参考[Apache Kylin Wiki](https://cwiki.apache.org/confluence/display/KYLIN/Comparison+of+Kylin+Job+scheduler)
-
-
+更多关于kylin任务调度器的细节可以参考[Apache Kylin Wiki](https://cwiki.apache.org/confluence/display/KYLIN/Comparison+of+Kylin+Job+scheduler).
### 安装负载均衡器
为了将查询请求发送给集群而非单个节点，您可以部署一个负载均衡器，如 [Nginx](http://nginx.org/en/)， [F5](https://www.f5.com/) 或 [cloudlb](https://rubygems.org/gems/cloudlb/) 等，使得客户端和负载均衡器通信代替和特定的 Kylin 实例通信。
-
-
### 读写分离部署
-
-为了达到更好的稳定性和最佳的性能，建议进行读写分离部署，将 Kylin 部署在两个集群上，如下：
-
-* 一个 Hadoop 集群用作 **Cube 构建**，这个集群可以是一个大的、与其它应用共享的集群；
-* 一个 HBase 集群用作 **SQL 查询**，通常这个集群是专门为 Kylin 配置的，节点数不用像 Hadoop 集群那么多，HBase 的配置可以针对 Kylin Cube 只读的特性而进行优化。
-
-这种部署策略是适合生产环境的最佳部署方案，关于如何进行读写分离部署，请参考 [Deploy Apache Kylin with Standalone HBase Cluster](/blog/2016/06/10/standalone-hbase-cluster/)。
\ No newline at end of file
+Kylin4 的读写分离部署方式与 Kylin3 存在一定的差异，请参考文档：[Read Write Separation Deployment for Kylin 4](https://cwiki.apache.org/confluence/display/KYLIN/Read-Write+Separation+Deployment+for+Kylin+4.0)
diff --git a/website/_docs/install/kylin_cluster.md b/website/_docs/install/kylin_cluster.md
index 3da1b94..8992bc5 100644
--- a/website/_docs/install/kylin_cluster.md
+++ b/website/_docs/install/kylin_cluster.md
@@ -6,16 +6,16 @@ permalink: /docs/install/kylin_cluster.html
---
-Kylin instances are stateless services, and runtime state information is stored in the HBase metastore. For load balancing purposes, you can enable multiple Kylin instances that share a metastore, so that each node shares query pressure and backs up each other, improving service availability. The following figure depicts a typical scenario for Kylin cluster mode deployment:
-
+Kylin instances are stateless services, and runtime state information is stored in the Mysql metastore. For load balancing purposes, you can enable multiple Kylin instances that share a metastore, so that each node shares query pressure and backs up each other, improving service availability. The following figure depicts a typical scenario for Kylin cluster mode deployment:
+
### Kylin Node Configuration
-If you need to cluster multiple Kylin nodes, make sure they use the same Hadoop cluster, HBase cluster. Then do the following steps in each node's configuration file `$KYLIN_HOME/conf/kylin.properties`:
+If you need to cluster multiple Kylin nodes, make sure they use the same Hadoop cluster. Then do the following steps in each node's configuration file `$KYLIN_HOME/conf/kylin.properties`:
-1. Configure the same `kylin.metadata.url` value to configure all Kylin nodes to use the same HBase metastore.
+1. Configure the same `kylin.metadata.url` value to configure all Kylin nodes to use the same Mysql metastore.
2. Configure the Kylin node list `kylin.server.cluster-servers`, including all nodes (the current node is also included). When the event changes, the node receiving the change needs to notify all other nodes (the current node is also included).
3. Configure the running mode `kylin.server.mode` of the Kylin node. Optional values include `all`, `job`, `query`. The default value is *all*.
The *job* mode means that the service is only used for job scheduling, not for queries; the *query* pattern means that the service is only used for queries, not for scheduling jobs; the *all* pattern represents the service for both job scheduling and queries.
@@ -32,12 +32,12 @@ To enable the distributed job scheduler, you need to set or update the configs i
```properties
kylin.job.scheduler.default=2
-kylin.job.lock=org.apache.kylin.storage.hbase.util.ZookeeperJobLock
+kylin.job.lock=org.apache.kylin.job.lock.zookeeper.ZookeeperJobLock
```
Then please add all job servers and query servers to the `kylin.server.cluster-servers`.
-### Use `CuratorScheduler`
+### Use `CuratorScheculer`
Since v3.0.0-alpha, kylin introduces the Leader/Follower mode multiple job engines scheduler based on Curator. Users can modify the following configuration to enable CuratorScheduler:
@@ -46,19 +46,11 @@ kylin.job.scheduler.default=100
kylin.server.self-discovery-enabled=true
```
-For more details about the kylin job scheduler, please refer to [Apache Kylin Wiki](https://cwiki.apache.org/confluence/display/KYLIN/Comparison+of+Kylin+Job+scheduler)
+For more details about the kylin job scheduler, please refer to [Apache Kylin Wiki](https://cwiki.apache.org/confluence/display/KYLIN/Comparison+of+Kylin+Job+scheduler).
### Installing a load balancer
-To send query requests to a cluster instead of a single node, you can deploy a load balancer such as [Nginx](http://nginx.org/en/), [F5](https://www.f5.com/) or [cloudlb](https://rubygems.org/gems/cloudlb/), etc., so that the client communicate with the load balancer instead of a specific Kylin instance.
-
-
+To send query requests to a cluster instead of a single node, you can deploy a load balancer such as [Nginx](http://nginx.org/en/), [F5](https://www.f5.com/) or [cloudlb](https://rubygems.org/gems/cloudlb/), etc., so that the client and load balancer communication instead communicate with a specific Kylin instance.
### Read and write separation deployment
-
-For better stability and optimal performance, it is recommended to perform a read-write separation deployment, deploying Kylin on two clusters as follows:
-
-* A Hadoop cluster used for *Cube build*, which can be a large cluster shared with other applications;
-* An HBase cluster used for *SQL query*. Usually this cluster is configured for Kylin. The number of nodes does not need to be as many as Hadoop clusters. HBase configuration can be optimized for Kylin Cube read-only features.
-
-This deployment strategy is the best deployment solution for the production environment. For how to perform read-write separation deployment, please refer to [Deploy Apache Kylin with Standalone HBase Cluster](/blog/2016/06/10/standalone-hbase-cluster/) .
\ No newline at end of file
+There are some differences between read and write separation deployment of kylin 4 and kylin 3, Please refer to : [Read Write Separation Deployment for Kylin 4](https://cwiki.apache.org/confluence/display/KYLIN/Read-Write+Separation+Deployment+for+Kylin+4.0)
\ No newline at end of file
diff --git a/website/_docs/install/kylin_docker.cn.md b/website/_docs/install/kylin_docker.cn.md
index e0bf234..a8141e7 100644
--- a/website/_docs/install/kylin_docker.cn.md
+++ b/website/_docs/install/kylin_docker.cn.md
@@ -1,27 +1,27 @@
---
-layout: docs
+layout: docs-cn
title: "用 Docker 运行 Kylin"
categories: install
permalink: /cn/docs/install/kylin_docker.html
-since: v3.0.0
+since: v4.0.0
---
为了让用户方便的试用 Kylin，以及方便开发者在修改了源码后进行验证及调试。我们提供了 Kylin 的 docker 镜像。该镜像中，Kylin 依赖的各个服务均已正确的安装及部署，包括：
- JDK 1.8
-- Hadoop 2.7.0
+- Hadoop 2.8.5
- Hive 1.2.1
-- Hbase 1.1.2 (with Zookeeper)
-- Spark 2.3.1
+- Spark 2.4.7
- Kafka 1.1.1
- MySQL 5.1.73
+- Zookeeper 3.4.6
## 快速试用 Kylin
我们已将面向用户的 Kylin 镜像上传至 docker 仓库，用户无需在本地构建镜像，直接执行以下命令从 docker 仓库 pull 镜像：
{% highlight Groff markup %}
-docker pull apachekylin/apache-kylin-standalone:3.1.0
+docker pull apachekylin/apache-kylin-standalone:4.0.0
{% endhighlight %}
pull 成功后，执行以下命令启动容器：
@@ -34,26 +34,23 @@ docker run -d \
-p 50070:50070 \
-p 8032:8032 \
-p 8042:8042 \
--p 16010:16010 \
-apachekylin/apache-kylin-standalone:3.1.0
+-p 2181:2181 \
+apachekylin/apache-kylin-standalone:4.0.0
{% endhighlight %}
在容器启动时，会自动启动以下服务：
- NameNode, DataNode
- ResourceManager, NodeManager
-- HBase
-- Kafka
- Kylin
-并自动运行 `$KYLIN_HOME/bin/sample.sh ` 及在 Kafka 中创建 "kylin_streaming_topic" topic 并持续向该 topic 中发送数据。这是为了让用户启动容器后，就能体验以批和流的方式的方式构建 Cube 并进行查询。
+并自动运行 `$KYLIN_HOME/bin/sample.sh `.
容器启动后，我们可以通过 "docker exec -it \<container_id\> bash" 命令进入容器内。当然，由于我们已经将容器内指定端口映射到本机端口，我们可以直接在本机浏览器中打开各个服务的页面，如：
- Kylin 页面：[http://127.0.0.1:7070/kylin/login](http://127.0.0.1:7070/kylin/login)
- HDFS NameNode 页面：[http://127.0.0.1:50070](http://127.0.0.1:50070/)
- YARN ResourceManager 页面：[http://127.0.0.1:8088](http://127.0.0.1:8088/)
-- HBase 页面：[http://127.0.0.1:16010](http://127.0.0.1:16010/)
## 容器资源建议
@@ -67,4 +64,4 @@ apachekylin/apache-kylin-standalone:3.1.0
---
-关于如何定制修改 Docker image，请参阅 Git 代码库的 [kylin/docker](https://github.com/apache/kylin/tree/master/docker/)
\ No newline at end of file
+关于如何定制修改 Docker image，请参阅 Git 代码库的 [kylin/docker](https://github.com/apache/kylin/tree/main/docker/)
\ No newline at end of file
diff --git a/website/_docs/install/kylin_docker.md b/website/_docs/install/kylin_docker.md
index aba06ec..701daf1 100644
--- a/website/_docs/install/kylin_docker.md
+++ b/website/_docs/install/kylin_docker.md
@@ -3,25 +3,25 @@ layout: docs
title: "Run Kylin with Docker"
categories: install
permalink: /docs/install/kylin_docker.html
-since: v3.0.0
+since: v4.0.0
---
In order to allow users to easily try Kylin, and to facilitate developers to verify and debug after modifying the source code. We provide Kylin's docker image. In this image, each service that Kylin relies on is properly installed and deployed, including:
- JDK 1.8
-- Hadoop 2.7.0
+- Hadoop 2.8.5
- Hive 1.2.1
-- Hbase 1.1.2 (with Zookeeper)
-- Spark 2.3.1
+- Spark 2.4.7
- Kafka 1.1.1
- MySQL 5.1.73
+- Zookeeper 3.4.6
## Quickly try Kylin
We have pushed the Kylin image for the user to the docker hub. Users do not need to build the image locally, just execute the following command to pull the image from the docker hub:
{% highlight Groff markup %}
-docker pull apachekylin/apache-kylin-standalone:3.1.0
+docker pull apachekylin/apache-kylin-standalone:4.0.0
{% endhighlight %}
After the pull is successful, execute the following command to start the container:
@@ -34,26 +34,23 @@ docker run -d \
-p 50070:50070 \
-p 8032:8032 \
-p 8042:8042 \
--p 16010:16010 \
-apachekylin/apache-kylin-standalone:3.1.0
+-p 2181:2181 \
+apachekylin/apache-kylin-standalone:4.0.0
{% endhighlight %}
The following services are automatically started when the container starts:
- NameNode, DataNode
- ResourceManager, NodeManager
-- HBase
-- Kafka
- Kylin
-and run automatically `$KYLIN_HOME/bin/sample.sh `, create a "kylin_streaming_topic" topic in Kafka and continue to send data to this topic. This is to let the users start the container and then experience the batch and streaming way to build the cube and query.
+and run automatically `$KYLIN_HOME/bin/sample.sh `.
After the container is started, we can enter the container through the `docker exec -it <container_id> bash` command. Of course, since we have mapped the specified port in the container to the local port, we can open the pages of each service directly in the native browser, such as:
- Kylin Web UI: [http://127.0.0.1:7070/kylin/login](http://127.0.0.1:7070/kylin/login)
- Hdfs NameNode Web UI: [http://127.0.0.1:50070](http://127.0.0.1:50070/)
- Yarn ResourceManager Web UI: [http://127.0.0.1:8088](http://127.0.0.1:8088/)
-- HBase Web UI: [http://127.0.0.1:16010](http://127.0.0.1:16010/)
## Container resource recommendation
@@ -66,4 +63,4 @@ For the resource setting method for the container, please refer to:
---
-For how to customize the image, please check the github page [kylin/docker](https://github.com/apache/kylin/tree/master/docker/).
\ No newline at end of file
+For how to customize the image, please check the github page [kylin/docker](https://github.com/apache/kylin/tree/main/docker/).
\ No newline at end of file
diff --git a/website/_docs/release_notes.md b/website/_docs/release_notes.md
index a4a10bd..85ce861 100644
--- a/website/_docs/release_notes.md
+++ b/website/_docs/release_notes.md
@@ -5,7 +5,7 @@ categories: gettingstarted
permalink: /docs/release_notes.html
---
-To download latest release, please visit: [http://kylin.apache.org/download/](http://kylin.apache.org/download/),
+To download the latest release, please visit: [http://kylin.apache.org/download/](http://kylin.apache.org/download/),
there are source code package, binary package and installation guide avaliable.
Any problem or issue, please report to Apache Kylin JIRA project: [https://issues.apache.org/jira/browse/KYLIN](https://issues.apache.org/jira/browse/KYLIN)
@@ -15,6 +15,59 @@ or send to Apache Kylin mailing list:
* User relative: [user@kylin.apache.org](mailto:user@kylin.apache.org)
* Development relative: [dev@kylin.apache.org](mailto:dev@kylin.apache.org)
+## v4.0.0 - 2021-08-30
+
+__New Feature__
+
+* [KYLIN-4498] - CubePlaner for Kylin on Parquet
+* [KYLIN-4895] - change spark deploy mode of kylin4.0 engine from local to cluster
+* [KYLIN-4905] - Support limit .. offset ... in spark query engine
+* [KYLIN-4925] - Use Spark 3 as build and query engine for Kylin 4
+* [KYLIN-4948] - Provide an API to allow users to adjust cuboids manually
+* [KYLIN-4966] - Refresh the existing segment according to the new cuboid list in kylin4
+* [KYLIN-5011] - Detect and scatter skewed data in dict encoding step
+* [KYLIN-5019] - Avoid building global dictionary from all data of fact table each time
+* [KYLIN-5059] - Fix error when using different HDFS cluster in cube building
+
+__Bug Fix__
+
+* [KYLIN-4729] - The hive table will be overwrited when add csv table with the same name
+* [KYLIN-4879] - The function of sql to remove comments is not perfect. In some cases, the sql query conditions used will be modified
+* [KYLIN-4887] - Segment pruner support string type partition col in spark query engine
+* [KYLIN-4889] - Query error when spark engine in local mode
+* [KYLIN-4935] - Existing JobOutput's extend info will be lost when it is updated
+* [KYLIN-4967] - Forbid to set 'spark.sql.adaptive.enabled' to true when building cube with Spark 2.X
+* [KYLIN-5013] - Write table_snapshot to wrong cluster in Kylin4.0
+* [KYLIN-5014] - Spark driver log is abnormal in yarn cluster mode
+* [KYLIN-5021] - FilePruner in Spark3 throws NPE when no partition columns in cube
+* [KYLIN-5040] - Cuboid should not be exactly matched when there is no group by time partition column and there are multiple segments in the query
+
+__Improvement__
+
+* [KYLIN-4554] - Validate "filter condition" on model saving
+* [KYLIN-4888] - Performance optimization of union query with spark engine
+* [KYLIN-4890] - Use numSlices = 1 to reduce task num when executing sparder canary
+* [KYLIN-4892] - Reduce the times of fetching files status from HDFS in FilePruner
+* [KYLIN-4893] - Optimize query performance when using shard by column
+* [KYLIN-4894] - Upgrade Apache Spark version to 2.4.7
+* [KYLIN-4897] - Add table snapshot and global dictionary cleaning in StorageCleanupJob
+* [KYLIN-4898] - Add automated test cases
+* [KYLIN-4903] - cache parent datasource to accelerate next layer's cuboid building
+* [KYLIN-4906] - support query/job server dynamic register and discovery in kylin4
+* [KYLIN-4908] - Segment pruner support integer partition col in spark query engine
+* [KYLIN-4910] - Return hostname as Sparder URL address when spark master is set to local
+* [KYLIN-4917] - Fix some problem of logger system in kylin4
+* [KYLIN-4923] - CubeMigration Tools support migrate meta from 2.x/3.x cluster to 4.0 cluster
+* [KYLIN-4926] - Optimize Global Dict building: replace operation 'mapPartitions.count()' with 'foreachPartitions'
+* [KYLIN-4927] - Forbid to use AE when building Global Dict
+* [KYLIN-4936] - Exactly aggregation can't transform to project
+* [KYLIN-4937] - Verify the uniqueness of the global dictionary after building global dictionary
+* [KYLIN-4944] - Upgrade CentOS version, Hadoop version and Spark version for Kylin Docker image
+* [KYLIN-4945] - Repartition encoded dataset to avoid data skew caused by a single column
+* [KYLIN-4980] - Support prunning segments from complex filter conditions
+* [KYLIN-5027] - Add the config of whether to build base cuboid in kylin4
+* [KYLIN-5031] - The last_build_job_id of segment is null when the semgent status is RUNNING or ERROR.
+
## v3.1.2 - 2021-04-26
__New Feature__
diff --git a/website/_docs/security.md b/website/_docs/security.md
index 2f0adce..405c4e6 100644
--- a/website/_docs/security.md
+++ b/website/_docs/security.md
@@ -5,32 +5,6 @@ categories: docs
permalink: /docs/security.html
---
-### [CVE-2020-13937](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13937)
-
-__Severity__
-
-Important
-
-__Versions Affected__
-
-Kylin 2.0.0, 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.5.2, 2.6.0, 2.6.1, 2.6.2, 2.6.3, 2.6.4, 2.6.5, 2.6.6, 3.0.0-alpha, 3.0.0-alpha2, 3.0.0-beta, 3.0.0, 3.0.1, 3.0.2, 3.1.0, 4.0.0-alpha.
-
-__Description__
-
-Kylin has one restful api which exposed Kylin's configuration information without any authentication, so it is dangerous because some confidential information entries will be disclosed to everyone.
-
-__Mitigation__
-
-Users of all previous versions after 2.0 should upgrade to 3.1.0.
-
-Users could edit `$KYLIN_HOME/WEB-INF/classes/kylinSecurity.xml`, and remove this line `<scr:intercept-url pattern="/api/admin/config" access="permitAll"/>`. After that, restart all Kylin instances to make it effective.
-
-Otherwise, you can upgrade Kylin to 3.1.1.
-
-__Credit__
-
-This issue was discovered by Ngo Wei Lin (@Creastery) of STAR Labs (@starlabs_sg).
-
### [CVE-2020-13926](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13926)
__Severity__
diff --git a/website/_docs/tutorial/create_cube.cn.md b/website/_docs/tutorial/create_cube.cn.md
index 43a62d3..074c345 100644
--- a/website/_docs/tutorial/create_cube.cn.md
+++ b/website/_docs/tutorial/create_cube.cn.md
@@ -46,10 +46,6 @@ since: v0.7.1
-6. 在后台，Kylin 将会执行 MapReduce 任务计算新同步表的基数（cardinality），任务完成后，刷新页面并点击表名，基数值将会显示在表信息中。
-
- 
-
### III. 新建 Data Model
创建 cube 前，需定义一个数据模型。数据模型定义了一个星型（star schema）或雪花（snowflake schema）模型。一个模型可以被多个 cube 使用。
@@ -121,7 +117,7 @@ cube 名字可以使用字母，数字和下划线（空格不允许）。`Notif
-2. 根据它的表达式共有8种不同类型的度量：`SUM`、`MAX`、`MIN`、`COUNT`、`COUNT_DISTINCT` `TOP_N`, `EXTENDED_COLUMN` 和 `PERCENTILE`。请合理选择 `COUNT_DISTINCT` 和 `TOP_N` 返回类型，它与 cube 的大小相关。
+2. 根据它的表达式共有7种不同类型的度量：`SUM`、`MAX`、`MIN`、`COUNT`、`COUNT_DISTINCT` `TOP_N` 和 `PERCENTILE`。请合理选择 `COUNT_DISTINCT` 和 `TOP_N` 返回类型，它与 cube 的大小相关。
* SUM
@@ -141,7 +137,7 @@ cube 名字可以使用字母，数字和下划线（空格不允许）。`Notif
* DISTINCT_COUNT
这个度量有两个实现：
1）近似实现 HyperLogLog，选择可接受的错误率，低错误率需要更多存储；
- 2）精确实现 bitmap（具体限制请看 https://issues.apache.org/jira/browse/KYLIN-1186）
+ 2）精确实现 bitmap（具体实现请看 [Global Dictionary on Kylin 4](https://cwiki.apache.org/confluence/display/KYLIN/Global+Dictionary+on+Spark)）
@@ -155,11 +151,6 @@ cube 名字可以使用字母，数字和下划线（空格不允许）。`Notif
- * EXTENDED_COLUMN
- Extended_Column 作为度量比作为维度更节省空间。一列和另一列可以生成新的列。
-
- 
-
* PERCENTILE
Percentile 代表了百分比。值越大，错误就越少。100为最合适的值。
@@ -191,23 +182,15 @@ cube 名字可以使用字母，数字和下划线（空格不允许）。`Notif
关于更多维度优化，请阅读这个博客: [新的聚合组](/blog/2016/02/18/new-aggregation-group/)
-`Rowkeys`: 是由维度编码值组成。"Dictionary" （字典）是默认的编码方式; 字典只能处理中低基数（少于一千万）的维度；如果维度基数很高（如大于1千万), 选择 "false" 然后为维度输入合适的长度，通常是那列的最大长度值; 如果超过最大值，会被截断。请注意，如果没有字典编码，cube 的大小可能会非常大。
+`Rowkeys`: 是由维度编码值组成。
你可以拖拽维度列去调整其在 rowkey 中位置; 位于rowkey前面的列，将可以用来大幅缩小查询的范围。通常建议将 mandantory 维度放在开头, 然后是在过滤 ( where 条件)中起到很大作用的维度；如果多个列都会被用于过滤，将高基数的维度（如 user_id）放在低基数的维度（如 age）的前面。
-`Mandatory Cuboids`: 维度组合白名单。确保你想要构建的 cuboid 能被构建。
-
-`Cube Engine`: cube 构建引擎。有两种：MapReduce 和 Spark。如果你的 cube 只有简单度量（SUM, MIN, MAX)，建议使用 Spark。如果 cube 中有复杂类型度量（COUNT DISTINCT, TOP_N），建议使用 MapReduce。
-
-`Advanced Dictionaries`: "Global Dictionary" 是用于精确计算 COUNT DISTINCT 的字典, 它会将一个非 integer的值转成 integer，以便于 bitmap 进行去重。如果你要计算 COUNT DISTINCT 的列本身已经是 integer 类型，那么不需要定义 Global Dictionary。 Global Dictionary 会被所有 segment 共享，因此支持在跨 segments 之间做上卷去重操作。请注意，Global Dictionary 随着数据的加载，可能会不断变大。
-
-"Segment Dictionary" 是另一个用于精确计算 COUNT DISTINCT 的字典，与 Global Dictionary 不同的是，它是基于一个 segment 的值构建的，因此不支持跨 segments 的汇总计算。如果你的 cube 不是分区的或者能保证你的所有 SQL 按照 partition_column 进行 group by, 那么你应该使用 "Segment Dictionary" 而不是 "Global Dictionary"，这样可以避免单个字典过大的问题。
+此外，你还可以在这里指定使用某一列作为 shardBy 列，kylin4.0 会根据 shardBy 列对存储文件进行分片，分片能够使查询引擎跳过不必要的文件，提高查询性能，最好选择高基列并且会在多个 cuboid 中出现的列作为 shardBy 列。
-请注意："Global Dictionary" 和 "Segment Dictionary" 都是单向编码的字典，仅用于 COUNT DISTINCT 计算(将非 integer 类型转成 integer 用于 bitmap计算)，他们不支持解码，因此不能为普通维度编码。
-
-`Advanced Snapshot Table`: 为全局 lookup 表而设计，提供不同的存储类型。
+`Mandatory Cuboids`: 维度组合白名单。确保你想要构建的 cuboid 能被构建。
-`Advanced ColumnFamily`: 如果有超过一个的COUNT DISTINCT 或 TopN 度量, 你可以将它们放在更多列簇中，以优化与HBase 的I/O。
+`Cube Engine`: cube 构建引擎。Spark构建。
**步骤6. 重写配置**
diff --git a/website/_docs/tutorial/create_cube.md b/website/_docs/tutorial/create_cube.md
index 5ec7365..7fd68c8 100644
--- a/website/_docs/tutorial/create_cube.md
+++ b/website/_docs/tutorial/create_cube.md
@@ -184,7 +184,7 @@ This step is designed for incremental cube build.
For more please read this blog: [New Aggregation Group](/blog/2016/02/18/new-aggregation-group/)
-`Rowkeys`: the rowkeys are composed by the dimension encoded values. "Dictionary" is the default encoding method; If a dimension is not fit with dictionary (e.g., cardinality > 10 million), select "false" and then enter the fixed length for that dimension, usually that is the max length of that column; if a value is longer than that size it will be truncated. Please note, without dictionary encoding, the cube size might be much bigger.
+`Rowkeys`: the rowkeys are composed by the dimension encoded values.
You can drag & drop a dimension column to adjust its position in rowkey; Put the mandatory dimension at the beginning, then followed the dimensions that heavily involved in filters (where condition). Put high cardinality dimensions ahead of low cardinality dimensions.
@@ -192,16 +192,6 @@ You can drag & drop a dimension column to adjust its position in rowkey; Put the
`Cube Engine`: The engine for building cube. There are 2 engines: MapReduce and Spark. If your cube only has simple measures (COUNT, SUM, MIN, MAX), Spark can gain better performance; If cube has complex measures (COUNT DISTINCT, TOP_N), MapReduce is more stable.
-`Advanced Dictionaries`: "Global Dictionary" is the default dictionary for precise count distinct measure, it can ensure one value always be encoded into one consistent integer, so it can support "COUNT DISTINCT" rollup among multiple segments. But global dictionary may grow to very big size as time go.
-
-"Segment Dictionary" is a special dictionary for precise count distinct measure, which is built on one segment and could not support rollup among segments. Its size can be much smaller than global dictionary. Specifically, if your cube isn't partitioned or you can ensure all your SQLs will group by your partition_column, you could use "Segment Dictionary" instead of "Global Dictionary".
-
-Please note: "Global Dictionary" and "Segment Dictionary" are one-way dictionary for COUNT DISTINCT (converting a non-integer value to integer for bitmap), they couldn't be used as the encoding for a dimension.
-
-`Advanced Snapshot Table`: design for global lookup table and provide different storage type.
-
-`Advanced ColumnFamily`: If there are more than one ultra-high cardinality precise count distinct or TopN measures, you could divide these measures to more column family to optimize the I/O from HBase.
-
**Step 6. Configuration Overwrites**
Kylin allows overwritting system configurations (conf/kylin.properties) at Cube level . You can add the key/values that you want to overwrite here. If you don't have anything to config, click `Next` button.
diff --git a/website/_docs/tutorial/cube_build_job.cn.md b/website/_docs/tutorial/cube_build_job.cn.md
index 1a2c271..9ec613c 100644
--- a/website/_docs/tutorial/cube_build_job.cn.md
+++ b/website/_docs/tutorial/cube_build_job.cn.md
@@ -21,19 +21,19 @@ since: v0.7.1
3. 点击 `Submit` 提交请求。成功之后，你将会在 `Monitor` 页面看到新建的 job。
- 
+ 
4. 新建的 job 是 “pending” 状态；一会儿，它就会开始运行并且你可以通过刷新 web 页面或者点击刷新按钮来查看进度。
- 
+ 
5. 等待 job 完成。期间如要放弃这个 job ，点击 `Actions` -> `Discard` 按钮。
-6. 等到 job 100%完成，cube 的状态就会变为 “Ready”, 意味着它已经准备好进行 SQL 查询。在 `Model` 页，找到 cube，然后点击 cube 名展开消息，在 “Storage” 标签下，列出 cube segments。每一个 segment 都有 start/end 时间；Hbase 表的信息也会列出。
+6. 等到 job 100%完成，cube 的状态就会变为 “Ready”, 意味着它已经准备好进行 SQL 查询。在 `Model` 页，找到 cube，然后点击 cube 名展开消息，在 “Storage” 标签下，列出 cube segments。每一个 segment 都有 start/end 时间；Parquet 文件相关的信息也会列出。
- 
+ 
如果你有更多的源数据，重复以上的步骤将它们构建进 cube。
@@ -41,29 +41,17 @@ since: v0.7.1
在 `Monitor` 页面，点击job详情按钮查看显示于右侧的详细信息。
-
+
job 详细信息为跟踪一个 job 提供了它的每一步记录。你可以将光标停放在一个步骤状态图标上查看基本状态和信息。
-
+
-点击每个步骤显示的图标按钮查看详情：`Parameters`、`Log`、`MRJob`。
+点击每个步骤显示的图标按钮查看日志。
-* Parameters
- 
+
- 
+
-* Log
-
- 
-
- 
-
-* MRJob(MapReduce Job)
-
- 
-
- 
diff --git a/website/_docs/tutorial/cube_build_job.md b/website/_docs/tutorial/cube_build_job.md
index d15251d..5b5931a 100644
--- a/website/_docs/tutorial/cube_build_job.md
+++ b/website/_docs/tutorial/cube_build_job.md
@@ -18,50 +18,37 @@ First of all, make sure that you have authority of the cube you want to build.
4. Click `Submit`, you will see the new job in the `Monitor` page.
- 
+ 
5. The new job is in "pending" status; after a while, it will be started to run and you will see the progress by refresh the web page or click the refresh button.
- 
+ 
6. Wait the job to finish. In the between if you want to discard it, click `Actions` -> `Discard` button. If the job is failed, you can click `Resume` to retry.
-7. After the job is 100% finished, the cube's status becomes to "Ready", means it is ready to serve SQL queries. In the `Model` tab, find the cube, click cube name to expand the section, in the "Storage" tab, it will list the cube segments. Each segment has a start/end time; Its underlying HBase table information is also listed.
+7. After the job is 100% finished, the cube's status becomes to "Ready", means it is ready to serve SQL queries. In the `Model` tab, find the cube, click cube name to expand the section, in the "Storage" tab, it will list the cube segments. Each segment has a start/end time; Its underlying Parquet file information is also listed.
- 
+ 
If you have more source data, repeat the steps above to build them into the cube.
### Job Monitoring
In the `Monitor` page, click the job detail button to see detail information show in the right side.
-
+
The detail information of a job provides a step-by-step record to trace a job. You can hover a step status icon to see the basic status and information.
-
+
-Click the icon buttons showing in each step to see the details: `Parameters`, `Log`, `MRJob`.
-
-* Parameters
-
- 
-
- 
-
-* Log
+Click the icon buttons showing in each step to see the Log.
- 
-
- 
-
-* MRJob(MapReduce Job)
+
- 
+
- 
diff --git a/website/_docs/tutorial/query_pushdown.cn.md b/website/_docs/tutorial/query_pushdown.cn.md
index 82b66a1..453931f 100644
--- a/website/_docs/tutorial/query_pushdown.cn.md
+++ b/website/_docs/tutorial/query_pushdown.cn.md
@@ -3,80 +3,18 @@ layout: docs-cn
title: 查询下压
categories: tutorial
permalink: /cn/docs/tutorial/query_pushdown.html
-since: v2.1
+since: v4.0.0
---
### Kylin 支持查询下压
-对于没有cube能查得结果的sql，Kylin支持将这类查询通过JDBC下压至备用查询引擎如Hive, SparkSQL, Impala等来查得结果。
+对于没有cube能查得结果的sql，Kylin4.0 支持将这类查询下压至 SparkSql 去查询 Hive 源数据。
-### 查询下压配置
+### 开启查询下压
-#### 下压至单个引擎
-以Hive为例说明开启步骤
-
-1. 修改配置文件`kylin.properties`打开Query Pushdown注释掉的配置项`kylin.query.pushdown.runner-class-name`，设置为`org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl`；
-如果你需要将查询下压到presto，请将这项配置设置为`org.apache.kylin.query.pushdown.PushdownRunnerSDKImpl`，其余配置参见本页末尾的[Pushdown to Presto](#pushdown-to-presto)中的介绍。
-
-2. 在配置文件`kylin.properties`添加如下配置项。若不设置，将使用默认配置项。请不要忘记将"hiveserver"和"10000"替换成环境中Hive运行的主机和端口。
-
- - *kylin.query.pushdown.jdbc.url*：Hive JDBC的URL.
-
- - *kylin.query.pushdown.jdbc.driver*：Hive Jdbc的driver类名
-
- - *kylin.query.pushdown.jdbc.username*：Hive Jdbc对应数据库的用户名
-
- - *kylin.query.pushdown.jdbc.password*：Hive Jdbc对应数据库的密码
-
- - *kylin.query.pushdown.jdbc.pool-max-total*：Hive Jdbc连接池的最大连接数，默认值为8
-
- - *kylin.query.pushdown.jdbc.pool-max-idle*：Hive Jdbc连接池的最大等待连接数，默认值为8
-
- - *kylin.query.pushdown.jdbc.pool-min-idle*：Hive Jdbc连接池的最小连接数，默认值为0
-
-下面是一个样例设置; 请记得将主机名"hiveserver"以及端口"10000"修改为您的集群设置。
-
-{% highlight Groff markup %} kylin.query.pushdown.runner-class-name=org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl kylin.query.pushdown.jdbc.url=jdbc:hive2://hiveserver:10000/default kylin.query.pushdown.jdbc.driver=org.apache.hive.jdbc.HiveDriver kylin.query.pushdown.jdbc.username=hive kylin.query.pushdown.jdbc.password= kylin.query.pushdown.jdbc.pool-max-total=8 kylin.query.pushdown.jdbc.pool-max-idle=8 kylin.query.pushdown.jdbc.pool-min-idle=0
-
-{% endhighlight %}
-
-3. 当 Pushdown 到的查询引擎不是`Hive` 时，请将对应查询引擎的 JDBC 驱动放在 $KYLIN_HOME/ext 目录下，没有此目录时请自行创建。
-
-然后，重启Kylin。
-
-#### 下压至多个引擎
-自 v3.0.0 起，开始支持通过JDBC下压至多个引擎进行查询。
-通过配置 `kylin.query.pushdown.runner.ids` 来指定多个引擎id，id 之间以 `,` 进行分隔，如：
-
-{% highlight Groff markup %}
-kylin.query.pushdown.runner.ids=id1,id2,id3
-{% endhighlight %}
-
-指定了三个下压引擎，这三个引擎可以是同类型或不同类型的引擎。
-
-多引擎下压同样支持指定具体的 jdbc 参数，参数含义和上述单个引擎下压相同，请看下面的配置
-
-{% highlight Groff markup %}
-kylin.query.pushdown.{id}.jdbc.url
-kylin.query.pushdown.{id}.jdbc.driver
-kylin.query.pushdown.{id}.jdbc.username
-kylin.query.pushdown.{id}.jdbc.password
-kylin.query.pushdown.{id}.jdbc.pool-max-total
-kylin.query.pushdown.{id}.jdbc.pool-max-idle
-kylin.query.pushdown.{id}.jdbc.pool-min-idle
-{% endhighlight %}
-
-当要为某个引擎指定具体的 jdbc 参数时，请将上面的 `{id}` 替换为真实的引擎 id，如替换为 `id1` 的各配置为:
-
-{% highlight Groff markup %}
-kylin.query.pushdown.id1.jdbc.url
-kylin.query.pushdown.id1.jdbc.driver
-kylin.query.pushdown.id1.jdbc.username
... 6895 lines suppressed ...