You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by xx...@apache.org on 2021/07/04 12:17:32 UTC
[kylin] branch document updated (cc6aefc -> 8b9dfbd)
This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a change to branch document
in repository https://gitbox.apache.org/repos/asf/kylin.git.
from cc6aefc Replace aws with amazon
new efe2b4f Add dev_40
new c40cc30 Update overview of kylin4
new 8b9dfbd Add kylin4 blog
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
website/_config.yml | 6 +-
.../{development-cn.yml => development40-cn.yml} | 2 -
.../_data/{development.yml => development40.yml} | 2 -
website/_dev/howto_hbase_branches.cn.md | 24 ---
website/_dev/howto_hbase_branches.md | 24 ---
website/_dev/index.cn.md | 3 +
website/_dev/index.md | 3 +
website/{_dev => _dev40}/about_temp_files.cn.md | 4 +-
website/{_dev => _dev40}/about_temp_files.md | 4 +-
.../coding_naming_convention.cn.md | 4 +-
.../{_dev => _dev40}/coding_naming_convention.md | 4 +-
website/_dev40/dev_env.cn.md | 166 ++++++++++++++++++++
website/_dev40/dev_env.md | 170 +++++++++++++++++++++
website/{_dev => _dev40}/doc_spec.cn.md | 6 +-
website/{_dev => _dev40}/doc_spec.md | 6 +-
.../howto_become_apache_committer.cn.md | 4 +-
.../howto_become_apache_committer.md | 4 +-
website/{_dev => _dev40}/howto_contribute.cn.md | 10 +-
website/{_dev => _dev40}/howto_contribute.md | 10 +-
website/{_dev => _dev40}/howto_docs.cn.md | 6 +-
website/{_dev => _dev40}/howto_docs.md | 6 +-
website/{_dev => _dev40}/howto_package.cn.md | 14 +-
website/{_dev => _dev40}/howto_package.md | 14 +-
website/{_dev => _dev40}/howto_release.cn.md | 4 +-
website/{_dev => _dev40}/howto_release.md | 4 +-
website/_dev40/howto_test.cn.md | 24 +++
website/_dev40/howto_test.md | 23 +++
website/_dev40/index.cn.md | 37 +++++
website/_dev40/index.md | 34 +++++
website/{_dev => _dev40}/plugin_arch.cn.md | 4 +-
website/{_dev => _dev40}/plugin_arch.md | 4 +-
website/{_dev => _dev40}/web_tech.cn.md | 4 +-
website/{_dev => _dev40}/web_tech.md | 4 +-
website/_docs/install/index.cn.md | 4 +-
website/_docs40/gettingstarted/quickstart.cn.md | 2 +-
website/_docs40/gettingstarted/quickstart.md | 2 +
website/_docs40/index.cn.md | 163 +++++++++++++++-----
website/_docs40/index.md | 168 ++++++++++++++------
website/_docs40/install/kylin_cluster.cn.md | 4 +-
website/_docs40/install/kylin_cluster.md | 3 +
.../{dev_nav.cn.html => dev40_nav.cn.html} | 4 +-
website/_includes/{dev_nav.html => dev40_nav.html} | 4 +-
.../_includes/{dev_ul.html => dev40_ul.cn.html} | 4 +-
website/_includes/{dev_ul.html => dev40_ul.html} | 4 +-
website/_layouts/{dev-cn.html => dev40-cn.html} | 2 +-
website/_layouts/{dev.html => dev40.html} | 2 +-
...ylin4-A-new-storage-and-compute-architecture.md | 92 +++++++++++
.../blog/kylin4/1 apache_kylin_introduction.png | Bin 0 -> 45730 bytes
.../blog/kylin4/10 spark_parquet_solution.png | Bin 0 -> 365968 bytes
.../blog/kylin4/11 spark_parquet_architecture.png | Bin 0 -> 79197 bytes
website/images/blog/kylin4/2 cube_vs_cuboid.png | Bin 0 -> 122901 bytes
.../blog/kylin4/3 cuboid_selected_for_query.png | Bin 0 -> 131322 bytes
.../blog/kylin4/4 apache_kylin_query_process.png | Bin 0 -> 87024 bytes
.../blog/kylin4/5 query_using_precomputed_data.png | Bin 0 -> 289496 bytes
.../images/blog/kylin4/6 on-site-computation.png | Bin 0 -> 278849 bytes
.../7 precomputation_using_aggregated_data.png | Bin 0 -> 323673 bytes
.../kylin4/8 diagram_of_calcite_executions.png | Bin 0 -> 219636 bytes
.../kylin4/9 diagram_of_bottleneck_on_HBase.png | Bin 0 -> 265462 bytes
website/images/develop40/debug_tomcat_config.png | Bin 0 -> 202507 bytes
.../tutorial/4.0/overview/build_duration_ssb.png | Bin 0 -> 115357 bytes
.../tutorial/4.0/overview/query_response_ssb.png | Bin 0 -> 105312 bytes
.../tutorial/4.0/overview/query_response_tpch.png | Bin 0 -> 114768 bytes
.../tutorial/4.0/overview/result_size_ssb.png | Bin 0 -> 111943 bytes
website/website.iml | 9 ++
64 files changed, 882 insertions(+), 223 deletions(-)
copy website/_data/{development-cn.yml => development40-cn.yml} (96%)
copy website/_data/{development.yml => development40.yml} (96%)
delete mode 100644 website/_dev/howto_hbase_branches.cn.md
delete mode 100644 website/_dev/howto_hbase_branches.md
copy website/{_dev => _dev40}/about_temp_files.cn.md (94%)
copy website/{_dev => _dev40}/about_temp_files.md (94%)
copy website/{_dev => _dev40}/coding_naming_convention.cn.md (97%)
copy website/{_dev => _dev40}/coding_naming_convention.md (97%)
create mode 100644 website/_dev40/dev_env.cn.md
create mode 100644 website/_dev40/dev_env.md
copy website/{_dev => _dev40}/doc_spec.cn.md (93%)
copy website/{_dev => _dev40}/doc_spec.md (93%)
copy website/{_dev => _dev40}/howto_become_apache_committer.cn.md (95%)
copy website/{_dev => _dev40}/howto_become_apache_committer.md (95%)
copy website/{_dev => _dev40}/howto_contribute.cn.md (94%)
copy website/{_dev => _dev40}/howto_contribute.md (94%)
copy website/{_dev => _dev40}/howto_docs.cn.md (98%)
copy website/{_dev => _dev40}/howto_docs.md (98%)
copy website/{_dev => _dev40}/howto_package.cn.md (73%)
copy website/{_dev => _dev40}/howto_package.md (72%)
copy website/{_dev => _dev40}/howto_release.cn.md (99%)
copy website/{_dev => _dev40}/howto_release.md (99%)
create mode 100644 website/_dev40/howto_test.cn.md
create mode 100644 website/_dev40/howto_test.md
create mode 100644 website/_dev40/index.cn.md
create mode 100644 website/_dev40/index.md
copy website/{_dev => _dev40}/plugin_arch.cn.md (96%)
copy website/{_dev => _dev40}/plugin_arch.md (97%)
copy website/{_dev => _dev40}/web_tech.cn.md (95%)
copy website/{_dev => _dev40}/web_tech.md (96%)
copy website/_includes/{dev_nav.cn.html => dev40_nav.cn.html} (91%)
copy website/_includes/{dev_nav.html => dev40_nav.html} (92%)
copy website/_includes/{dev_ul.html => dev40_ul.cn.html} (78%)
copy website/_includes/{dev_ul.html => dev40_ul.html} (79%)
copy website/_layouts/{dev-cn.html => dev40-cn.html} (97%)
copy website/_layouts/{dev.html => dev40.html} (97%)
create mode 100644 website/_posts/blog/2021-07-02-Apache-Kylin4-A-new-storage-and-compute-architecture.md
create mode 100644 website/images/blog/kylin4/1 apache_kylin_introduction.png
create mode 100644 website/images/blog/kylin4/10 spark_parquet_solution.png
create mode 100644 website/images/blog/kylin4/11 spark_parquet_architecture.png
create mode 100644 website/images/blog/kylin4/2 cube_vs_cuboid.png
create mode 100644 website/images/blog/kylin4/3 cuboid_selected_for_query.png
create mode 100644 website/images/blog/kylin4/4 apache_kylin_query_process.png
create mode 100644 website/images/blog/kylin4/5 query_using_precomputed_data.png
create mode 100644 website/images/blog/kylin4/6 on-site-computation.png
create mode 100644 website/images/blog/kylin4/7 precomputation_using_aggregated_data.png
create mode 100644 website/images/blog/kylin4/8 diagram_of_calcite_executions.png
create mode 100644 website/images/blog/kylin4/9 diagram_of_bottleneck_on_HBase.png
create mode 100644 website/images/develop40/debug_tomcat_config.png
create mode 100644 website/images/tutorial/4.0/overview/build_duration_ssb.png
create mode 100644 website/images/tutorial/4.0/overview/query_response_ssb.png
create mode 100644 website/images/tutorial/4.0/overview/query_response_tpch.png
create mode 100644 website/images/tutorial/4.0/overview/result_size_ssb.png
create mode 100644 website/website.iml
[kylin] 02/03: Update overview of kylin4
Posted by xx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a commit to branch document
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit c40cc301f8dd59cde13d9b64ec9624a9485a5cb9
Author: yaqian.zhang <59...@qq.com>
AuthorDate: Fri Jul 2 18:39:04 2021 +0800
Update overview of kylin4
---
website/_docs/install/index.cn.md | 4 +-
website/_docs40/gettingstarted/quickstart.cn.md | 2 +-
website/_docs40/gettingstarted/quickstart.md | 2 +
website/_docs40/index.cn.md | 163 ++++++++++++++------
website/_docs40/index.md | 168 +++++++++++++++------
website/_docs40/install/kylin_cluster.cn.md | 4 +-
website/_docs40/install/kylin_cluster.md | 3 +
.../tutorial/4.0/overview/build_duration_ssb.png | Bin 0 -> 115357 bytes
.../tutorial/4.0/overview/query_response_ssb.png | Bin 0 -> 105312 bytes
.../tutorial/4.0/overview/query_response_tpch.png | Bin 0 -> 114768 bytes
.../tutorial/4.0/overview/result_size_ssb.png | Bin 0 -> 111943 bytes
11 files changed, 252 insertions(+), 94 deletions(-)
diff --git a/website/_docs/install/index.cn.md b/website/_docs/install/index.cn.md
index 743d7cc..2843535 100644
--- a/website/_docs/install/index.cn.md
+++ b/website/_docs/install/index.cn.md
@@ -42,7 +42,7 @@ Kylin 可以在 Hadoop 集群的任意节点上启动。方便起见，您可以
cd /usr/local/
wget http://mirror.bit.edu.cn/apache/kylin/apache-kylin-2.5.0/apache-kylin-2.5.0-bin-hbase1x.tar.gz
```
-
+
2. 解压 tar 包，配置环境变量 `$KYLIN_HOME` 指向 Kylin 文件夹。
```shell
@@ -51,7 +51,7 @@ cd apache-kylin-2.5.0-bin-hbase1x
export KYLIN_HOME=`pwd`
```
-从 v2.6.1 开始， Kylin 不再包含 Spark 二进制包; 您需要另外下载 Spark，然后设置 `SPARK_HOME` 系统变量到 Spark 安装目录：
+3. 从 v2.6.1 开始， Kylin 不再包含 Spark 二进制包; 您需要另外下载 Spark，然后设置 `SPARK_HOME` 系统变量到 Spark 安装目录：
```shell
export SPARK_HOME=/path/to/spark
diff --git a/website/_docs40/gettingstarted/quickstart.cn.md b/website/_docs40/gettingstarted/quickstart.cn.md
index e840919..78e76e0 100644
--- a/website/_docs40/gettingstarted/quickstart.cn.md
+++ b/website/_docs40/gettingstarted/quickstart.cn.md
@@ -67,7 +67,7 @@ $KYLIN_HOME/bin/download-spark.sh
脚本会将解压好的spark放在$KYLIN_HOME目录下，如果系统中没有设置SPARK_HOME，启动kylin时会自动找到$KYLIN_HOME目录下的spark。
-### ste4、配置 Mysql 元数据
+### step4、配置 Mysql 元数据
Kylin 4.0 使用 Mysql 作为元数据存储，需要在kylin.properties做如下配置：
diff --git a/website/_docs40/gettingstarted/quickstart.md b/website/_docs40/gettingstarted/quickstart.md
index 66d0558..6ca9c75 100644
--- a/website/_docs40/gettingstarted/quickstart.md
+++ b/website/_docs40/gettingstarted/quickstart.md
@@ -43,6 +43,8 @@ When your environment meets the above prerequisites, you can install and start u
#### Step1. Download the Kylin Archive
Download a kylin4.0 binary package from [Apache Kylin Download Site](https://kylin.apache.org/download/).
+
+```
cd /usr/local/
wget http://apache.website-solution.net/kylin/apache-kylin-4.0.0/apache-kylin-4.0.0-bin.tar.gz
```
diff --git a/website/_docs40/index.cn.md b/website/_docs40/index.cn.md
index 495853c..5bd4b4c 100644
--- a/website/_docs40/index.cn.md
+++ b/website/_docs40/index.cn.md
@@ -1,6 +1,6 @@
---
layout: docs40-cn
-title: 概述
+title: Apache Kylin4 概述
categories: docs
permalink: /cn/docs40/index.html
---
@@ -16,48 +16,127 @@ Apache Kylin™是一个开源的、分布式的分析型数据仓库，提供 H
* [v2.4 document](/cn/docs24/)
* [归档](/archive/)
-安装
-------------
-1. [安装指南](install/index.html)
-2. [Kylin 配置](install/configuration.html)
-3. [集群模式部署](install/kylin_cluster.html)
-4. [高级配置](install/advance_settings.html)
-5. [用 Docker 运行 Kylin](install/kylin_docker.html)
+Apache Kylin4.0 是 Apache Kylin3.x 之后一次重大的版本更新，它采用了全新的 Spark 构建引擎和 Parquet 作为存储，同时使用 Spark 作为查询引擎。
+
+Apache Kylin4.0 的第一个版本 kylin4.0.0-alpha 于 2020 年 7 月份发布，此后相继发布 kylin4.0.0-beta 以及正式版本。
+
+为了方便用户对 Kylin4.x 有更全面更深层的了解，本篇文档会着重从 Kylin4.x 与之前版本有何异同的角度对 Kylin4.x 做全面概述。文章分为以下几个部分：
+
+- 为什么选择 Parquet 替换 HBase
+- 预计算结果在 Kylin4.0 中如何存储
+- Kylin 4.0 的构建引擎
+- Kylin 4.0 的查询引擎
+- Kylin4.0 与 Kylin3.1 功能对比
+- Kylin 4.0 性能表现
+- Kylin 4.0 查询和构建调优
+- Kylin 4.0 用户案例
+
+## 为什么选择 Parquet 替换 HBase
+在 3.x 以及之前的版本中，kylin 一直使用 HBase 作为存储引擎来保存 cube 构建后产生的预计算结果。HBase 作为 HDFS 之上面向列族的数据库，查询表现已经算是比较优秀，但是它仍然存在以下几个缺点：
+1. HBase 不是真正的列式存储；
+2. HBase 没有二级索引，Rowkey 是它唯一的索引；
+3. HBase 没有对存储的数据进行编码，kylin 必须自己进行对数据编码的过程；
+4. HBase 不适合云上部署和自动伸缩；
+5. HBase 不同版本之间的 API 版本不同，存在兼容性问题（比如，0.98，1.0，1.1，2.0）；
+6. HBase 存在不同的供应商版本，他们之间有兼容性问题。
+
+针对以上问题，社区提出了对使用 Apache Parquet + Spark 来代替 HBase 的提议，理由如下：
+1. Parquet 是一种开源并且已经成熟稳定的列式存储格式；
+2. Parquet 对云更加友好，可以兼容各种文件系统，包括 HDFS、S3、Azure Blob store、Ali OSS 等；
+3. Parquet 可以很好地与 Hadoop、Hive、Spark、Impala 等集成；
+4. Parquet 支持自定义索引。
+
+## 预计算结果在 Kylin4.0 中如何存储
+在 Kylin4.x 中，预计算结果以 Parquet 格式存储在文件系统中，文件存储结构对于 I/O 优化很重要，提前对存储目录结构进行设计，就能够在查询时通过目录或者文件名过滤数据文件，避免不必要的扫描。
+Kylin4 对 cube 进行构建得到的预计算结果的 Parquet 文件在文件系统中存储的目录结构如下：
+- cube_name
+ - SegmentA
+ - Cuboid-111
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - Cuboid-222
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - SegmentB
+ - Cuboid-111
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - Cuboid-222
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+
+可以看出，与 HBase 相比，采用 Parquet 存储可以很方便地增删 cuboid 而不影响其他数据。利用这种特点，Kylin4 中实现了支持用户手动增删 cuboid 的功能，请参考：[How to update cuboid list for a cube](https://cwiki.apache.org/confluence/display/KYLIN/How+to+update+cuboid+list+for+a+cube)
+
+## Kylin 4.0 的构建引擎
+在 Kylin4 中，Spark Engine 是唯一的构建引擎，与之前版本中的构建引擎相比，存在如下特点：
+
+1、Kylin4 的构建简化了很多步骤。比如在 Cube Build Job 中， kylin4 只需要资源探测和 cubing 两个步骤，就可以完成构建；
+2、由于 Parquet 会对存储的数据进行编码，所以在 kylin4 中不再需要维度字典和对维度列编码的过程；
+3、Kylin4 对全局字典做了全新的实现，更多细节请参考：[Kylin4 全局字典](https://cwiki.apache.org/confluence/display/KYLIN/Global+Dictionary+on+Spark+CN) ；
+4、Kylin4 会根据集群资源、构建任务情况等对 Spark 进行自动调参；
+5、Kylin4 提高了构建速度。
+
+用户可以通过 `kylin.build.spark-conf` 开头的配置项手动修改构建相关的 Spark 配置，经过用户手动修改的 Spark 配置项不会再参与自动调参。
+
+## Kylin 4.0 的查询引擎
+Kylin4 的查询引擎 `Sparder(SparderContext)` 是由 spark application 后端实现的新型分布式查询引擎，相比于原来的查询引擎，Sparder 的优势体现在以下几点：
+- 分布式的查询引擎，有效避免单点故障；
+- 与构建所使用的计算引擎统一为 Spark；
+- 对于复杂查询的性能有很大提高；
+- 可以从 Spark 的新功能及其生态中获益。
+
+在 Kylin4 中，Sparder 是作为一个 long-running 的 spark application 存在的。 Sparder 会根据 `kylin.query.spark-conf` 开头的配置项中配置的 Spark 参数来获取 Yarn 资源，如果配置的资源参数过大，可能会影响构建任务甚至无法成功启动 Sparder，如果 Sparder 没有成功启动，则所有查询任务都会失败，用户可以在 kylin WebUI 的 System 页面中检查 Sparder 状态。
+
+默认情况下，用于查询的 spark 参数会设置的比较小，在生产环境中，大家可以适当把这些参数调大一些，以提升查询性能。
+`kylin.query.auto-sparder-context` 参数用于控制是否在启动 kylin 的同时启动 Sparder，默认值为 false，即默认情况下会在执行第一条 SQL 的时候才启动 Sparder，由于这个原因，执行第一条 SQL 的时候的会花费较长时间。
+如果你不希望第一条 SQL 的查询速度低于预期，可以设置 `kylin.query.auto-sparder-context` 为 `true`，此时 Sparder 会随 Kylin 一起启动。
+
+## Kylin 4.0 与 Kylin 3.1 功能对比
+
+| Feature | Kylin 3.1.0 | Kylin 4.0 |
+| --------------------- | :------------------------------------------- | :----------------------------------------------|
+| Storage | HBase | Parquet |
+| BuildEngine | MapReduce/Spark/Flink | New Spark Engine |
+| Metastore | HBase(Default)/Mysql | Mysql(Default) |
+| DataSource | Kafka/Hive/JDBC | Hive/CSV |
+| Global Dictionary | Two implementation | New implementation |
+| Cube Optimization Tool | Cube Planner | Cube Planner phase1 and Optimize cube manually |
+| Self-monitoring | System cube and Dashboard | System cube and Dashboard |
+| PushDown Engine | Hive/JDBC | Spark SQL |
+| Hadoop platform | HDP2/HDP3/CDH5/CDH6/EMR5 | HDP2/CDH5/CDH6/EMR5/EMR6/HDI |
+| Deployment mode | Single node/Cluster/Read and write separation| Single node/Cluster/Read and write separation |
+
+## Kylin 4.0 性能表现
+为了测试 Kylin4.0 的性能，我们分别在 SSB 数据集和 TPC-H 数据集上做了 benchmark，与 Kylin3.1.0 进行对比。测试环境为 4 个节点的 CDH 集群，所使用的 yarn 队列分配了 400G 内存和 128 cpu cores。
+性能测试对比结果如下：
+- Comparison of build duration and result size（SSB）
+
+
+
+测试结果可以体现以下两点：
+- kylin4 的构建速度与 kylin3.1.0 的 Spark Engine 相比有明显提升；
+- Kylin4 构建后得到的预计算结果 Parquet 文件大小与 HBase 相比有明显减小；
+
+- Comparison of query response(SSB and TPC-H)
+
+
+
+从查询结果对比中可以看出，对于***简单查询***，kylin3 与 Kylin4 不相上下，kylin4 略有不足；而对于***复杂查询***，kylin4 则体现出了明显的优势，查询速度比 kylin3 快很多。
+并且，Kylin4 中的***简单查询***的性能还存在很大的优化空间。在有赞使用 Kylin4 的实践中，对于***简单查询***的性能可以优化到 1 秒以内。
+
+## Kylin 4.0 查询和构建调优
+对于 Kylin4 的调优，请参考：[How to improve cube building and query performance](/docs40/howto/howto_optimize_build_and_query.html)
+
+## Kylin 4.0 用户案例
+[Why did Youzan choose Kylin4](/blog/2021/06/17/Why-did-Youzan-choose-Kylin4)
+
+参考链接：
+[Kylin Improvement Proposal 1: Parquet Storage](https://cwiki.apache.org/confluence/display/KYLIN/KIP-1%3A+Parquet+storage)
-教程
-------------
-1. [样例 Cube 快速入门](tutorial/kylin_sample.html)
-2. [Web 界面](tutorial/web.html)
-3. [Cube 创建](tutorial/create_cube.html)
-4. [Cube 构建和 Job 监控](tutorial/cube_build_job.html)
-5. [SQL 快速参考](tutorial/sql_reference.html)
-6. [优化 Cube 构建](tutorial/cube_build_performance.html)
-7. [查询下压](tutorial/query_pushdown.html)
-8. [建立 System Cube](tutorial/setup_systemcube.html)
-9. [使用 Cube Planner](tutorial/use_cube_planner.html)
-10. [使用 Dashboard](tutorial/use_dashboard.html)
-11. [优化构建和查询性能](howto/howto_optimize_build_and_query.html)
-
-
-工具集成
-------------
-1. [ODBC 驱动](tutorial/odbc.html)
-2. [JDBC 驱动](howto/howto_jdbc.html)
-3. [RESTful API 列表](howto/howto_use_restapi.html)
-4. [用 API 构建 Cube](howto/howto_build_cube_with_restapi.html)
-5. [MS Excel 及 PowerBI 教程](tutorial/powerbi.html)
-6. [Tableau 8](tutorial/tableau.html)
-7. [Tableau 9](tutorial/tableau_91.html)
-9. [Qlik Sense 集成](tutorial/Qlik.html)
-10. [Apache Superset](tutorial/superset.html)
-11. [Redash](/blog/2018/05/08/redash-kylin-plugin-strikingly/)
-12. [Davinci](/cn_blog/2019/11/29/Davinci-Kylin-Insight/)
-
-
-帮助
-------------
-1. [备份元数据](howto/howto_backup_metadata.html)
-2. [清理存储](howto/howto_cleanup_storage.html)
diff --git a/website/_docs40/index.md b/website/_docs40/index.md
index e41b5bf..ccaf410 100644
--- a/website/_docs40/index.md
+++ b/website/_docs40/index.md
@@ -1,6 +1,6 @@
---
layout: docs40
-title: Overview
+title: Overview of Apache Kylin4.x
categories: docs
permalink: /docs40/index.html
---
@@ -16,53 +16,127 @@ This is the document for Apache Kylin4.0. Document of other versions:
* [v2.4 document](/docs24)
* [Archived](/archive/)
-Installation & Setup
-------------
-1. [Installation Guide](install/index.html)
-2. [Configurations](install/configuration.html)
-3. [Deploy in cluster mode](install/kylin_cluster.html)
-4. [Advanced settings](install/advance_settings.html)
-5. [Run Kylin with Docker](install/kylin_docker.html)
+Apache kylin 4.0 is a major version after Apache kylin 3.x. Kylin4 uses a new spark build engine and parquet as storage, and uses spark as query engine.
-Tutorial
-------------
-1. [Quick Start with Sample Cube](tutorial/kylin_sample.html)
-2. [Web Interface](tutorial/web.html)
-3. [Cube Wizard](tutorial/create_cube.html)
-4. [Cube Build and Job Monitoring](tutorial/cube_build_job.html)
-5. [SQL reference](tutorial/sql_reference.html)
-6. [Cube Build Tuning](tutorial/cube_build_performance.html)
-7. [Enable Query Pushdown](tutorial/query_pushdown.html)
-8. [Setup System Cube](tutorial/setup_systemcube.html)
-9. [Optimize with Cube Planner](tutorial/use_cube_planner.html)
-10. [Use System Dashboard](tutorial/use_dashboard.html)
-11. [Optimize build and query](howto/howto_optimize_build_and_query.html)
-12. [Config Spark Pool](howto/howto_config_spark_pool.html)
-
-Connectivity and APIs
-------------
-1. [ODBC driver](tutorial/odbc.html)
-2. [JDBC driver](howto/howto_jdbc.html)
-3. [RESTful API list](howto/howto_use_restapi.html)
-4. [Build cube with RESTful API](howto/howto_build_cube_with_restapi.html)
-5. [Connect from MS Excel and PowerBI](tutorial/powerbi.html)
-6. [Connect from Tableau 8](tutorial/tableau.html)
-7. [Connect from Tableau 9](tutorial/tableau_91.html)
-8. [Connect from MicroStrategy](tutorial/microstrategy.html)
-9. [Connect from SQuirreL](tutorial/squirrel.html)
-10. [Connect from Apache Flink](tutorial/flink.html)
-11. [Connect from Apache Spark](tutorial/spark.html)
-12. [Connect from Hue](tutorial/hue.html)
-13. [Connect from Qlik Sense](tutorial/Qlik.html)
-14. [Connect from Apache Superset](tutorial/superset.html)
-15. [Connect from Redash](/blog/2018/05/08/redash-kylin-plugin-strikingly/)
-
-
-Operations
-------------
-1. [Backup/restore Kylin metadata](howto/howto_backup_metadata.html)
-2. [Cleanup storage](howto/howto_cleanup_storage.html)
-3. [Upgrade from old version](howto/howto_upgrade.html)
+Kylin 4.0.0-alpha, the first version of Apache kylin 4.0, was released in July 2020, and then kylin 4.0.0-beta and official version were released.
+
+In order to facilitate users to have a more comprehensive and deeper understanding of kylin4.x, this document will focus on a comprehensive overview of kylin4.x from the perspective of the similarities and differences between kylin4.x and previous versions.
+
+The article includes the following parts:
+
+- Why replace HBase with Parquet
+- How to store pre calculation results in kylin 4.0
+- Build engine of Kylin 4.0
+- Query engine of kylin 4.0
+- Feature comparison between kylin 4.0 and kylin 3.1
+- Kylin 4.0 performance
+- Kylin 4.0 query and build tuning
+- Kylin 4.0 use case
+
+## Why replace HBase with Parquet
+In versions 3.x and before, Kylin has been using HBase as a storage engine to save the precomputing results generated after cube builds. HBase, as the database of HDFS, has been excellent in query performance, but it still has the following disadvantages:
+
+1. HBase is not real columnar storage;
+2. HBase has no secondary index; Rowkey is the only index;
+3. HBase has no encoding, Kylin has to do the encoding by itself;
+4. HBase does not fit for cloud deployment and auto-scaling;
+5. HBase has different API versions and has compatible issues (e.g, 0.98, 1.0, 1.1, 2.0);
+6. HBase has different vendor releases and has compatible issues (e.g, Cloudera's is not compatible with others);
+
+In view of the above problems, Kylin community proposed to replace HBase with Apache parquet + spark, for the following reasons:
+
+1. parquet is an open source and mature and stable column storage format;
+2. Parquet is more cloud-friendly, can work with most FS including HDFS, S3, Azure Blob store, Ali OSS, etc;
+3. parquet can be well integrated with Hadoop, hive, spark, impala, etc;
+4. parquet supports custom index.
+
+
+## How to store pre calculation results in kylin 4.0
+In kylin4.x, the pre calculation results are stored in the file system in parquet format. The file storage structure is very important for I/O optimization. If the storage directory structure is designed in advance, the data files can be filtered through the directory or file name during query to avoid unnecessary file scan.
+
+The directory structure of parquet file stored in the file system is as follows:
+- cube_name
+ - SegmentA
+ - Cuboid-111
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - Cuboid-222
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - SegmentB
+ - Cuboid-111
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+ - Cuboid-222
+ - part-0000-XXX.snappy.parquet
+ - part-0001-XXX.snappy.parquet
+ - ...
+
+It can be seen that, using parquet storage can add and delete cuboid easily without affecting other data. With this feature, kylin4 realizes the feature of supporting users to add and delete cuboid manually. Please refer to: [how to update cuboid list for a cube]（ https://cwiki.apache.org/confluence/display/KYLIN/How+to+update+cuboid+list+for+a+cube )
+
+## Build engine of Kylin 4.0
+In kylin4, spark engine is the only build engine. Compared with the build engine in previous versions, it has the following characteristics:
+
+1. Building kylin4 simplifies many steps. For example, in cube build job, kylin4 only needs two steps: resource detection and cubing;
+2. Since parquet encodes the stored data, the process of dimension dictionary and dimension column encoding is no longer needed in kylin4;
+3. Kylin4 implements a new global dictionary. For more details, please refer to [kylin4 global dictionary](https://cwiki.apache.org/confluence/display/KYLIN/Global+Dictionary+on+Spark);
+4. Kylin4 will automatically adjust parameters of spark according to cluster resources and build job;
+5. Kylin4 can improve the build performance.
+
+Users can manually modify the build the relevant spark configuration through the configuration item beginning with `kylin.build.spark-conf`. The manually modified spark configuration item will no longer participate in automatic parameter adjustment.
+
+## Query engine of kylin 4.0
+
+`Sparder (spardercontext)`, the query engine of Kylin4, is a new distributed query engine implemented by the back end of spark application. Compared with the original query engine, sparder has the following advantages:
+1. Distributed query engine，avoid single-point-of-failure;
+2. Unified calculation engine for building and querying;
+3. There is a substantial increase in complex query performance;
+4. Can benefit from spark new features and ecology.
+
+In kylin4, sparder exists as a long-running spark application. Sparder will obtain the horn resource according to the spark parameter configured in the configuration item beginning with `kylin.query.spark-conf`. If the configured resource parameter is too large, the build engine may be affected, and even sparder cannot be started successfully. If sparder is not started successfully, all query tasks will fail, Users can check the sparder status in the system page of kylin webui.
+By default, the spark parameter used for query will be set smaller. In the production environment, you can increase these parameters appropriately to improve query performance.
+`Kylin.query.auto-sparder-context` parameter is used to control whether to start sparder when kylin is started. The default value is `false`, that is, sparder will be started only when the first SQL is executed by default. For this reason, it will take a long time to execute the first SQL.
+If you don't want the query speed of the first SQL to be lower than expected, you can set `kylin.query.auto-sparder-context` to `true`, and sparder will start with kylin.
+
+## Feature comparison between kylin 4.0 and kylin 3.1
+
+| Feature | Kylin 3.1.0 | Kylin 4.0 |
+| --------------------- | :------------------------------------------- | :----------------------------------------------|
+| Storage | HBase | Parquet |
+| BuildEngine | MapReduce/Spark/Flink | New Spark Engine |
+| Metastore | HBase(Default)/Mysql | Mysql(Default) |
+| DataSource | Kafka/Hive/JDBC | Hive/CSV |
+| Global Dictionary | Two implementation | New implementation |
+| Cube Optimization Tool | Cube Planner | Cube Planner phase1 and Optimize cube manually |
+| Self-monitoring | System cube and Dashboard | System cube and Dashboard |
+| PushDown Engine | Hive/JDBC | Spark SQL |
+| Hadoop platform | HDP2/HDP3/CDH5/CDH6/EMR5 | HDP2/CDH5/CDH6/EMR5/EMR6/HDI |
+| Deployment mode | Single node/Cluster/Read and write separation| Single node/Cluster/Read and write separation |
+
+## Kylin 4.0 performance
+In order to test the performance of kylin 4.0, we benchmark SSB dataset and TPC-H dataset respectively, and compare with kylin 3.1.0. The test environment is a 4-node CDH cluster, and the yarn queue is used to allocate 400G memory and 128 CPU cores.
+The results of performance test are as follows:
+
+- Comparison of build duration and result size（SSB）
+
+
+
+
+The test results can reflect the following two points:
+- The build speed of kylin4 is significantly higher than that of kylin3.1.0 spark engine;
+- Compared with HBase, the parquet file size of kylin4 is significantly reduced;
+
+From the comparison of query results, it can be seen that kylin3 and kylin4 are the same for ***simple query***, kylin4 is slightly insufficient; However, kylin4 has obvious advantages over kylin3 for ***complex query***.
+Moreover, there is still a lot of room to optimize the performance of ***simple query*** in kylin4. In the practice of Youzan using kylin4, the performance of ***simple query*** can be optimized to less than 1 second.
+## Kylin 4.0 query and build tuning
+For kylin4 tuning, please refer to: [How to improve cube building and query performance](/docs40/howto/howto_optimize_build_and_query.html)
+## Kylin 4.0 use case
+[Why did Youzan choose Kylin4](/blog/2021/06/17/Why-did-Youzan-choose-Kylin4)
+Reference link:
+[Kylin Improvement Proposal 1: Parquet Storage](https://cwiki.apache.org/confluence/display/KYLIN/KIP-1%3A+Parquet+storage)
diff --git a/website/_docs40/install/kylin_cluster.cn.md b/website/_docs40/install/kylin_cluster.cn.md
index f3dc8c2..a021c68 100644
--- a/website/_docs40/install/kylin_cluster.cn.md
+++ b/website/_docs40/install/kylin_cluster.cn.md
@@ -47,9 +47,9 @@ kylin.server.self-discovery-enabled=true
```
更多关于kylin任务调度器的细节可以参考[Apache Kylin Wiki](https://cwiki.apache.org/confluence/display/KYLIN/Comparison+of+Kylin+Job+scheduler).
-
-
### 安装负载均衡器
为了将查询请求发送给集群而非单个节点，您可以部署一个负载均衡器，如 [Nginx](http://nginx.org/en/)， [F5](https://www.f5.com/) 或 [cloudlb](https://rubygems.org/gems/cloudlb/) 等，使得客户端和负载均衡器通信代替和特定的 Kylin 实例通信。
+### 读写分离部署
+Kylin4 的读写分离部署方式与 Kylin3 存在一定的差异，请参考文档：[Read Write Separation Deployment for Kylin 4](https://cwiki.apache.org/confluence/display/KYLIN/Read-Write+Separation+Deployment+for+Kylin+4.0)
diff --git a/website/_docs40/install/kylin_cluster.md b/website/_docs40/install/kylin_cluster.md
index e4f7143..a2dd647 100644
--- a/website/_docs40/install/kylin_cluster.md
+++ b/website/_docs40/install/kylin_cluster.md
@@ -51,3 +51,6 @@ For more details about the kylin job scheduler, please refer to [Apache Kylin Wi
### Installing a load balancer
To send query requests to a cluster instead of a single node, you can deploy a load balancer such as [Nginx](http://nginx.org/en/), [F5](https://www.f5.com/) or [cloudlb](https://rubygems.org/gems/cloudlb/), etc., so that the client and load balancer communication instead communicate with a specific Kylin instance.
+
+### Read and write separation deployment
+There are some differences between read and write separation deployment of kylin 4 and kylin 3, Please refer to : [Read Write Separation Deployment for Kylin 4](https://cwiki.apache.org/confluence/display/KYLIN/Read-Write+Separation+Deployment+for+Kylin+4.0)
\ No newline at end of file
diff --git a/website/images/tutorial/4.0/overview/build_duration_ssb.png b/website/images/tutorial/4.0/overview/build_duration_ssb.png
new file mode 100644
index 0000000..2484192
Binary files /dev/null and b/website/images/tutorial/4.0/overview/build_duration_ssb.png differ
diff --git a/website/images/tutorial/4.0/overview/query_response_ssb.png b/website/images/tutorial/4.0/overview/query_response_ssb.png
new file mode 100644
index 0000000..4bacf1a
Binary files /dev/null and b/website/images/tutorial/4.0/overview/query_response_ssb.png differ
diff --git a/website/images/tutorial/4.0/overview/query_response_tpch.png b/website/images/tutorial/4.0/overview/query_response_tpch.png
new file mode 100644
index 0000000..5b3cbc4
Binary files /dev/null and b/website/images/tutorial/4.0/overview/query_response_tpch.png differ
diff --git a/website/images/tutorial/4.0/overview/result_size_ssb.png b/website/images/tutorial/4.0/overview/result_size_ssb.png
new file mode 100644
index 0000000..8fd5e1c
Binary files /dev/null and b/website/images/tutorial/4.0/overview/result_size_ssb.png differ
[kylin] 01/03: Add dev_40
Posted by xx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a commit to branch document
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit efe2b4fcd63ba7207eddf060b5cd1d53ca134fef
Author: yaqian.zhang <59...@qq.com>
AuthorDate: Wed Jun 30 10:59:17 2021 +0800
Add dev_40
---
website/_config.yml | 6 +-
website/_data/development40-cn.yml | 34 ++
website/_data/development40.yml | 34 ++
website/_dev/howto_hbase_branches.cn.md | 24 -
website/_dev/howto_hbase_branches.md | 24 -
website/_dev/index.cn.md | 3 +
website/_dev/index.md | 3 +
website/_dev40/about_temp_files.cn.md | 22 +
website/_dev40/about_temp_files.md | 21 +
website/_dev40/coding_naming_convention.cn.md | 62 +++
website/_dev40/coding_naming_convention.md | 62 +++
website/_dev40/dev_env.cn.md | 166 +++++++
website/_dev40/dev_env.md | 170 ++++++++
website/_dev40/doc_spec.cn.md | 89 ++++
website/_dev40/doc_spec.md | 96 ++++
website/_dev40/howto_become_apache_committer.cn.md | 24 +
website/_dev40/howto_become_apache_committer.md | 24 +
website/_dev40/howto_contribute.cn.md | 117 +++++
website/_dev40/howto_contribute.md | 117 +++++
website/_dev40/howto_docs.cn.md | 202 +++++++++
website/_dev40/howto_docs.md | 204 +++++++++
website/_dev40/howto_package.cn.md | 33 ++
website/_dev40/howto_package.md | 33 ++
website/_dev40/howto_release.cn.md | 475 ++++++++++++++++++++
website/_dev40/howto_release.md | 485 +++++++++++++++++++++
website/_dev40/howto_test.cn.md | 24 +
website/_dev40/howto_test.md | 23 +
website/_dev40/index.cn.md | 37 ++
website/_dev40/index.md | 34 ++
website/_dev40/plugin_arch.cn.md | 42 ++
website/_dev40/plugin_arch.md | 42 ++
website/_dev40/web_tech.cn.md | 46 ++
website/_dev40/web_tech.md | 46 ++
website/_includes/dev40_nav.cn.html | 33 ++
website/_includes/dev40_nav.html | 33 ++
website/_includes/dev40_ul.cn.html | 27 ++
website/_includes/dev40_ul.html | 27 ++
website/_layouts/dev40-cn.html | 47 ++
website/_layouts/dev40.html | 47 ++
website/images/develop40/debug_tomcat_config.png | Bin 0 -> 202507 bytes
website/website.iml | 9 +
41 files changed, 2998 insertions(+), 49 deletions(-)
diff --git a/website/_config.yml b/website/_config.yml
index f4c8c1e..763f5a8 100644
--- a/website/_config.yml
+++ b/website/_config.yml
@@ -27,7 +27,7 @@ encoding: UTF-8
timezone: America/Dawson
exclude: ["README.md", "Rakefile", "*.scss", "*.haml", "*.sh", "Dockerfile", "Makefile"]
-include: [_docs,_docs20,_docs21,_docs23,_docs24,_docs30,_docs31,_docs40,_dev,community,blog]
+include: [_docs,_docs20,_docs21,_docs23,_docs24,_docs30,_docs31,_docs40,_dev,_dev40,community,blog]
# Build settings
markdown: kramdown
@@ -90,6 +90,10 @@ collections:
output: true
dev-cn:
output: true
+ dev40:
+ output: true
+ dev40-cn:
+ output: true
community:
output: true
blog:
diff --git a/website/_data/development40-cn.yml b/website/_data/development40-cn.yml
new file mode 100644
index 0000000..44c9dd1
--- /dev/null
+++ b/website/_data/development40-cn.yml
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Development menu items, for English one, dev-cn.yml is for Chinese one
+# The development menu is constructed in dev_nav.html with these data
+
+- title: 开发指南
+ dev:
+ - index
+ - dev_env
+ - coding_naming_convention
+ - howto_test
+ - howto_contribute
+ - howto_become_apache_committer
+ - howto_docs
+ - doc_spec
+ - howto_package
+ - howto_release
+ - plugin_arch
+ - new_metadata
+ - web_tech
+ - about_temp_files
diff --git a/website/_data/development40.yml b/website/_data/development40.yml
new file mode 100644
index 0000000..8090c64
--- /dev/null
+++ b/website/_data/development40.yml
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to you under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Development menu items, for English one, dev-cn.yml is for Chinese one
+# The development menu is constructed in dev_nav.html with these data
+
+- title: Development Guide
+ dev:
+ - index
+ - dev_env
+ - coding_naming_convention
+ - howto_test
+ - howto_contribute
+ - howto_become_apache_committer
+ - howto_docs
+ - doc_spec
+ - howto_package
+ - howto_release
+ - plugin_arch
+ - new_metadata
+ - web_tech
+ - about_temp_files
diff --git a/website/_dev/howto_hbase_branches.cn.md b/website/_dev/howto_hbase_branches.cn.md
deleted file mode 100644
index 18cc673..0000000
--- a/website/_dev/howto_hbase_branches.cn.md
+++ /dev/null
@@ -1,24 +0,0 @@
----
-layout: dev-cn
-title: 如何维护 HBase 分支
-categories: development
-permalink: /cn/development/howto_hbase_branches.html
----
-
-### Kylin 不同版本的 HBase 分支
-
-因为 HBase API 基于版本和供应商的不同，因此必须针对不同的 HBase 版本维护不同的代码分支。
-
-分支设计为：
-
-- `master` 分支编译的是 HBase 1.1，也是开发的主要分支。 所有错误修复和新功能仅提交给 `master`。
-- `master-hadoop3.1` 分支编译的是 Hadoop 3.1 + HBase 2.x。通过在 `master` 上应用若干个 patch 来创建此分支。换句话说，`master-hadoop3.1` = `master` + `patches to support HBase 2.x`。
-- `master-hbase0.98` 已经弃之不用，0.98 用户建议升级 HBase。
-- 另外有若干个 Kylin 版本维护分支，如 2.5.x，2.4.x 等；如果你提了一个 patch 或 Pull Request，请告知 reviewer 哪几个版本需要此 patch， reviewer 会把 patch 合并到除 master 以外的其它分支。
-- 在 `master-hadoop3.1` 上不会直接发生代码更改（除非分支上最后一次提交采用了 HBase 调用）。
-
-有一个脚本有助于保持这些分支同步：`dev-support/sync_hbase_cdh_branches.sh`。
-
-
-
-
diff --git a/website/_dev/howto_hbase_branches.md b/website/_dev/howto_hbase_branches.md
deleted file mode 100644
index 48a2dd5..0000000
--- a/website/_dev/howto_hbase_branches.md
+++ /dev/null
@@ -1,24 +0,0 @@
----
-layout: dev
-title: How to Maintain Hadoop/HBase Branches
-categories: development
-permalink: /development/howto_hbase_branches.html
----
-
-### Kylin Branches for Different Hadoop/HBase Versions
-
-Because HBase API diverges based on versions and vendors, different code branches have to be maintained for different HBase versions.
-
-The branching design is
-
-- The `master` branch compiles with HBase 1.1, and is also the main branch for development. All bug fixes and new features commits to `master` only.
-- The `master-hadoop3.1` branch compiles with Hadoop 3.1 and HBase 1.x. This branch is created by applying several patches on top of `master`. In other word, `master-hadoop3.1` = `master` + `patches to support Hadoop 3 and HBase 2.x`.
-- The `master-hbase0.98` is deprecated;
-- There are several release maintenance branches like `2.5.x`, `2.4.x`. If you have a PR or patch, please let reviewer knows which branch it need be applied. The reviewer should cherry-pick the patch to a specific branch after be merged into master.
-- No code changes should happen on `master-hadoop3.1`, `master-hbase0.98` directly (apart from the last commit on the branch that adapts HBase calls).
-
-There is a script helps to keep these branches in sync: `dev-support/sync_hbase_cdh_branches.sh`.
-
-
-
-
diff --git a/website/_dev/index.cn.md b/website/_dev/index.cn.md
index b67591b..cf9d7d0 100644
--- a/website/_dev/index.cn.md
+++ b/website/_dev/index.cn.md
@@ -6,6 +6,9 @@ permalink: /cn/development/index.html
Apache Kylin 一直寻求的不只是代码的贡献，还寻求使用文档，性能报告，问答等方面的贡献。所有类型的贡献都为成为 Kylin Committer 铺平了道路。每个人都有机会，尤其是那些有分析和解决方案背景的，因为缺少来自于用户和解决方案视角的内容。
+这里是适用于 Apache Kylin3.x 及以前版本的开发文档，查看其他版本开发文档：
+* [v4.x 版本开发文档](/cn/development40/)
+
### 如何贡献
查看[如何贡献](/cn/development/howto_contribute.html)文档。
diff --git a/website/_dev/index.md b/website/_dev/index.md
index 6265c2b..4a1765d 100644
--- a/website/_dev/index.md
+++ b/website/_dev/index.md
@@ -6,6 +6,9 @@ permalink: /development/index.html
Apache Kylin is always looking for contributions of not only code, but also usage document, performance report, Q&A etc. All kinds of contributions pave the way towards a Kylin Committer. There is opportunity for everyone, especially for those come from analysis and solution background, due to the lacking of content from user and solution perspective.
+Here is the development document for Apache kylin 3.x and earlier version. Check the development documents of other versions:
+* [v4.x development documents](/development40/)
+
### How to Contribute
Check out the [How to Contribute](/development/howto_contribute.html) document.
diff --git a/website/_dev40/about_temp_files.cn.md b/website/_dev40/about_temp_files.cn.md
new file mode 100644
index 0000000..0e117c2
--- /dev/null
+++ b/website/_dev40/about_temp_files.cn.md
@@ -0,0 +1,22 @@
+---
+layout: dev40-cn
+title: 关于临时文件
+categories: development
+permalink: /cn/development40/about_temp_files.html
+---
+
+在我们查看代码时，我们发现 Kylin 在下列位置留下了大量垃圾文件：
+
+* 客户端的本地文件系统
+* HDFS
+* hadoop 节点的本地文件系统
+
+开了一个单子来追踪这个 issue：
+[https://issues.apache.org/jira/browse/KYLIN-926](https://issues.apache.org/jira/browse/KYLIN-926)
+
+为了将来的开发，请：
+
+* 每当您想在本地创建临时文件时，请选择 `File.createTempFile` 或使用文件夹：`BatchConstants.CFG_KYLIN_LOCAL_TEMP_DIR(/tmp/kylin)`，不要随意使用
+ `/tmp` 中的另一个文件夹，它会变得一团糟，看起来不专业。
+* 每当您在本地创建临时文件时，请记得在使用之后删除它。最好使用 `FileUtils.forceDelete`，因为它也适用于删除文件夹。如果 Kylin 异常退出，请尽量避免使用 `deleteOnExit` 方法。
+* 每当您想要在 HDFS 中创建文件，请尝试在 `kylin.hdfs.working.dir` 或 `BatchConstants.CFG_KYLIN_HDFS_TEMP_DIR` 下创建文件，以及记得在它不再有用之后删除它。尽量避免投掷一切到 hdfs:///tmp 并将其留作垃圾。
diff --git a/website/_dev40/about_temp_files.md b/website/_dev40/about_temp_files.md
new file mode 100644
index 0000000..43c21d6
--- /dev/null
+++ b/website/_dev40/about_temp_files.md
@@ -0,0 +1,21 @@
+---
+layout: dev40
+title: "About Temp Files"
+categories: development
+permalink: /development40/about_temp_files.html
+---
+
+As we reviewed the code we found that Kylin left lots of garbage files in:
+
+* Local file system of the CLI
+* HDFS
+* Local file system of the hadoop nodes.
+
+A ticket was opened to track this issue:
+[https://issues.apache.org/jira/browse/KYLIN-926](https://issues.apache.org/jira/browse/KYLIN-926)
+
+For future developments, please:
+
+* Whenever you want to create temp files at Local, choose `File.createTempFile` or use the folder: `BatchConstants.CFG_KYLIN_LOCAL_TEMP_DIR(/tmp/kylin)`, do not randomly use another folder in `/tmp`, it will end up a mess, and look unprofessional.
+* Whenever you create temp files at Local, remember to delete it after using it. It's best to use `FileUtils.forceDelete`, as it also works for deleting folders. Try avoid `deleteOnExit`, in case Kylin exits abnormally.
+* Whenever you want to create files in HDFS, try to create it under `kylin.hdfs.working.dir` or `BatchConstants.CFG_KYLIN_HDFS_TEMP_DIR`, and remember to delete it after it is no longer useful. Try avoid throwing everything into hdfs:///tmp and leave it as garbage.
diff --git a/website/_dev40/coding_naming_convention.cn.md b/website/_dev40/coding_naming_convention.cn.md
new file mode 100644
index 0000000..057dbfe
--- /dev/null
+++ b/website/_dev40/coding_naming_convention.cn.md
@@ -0,0 +1,62 @@
+---
+layout: dev40-cn
+title: 编码和命名惯例
+categories: development
+permalink: /cn/development40/coding_naming_convention.html
+---
+
+## 编码惯例
+
+团队合作中编码管理是非常重要的。它不仅使得代码整齐及整洁，它也节省了许多的工作。不同的代码惯例（自动格式化）将会导致不必要的代码改动使得在代码复审和合并时需要更多的努力。
+
+对于 Java 代码，我们使用 Eclipse 默认的格式化配置，其中一个改动允许长行。
+
+- 对于 Eclipse 开发者，无需手动设置。代码格式化配置 `.settings/org.eclipse.jdt.core.prefs` 在 git 仓库。当项目被引入时，您的 IDE 应该自动配置。
+- 对于 intellij IDEA 开发者，您需要安装 "Eclipse Code Formatter" 并手动将 Eclipse 格式化配置加载到您的 IDE。详细内容请看[搭建开发环境](dev_env.html)。
+- 我们在 maven 中有 *checkstyle plugin* 能强制进行规范检查。
+
+对于 JavaScript，XML 和其它编码，请使用空格来缩进。作为常规，请保持代码格式与现有行一致。目前没有其他的强制措施。
+
+
+
+## 配置命名惯例
+
+Kylin 配置名称（在 `kylin.properties` 中的那些)
+
+- 惯例是 `dot.separated.namespace.config-name-separated-by-dash`，所有的字符都是小写。
+- 基本原理：点分隔前缀用于命名空间，如 java 包名。最后一级与类名相同，但要小写并以短划线分隔。结果与常见的 hadoop 配置名称一致，即 `dfs.namenode.servicerpc-bind-host`。
+- 正面例子：`kylin.metadata.url`， `kylin.snapshot.max-mb`。
+- 反面例子：`kylin.cube.customEncodingFactories`，应该为 `kylin.cube.custom-encoding-factories`。
+- 名称的命名空间（或包）应映射来自配置中使用的 Java 项目和包。 下面是当前名称空间的列表：
+ - kylin.env
+ - kylin.metadata
+ - kylin.snapshot
+ - kylin.dictionary
+ - kylin.cube
+ - kylin.job
+ - kylin.engine
+ - kylin.engine.mr
+ - kylin.engine.spark
+ - kylin.source
+ - kylin.source.hive
+ - kylin.source.kafka
+ - kylin.storage
+ - kylin.storage.hbase
+ - kylin.storage.partition
+ - kylin.query
+ - kylin.security
+ - kylin.server
+ - kylin.web
+
+
+
+## 配置文件命名惯例
+
+对于配置文件如日志配置，spring 配置，mapreduce job 配置等。
+
+- 惯例是 `words-separated-by-dash.ext`，所有的字符要小写。
+- 基本原理：要和 hadoop 配置文件命名一致，即 `hdfs-site.xml`。
+- 正面例子：`kylin-server-log4j.properties`。
+- 反面例子：`kylin_hive_conf.xml`，应该为 `kylin-hive-conf.xml`。
+
+
diff --git a/website/_dev40/coding_naming_convention.md b/website/_dev40/coding_naming_convention.md
new file mode 100644
index 0000000..0ec37f8
--- /dev/null
+++ b/website/_dev40/coding_naming_convention.md
@@ -0,0 +1,62 @@
+---
+layout: dev40
+title: Coding and Naming Convention
+categories: development
+permalink: /development40/coding_naming_convention.html
+---
+
+## Coding Convention
+
+Coding convention is very important for teamwork. Not only it keeps code neat and tidy, it saves a lot of work too. Different coding convention (and auto formatter) will cause unnecessary code changes that requires more effort at code review and code merge.
+
+For Java code, we use Eclipse default formatter setting, with one change that to allow long lines.
+
+- For Eclipse developers, no manual setting is required. Code formatter configurations `.settings/org.eclipse.jdt.core.prefs` is on git repo. Your IDE should be auto configured when the projects are imported.
+- For intellij IDEA developers, you need to install "Eclipse Code Formatter" and load the Eclipse formatter settings into your IDE manually. See [Setup Development Env](dev_env.html) for details.
+- We have *checkstyle plugin* enabled in maven to enforce some convention checks.
+
+For JavaScript, XML, and other code, please use space for indent. And as a general rule, keep your code format consistent with existing lines. No other enforcement at the moment.
+
+
+
+## Configuration Naming Convention
+
+For Kylin configuration names (those in `kylin.properties`)
+
+- The convention is `dot.separated.namespace.config-name-separated-by-dash`, all chars in lower case.
+- Rationale: The dot separated prefix is for namespace, like java packages. The last level is like class name, but in lower case and separated by dash. The result is consistent with common hadoop config names, i.e. `dfs.namenode.servicerpc-bind-host`.
+- Good examples: `kylin.metadata.url`; `kylin.snapshot.max-mb`
+- Bad exmaples: `kylin.cube.customEncodingFactories`, should be `kylin.cube.custom-encoding-factories`
+- Namespaces (or packages) of the names should map from the Java project and package where the configuration is used. Below is a list of current namespaces.
+ - kylin.env
+ - kylin.metadata
+ - kylin.snapshot
+ - kylin.dictionary
+ - kylin.cube
+ - kylin.job
+ - kylin.engine
+ - kylin.engine.mr
+ - kylin.engine.spark
+ - kylin.source
+ - kylin.source.hive
+ - kylin.source.kafka
+ - kylin.storage
+ - kylin.storage.hbase
+ - kylin.storage.partition
+ - kylin.query
+ - kylin.security
+ - kylin.server
+ - kylin.web
+
+
+
+## Configuration File Naming Convention
+
+For configuration files like logging config, spring config, mapreduce job config etc.
+
+- The convention is `words-separated-by-dash.ext`, all chars in lower case.
+- Rationale: Be consistent with hadoop config file names, i.e. `hdfs-site.xml`.
+- Good example: `kylin-server-log4j.properties`.
+- Bad example: `kylin_hive_conf.xml`, should be `kylin-hive-conf.xml`.
+
+
diff --git a/website/_dev40/dev_env.cn.md b/website/_dev40/dev_env.cn.md
new file mode 100644
index 0000000..8c031e0
--- /dev/null
+++ b/website/_dev40/dev_env.cn.md
@@ -0,0 +1,166 @@
+---
+layout: dev40-cn
+title: 搭建开发环境
+categories: development
+permalink: /cn/development40/dev_env.html
+---
+
+开发者想要在他们的开发机器上运行 Kylin4 的测试用例或应用。
+
+跟随这个教程，您可以很方便的在本地机器上搭建一个 Kylin4 的开发环境，不需要连接 Hadoop 客户端或者沙箱。
+
+## 开发机器的环境
+
+### 安装 Maven
+
+最新的 Maven 下载地址：<http://maven.apache.org/download.cgi>，然后创建一个软链接，以便 `mvn` 可以在任何地方运行。
+
+{% highlight Groff markup %}
+cd ~
+wget http://xenia.sote.hu/ftp/mirrors/www.apache.org/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
+tar -xzvf apache-maven-3.2.5-bin.tar.gz
+ln -s /root/apache-maven-3.2.5/bin/mvn /usr/bin/mvn
+{% endhighlight %}
+
+### 安装 Spark
+
+在像 /usr/local/spark 这样的本地文件夹下手动安装 Spark；Kylin4 支持 Spark2.4.6，你需要从 Spark 下载页面获取下载链接。
+
+{% highlight Groff markup %}
+wget -O /tmp/spark-2.4.6-bin-hadoop2.7.tgz https://archive.apache.org/dist/spark/spark-2.4.6/spark-2.4.6-bin-hadoop2.7.tgz
+cd /usr/local
+tar -zxvf /tmp/spark-2.4.6-bin-hadoop2.7.tgz
+ln -s spark-2.4.6-bin-hadoop2.7 spark
+{% endhighlight %}
+
+### 编译
+
+首先将 Kylin 工程下载到本地：
+
+{% highlight Groff markup %}
+git clone https://github.com/apache/kylin.git
+{% endhighlight %}
+
+将 Kylin 工件安装到 Maven 仓库：
+
+{% highlight Groff markup %}
+mvn clean install -DskipTests
+{% endhighlight %}
+
+
+### 运行单元测试
+运行单元测试来测试每一个 classes 基本功能的有效性。
+
+{% highlight Groff markup %}
+mvn clean test
+{% endhighlight %}
+
+### 运行集成测试
+执行以下命令，将会运行单元测试和集成测试。提交代码前，单元测试和集成测试都需要运行通过。
+
+{% highlight Groff markup %}
+mvn clean test -DskipRunIt=false
+{% endhighlight %}
+
+关于测试想要了解更多，请参考[如何测试](/cn/development40/howto_test.html).
+
+### 本地运行 Kylin Web 服务器
+
+拷贝 server/src/main/webapp/WEB-INF 到 webapp/app/WEB-INF
+
+{% highlight Groff markup %}
+cp -r server/src/main/webapp/WEB-INF webapp/app/WEB-INF
+{% endhighlight %}
+
+为 Kylin web GUI 下载 JS。`npm` 是 `Node.js` 的一部分，请搜索有关如何在您的操作系统上安装它的信息。
+
+{% highlight Groff markup %}
+cd webapp
+npm install -g bower
+bower --allow-root install
+{% endhighlight %}
+
+如果在 bower install 的过程当中遇到问题，可以尝试命令：
+
+{% highlight Groff markup %}
+git config --global url."git://".insteadOf https://
+{% endhighlight %}
+
+注意，如果是在 Windows 上，安装完 bower，需要将 "bower.cmd" 的路径加入系统环境变量 'PATH' 中，然后运行：
+
+{% highlight Groff markup %}
+bower.cmd --allow-root install
+{% endhighlight %}
+
+在配置文件 **examples/test_case_data/sandbox/kylin.properties** 中找到以下配置，并按照下面的的示例修改：
+
+```
+# Need to use absolute pat
+kylin.metadata.url=${KYLIN_SOURCE_DIR}/examples/test_case_data/sample_local
+kylin.storage.url=${KYLIN_SOURCE_DIR}/examples/test_case_data/sample_local
+kylin.env.zookeeper-is-local=true
+kylin.env.hdfs-working-dir=file://$KYLIN_SOURCE_DIR/examples/test_case_data/sample_local
+kylin.engine.spark-conf.spark.master=local
+# Need to create `/path/to/local/dir` manually
+kylin.engine.spark-conf.spark.eventLog.dir=/path/to/local/dir
+kylin.engine.spark-conf.spark.sql.shuffle.partitions=1
+kylin.env=LOCAL
+```
+
+在 IDE，运行 `org.apache.kylin.rest.DebugTomcat`。将工作目录设置为 /server 文件夹，使用 "kylin-server" 的 classpath。在运行之前，请在 IDE 安装 Scala 插件，以保证能够编译 Spark 代码。对于 IntelliJ IDEA 2017 或之前的用户，需要修改 "server/kylin-server.iml" 文件，将所有的 "PROVIDED" 替换为 "COMPILE"；对于 IntelliJ IDEA 2018 用户，请勾选 “Include dependencies with 'Provided' scope”，否则可能会抛出 "java.lang.NoClassDefFoundError: org/apache/catalina/LifecycleListener" 错误。
+
+并调节 VM options:
+
+```
+-Dspark.local=true
+```
+
+
+
+`DebugTomcat` 运行成功后，查看 Kylin Web `http://localhost:7070/kylin`（用户名：ADMIN，密码：KYLIN)
+
+## 安装 IDE 编码格式化工具
+
+如果你正在为 Kylin 编写代码，你应该确保你的代码符合预期的格式。
+
+对于 Eclipse 用户，只需在提交代码之前格式化代码。
+
+对于 intellij IDEA 用户，您还需要执行一些额外步骤：
+
+1. 安装 "Eclipse Code Formatter" 并在 core-common/.settings 中使用 "org.eclipse.jdt.core.prefs" 和 "org.eclipse.jdt.ui.prefs" 来配置 "Eclipse Java Formatter config file" 和 "Import order"
+
+ 
+
+2. 去 Preference => Code Style => Java，将 "Scheme" 设为默认，并设置 "Class count to use import with '\*'" 和 "Names count to use static import with '\*'" 为 99。
+
+ 
+
+3. 禁用 intellij IDEA 的 "Optimize imports on the fly"
+
+ 
+
+3. 提交代码前格式化代码。
+
+## 设置 IDE license 头部模板
+
+每一个源文件都应该包括以下的 Apache License 头部
+{% highlight Groff markup %}
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endhighlight %}
+
+当打包时 checkstyle 插件将会检查头部规则。license 文件在 `dev-support/checkstyle-apache-header.txt`。为了方便开发人员，请将头部添加为 `Copyright Profile`，并将其设置为 Kylin 项目的默认值。
+
diff --git a/website/_dev40/dev_env.md b/website/_dev40/dev_env.md
new file mode 100644
index 0000000..8a88f2d
--- /dev/null
+++ b/website/_dev40/dev_env.md
@@ -0,0 +1,170 @@
+---
+layout: dev40
+title: Setup Development Env
+categories: development
+permalink: /development40/dev_env.html
+---
+
+Developers want to run Kylin4 test cases or applications at their development machine.
+
+Following this tutorial, you can easily build a kylin4 development environment on your local machine without connecting to a Hadoop client or sandbox.
+
+## Environment on the dev machine
+
+
+### Install Maven
+
+The latest maven can be found at <http://maven.apache.org/download.cgi>, we create a symbolic so that `mvn` can be run anywhere.
+
+{% highlight Groff markup %}
+cd ~
+wget http://xenia.sote.hu/ftp/mirrors/www.apache.org/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
+tar -xzvf apache-maven-3.2.5-bin.tar.gz
+ln -s /root/apache-maven-3.2.5/bin/mvn /usr/bin/mvn
+{% endhighlight %}
+
+### Install Spark
+
+Manually install the Spark binary in a local folder like /usr/local/spark. Kylin4 supports spark 2.4.6, you need to get the download link from the spark download page.
+
+{% highlight Groff markup %}
+wget -O /tmp/spark-2.4.6-bin-hadoop2.7.tgz https://archive.apache.org/dist/spark/spark-2.4.6/spark-2.4.6-bin-hadoop2.7.tgz
+cd /usr/local
+tar -zxvf /tmp/spark-2.4.6-bin-hadoop2.7.tgz
+ln -s spark-2.4.6-bin-hadoop2.7 spark
+{% endhighlight %}
+
+### Compile
+
+First clone the Kylin project to your local:
+
+{% highlight Groff markup %}
+git clone https://github.com/apache/kylin.git
+{% endhighlight %}
+
+Install Kylin artifacts to the maven repository.
+
+{% highlight Groff markup %}
+mvn clean install -DskipTests
+{% endhighlight %}
+
+### Run unit tests
+Run unit tests to validate basic function of each classes.
+
+{% highlight Groff markup %}
+mvn clean test
+{% endhighlight %}
+
+### Run integration tests
+Executing the following command will run the unit test and integration test. Both unit and integration tests need to be passed before code is submitted.
+
+{% highlight Groff markup %}
+mvn clean test -DskipRunIt=false
+{% endhighlight %}
+
+To learn more about test, please refer to [How to test](/development40/howto_test.html).
+
+### Launch Kylin Web Server locally
+
+Copy server/src/main/webapp/WEB-INF to webapp/app/WEB-INF
+
+{% highlight Groff markup %}
+cp -r server/src/main/webapp/WEB-INF webapp/app/WEB-INF
+{% endhighlight %}
+
+Download JS for Kylin web GUI. `npm` is part of `Node.js`, please search about how to install it on your OS.
+
+{% highlight Groff markup %}
+cd webapp
+npm install -g bower
+bower --allow-root install
+{% endhighlight %}
+
+If you encounter network problem when run "bower install", you may try:
+
+{% highlight Groff markup %}
+git config --global url."git://".insteadOf https://
+{% endhighlight %}
+
+Note, if on Windows, after install bower, need to add the path of "bower.cmd" to system environment variable 'PATH', and then run:
+
+{% highlight Groff markup %}
+bower.cmd --allow-root install
+{% endhighlight %}
+
+Find the following configuration in **examples/test_case_data/sandbox/kylin.properties** and modify them according to the following example:
+
+```
+# Need to use absolute pat
+kylin.metadata.url=${KYLIN_SOURCE_DIR}/examples/test_case_data/sample_local
+kylin.storage.url=${KYLIN_SOURCE_DIR}/examples/test_case_data/sample_local
+kylin.env.zookeeper-is-local=true
+kylin.env.hdfs-working-dir=file://$KYLIN_SOURCE_DIR/examples/test_case_data/sample_local
+kylin.engine.spark-conf.spark.master=local
+# Need to create `/path/to/local/dir` manually
+kylin.engine.spark-conf.spark.eventLog.dir=/path/to/local/dir
+kylin.engine.spark-conf.spark.sql.shuffle.partitions=1
+kylin.env=LOCAL
+```
+
+In IDE, launch `org.apache.kylin.rest.DebugTomcat`. Please set the path of "server" module as the "Working directory", set "kylin-server" for "Use classpath of module", and check "Include dependencies with 'Provided' scope" option in IntelliJ IDEA 2018. If you're using IntelliJ IDEA 2017 and older, you need modify "server/kylin-server.iml" file, replace all "PROVIDED" to "COMPILE", otherwise an "java.lang.NoClassDefFoundError: org/apache/catalina/LifecycleListener" error may be thrown..
+
+And adjust VM options:
+
+```
+-Dspark.local=true
+```
+
+
+
+
+By default Kylin server will listen on 7070 port; If you want to use another port, please specify it as a parameter when run `DebugTomcat`.
+
+Check Kylin Web at `http://localhost:7070/kylin` (user:ADMIN, password:KYLIN)
+
+
+## Setup IDE code formatter
+
+In case you're writting code for Kylin, you should make sure that your code in expected formats.
+
+For Eclipse users, just format the code before committing the code.
+
+For intellij IDEA users, you have to do a few more steps:
+
+1. Install "Eclipse Code Formatter" and use "org.eclipse.jdt.core.prefs" and "org.eclipse.jdt.ui.prefs" in core-common/.settings to configure "Eclipse Java Formatter config file" and "Import order"
+
+ 
+
+2. Go to Preference => Code Style => Java, set "Scheme" to Default, and set both "Class count to use import with '\*'" and "Names count to use static import with '\*'" to 99.
+
+ 
+
+3. Disable intellij IDEA's "Optimize imports on the fly"
+
+ 
+
+3. Format the code before committing the code.
+
+## Setup IDE license header template
+
+Each source file should include the following Apache License header
+{% highlight Groff markup %}
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endhighlight %}
+
+The checkstyle plugin will check the header rule when packaging also. The license file locates under `dev-support/checkstyle-apache-header.txt`. To make it easy for developers, please add the header as `Copyright Profile` and set it as default for Kylin project.
+
\ No newline at end of file
diff --git a/website/_dev40/doc_spec.cn.md b/website/_dev40/doc_spec.cn.md
new file mode 100644
index 0000000..5448f83
--- /dev/null
+++ b/website/_dev40/doc_spec.cn.md
@@ -0,0 +1,89 @@
+---
+layout: dev40-cn
+title: Kylin 文档撰写规范
+categories: development
+permalink: /cn/development40/doc_spec.html
+---
+
+
+本文从章节结构、元素标记、用语规范、文件/路径规范等方面对 Kylin 文档的撰写规范进行了详述。
+
+### 准备工作
+
+1. 请您根据 [如何写文档](/cn/development40/howto_docs.html) 准备撰写文档有关的环境，了解 Kylin 文档结构。
+2. Kylin 文档使用 Markdown 语法书写，以下简称 md。请您确保您熟悉 [Markdown 语法](https://guides.github.com/features/mastering-markdown/)。
+
+### 章节结构
+
+- 每个章节的内容以多个小节的形式组织，每个小节的标题使用 **Heading 3 样式**。如：
+ \#\#\# 安装 Kylin
+- 如果需要在小节内进一步对内容进行组织，请使用 **无序 / 有序 列表**，尽量不使用 **Heading 4**，完全避免 **Heading 5**。如：
+ \### 安装 Kylin
+ 1. 首先，……
+ \* 运行……
+ \* 解压……
+
+### 元素标记
+
+- 粗体
+ 使用粗体标记您需要强调的内容。如：
+ 1. 强调 GUI 上某个组件的名称。
+ 2. 强调一个新概念。
+ 3. 强调用户在阅读时容易忽略的否定词。
+
+- 斜体
+ 1. 中文文档中一般不使用斜体。
+ 2. 英文文档中对于以下情形可以使用斜体，如数据库表名、列名等。
+
+- 引用
+ 1. 使用引用来标记 次要信息 / 补充信息，即不影响正常理解和使用的扩展信息。如：
+ > 您可以继续阅读以获得更多关于……的信息。
+ 2. 使用引用来标记 提示信息。
+ - 对于一般性提示信息，使用 **提示 / Note** 开头。
+ - 对于关键或警示的提示信息，使用 **注意 / Caution** 开头。
+
+- 行内代码
+ 使用行内代码标记一切**可能**会**被用户输入到 shell / config 中的内容**，比如文件路径、 Unix 账户、配置项和值等。
+
+- 代码段
+ 使用代码段标记**所有用户需要执行的 shell 命令和 config 配置**，统一格式且需要足够凸显。如：
+
+ 1. shell 命令
+ \`\`\`sh
+ $KYLIN_HOME/bin/kylin.sh start
+ \`\`\`
+
+ 2. config 配置
+ \`\`\`properties
+ kylin.env.hdfs-working-dir=/kylin
+ \`\`\`
+ \`\`\` xml
+ <property>
+ <name>mapreduce.map.memory.mb</name>
+ <value>2048</value>
+ </property>
+ \`\`\`
+
+
+### 用语规范
+
+- 英文专用词汇
+ - 中文文档中，一般出现的英文词汇都需要使用首字母大写。如：
+ Cube 概念是指一个 Cuboid 的集合，其中……。
+ - 英文文档中，当第一次出现某个英语专有词汇时，需要将首字母大写，并且用粗体强调，其他时候不需要大写 ”cube“ 或者 ”model“ 等词语。
+
+- 中英文（数字）混合
+ 在中文版中，所有出现的英文（数字）需要在两端中英文交界处添加一个**额外英文半角空格**，以增强中英文混排的美观性和可读性。
+- 标点符号
+ - 在中文文档中，**请一律使用中文标点符号**。
+
+- UI 交互的描写
+ 1. 统一对页面元素的称呼。
+ 顶部状态栏 / the top header
+ 左侧导航栏 / the left navigation
+ xxx 页面 / the xxx page
+ xxx 面板 / the xxx panel
+ xxx 对话框 / the xxx dialog
+ 2. 用**加粗样式**强调交互元素。如：
+ 点击\*\*提交\*\*按钮。
+ 3. 用 **->** 说明连续操作。
\ No newline at end of file
diff --git a/website/_dev40/doc_spec.md b/website/_dev40/doc_spec.md
new file mode 100644
index 0000000..b06d944
--- /dev/null
+++ b/website/_dev40/doc_spec.md
@@ -0,0 +1,96 @@
+---
+layout: dev40
+title: Kylin Document Writing Specification
+categories: development
+permalink: /development40/doc_spec.html
+---
+
+This article describes the Kylin document writing specifications. We will introduce the chapter structure, element tag, term specification, file/path specification, etc.
+
+### Preparation
+
+- Please prepare the environment related to writing documents according to [How to Write Document](/development40/howto_docs.html) and understand the Kylin document structure.
+
+- Kylin documents are written in Markdown syntax, hereinafter referred to as md. Please make sure you are familiar with [Markdown Syntax](https://guides.github.com/features/mastering-markdown/).
+
+### Chapter Structure
+
+- The content of each chapter is organized in multiple subsections, and the heading of each subsection uses **Heading 3 style**. For example:
+ \#\#\# Install Kylin
+- If you need to further organize the content within the subsections, please use **Unordered / Ordered List**, try not to use **Heading 4**, and avoid **Heading 5** completely. For example:
+ \### Install Kylin
+ 1. First, ...
+ \* Run...
+ \* Unzip...
+
+### Element Tag
+
+- Bold
+ Use bold to mark the content you need to emphasize. Such as:
+
+ - Emphasize the name of a component on the GUI.
+ - Emphasize a new concept.
+ - Emphasize negative words that users tend to ignore when reading.
+
+- Italic
+
+ - Italics are generally not used in Chinese documents.
+
+ - Italics can be used in English documents for the following situations, such as database name, table name, column name, etc.
+
+- Quote
+
+ - Use quote to mark secondary information / supplementary information, that is, extended information that does not affect normal understanding and use. Such as:
+
+ \> You can continue reading to get more information about...
+
+ - Use quote to mark reminders.
+ - For general prompt information, use **Note** at the beginning.
+ - For critical or warning messages, start with **Caution**.
+
+- Inline code
+ Use inline code to mark everything that **may** be **entered by the user into the shell / config**, such as file paths, Unix accounts, configuration items, and values.
+
+- Code snippet
+ Use code snippets to mark **shell commands and config configurations that all users need to execute**, in a unified format and need to be sufficiently prominent. Such as:
+
+ - shell commands
+
+ \`\`\`sh
+ $KYLIN_HOME/bin/kylin.sh start
+ \`\`\`
+
+ - config configurations
+
+ \`\`\`properties
+ kylin.env.hdfs-working-dir=/kylin
+ \`\`\`
+ \`\`\` xml
+ <property>
+ <name>mapreduce.map.memory.mb</name>
+ <value>2048</value>
+ </property>
+ \`\`\`
+
+### Term Specification
+
+- English vocabulary
+ - In Chinese documents, the first letter of the English words must be capitalized. For example:
+ Cube 概念是指一个 Cuboid 的集合，其中……。
+ - In English documents, when an English-specific vocabulary appears for the first time, the first letter must be capitalized and emphasized in bold. Other times, words such as "cube" or "model" are not required to be capitalized.
+
+- Punctuation
+ - In Chinese documents, **Please always use Chinese punctuation**.
+
+- Description of UI interaction
+ - Unify the title of page elements.
+ - the top header
+ - the left navigation
+ - the xxx page
+ - the xxx panel
+ - the xxx dialog
+
+- Use **bold style** to emphasize interactive elements. For example:
+ Click the \*\*Submit\*\* button.
+
+- Use **->** to illustrate continuous operation.
\ No newline at end of file
diff --git a/website/_dev40/howto_become_apache_committer.cn.md b/website/_dev40/howto_become_apache_committer.cn.md
new file mode 100644
index 0000000..babfe77
--- /dev/null
+++ b/website/_dev40/howto_become_apache_committer.cn.md
@@ -0,0 +1,24 @@
+---
+layout: dev40-cn
+title: 如何成为 Apache Committer
+categories: development
+permalink: /cn/development40/howto_become_apache_committer.html
+---
+
+## 宽泛的标准
+这篇文章将介绍如何成为一名 Apache Committer，简单来说可分为以下几个步骤：
+
+1. 理解和认同 Apache 运作方式和理念（Apache Way），并以此理念来与其他人协同工作；
+2. 参与项目，可以从贡献文档，回答社区问题（礼貌地），贡献 patch 等做起；到这个时候，你就是一名 contributor 了；
+3. 成为开源项目某个模块的专家，能够贡献代码、添加测试案例并不 break 其他人的功能，与其他 committer 良好沟通建立互信；
+
+**另外，这些行为最好是自愿的，候选人喜爱此项目并自我激励，把项目当作是自己的事情。**
+
+如果做到了上述的全部或大部分，那么接下来就是找到一个项目的 PMC 愿意来提名你为 committer。他需要陈述你的贡献给所有 PMC，并获得至少 3 个 +1 票。
+
+总之，Apache 强调 “**Community over code**” （社区第一，胜过代码），你需要在 Kylin 社区树立影响力和发言权，大胆发言，让其他人认可你的贡献，并愿意与你建立信任关系。成为 committer 是一个荣誉与责任共存的事情；成为 committer 不是终点，而是一个更高的起点。
+
+参考资料：
+[https://community.apache.org/contributors/](https://community.apache.org/contributors/)
+[https://www.quora.com/How-can-I-become-an-Apache-committer](https://www.quora.com/How-can-I-become-an-Apache-committer)
+[https://mahout.apache.org/developers/how-to-become-a-committer](https://mahout.apache.org/developers/how-to-become-a-committer)
diff --git a/website/_dev40/howto_become_apache_committer.md b/website/_dev40/howto_become_apache_committer.md
new file mode 100644
index 0000000..66aee08
--- /dev/null
+++ b/website/_dev40/howto_become_apache_committer.md
@@ -0,0 +1,24 @@
+---
+layout: dev40
+title: How to become an Apache Committer
+categories: development
+permalink: /development40/howto_become_apache_committer.html
+---
+
+## The broad standard
+This article will show you how to become an Apache Committer; in simple terms, it can be divided into the following steps:
+
+1. Understand and agree with Apache's way of working and idea (Apache Way) and use it to work with others;
+2. Participate in the project, you can start with contributing documents, answering community questions (courtesy), contributing patches, etc. At this time, you are a contributor;
+3. Be an expert in a module of an open source project, able to contribute code, add test cases and not break other people's functions, and establish good mutual trust with other committers;
+
+**In addition, these behaviours are best voluntary. Candidates love the project and are self-motivated, treating the project as their own.**
+
+If you do all or most of the above, then the next step is to find a project PMC willing to nominate you as a committer. He needs to state your contribution to all PMCs and get at least 3 +1 votes.
+
+In short, Apache emphasizes "**Community over code**" (community first, better than code), you need to establish influence and voice in the Kylin community, speak boldly, let others recognize your contribution, and are willing to build trust with you. Being a committer is a matter of coexistence of honour and responsibility; becoming a committer is not an end, but a higher starting point.
+
+Reference:
+[https://community.apache.org/contributors/](https://community.apache.org/contributors/)
+[https://www.quora.com/How-can-I-become-an-Apache-committer](https://www.quora.com/How-can-I-become-an-Apache-committer)
+[https://mahout.apache.org/developers/how-to-become-a-committer](https://mahout.apache.org/developers/how-to-become-a-committer)
diff --git a/website/_dev40/howto_contribute.cn.md b/website/_dev40/howto_contribute.cn.md
new file mode 100644
index 0000000..15ab30f
--- /dev/null
+++ b/website/_dev40/howto_contribute.cn.md
@@ -0,0 +1,117 @@
+---
+layout: dev40-cn
+title: 如何贡献
+categories: development
+permalink: /cn/development40/howto_contribute.html
+---
+
+Apache Kylin 一直寻求的不只是代码的贡献，还寻求使用文档，性能报告，问答等方面的贡献。所有类型的贡献都为成为 Kylin Committer 铺平了道路。每个人都有机会，尤其是那些有分析和解决方案背景的，因为缺少来自于用户和解决方案视角的内容。
+
+
+## 源分支
+代码和文档都在 Git 源代码控制之下。注意不同分支的用途。
+
+* `master`：新功能的主开发分支
+* `2.[n].x`：一些 2.x 主要版本的维护分支
+* `3.[n].x`：一些 3.x 主要版本的维护分支
+* `document`: 文档分支
+
+## 组件及拥有者
+Apache Kylin 有几个子组件。为了更好地帮助社区的发展，我们为每个组件安排了一个或多个组件负责人。
+
+- 组件负责人是志愿者（组件领域的专家）。负责人需要成为 Apache Kylin 提交者或 PMC。
+
+- 负责人将尝试审查其组件范围内的补丁。
+
+- 负责人可以根据他的愿望和社区需求进行轮换。
+
+- 在提名或投票新提交者时，提名者需要说明候选人可以成为哪个组件的负责人。
+
+- 如果您已经是 Apache Kylin 提交者或 PMC 成员并希望成为组件负责人的志愿者，请给 dev 列表写信，我们将为您注册。
+
+- 所有项目计划，决策仍由 Apache Kylin PMC 管理。
+
+- 如果您认为组件列表需要更新（添加，删除，重命名等），请给 dev 列表写信，我们将对其进行审核。
+
+组件负责人列在了这个 Apache Kylin [JIRA components page](https://issues.apache.org/jira/projects/KYLIN?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page) 页面中的 Description 字段位置。负责人列在 “Description” 字段中而不是 “Component Lead” 字段中，因为后者仅允许我们列出一个人，然而其鼓励组件具有多个负责人。
+
+## 选择一个任务
+这里有新创建的任务等待被完成，由 JIRA 追踪。为了让其容易被搜索，这里有一些过滤条件。
+
+* 由李扬管理的[任务列表](https://issues.apache.org/jira/issues/?filter=12339895)。
+* 由 Ted Yu 创建的[任务列表](https://issues.apache.org/jira/issues/?filter=12341496)，重要的小的 bugs 且其中一些很容易被修复。
+* 您也可以在 Kylin JIRA 中搜索标签 “newbie”。
+
+在做大任务之前别忘了在[邮箱列表](/community/index.html)中讨论。
+
+如果为 bug 或功能创建了新的 JIRA，请记住为社区提供足够的信息：
+
+* 问题或功能的良好摘要
+* 详细说明，可能包括：
+ - 这个问题发生的环境
+ - 重现问题的步骤
+ - 错误跟踪或日志文件（作为附件）
+ - model 或 cube 的元数据
+* 相关组件：我们将根据此选择安排审核人员。
+* 受影响的版本：您正在使用的 Kylin 版本。
+
+## 进行代码更改
+* [搭建开发环境](/cn/development40/dev_env.html)
+* 提出 JIRA，描述功能/提升/bug
+* 在邮件列表或 issue 评论中与其他人讨论，确保提议的更改符合其他人正在做的事情以及为项目规划的内容
+* 在您的 fork 中进行修改
+ * 目前没有严格的代码样式，但一般规则与现有文件保持一致。例如，对 java 文件使用 4 空格缩进。
+ * 尽可能为代码更改添加测试用例。
+ * 确保 “mvn clean package” 和 “mvn test” 能够获得成功。
+ * 充分的单元测试和集成测试是代码更改的必要部分。
+* [运行测试](/cn/development40/howto_test.html) 以确保您的更改质量良好且不会破坏任何内容。如果您的补丁生成不正确或您的代码不符合代码指南，则可能会要求您重做某些工作。
+* 生成补丁并将其附加到相关的 JIRA。
+
+## 生成 Patch
+* 使用 `submit-patch.py`（推荐）创建 patches，上传到 jira 并可选择在 Review Board 上创建/更新评论。 Patch 名称自动格式化为(JIRA).(分支名称).(补丁号).patch，遵循 Yetus 的命名规则。
+
+```
+$ ./dev-support/submit-patch.py -jid KYLIN-xxxxx -b master -srb
+```
+
+* 用 -h 标志可以了解此脚本的详细用法信息。最有用的选项是：
+ * -b BRANCH, --branch BRANCH : 指定用于生成 diff 的基本分支。如果未指定，则使用跟踪分支。如果没有跟踪分支，则会抛出错误。
+ * -jid JIRA_ID, --jira-id JIRA_ID : 如果使用，则从 jira 中的附件推断下一个补丁版本并上传新补丁。脚本将要求 jira 用户名/密码进行身份验证。如果未设置，则将补丁命名为 .patch。
+* 默认情况下，它还会创建/更新 review board。要跳过该操作，请使用 -srb 选项。它使用 jira 中的“Issue Links”来确定审核请求是否已存在。如果没有审核请求，则创建一个新请求并使用 jira 摘要，patch 说明等填充所有必填字段。此外，还将此评论的链接添加到 jira。
+* 安装需要的 python 依赖，从 master 分支执行 `pip install -r dev-support/python-requirements.txt`。
+
+* 或者，您也可以手动生成 patch。请首先使用 `git rebase -i`，将较小的提交组合（squash）为一个较大的提交。然后使用 `git format-patch` 命令生成 patch，有关详细指南，请参阅[如何使用 Git 创建和应用补丁](https://ariejan.net/2009/10/26/how-to-create-and-apply-a-patch-with-git/)。
+
+## 代码审查
+审核人员需要从以下角度审核 patch：
+
+* _功能性_：patch 必须解决问题，并在提交审核之前已经过贡献者的验证。
+* _测试范围_：更改必须由 UT 或集成测试覆盖，否则无法维护。执行案例包括 GUI，shell 脚本等。
+* _性能_：改变不应该降低 Kylin 的性能。
+* _元数据兼容性_：更改应支持旧元数据定义。否则，需要元数据迁移工具和文档。
+* _API 兼容性_：更改不应该破坏公共 API 的功能和行为；如果需要用新 API 替换旧 API，请在那里打印警告消息。
+* _文档_：如果需要同时更新 Kylin 文档，请创建另一个 JIRA，并将 “Document” 作为要跟踪的组件。在 JIRA 文档中，附加 “文档” 分支的文档更改 patch。
+
+不符合上述规则的补丁可能无法合并。
+
+## Patch +1 政策
+
+在提交之前，适合单个组件范围的修补程序至少需要一个组件负责人的 +1。如果负责人不在 — 在忙或其他 — 两个非负责人（即两个提交者）的 +1，就足够了。
+
+跨组件的 patch 在提交之前至少需要两个 +1，最好由 x-component patch 涉及的组件负责人的 +1。
+
+任何人都可以在 patch 上 -1，任何 -1 都可以否决补丁；在解决 -1 的理由之前，它不能被提交。
+
+
+## 应用 Patch
+* Committer 将审核 JIRA 中的 Pull Requests 和 Patches 的正确性，性能，设计，编码风格，测试覆盖率；
+* 必要时进行讨论和修改；
+* Committer 将代码合并到目标分支中
+ * 对于 git patch，请使用 “git am -s -3 patch-file” 命令进行应用；
+ * 如果是来自 github Pull Request，则需要添加 “This closing＃” 作为提交消息的一部分。这将允许 ASF Git bot 关闭 PR；
+ * 使用 `git rebase` 确保合并结果是提交的简化。
+
+
+## 进行文档更改
+查看[如何写文档](/cn/development40/howto_docs.html)。
+
diff --git a/website/_dev40/howto_contribute.md b/website/_dev40/howto_contribute.md
new file mode 100644
index 0000000..292c5ab
--- /dev/null
+++ b/website/_dev40/howto_contribute.md
@@ -0,0 +1,117 @@
+---
+layout: dev40
+title: How to Contribute
+categories: development
+permalink: /development40/howto_contribute.html
+---
+
+Apache Kylin is always looking for contributions of not only code, but also usage document, performance report, Q&A etc. All kinds of contributions pave the way towards a Kylin Committer. There is opportunity for everyone, especially for those come from analysis and solution background, due to the lacking of content from user and solution perspective.
+
+
+## Source Branches
+Both code and document are under Git source control. Note the purpose of different branches.
+
+* `master`: Main development branch for new features
+* `2.[n].x`: Maintenance branch for a certain major release for v2.x
+* `3.[n].x`: Maintenance branch for a certain major release for v3.x
+* `document`: Document branch
+
+## Components and owners
+Apache Kylin has several sub-components. To better help the community's growth, we arrange one or multiple component owners for each component.
+
+- Component owners are volunteers who are expert in their component domain. The owner needs to be an Apache Kylin committer or PMC at this moment.
+
+- Owners will try and review patches that land within their component’s scope.
+
+- Owners can rotate, based on his aspiration and community need.
+
+- When nominate or vote a new committer, the nominator needs to state which component the candidate can be the owner.
+
+- If you're already an Apache Kylin committer or PMC memeber and would like to be a volunteer as a component owner, just write to the dev list and we’ll sign you up.
+
+- All the project plan, decisions are still managed by Apache Kylin PMC.
+
+- If you think the component list need be updated (add, remove, rename, etc), write to the dev list and we’ll review that.
+
+Component owners is listed in the description field on this Apache Kylin [JIRA components page](https://issues.apache.org/jira/projects/KYLIN?selectedItem=com.atlassian.jira.jira-projects-plugin:components-page). The owners are listed in the 'Description' field rather than in the 'Component Lead' field because the latter only allows us to list one individual whereas it is encouraged that components have multiple owners.
+
+## Pick a task
+There are open tasks waiting to be done, tracked by JIRA. To make it easier to search, there are a few JIRA filters.
+
+* [A list of tasks](https://issues.apache.org/jira/issues/?filter=12339895) managed by Yang Li.
+* [A list of tasks](https://issues.apache.org/jira/issues/?filter=12341496) opened by Ted Yu, important small bugs and some are easy fixes.
+* Also you can search for tag "newbie" in Kylin JIRA.
+
+Do not forget to discuss in [mailing list](/community/index.html) before working on a big task.
+
+If create a new JIRA for bug or feature, remember to provide enough information for the community:
+
+* A well summary for the problem or feature
+* A detail description, which may include:
+ - the environment of this problem occurred
+ - the steps to reproduce the problem
+ - the error trace or log files (as attachment)
+ - the metadata of the model or cube
+* Related components: we will arrange reviewer based on this selection.
+* Affected version: which Kylin you're using.
+
+## Making Code Changes
+* [Setup development environment](/development40/dev_env.html)
+* Raise a JIRA, describe the feature/enhancement/bug
+* Discuss with others in mailing list or issue comments, make sure the proposed changes fit in with what others are doing and have planned for the project
+* Make changes in your fork
+ * No strict code style at the moment, but the general rule is keep consistent with existing files. E.g. use 4-space indent for java files.
+ * Add test case for your code change as much as possible.
+ * Make sure "mvn clean package" and "mvn test" can get success.
+ * Sufficient unit test and integration test is a mandatory part of code change.
+* [Run tests](/development40/howto_test.html) to ensure your change is in good quality and does not break anything. If your patch was generated incorrectly or your code does not adhere to the code guidelines, you may be asked to redo some work.
+* Generate a patch and attach it to relative JIRA.
+
+## Generate Patch
+* Using `submit-patch.py` (recommended) to create patches, upload to jira and optionally create/update reviews on Review Board. Patch name is automatically formatted as (JIRA).(branch name).(patch number).patch to follow Yetus' naming rules.
+
+```
+$ ./dev-support/submit-patch.py -jid KYLIN-xxxxx -b master -srb
+```
+
+* Use -h flag for this script to know detailed usage information. Most useful options are:
+ * -b BRANCH, --branch BRANCH : Specify base branch for generating the diff. If not specified, tracking branch is used. If there is no tracking branch, error will be thrown.
+ * -jid JIRA_ID, --jira-id JIRA_ID : If used, deduces next patch version from attachments in the jira and uploads the new patch. Script will ask for jira username/password for authentication. If not set, patch is named <branch>.patch.
+* By default, it'll also create/update review board. To skip that action, use -srb option. It uses 'Issue Links' in the jira to figure out if a review request already exists. If no review request is present, then creates a new one and populates all required fields using jira summary, patch description, etc. Also adds this review’s link to the jira.
+* To install required python dependencies, execute `pip install -r dev-support/python-requirements.txt` from the master branch.
+
+* Alternatively, you can also manually generate a patch. Please use `git rebase -i` first, to combine (squash) smaller commits into a single larger one. Then use `git format-patch` command to generate the patch, for a detail guide you can refer to [How to create and apply a patch with Git](https://ariejan.net/2009/10/26/how-to-create-and-apply-a-patch-with-git/)
+
+## Code Review
+The reviewer need to review the patch from the following perspectives:
+
+* _Functionality_: the patch MUST address the issue and has been verified by the contributor before submitting for review.
+* _Test coverage_: the change MUST be covered by a UT or the Integration test, otherwise it is not maintainable. Execptional case includes GUI, shell script, etc.
+* _Performance_: the change SHOULD NOT downgrade Kylin's performance.
+* _Metadata compatibility_: the change should support old metadata definition. Otherwise, a metadata migration tool and documentation is required.
+* _API compatibility_: the change SHOULD NOT break public API's functionality and behavior; If an old API need be replaced by the new one, print warning message there.
+* _Documentation_: if the Kylin document need be updated together, create another JIRA with "Document" as the component to track. In the document JIRA, attach the doc change patch which is againt the "document" branch.
+
+A patch which doesn't comply with the above rules may not get merged.
+
+## Patch +1 Policy
+
+Patches that fit within the scope of a single component require, at least, a +1 by one of the component’s owners before commit. If owners are absent — busy or otherwise — two +1s by non-owners but committers will suffice.
+
+Patches that span components need at least two +1s before they can be committed, preferably +1s by owners of components touched by the x-component patch.
+
+Any -1 on a patch by anyone vetoes a patch; it cannot be committed until the justification for the -1 is addressed.
+
+
+## Apply Patch
+* Committer will review Pull Requests and Patches in JIRA regarding correctness, performance, design, coding style, test coverage;
+* Discuss and revise if necessary;
+* Finally committer merge code into target branch
+ * For a git patch, use "git am -s -3 patch-file" command to apply;
+ * If it is from a github Pull Request, need add "This closes #<PR NUMBER>" as part of the commit messages. This will allow ASF Git bot to close the PR;
+ * Use `git rebase` to ensure the merged result is a streamline of commits.
+
+
+## Making Document Changes
+Check out [How to Write Document](/development40/howto_docs.html).
+
diff --git a/website/_dev40/howto_docs.cn.md b/website/_dev40/howto_docs.cn.md
new file mode 100644
index 0000000..105ace4
--- /dev/null
+++ b/website/_dev40/howto_docs.cn.md
@@ -0,0 +1,202 @@
+---
+layout: dev40-cn
+title: 如何写文档
+categories: development
+permalink: /cn/development40/howto_docs.html
+---
+
+我们以 MD 格式编写文档并使用 [Jekyll](http://jekyllrb.com) 转换为 HTML。Jekyll 生成的 HTML 上传到 apache SVN 并成为 Kylin 网站。所有 MD 源文件都在 git 中管理，因此可以清楚地跟踪所有更改和贡献者。
+
+## 工作前
+
+在您添加或修改文档前，请部署文档编译所需的环境，我们提供以下两种方法：
+
+- [本地部署文档编译环境](#本地部署文档编译环境)
+- [使用 Docker 部署文档编译环境（推荐）](#使用 Docker 部署文档编译环境)
+
+### <span id="本地部署文档编译环境">本地部署文档编译环境</span>
+
+在您添加或修改文档前请安装以下工具：
+
+1. 首先，确保 Ruby 和 Gem 能在您的机器上工作
+ * 对于 Mac 用户，请参考[这个](https://github.com/sstephenson/rbenv#homebrew-on-mac-os-x)来搭建 ruby 环境。
+ * 对于 Windows 用户，使用 [ruby installer](http://rubyinstaller.org/downloads/)。
+ * 对于 China 用户，考虑使用一个[本地 gem 仓库](https://ruby.taobao.org/)以防止网络问题。
+
+2. 然后，安装 [Jekyll](http://jekyllrb.com)，以及需要的插件
+ * `gem install jekyll jekyll-multiple-languages kramdown rouge`
+ * __注意__：一些特定的 jekyll 和 jekyll-multiple-languages 版本不能一起使用（使用 jekyll 3.0.1 和 jekyll-multiple-languages 2.0.3 时我遇到一个 "undefined method" 错误)。这种情况下，`jekyll 2.5.3` 和 `jekyll-multiple-languages 1.0.8` 是已知可运行的版本。
+ * 例如，使用 `gem install jekyll --version "=2.5.3"` 来安装具体的版本。
+ * __注意__：对于 Mac 用户，如果 gem 安装时遇到类似这样的错误 'ERROR: While executing gem ... (Gem::FilePermissionError)'。您可以使用 'brew install ruby' 的方式解决这个问题，然后重启您的终端。
+3. 您可以选择任何 markdown 编辑器
+
+下面是一个可以工作的 gem 列表。如果 jekyll 安装成为问题，请坚持使用这些版本。
+
+```
+$ gem list
+
+...
+jekyll (2.5.3)
+jekyll-coffeescript (1.0.1)
+jekyll-gist (1.4.0)
+jekyll-multiple-languages (1.0.8)
+jekyll-paginate (1.1.0)
+jekyll-sass-converter (1.4.0)
+jekyll-watch (1.3.1)
+json (1.8.1)
+kramdown (1.9.0)
+...
+rouge (1.10.1)
+...
+```
+
+### <span id="使用 Docker 部署文档编译环境">使用 Docker 部署文档编译环境</span>
+
+最新版的 kylin 发布提供了 dockerfile，来减少构建复杂性使用 docker 和 Makefile 能调用 docker 命令。
+
+```
+$ pwd
+/Users/<username>/kylin/website
+$ make docker.build
+docker build -f Dockerfile -t kylin-document:latest .
+Sending build context to Docker daemon 82.44MB
+Step 1/3 : FROM jekyll/jekyll:2.5.3
+ ---> e81842c29599
+Step 2/3 : RUN gem install jekyll-multiple-languages -v 1.0.11
+ ---> Using cache
+ ---> e9e8b0f1d388
+Step 3/3 : RUN gem install rouge -v 3.0.0
+ ---> Using cache
+ ---> 1bd42c6b93c0
+Successfully built 1bd42c6b93c0
+Successfully tagged kylin-document:latest
+$ make runserver
+docker run --volume="/Users/<username>/kylin/website:/srv/jekyll" -p 4000:4000 --rm -it kylin-document:latest jekyll server --watch
+Configuration file: /srv/jekyll/_config.yml
+ Source: /srv/jekyll
+ Destination: /srv/jekyll/_site
+ Generating...
+...
+```
+
+## 关于 Jekyll
+Jekyll 是一个用于从源文本和主题生成静态 HTML 网站的 Ruby 脚本，HTML 在部署到 Web 服务器之前生成。Jekyll 恰好也是 GitHub 页面背后的引擎。
+
+Apache Kylin 的网站和文档使用 Jekyll 来管理和生成，可在 [http://kylin.apache.org](http://kylin.apache.org) 上看到最终内容。
+
+## Multi-Language
+要草拟中文版文档或翻译现有文档，只需添加或复制该文档，名称以 .cn.md 作为后缀。它将在 /cn 文件夹下生成与 html 同名的文件。
+要添加其他语言，请更新 _config.yml 并遵循与中文版相同的模式。
+
+## Kylin 文档结构以及导航菜单
+
+[作为 Jekyll 源的 Kylin 网站](https://github.com/apache/kylin/tree/document/website)是在 `doucment` 分支下维护的。
+
+1. __Home Page__：_"index.md"_ 文档的主页
+2. __Getting Started__：_"gettingstarted"_ 生成 Apache Kylin 的文档，包括 FAQ，术语
+3. __Installation__：_"install"_ Apache Kylin 安装指南
+4. __Tutorial__：_"tutorial"_ 关于用户如何使用 Apache Kylin 的教程
+5. __How To__：_"howto"_ 更细节的帮助指南
+6. __Development__：_"development"_ 为了开发者贡献，集成其它应用和扩展 Apache Kylin
+7. __Others__：其它文档。
+
+菜单由 Jekyll 集合管理：
+
+* ___data/docs.yml__：英文版本菜单结构
+* ___data/docs-cn.yml__：中文版本菜单结构
+* __add new menu item__：添加新的条目：在相关文件夹下创建新文档，例如 howto_example.md。添加如下的前标记：
+
+```
+---
+layout: docs
+title: How to expamle
+categories: howto
+permalink: /docs/howto/howto_example.html
+version: v0.7.2
+since: v0.7.2
+---
+```
+
+将链接更改为完全链接
+然后将条目添加到 docs.yml，如：
+
+```
+- title: How To
+ docs:
+ - howto/howto_contribute
+ - howto/howto_jdbc
+ - howto/howto_example
+```
+
+## 如何编写文档
+使用任何 markdown 编辑器打开文档，草拟内容并在本地预览。
+
+样例文档：
+
+```
+---
+layout: docs
+title: How to example
+categories: howto
+permalink: /docs/howto/howto_example.html
+version: v0.7.2
+since: v0.7.2
+---
+
+## This is example doc
+The quick brown fox jump over the lazy dog.
+```
+
+## 如何添加图片
+所有的图片请放到 _images_ 文件夹下，在你的文件中，请使用以下样式引入图片：
+
+```
+
+```
+
+## 如何添加连接
+使用站点链接的相对路径，例如
+
+```
+[REST API](docs/development40/rest_api.html).
+```
+
+## 如何添加代码高亮
+我们使用 [Rouge](https://github.com/jneen/rouge) 突出显示代码语法。
+查看此 doc 的源代码以获取更多详细信息示例。
+
+## 如何在本地预览
+您可以在 markdown 编辑器中预览，要检查网站上的确切内容，请从 `website` 文件夹中运行 Jekyll：
+
+```
+jekyll server
+```
+然后在浏览器中访问 http://127.0.0.1:4000。
+
+## 如何发布到网站（只适用于 committer)
+
+### 搭建
+
+1. `cd website`
+2. `svn co https://svn.apache.org/repos/asf/kylin/site _site`
+
+___site__ 文件夹是工作目录，将由 maven 或 git 随时删除，请确保只有当你想要发布到网站时从 svn 检出。
+
+### 本地运行
+在创建一个 PR 或推送到 git 仓库之前，您可以通过以下方式在本地预览更改：
+
+1. `cd website`
+2. `jekyll s`
+3. 在您的浏览器打开 [http://127.0.0.1:4000](http://127.0.0.1:4000)
+
+### 推到网站
+
+1. 拷贝 jekyll 生成的 `_site` 到 svn 的 `website/_site`
+2. `cd website/_site`
+3. `svn status`
+4. 您需要使用 `svn add` 添加任意新的文件
+5. `svn commit -m 'UPDATE MESSAGE'`
+
+在几分钟内，svnpubsub 应该开始且您将能够在 [http://kylin.apache.org](http://kylin.apache.org/) 看到结果。
+
+
diff --git a/website/_dev40/howto_docs.md b/website/_dev40/howto_docs.md
new file mode 100644
index 0000000..01acd3a
--- /dev/null
+++ b/website/_dev40/howto_docs.md
@@ -0,0 +1,204 @@
+---
+layout: dev40
+title: How to Write Document
+categories: development
+permalink: /development40/howto_docs.html
+---
+
+We write documents in MD format and convert to HTML using [Jekyll](http://jekyllrb.com). The Jekyll generated HTML gets uploaded to apache SVN and becomes Kylin website. All MD source files are managed in git so all changes and contributors are clearly tracked.
+
+## Before your work
+
+Before you add or edit documentation, please deploy the document compilation environment. We provide two ways:
+
+- [Deploy a local document compilation environment](#Deploy a local document compilation environment)
+- [Use Docker to deploy document compilation environment](#Use Docker to deploy document compilation environment)
+
+### <span id="Deploy a local document compilation environment">Deploy a local document compilation environment</span>
+
+Install following tools before you add or edit documentation:
+
+1. First, make sure Ruby and Gem work on your machine
+ * For Mac user, please refer [this](https://github.com/sstephenson/rbenv#homebrew-on-mac-os-x) to setup ruby env.
+ * For Windows user, use the [ruby installer](http://rubyinstaller.org/downloads/).
+ * For China user, consider use a [local gem repository](https://ruby.taobao.org/) in case of network issues.
+
+2. Then, install [Jekyll](http://jekyllrb.com), and required plugins
+ * `gem install jekyll jekyll-multiple-languages kramdown rouge`
+ * __Note__: Some specific version of jekyll and jekyll-multiple-languages does not work together (I got a "undefined method" error with jekyll 3.0.1 and jekyll-multiple-languages 2.0.3). In that case, `jekyll 2.5.3` and `jekyll-multiple-languages 1.0.8` is the known working version.
+ * eg. Use `gem install jekyll --version "=2.5.3"` to install a specific version.
+ * __Note__: For Mac user, if gem install raise error like this 'ERROR: While executing gem ... (Gem::FilePermissionError)'. To solve this problem you can use 'brew install ruby', then restart you terminal.
+3. And optionally any markdown editor you prefer
+
+Below is a gem list that works. Stick to these versions if jekyll installation becomes a problem.
+
+```
+$ gem list
+
+...
+jekyll (2.5.3)
+jekyll-coffeescript (1.0.1)
+jekyll-gist (1.4.0)
+jekyll-multiple-languages (1.0.8)
+jekyll-paginate (1.1.0)
+jekyll-sass-converter (1.4.0)
+jekyll-watch (1.3.1)
+json (1.8.1)
+kramdown (1.9.0)
+...
+rouge (1.10.1)
+...
+```
+
+### <span id="Use Docker to deploy document compilation environment">Use Docker to deploy document compilation environment</span>
+
+The latest kylin release provides dockerfile, to reduce build complexity using docker and Makefile can call docker command.
+
+```
+$ pwd
+/Users/<username>/kylin/website
+$ make docker.build
+docker build -f Dockerfile -t kylin-document:latest .
+Sending build context to Docker daemon 82.44MB
+Step 1/3 : FROM jekyll/jekyll:2.5.3
+ ---> e81842c29599
+Step 2/3 : RUN gem install jekyll-multiple-languages -v 1.0.11
+ ---> Using cache
+ ---> e9e8b0f1d388
+Step 3/3 : RUN gem install rouge -v 3.0.0
+ ---> Using cache
+ ---> 1bd42c6b93c0
+Successfully built 1bd42c6b93c0
+Successfully tagged kylin-document:latest
+$ make runserver
+docker run --volume="/Users/<username>/kylin/website:/srv/jekyll" -p 4000:4000 --rm -it kylin-document:latest jekyll server --watch
+Configuration file: /srv/jekyll/_config.yml
+ Source: /srv/jekyll
+ Destination: /srv/jekyll/_site
+ Generating...
+...
+```
+
+## About Jekyll
+Jekyll is a Ruby script to generate a static HTML website from source text and themes, the HTML is generated before being deployed to the web server. Jekyll also happens to be the engine behind GitHub Pages.
+
+Apache Kylin's website and documentation is using Jekyll to manage and generate final content which avaliable at [http://kylin.apache.org](http://kylin.apache.org).
+
+## Multi-Language
+To draft Chinese version document or translate existing one, just add or copy that doc and name with .cn.md as sufffix. It will generate under /cn folder with same name as html file.
+To add other language, please update _config.yml and follow the same pattern as Chinese version.
+
+## Kylin document structure and navigation menu
+
+The Kylin [website as the Jekyll source](https://github.com/apache/kylin/tree/document/website) is maintained under the `doucment` branch.
+
+1. __Home Page__: _"index.md"_ Home page of Docs
+2. __Getting Started__: _"gettingstarted"_ General docs about Apache Kylin, including FAQ, Terminology
+3. __Installation__: _"install"_ Apache Kylin installation guide
+4. __Tutorial__: _"tutorial"_ User tutorial about how to use Apache Kylin
+5. __How To__: _"howto"_ Guide for more detail help
+6. __Development__: _"development"_ For developer to contribute, integration with other application and extend Apache Kylin
+7. __Others__: Other docs.
+
+The menu is managed by Jekyll collection:
+
+* ___data/docs.yml__: English version menu structure
+* ___data/docs-cn.yml__: Chinese version menu structure
+* __add new menu item__: To add new item: create new docs under relative folder, e.g howto_example.md. add following Front Mark:
+
+```
+---
+layout: docs
+title: How to expamle
+categories: howto
+permalink: /docs/howto/howto_example.html
+version: v0.7.2
+since: v0.7.2
+---
+```
+
+change the __permalink__ to exactly link
+Then add item to docs.yml like:
+
+```
+- title: How To
+ docs:
+ - howto/howto_contribute
+ - howto/howto_jdbc
+ - howto/howto_example
+```
+
+## How to edit document
+Open doc with any markdown editor, draft content and preview in local.
+
+Sample Doc:
+
+```
+---
+layout: docs
+title: How to example
+categories: howto
+permalink: /docs/howto/howto_example.html
+version: v0.7.2
+since: v0.7.2
+---
+
+## This is example doc
+The quick brown fox jump over the lazy dog.
+```
+
+## How to add image
+All impage please put under _images_ folder, in your document, please using below sample to include image:
+
+```
+
+```
+
+## How to add link
+Using relative path for site links, for example:
+
+```
+[REST API](docs/development40/rest_api.html).
+```
+
+## How to add code highlight
+We are using [Rouge](https://github.com/jneen/rouge) to highlight code syntax.
+check this doc's source code for more detail sample.
+
+## How to preview in your local
+You can preview in your markdown editor, to check exactly what it will looks like on website, please run Jekyll from `website` folder:
+
+```
+jekyll server
+```
+Then access http://127.0.0.1:4000 in your browser.
+
+## How to publish to website (for committer only)
+
+### Setup
+
+1. `cd website`
+2. `svn co https://svn.apache.org/repos/asf/kylin/site _site`
+
+___site__ folder is working dir which will be removed anytime by maven or git, please make sure only check out from svn when you want to publish to website.
+
+### Running locally
+Before opening a pull request or push to git repo, you can preview changes from your local box with following:
+
+1. `cd website`
+2. `jekyll s`
+3. Open [http://127.0.0.1:4000](http://127.0.0.1:4000) in your browser
+
+### Pushing to site
+
+1. Copy jekyll generated `_site` to svn `website/_site`
+2. `cd website/_site`
+3. `svn status`
+4. You will need to `svn add` any new files
+5. `svn commit -m 'UPDATE MESSAGE'`
+
+Within a few minutes, svnpubsub should kick in and you'll be able to
+see the results at
+[http://kylin.apache.org](http://kylin.apache.org/).
+
+
diff --git a/website/_dev40/howto_package.cn.md b/website/_dev40/howto_package.cn.md
new file mode 100644
index 0000000..4ae2b18
--- /dev/null
+++ b/website/_dev40/howto_package.cn.md
@@ -0,0 +1,33 @@
+---
+layout: dev40-cn
+title: 如何打二进制包
+categories: development
+permalink: /cn/development40/howto_package.html
+---
+
+### 生成二进制包
+本文档讲述的是如何从源码构建 Kylin4 的二进制包
+
+#### 下载源码
+您可以从 github 仓库下载 Apache Kylin 源码。
+
+```
+git clone https://github.com/apache/kylin kylin
+```
+
+#### 构建二进制包
+
+为了生成二进制包，需要预先准备好 **maven** 和 **npm**。
+
+**(可选)** 如果您在代理服务器后面，在运行 ./script/package.sh 之前，需要将代理信息告知 npm 和 bower：
+
+```
+export http_proxy=http://your-proxy-host:port
+npm config set proxy http://your-proxy-host:port
+```
+
+##### 运行打包命令
+```
+cd kylin
+build/script/package.sh
+```
diff --git a/website/_dev40/howto_package.md b/website/_dev40/howto_package.md
new file mode 100644
index 0000000..ec3fdd6
--- /dev/null
+++ b/website/_dev40/howto_package.md
@@ -0,0 +1,33 @@
+---
+layout: dev40
+title: How to Build Binary Package
+categories: development
+permalink: /development40/howto_package.html
+---
+
+### Generate Binary Package
+This document talks about how to build binary package of kylin4 from source code.
+
+#### Download source code
+You can download Apache Kylin source code from github repository.
+
+```
+git clone https://github.com/apache/kylin kylin
+```
+
+#### Build Binary Package
+
+In order to generate binary package, **maven** and **npm** are pre-requisites.
+
+**(Optional)** If you're behind a proxy server, both npm and bower need be told with the proxy info before running ./script/package.sh:
+
+```
+export http_proxy=http://your-proxy-host:port
+npm config set proxy http://your-proxy-host:port
+```
+
+##### Build Package for kylin4
+```
+cd kylin
+build/script/package.sh
+```
diff --git a/website/_dev40/howto_release.cn.md b/website/_dev40/howto_release.cn.md
new file mode 100644
index 0000000..ef4f61d
--- /dev/null
+++ b/website/_dev40/howto_release.cn.md
@@ -0,0 +1,475 @@
+---
+layout: dev40-cn
+title: 如何发布
+categories: development
+permalink: /cn/development40/howto_release.html
+---
+
+_本教程只适用于 Apache Kylin Committers。_
+_以在 Mac OS X 上的 Shell 命令作为样例。_
+_对于中国用户，请谨慎使用代理以避免潜在的防火墙问题。_
+
+## 建立账户
+确保您有可使用的账号且对以下应用有权限:
+
+* Apache 账户：[https://id.apache.org](https://id.apache.org/)
+* Apache Kylin git repo (main cobe base)： [https://github.com/apache/kylin](https://github.com/apache/kylin)
+* Apache Kylin svn 仓库（只针对网站）：https://svn.apache.org/repos/asf/kylin](https://svn.apache.org/repos/asf/kylin)
+* Apache Nexus （maven 仓库）：[https://repository.apache.org](https://repository.apache.org)
+* Apache Kylin dist 仓库：[https://dist.apache.org/repos/dist/dev/kylin](https://dist.apache.org/repos/dist/dev/kylin)
+
+## 软件要求
+* Java 8 或更高版本；
+* Maven 3.5.3 或更高版本；
+* 如果你是用 Mac OS X 做发布，请按照[此文章](http://macappstore.org/gnu-tar/)安装 GNU TAR。
+
+## 设置 GPG 签名密钥
+按照 [http://www.apache.org/dev/release-signing](http://www.apache.org/dev/release-signing) 上的说明创建密钥对
+安装 gpg（以 Mac OS X 为例）：
+`brew install gpg`
+
+生成 gpg 密钥：
+参考: [https://www.gnupg.org/gph/en/manual/c14.html](https://www.gnupg.org/gph/en/manual/c14.html)
+_生成的所有新 RSA 密钥应至少为 4096 位。不要生成新的 DSA 密钥_
+`gpg --full-generate-key`
+
+验证您的密钥：
+`gpg --list-sigs YOUR_NAME`
+
+获取密钥的指纹：
+`gpg --fingerprint YOUR_NAME`
+
+它将显示指纹，如 "Key fingerprint = XXXX XXXX ..."，然后在 [https://id.apache.org/](https://id.apache.org/) 上的 "OpenPGP Public Key Primary Fingerprint" 字段处将指纹添加到您的 apache 帐户；等待几个小时，密钥将添加到 [https://people.apache.org/keys/](https://people.apache.org/keys/)，例如:
+[https://people.apache.org/keys/committer/lukehan.asc](https://people.apache.org/keys/committer/lukehan.asc)
+
+生成 ASCII Amromed 键：
+`gpg -a --export YOUR_MAIL_ADDRESS > YOUR_NAME.asc &`
+
+上传密钥到公共服务器：
+`gpg --send-keys YOUR_KEY_HASH`
+
+或通过 web 提交密钥：
+打开并提交到 [http://pool.sks-keyservers.net:11371](http://pool.sks-keyservers.net:11371)（您可以选择任意一个有效的公钥服务器）
+
+一旦您的密钥提交到服务器，您可以通过使用以下命令验证：
+`gpg --recv-keys YOUR_KEY_HASH`
+举例：
+`gpg --recv-keys 027DC364`
+
+按照 KEYS 文件中的说明将公钥添加到 KEYS 文件：
+_KEYS 文件位于：_ __${kylin}/KEYS__
+例如：
+`(gpg --list-sigs YOURNAME && gpg --armor --export YOURNAME) >> KEYS`
+
+提交您的改动。
+
+## 准备 release 的工件
+__开始前：__
+
+* 如上所述设置签名密钥。
+* 确保您使用的是 JDK 1.8。
+* 确保您使用的是 GIT 2.7.2 或更高版本。
+* 确保您使用的是正确的 release 版本号。
+* 确保每个 “resolved” 的 JIRA 案例（包括重复案例）都分配了修复版本。
+* 确保你在干净的目录工作。
+
+__在 Maven 中配置 Apache 存储库服务器__
+如果您是第一次发布，您需要在 ~/.m2/settings.xml 中服务器授权信息；如果该文件不存在，从 $M2_HOME/conf/settings.xml 拷贝一个模板；
+
+在 “服务器” 部分中，确保添加以下服务器，并将 #YOUR_APACHE_ID#, #YOUR_APACHE_PWD#, #YOUR_GPG_PASSPHRASE# 替换为您的 ID，密码和口令：
+{% highlight bash %}
+ <servers>
+ <!-- To publish a snapshot of some part of Maven -->
+ <server>
+ <id>apache.snapshots.https</id>
+ <username>#YOUR_APACHE_ID#</username>
+ <password>#YOUR_APACHE_PWD#</password>
+ </server>
+ <!-- To stage a release of some part of Maven -->
+ <server>
+ <id>apache.releases.https</id>
+ <username>#YOUR_APACHE_ID#</username>
+ <password>#YOUR_APACHE_PWD#</password>
+ </server>
+
+
+ <!-- To publish a website of some part of Maven -->
+ <server>
+ <id>apache.website</id>
+ <username>#YOUR_APACHE_ID#</username>
+ <password>#YOUR_APACHE_PWD#</password>
+ <!-- Either
+ <privateKey>...</privateKey>
+ -->
+ <filePermissions>664</filePermissions>
+ <directoryPermissions>775</directoryPermissions>
+ </server>
+
+ <!-- To stage a website of some part of Maven -->
+ <server>
+ <id>stagingSite</id>
+ <!-- must match hard-coded repository identifier in site:stage-deploy -->
+ <username>#YOUR_APACHE_ID#</username>
+ <filePermissions>664</filePermissions>
+ <directoryPermissions>775</directoryPermissions>
+ </server>
+ <server>
+ <id>gpg.passphrase</id>
+ <passphrase>#YOUR_GPG_PASSPHRASE#</passphrase>
+ </server>
+ </servers>
+{% endhighlight %}
+
+__修复许可证问题__
+{% highlight bash %}
+
+# 设置密码变量而不将其放入shell历史记录中
+$ read -s GPG_PASSPHRASE
+
+# 确保沙箱中没有垃圾文件
+$ git clean -xf
+$ mvn clean
+
+# 确保所有单元测试均通过
+$ mvn test
+
+# 检查 `org.apache.kylin.common.KylinVersion` 类，确保 `CURRENT_KYLIN_VERSION`的值是发行版本
+
+# 修复 target / rat.txt 报告的所有许可证问题
+$ mvn -Papache-release -DskipTests -Dgpg.passphrase=${GPG_PASSPHRASE} install
+{% endhighlight %}
+
+可选的，当 dry-run 成功了，将安装变为部署:
+{% highlight bash %}
+$ mvn -Papache-release -DskipTests -Dgpg.passphrase=${GPG_PASSPHRASE} deploy
+{% endhighlight %}
+
+__准备__
+检查并确保你可以 ssh 连接到 github:
+{% highlight bash %}
+ssh -T git@github.com
+{% endhighlight %}
+
+基于要当前的开发分支，创建一个以 release 版本号命名的发布分支，例如，v2.5.0-release（注意分支名字不能与 tag 名字相同），并将其推到服务器端。
+{% highlight bash %}
+$ git checkout -b vX.Y.Z-release
+$ git push -u origin vX.Y.Z-release
+{% endhighlight %}
+
+如果任何步骤失败，请清理（见下文），解决问题，然后从头重新开始。
+{% highlight bash %}
+# 设置密码变量而不将其放入shell历史记录中
+$ read -s GPG_PASSPHRASE
+
+# 确保沙箱中没有垃圾文件
+$ git clean -xf
+$ mvn clean
+
+# （可选的）试运行 release：准备步骤，该步骤设置版本号
+$ mvn -DdryRun=true -DskipTests -DreleaseVersion=X.Y.Z -DdevelopmentVersion=(X.Y.Z+1)-SNAPSHOT -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests" release:prepare 2>&1 | tee /tmp/prepare-dry.log
+{% endhighlight %}
+
+__查看 dry run 输出:__
+
+* 在 `target` 目录中应该是这 8 个文件，其中包括：
+ * apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip
+ * apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip.asc
+ * apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip.asc.sha256
+ * apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip.sha256
+* 移除 .zip.asc.sha256 文件因为不需要
+* 注意文件名以 `apache-kylin-` 开始
+* 在源发行版 `.zip` 文件中，检查所有文件是否属于名为 `apache-kylin-X.Y.Z-SNAPSHOT` 的目录
+* 该目录必须包含 `NOTICE`，`LICENSE`，`README.md` 文件
+* 按[此](https://httpd.apache.org/dev/verification.html)检查 PGP。
+
+__运行真实的 release：__
+现在真正开始 release
+{% highlight bash %}
+# 如果之前做了dry run，在真正开始 release 之前需要做清理
+# 准备设置版本号，创建 tag，并将其推送到git
+$ mvn -DskipTests -DreleaseVersion=X.Y.Z -DdevelopmentVersion=(X.Y.Z+1)-SNAPSHOT -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests" release:prepare
+
+# 挑选出标记的版本，构建并部署到登台存储库
+$ mvn -DskipTests -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests" release:perform
+{% endhighlight %}
+
+__一个失败的 release 后进行清理:__
+{% highlight bash %}
+# 确保您要生成的 tag 不存在（由于一个失败的发布）
+$ git tag
+
+# 如果 tag 存在，请在本地和远程删除它
+$ git tag -d kylin-X.Y.Z
+$ git push origin :refs/tags/kylin-X.Y.Z
+
+# 删除修改的文件
+$ mvn release:clean
+
+# 检查是否有修改的文件，如果有，请返回原始的 git commit
+$ git status
+$ git reset --hard HEAD
+{% endhighlight %}
+
+__关闭 Nexus 仓库中的阶段性工件:__
+
+* 输入 [https://repository.apache.org/](https://repository.apache.org/) 并登陆
+* 在 `Build Promotion` 下，点击 `Staging Repositories`
+* 在 `Staging Repositories` 选项卡中，应该有一个包含配置文件 `org.apache.kylin` 的行
+* 浏览工件树并确保存在 .jar，.pom，.asc 文件
+* 选中行第一列中的复选框，点击 'Close' 按钮发布仓库到
+ [https://repository.apache.org/content/repositories/orgapachekylin-1006](https://repository.apache.org/content/repositories/orgapachekylin-1006)
+ (或相似的 URL)
+
+__上传到临时区域:__
+通过 subversion 将工件上传到临时区域，https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-X.Y.Z-rcN:
+{% highlight bash %}
+# 创建 subversion 工作区（如果尚未创建）
+$ mkdir -p ~/dist/dev
+$ pushd ~/dist/dev
+$ svn co https://dist.apache.org/repos/dist/dev/kylin
+$ popd
+
+# 将文件移到目录中
+$ cd target
+$ mkdir ~/dist/dev/kylin/apache-kylin-X.Y.Z-rcN
+$ mv apache-kylin-* ~/dist/dev/kylin/apache-kylin-X.Y.Z-rcN
+
+# 删除 .zip.asc.sha256 文件，因为它不再需要
+$ rm ~/dist/dev/kylin/apache-kylin-X.Y.Z-rcN/apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip.asc.sha256
+
+# Check in
+$ cd ~/dist/dev/kylin
+$ svn add apache-kylin-X.Y.Z-rcN
+$ svn commit -m 'Upload release artifacts to staging' --username <YOUR_APACHE_ID>
+{% endhighlight %}
+
+## 验证 release
+{% highlight bash %}
+# 检查单元测试
+$ mvn test
+
+# Check that the signing key (e.g. 2AD3FAE3) is pushed
+$ gpg --recv-keys key
+
+# Check keys
+$ curl -O https://dist.apache.org/repos/dist/release/kylin/KEYS
+
+# Sign/check sha256 hashes（假设您的操作系统具有“ shasum”命令）
+function checkHash() {
+ cd "$1"
+ for i in *.{pom,gz}; do
+ if [ ! -f $i ]; then
+ continue
+ fi
+ if [ -f $i.sha256 ]; then
+ if [ "$(cat $i.sha256)" = "$(shasum -a 256 $i)" ]; then
+ echo $i.sha256 present and correct
+ else
+ echo $i.sha256 does not match
+ fi
+ else
+ shasum -a 256 $i > $i.sha256
+ echo $i.sha256 created
+ fi
+ done
+};
+$ checkHash apache-kylin-X.Y.Z-rcN
+{% endhighlight %}
+
+## Apache 投票过程
+
+__在 Apache Kylin dev 邮件列表上投票__
+在 dev 邮件列表上进行 release 投票，使用由 Maven release plugin 生成的 commit id，其消息看起来像 "[maven-release-plugin] prepare release kylin-x.x.x"：
+
+{% highlight text %}
+To: dev@kylin.apache.org
+Subject: [VOTE] Release apache-kylin-X.Y.Z (RC[N])
+
+Hi all,
+
+I have created a build for Apache Kylin X.Y.Z, release candidate N.
+
+Changes highlights:
+...
+
+Thanks to everyone who has contributed to this release.
+Here’s release notes:
+https://github.com/apache/kylin/blob/XXX/docs/release_notes.md
+
+The commit to be voted upon:
+
+https://github.com/apache/kylin/commit/xxx
+
+Its hash is xxx.
+
+The artifacts to be voted on are located here:
+https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-X.Y.Z-rcN/
+
+The hash of the artifact is as follows:
+apache-kylin-X.Y.Z-source-release.zip.sha256 xxx
+
+A staged Maven repository is available for review at:
+https://repository.apache.org/content/repositories/orgapachekylin-XXXX/
+
+Release artifacts are signed with the following key:
+https://people.apache.org/keys/committer/lukehan.asc
+
+Please vote on releasing this package as Apache Kylin X.Y.Z.
+
+The vote is open for the next 72 hours and passes if a majority of
+at least three +1 PMC votes are cast.
+
+[ ] +1 Release this package as Apache Kylin X.Y.Z
+[ ] 0 I don't feel strongly about it, but I'm okay with the release
+[ ] -1 Do not release this package because...
+
+
+Here is my vote:
+
++1 (binding)
+
+
+{% endhighlight %}
+
+投票完成后，发出结果：
+{% highlight text %}
+Subject: [RESULT][VOTE] Release apache-kylin-X.Y.Z (RC[N])
+To: dev@kylin.apache.org
+
+Thanks to everyone who has tested the release candidate and given
+their comments and votes.
+
+The tally is as follows.
+
+N binding +1s:
+
+N non-binding +1s:
+
+No 0s or -1s.
+
+Therefore I am delighted to announce that the proposal to release
+Apache-Kylin-X.Y.Z has passed.
+
+
+{% endhighlight %}
+
+## 发布
+成功发布投票后，我们需要推动发行到镜像，以及其它任务。
+
+在 JIRA 中，搜索 [all issues resolved in this release](https://issues.apache.org/jira/issues/?jql=project%20%3D%20KYLIN%20)，并进行批量更新，将它们的状态更改为 "关闭"，并加上更改的评论 "Resolved in release X.Y.Z (YYYY-MM-DD)"
+(填写适当的发布号和日期)。
+__取消 "Send mail for this update"。__
+
+标记 JIRA 系统中发布的版本，[管理版本](https://issues.apache.org/jira/plugins/servlet/project-config/KYLIN/versions)。
+
+推广分阶段的 nexus 工件。
+
+* 转到 [https://repository.apache.org/](https://repository.apache.org/) 并登陆
+* 在 "Build Promotion" 下点击 "Staging Repositories"
+* 在 "orgapachekylin-xxxx" 行中，选中框
+* 点击 "Release" 按钮
+
+将工件检入 svn。
+{% highlight bash %}
+# 获取候选版本
+$ mkdir -p ~/dist/dev
+$ cd ~/dist/dev
+$ svn co https://dist.apache.org/repos/dist/dev/kylin
+
+# 复制工件，请注意，副本没有后缀 '-rcN'
+$ mkdir -p ~/dist/release
+$ cd ~/dist/release
+$ svn co https://dist.apache.org/repos/dist/release/kylin
+$ cd kylin
+$ mkdir apache-kylin-X.Y.Z
+$ cp -rp ../../dev/kylin/apache-kylin-X.Y.Z-rcN/apache-kylin* apache-kylin-X.Y.Z/
+$ svn add apache-kylin-X.Y.Z
+
+# Check in
+svn commit -m 'checkin release artifacts'
+{% endhighlight %}
+
+Svnpubsub 将会发布到
+[https://dist.apache.org/repos/dist/release/kylin](https://dist.apache.org/repos/dist/release/kylin) 并会在 24 小时内传播到
+[http://www.apache.org/dyn/closer.cgi/kylin](http://www.apache.org/dyn/closer.cgi/kylin)。
+
+如果现在有超过 2 个版本，请清除最旧的版本：
+
+{% highlight bash %}
+cd ~/dist/release/kylin
+svn rm apache-kylin-X.Y.Z
+svn commit -m 'Remove old release'
+{% endhighlight %}
+
+旧版本将保留在 [release archive](http://archive.apache.org/dist/kylin/)。
+
+在 JIRA 中发布相同版本，检查最新发布版本的更改日志。
+
+## 构建和上传二进制包
+发布后，您需要生成二进制包并将它们放入到 VPN 发布库中：
+
+* 使用 `git fetch --all --prune --tags` 来同步您本地和远程的仓库；
+* Git 检出当前发布的标签；
+* 通过参考[此文档](howto_package.html)制作二进制包;
+* 使用 gpg 对生成的二进制包进行签名，例如：
+ {% highlight bash %}
+ gpg --armor --output apache-kylin-2.5.0-bin.tar.gz.asc --detach-sig apache-kylin-2.5.0-bin.tar.gz
+ {% endhighlight %}
+* 生成二进制包的 sha256 文件，例如：
+ {% highlight bash %}
+ shasum -a 256 apache-kylin-2.5.0-bin.tar.gz > apache-kylin-2.5.0-bin.tar.gz.sha256
+
+ on Linux:
+ openssl sha256 apache-kylin-2.5.0-bin.tar.gz > apache-kylin-2.5.0-bin.tar.gz.sha256
+ {% endhighlight %}
+* 将二进制包，签名文件和 sha256 文件推送到 svn __dev__ 仓库，然后运行 `svn mv <files-in-dev> <files-in-release>` 命令将他们移动到 svn __release__ 仓库；
+* 对于不同的 Hadoop/HBase 版本，您可能需要上述步骤；
+* 添加文件，然后将更改提交 svn。
+
+
+## 更新源码
+发布后，您需要更新一些源代码：
+
+* 把分布分支，如 v2.5.0-release，合并到开发分支中，如 2.5.x，以便开始下个版本的开发。
+
+* 手动更新 `KylinVersion` 类，将 `CURRENT_KYLIN_VERSION` 的值更改为当前开发版本。
+
+## 发布网站
+更多细节参考[如何写文档](howto_docs.html)。
+
+## 发送通知邮件到邮件列表
+发送一个邮件主题如 "[Announce] Apache Kylin x.y.z released" 到以下列表：
+
+* Apache Kylin Dev 邮箱列表：dev@kylin.apache.org
+* Apache Kylin User 邮箱列表：user@kylin.apache.org
+* Apache Announce 邮箱列表：announce@apache.org
+ 请注意始终使用您的 Apache 邮件地址发送；
+
+这是一个公告电子邮件的样本（通过研究 Kafka):
+
+{% highlight text %}
+The Apache Kylin team is pleased to announce the immediate availability of the 2.5.0 release.
+
+This is a major release after 2.4, with more than 100 bug fixes and enhancements; All of the changes in this release can be found in:
+https://kylin.apache.org/docs/release_notes.html
+
+You can download the source release and binary packages from Apache Kylin's download page: https://kylin.apache.org/download/
+
+Apache Kylin is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Apache Hadoop, supporting extremely large datasets.
+
+Apache Kylin lets you query massive data set at sub-second latency in 3 steps:
+1. Identify a star schema or snowflake schema data set on Hadoop.
+2. Build Cube on Hadoop.
+3. Query data with ANSI-SQL and get results in sub-second, via ODBC, JDBC or RESTful API.
+
+Thanks everyone who have contributed to the 2.1.0 release.
+
+We welcome your help and feedback. For more information on how to
+report problems, and to get involved, visit the project website at
+https://kylin.apache.org/
+
+{% endhighlight %}
+
+## 感谢
+本指南起草于 [Apache Calcite](http://calcite.apache.org) Howto doc 的参考资料，非常感谢。
+
diff --git a/website/_dev40/howto_release.md b/website/_dev40/howto_release.md
new file mode 100644
index 0000000..88d6900
--- /dev/null
+++ b/website/_dev40/howto_release.md
@@ -0,0 +1,485 @@
+---
+layout: dev40
+title: How to Make Release
+categories: development
+permalink: /development40/howto_release.html
+---
+
+_This guide is for Apache Kylin Committers only._
+_Shell commands are on Mac OS X as sample._
+_For people in China, please aware using proxy to avoid potential firewall issue._
+
+## Setup Account
+Make sure you have avaliable account and privilege for following applications:
+
+* Apache account: [https://id.apache.org](https://id.apache.org/)
+* Apache Kylin git repo (main cobe base): [https://github.com/apache/kylin](https://github.com/apache/kylin)
+* Apache Kylin svn repo (for website only): [https://svn.apache.org/repos/asf/kylin](https://svn.apache.org/repos/asf/kylin)
+* Apache Nexus (maven repo): [https://repository.apache.org](https://repository.apache.org)
+* Apache Kylin dist repo: [https://dist.apache.org/repos/dist/dev/kylin](https://dist.apache.org/repos/dist/dev/kylin)
+
+## Software requirement
+* Java 8 or above;
+* Maven 3.5.3 or above;
+* If you're on Apple Mac OS X, please install GNU TAR, check [this post](http://macappstore.org/gnu-tar/).
+
+## Setup GPG signing keys
+Follow instructions at [http://www.apache.org/dev/release-signing](http://www.apache.org/dev/release-signing) to create a key pair
+Install gpg (On Mac OS X as sample):
+`brew install gpg`
+
+Generate gpg key:
+Reference: [https://www.gnupg.org/gph/en/manual/c14.html](https://www.gnupg.org/gph/en/manual/c14.html)
+_All new RSA keys generated should be at least 4096 bits. Do not generate new DSA keys_
+`gpg --full-generate-key`
+
+Verify your key:
+`gpg --list-sigs YOUR_NAME`
+
+Get the fingerprint of your key:
+`gpg --fingerprint YOUR_NAME`
+
+It will display the fingerprint like "Key fingerprint = XXXX XXXX ...", then add the fingerprint to your apache account at [https://id.apache.org/](https://id.apache.org/) in "OpenPGP Public Key Primary Fingerprint" field; wait for a few hours the key will added to [https://people.apache.org/keys/](https://people.apache.org/keys/), for example:
+[https://people.apache.org/keys/committer/lukehan.asc](https://people.apache.org/keys/committer/lukehan.asc)
+
+Generate ASCII Amromed Key:
+`gpg -a --export YOUR_MAIL_ADDRESS > YOUR_NAME.asc &`
+
+Upload key to public server:
+`gpg --send-keys YOUR_KEY_HASH`
+
+or Submit key via web:
+Open and Submit to [http://pool.sks-keyservers.net:11371](http://pool.sks-keyservers.net:11371) (you can pickup any available public key server)
+
+Once your key submitted to server, you can verify using following command:
+`gpg --recv-keys YOUR_KEY_HASH`
+for example:
+`gpg --recv-keys 027DC364`
+
+Add your public key to the KEYS file by following instructions in the KEYS file.:
+_KEYS file location:_ __${kylin}/KEYS__
+For example:
+`(gpg --list-sigs YOURNAME && gpg --armor --export YOURNAME) >> KEYS`
+
+Commit your changes.
+
+## Prepare artifacts for release
+__Before you start:__
+
+* Set up signing keys as described above.
+* Make sure you are using JDK 1.8.
+* Make sure you are using GIT 2.7.2 or above.
+* Make sure you are working on right release version number.
+* Make sure that every “resolved” JIRA case (including duplicates) has a fix version assigned.
+* Make sure you are working in clean dir.
+
+__Configure Apache repository server in Maven__
+If you're the first time to do release, you need update the server authentication information in ~/.m2/settings.xml; If this file doesn't exist, copy a template from $M2_HOME/conf/settings.xml;
+
+In the "servers" section, make sure the following servers be added, and replace #YOUR_APACHE_ID#, #YOUR_APACHE_PWD#, #YOUR_GPG_PASSPHRASE# with your ID, password, and passphrase:
+{% highlight bash %}
+ <servers>
+ <!-- To publish a snapshot of some part of Maven -->
+ <server>
+ <id>apache.snapshots.https</id>
+ <username>#YOUR_APACHE_ID#</username>
+ <password>#YOUR_APACHE_PWD#</password>
+ </server>
+ <!-- To stage a release of some part of Maven -->
+ <server>
+ <id>apache.releases.https</id>
+ <username>#YOUR_APACHE_ID#</username>
+ <password>#YOUR_APACHE_PWD#</password>
+ </server>
+
+ <!-- To publish a website of some part of Maven -->
+ <server>
+ <id>apache.website</id>
+ <username>#YOUR_APACHE_ID#</username>
+ <password>#YOUR_APACHE_PWD#</password>
+ <!-- Either
+ <privateKey>...</privateKey>
+ -->
+ <filePermissions>664</filePermissions>
+ <directoryPermissions>775</directoryPermissions>
+ </server>
+
+ <!-- To stage a website of some part of Maven -->
+ <server>
+ <id>stagingSite</id>
+ <!-- must match hard-coded repository identifier in site:stage-deploy -->
+ <username>#YOUR_APACHE_ID#</username>
+ <filePermissions>664</filePermissions>
+ <directoryPermissions>775</directoryPermissions>
+ </server>
+ <server>
+ <id>gpg.passphrase</id>
+ <passphrase>#YOUR_GPG_PASSPHRASE#</passphrase>
+ </server>
+ </servers>
+{% endhighlight %}
+
+__Fix license issues__
+{% highlight bash %}
+Set passphrase variable without putting it into shell history
+$ read -s GPG_PASSPHRASE
+
+Make sure that there are no junk files in the sandbox
+$ git clean -xf
+$ mvn clean
+
+Make sure all unit tests are passed
+$ mvn test
+
+Check the `org.apache.kylin.common.KylinVersion` class, ensure the value of `CURRENT_KYLIN_VERSION` is the release version.
+
+Fix any license issues as reported by target/rat.txt
+$ mvn -Papache-release -DskipTests -Dgpg.passphrase=${GPG_PASSPHRASE} install
+{% endhighlight %}
+
+Optionally, when the dry-run has succeeded, change install to deploy:
+{% highlight bash %}
+$ mvn -Papache-release -DskipTests -Dgpg.passphrase=${GPG_PASSPHRASE} deploy
+{% endhighlight %}
+
+__Prepare__
+
+Make sure your can ssh connection to github:
+{% highlight bash %}
+ssh -T git@github.com
+{% endhighlight %}
+
+Create a branch for release work from your current development branch, named with this release version, e.g. v2.5.0-release (should not be the same name with the tag name), and then push it to Apache.
+{% highlight bash %}
+$ git checkout -b vX.Y.Z-release
+$ git push -u origin vX.Y.Z-release
+{% endhighlight %}
+
+If any of the steps fail, clean up (see below), fix the problem, and start again from the top.
+{% highlight bash %}
+Set passphrase variable without putting it into shell history
+$ read -s GPG_PASSPHRASE
+
+Make sure that there are no junk files in the sandbox
+$ git clean -xf
+$ mvn clean
+
+Optionally, do a dry run of the release:prepare step, which sets version numbers. e.g. releaseVersion=2.5.0, developmentVersion=2.5.1-SNAPSHOT, use default tag kylin-2.5.0
+$ mvn -DdryRun=true -DskipTests -DreleaseVersion=X.Y.Z -DdevelopmentVersion=(X.Y.Z+1)-SNAPSHOT -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests" release:prepare 2>&1 | tee /tmp/prepare-dry.log
+{% endhighlight %}
+
+__Check the dry run output:__
+
+* In the `target` directory should be these 8 files, among others:
+ * apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip
+ * apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip.asc
+ * apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip.asc.sha256
+ * apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip.sha256
+* Remove the .zip.asc.sha256 file as it is not needed.
+* Note that the file names start `apache-kylin-`.
+* In the source distro `.zip`, check that all files belong to a directory called
+ `apache-kylin-X.Y.Z-SNAPSHOT`.
+* That directory must contain files `NOTICE`, `LICENSE`, `README.md`
+* Check PGP, per [this](https://httpd.apache.org/dev/verification.html).
+
+__Run real release:__
+Now, run the release for real.
+{% highlight bash %}
+# Note that if a dry run is done previously, need to do the cleanup first before run the release for real
+# Prepare sets the version numbers, creates a tag, and pushes it to git.
+$ mvn -DskipTests -DreleaseVersion=X.Y.Z -DdevelopmentVersion=(X.Y.Z+1)-SNAPSHOT -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests" release:prepare
+
+# Perform checks out the tagged version, builds, and deploys to the staging repository
+$ mvn -DskipTests -Papache-release -Darguments="-Dgpg.passphrase=${GPG_PASSPHRASE} -DskipTests" release:perform
+{% endhighlight %}
+
+__Close the staged artifacts in the Nexus repository:__
+
+* Go to [https://repository.apache.org/](https://repository.apache.org/) and login
+* Under `Build Promotion`, click `Staging Repositories`
+* In the `Staging Repositories` tab there should be a line with profile `org.apache.kylin`
+* Navigate through the artifact tree and make sure the .jar, .pom, .asc files are present
+* Check the box on in the first column of the row, and press the 'Close' button to publish the repository at
+ [https://repository.apache.org/content/repositories/orgapachekylin-1055](https://repository.apache.org/content/repositories/orgapachekylin-1055)
+ (or a similar URL)
+
+__Upload to staging area:__
+Upload the artifacts via subversion to a staging area, https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-X.Y.Z-rcN:
+{% highlight bash %}
+# Create a subversion workspace, if you haven't already
+$ mkdir -p ~/dist/dev
+$ pushd ~/dist/dev
+$ svn co https://dist.apache.org/repos/dist/dev/kylin
+$ popd
+
+## Move the files into a directory
+$ cd target
+$ mkdir ~/dist/dev/kylin/apache-kylin-X.Y.Z-rcN
+$ mv apache-kylin-* ~/dist/dev/kylin/apache-kylin-X.Y.Z-rcN
+
+## Remove the .zip.asc.sha256 file as it is not needed.
+$ rm ~/dist/dev/kylin/apache-kylin-X.Y.Z-rcN/apache-kylin-X.Y.Z-SNAPSHOT-source-release.zip.asc.sha256
+
+## Check in
+$ cd ~/dist/dev/kylin
+$ svn add apache-kylin-X.Y.Z-rcN
+$ svn commit -m 'Upload release artifacts to staging' --username <YOUR_APACHE_ID>
+{% endhighlight %}
+
+__Cleaning up after a failed release attempt:__
+{% highlight bash %}
+# Make sure that the tag you are about to generate does not already
+# exist (due to a failed release attempt)
+$ git tag
+
+# If the tag exists, delete it locally and remotely
+$ git tag -d kylin-X.Y.Z
+$ git push origin :refs/tags/kylin-X.Y.Z
+
+# Remove modified files
+$ mvn release:clean
+
+# Check whether there are modified files and if so, go back to the
+# original git commit
+$ git status
+$ git reset --hard HEAD
+{% endhighlight %}
+
+## Validate a release
+{% highlight bash %}
+# Check unit test
+$ mvn test
+
+# Check that the signing key (e.g. 2AD3FAE3) is pushed
+$ gpg --recv-keys key
+
+# Check keys
+$ curl -O https://dist.apache.org/repos/dist/release/kylin/KEYS
+
+# Sign/check sha256 hashes
+# (Assumes your O/S has a 'shasum' command.)
+function checkHash() {
+ cd "$1"
+ for i in *.{pom,gz}; do
+ if [ ! -f $i ]; then
+ continue
+ fi
+ if [ -f $i.sha256 ]; then
+ if [ "$(cat $i.sha256)" = "$(shasum -a 256 $i)" ]; then
+ echo $i.sha256 present and correct
+ else
+ echo $i.sha256 does not match
+ fi
+ else
+ shasum -a 256 $i > $i.sha256
+ echo $i.sha256 created
+ fi
+ done
+};
+$ checkHash apache-kylin-X.Y.Z-rcN
+{% endhighlight %}
+
+## Apache voting process
+
+__Vote on Apache Kylin dev mailing list__
+Release vote on dev list, use the commit id that generated by Maven release plugin, whose message looks like "[maven-release-plugin] prepare release kylin-x.x.x":
+
+{% highlight text %}
+To: dev@kylin.apache.org
+Subject: [VOTE] Release apache-kylin-X.Y.Z (RC[N])
+
+Hi all,
+
+I have created a build for Apache Kylin X.Y.Z, release candidate N.
+
+Changes highlights:
+...
+
+Thanks to everyone who has contributed to this release.
+Here’s release notes:
+https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121&version=12343540
+
+The commit to be voted upon:
+
+https://github.com/apache/kylin/commit/xxx
+
+Its hash is xxx.
+
+The artifacts to be voted on are located here:
+https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-X.Y.Z-rcN/
+
+The hash of the artifact is as follows:
+apache-kylin-X.Y.Z-source-release.zip.sha256 xxx
+
+A staged Maven repository is available for review at:
+https://repository.apache.org/content/repositories/orgapachekylin-XXXX/
+
+Release artifacts are signed with the following key:
+https://people.apache.org/keys/committer/lukehan.asc
+
+Please vote on releasing this package as Apache Kylin X.Y.Z.
+
+The vote is open for the next 72 hours and passes if a majority of
+at least three +1 PMC votes are cast.
+
+[ ] +1 Release this package as Apache Kylin X.Y.Z
+[ ] 0 I don't feel strongly about it, but I'm okay with the release
+[ ] -1 Do not release this package because...
+
+
+Here is my vote:
+
++1 (binding)
+
+
+{% endhighlight %}
+
+After vote finishes, send out the result:
+{% highlight text %}
+Subject: [RESULT][VOTE] Release apache-kylin-X.Y.Z (RC[N])
+To: dev@kylin.apache.org
+
+Thanks to everyone who has tested the release candidate and given
+their comments and votes.
+
+The tally is as follows.
+
+N binding +1s:
+
+N non-binding +1s:
+
+No 0s or -1s.
+
+Therefore I am delighted to announce that the proposal to release
+Apache-Kylin-X.Y.Z has passed.
+
+
+{% endhighlight %}
+
+## Publishing a release
+After a successful release vote, we need to push the release
+out to mirrors, and other tasks.
+
+In JIRA, search for
+[all issues resolved in this release](https://issues.apache.org/jira/issues/?jql=project%20%3D%20KYLIN%20),
+and do a bulk update changing their status to "Closed",
+with a change comment
+"Resolved in release X.Y.Z (YYYY-MM-DD)"
+(fill in release number and date appropriately).
+__Uncheck "Send mail for this update".__
+
+Mark the version released in JIRA system, [Manage Versions](https://issues.apache.org/jira/plugins/servlet/project-config/KYLIN/versions).
+
+Promote the staged nexus artifacts.
+
+* Go to [https://repository.apache.org/](https://repository.apache.org/) and login
+* Under "Build Promotion" click "Staging Repositories"
+* In the line with "orgapachekylin-xxxx", check the box
+* Press "Release" button
+
+Check the artifacts into svn.
+{% highlight bash %}
+# Get the release candidate.
+$ mkdir -p ~/dist/dev
+$ cd ~/dist/dev
+$ svn co https://dist.apache.org/repos/dist/dev/kylin
+
+# Copy the artifacts. Note that the copy does not have '-rcN' suffix.
+$ mkdir -p ~/dist/release
+$ cd ~/dist/release
+$ svn co https://dist.apache.org/repos/dist/release/kylin
+$ cd kylin
+$ mkdir apache-kylin-X.Y.Z
+$ cp -rp ../../dev/kylin/apache-kylin-X.Y.Z-rcN/apache-kylin* apache-kylin-X.Y.Z/
+$ svn add apache-kylin-X.Y.Z
+
+# Check in.
+svn commit -m 'checkin release artifacts'
+{% endhighlight %}
+
+Svnpubsub will publish to
+[https://dist.apache.org/repos/dist/release/kylin](https://dist.apache.org/repos/dist/release/kylin) and propagate to
+[http://www.apache.org/dyn/closer.cgi/kylin](http://www.apache.org/dyn/closer.cgi/kylin) within 24 hours.
+
+If there are more than 2 releases, clear out the oldest ones:
+
+{% highlight bash %}
+cd ~/dist/release/kylin
+svn rm apache-kylin-X.Y.Z
+svn commit -m 'Remove old release'
+{% endhighlight %}
+
+The old releases will remain available in the
+[release archive](http://archive.apache.org/dist/kylin/).
+
+Release same version in JIRA, check Change Log.
+
+## Build and upload binary package
+After publish the release, you need generate the binary packages and then put them to the svn release repository;
+
+* Do `git fetch --all --prune --tags` to sync your local repo with remote.
+* Git checkout the tag for current release;
+* Make a binary package by refering to [this doc](howto_package.html);
+* Sign the generated binary package with gpg, e.g,:
+ {% highlight bash %}
+ gpg --armor --output apache-kylin-2.5.0-bin.tar.gz.asc --detach-sig apache-kylin-2.5.0-bin.tar.gz
+ {% endhighlight %}
+* Generate the sha256 file for the binary package, e.g,:
+ {% highlight bash %}
+ shasum -a 256 apache-kylin-2.5.0-bin.tar.gz > apache-kylin-2.5.0-bin.tar.gz.sha256
+
+ on Linux:
+ openssl sha256 apache-kylin-2.5.0-bin.tar.gz > apache-kylin-2.5.0-bin.tar.gz.sha256
+
+ {% endhighlight %}
+* Push the binary package, the signature file and the sha256 file to the svn __dev__ repo, then run `svn mv <files-in-dev> <files-in-release>` to move them to svn __release__ repo.
+* For different Hadoop/HBase version, you may need repeat the above steps;
+* Add the files and then commit the svn changes.
+
+
+## Update source code
+After publish the release, you need to do update some source code:
+
+* Merge the release branch, e.g. v2.5.0-release, to the current development branch, e.g. 2.5.x, for preparing the next development iteration.
+
+* Manually update `KylinVersion` class, change value of `CURRENT_KYLIN_VERSION` to current development version.
+
+## Publishing the web site
+Refer to [How to document](howto_docs.html) for more detail.
+
+## Send announcement mail to mailing list
+Send one mail with subject like "[Announce] Apache Kylin X.Y.Z released" to following list:
+
+* Apache Kylin Dev mailing list: dev@kylin.apache.org
+* Apache Kylin User mailing list: user@kylin.apache.org
+* Apache Announce mailing list: announce@apache.org
+ Please notice to always use your Apache mail address to send this;
+
+Here is a sample of announcement email (by studying Kafka's):
+
+{% highlight text %}
+The Apache Kylin team is pleased to announce the immediate availability of the 2.5.0 release.
+
+This is a major release after 2.4, with more than 100 bug fixes and enhancements; All of the changes in this release can be found in:
+https://kylin.apache.org/docs/release_notes.html
+
+You can download the source release and binary packages from Apache Kylin's download page: https://kylin.apache.org/download/
+
+Apache Kylin is an open source Distributed Analytics Engine designed to provide SQL interface and multi-dimensional analysis (OLAP) on Apache Hadoop, supporting extremely large datasets.
+
+Apache Kylin lets you query massive dataset at sub-second latency in 3 steps:
+1. Identify a star schema or snowflake schema data set on Hadoop.
+2. Build Cube on Hadoop.
+3. Query data with ANSI-SQL and get results in sub-second, via ODBC, JDBC or RESTful API.
+
+Thanks to everyone who has contributed to the 2.5.0 release.
+
+We welcome your help and feedback. For more information on how to
+report problems, and to get involved, visit the project website at
+https://kylin.apache.org/
+
+{% endhighlight %}
+
+## Thanks
+This guide drafted with reference from [Apache Calcite](http://calcite.apache.org) Howto doc, Thank you very much.
+
diff --git a/website/_dev40/howto_test.cn.md b/website/_dev40/howto_test.cn.md
new file mode 100644
index 0000000..00d82b9
--- /dev/null
+++ b/website/_dev40/howto_test.cn.md
@@ -0,0 +1,24 @@
+---
+layout: dev40-cn
+title: "如何测试"
+categories: development
+permalink: /cn/development40/howto_test.html
+---
+
+一般来说，应该有单元测试来涵盖个别 classes；必须有集成测试来涵盖端到端的场景，如构建，合并和查询。单元测试必须独立运行（不需要外部沙箱）。
+
+## 测试 v4.x
+
+* `mvn clean test` 运行单元测试，它的测试覆盖范围有限。
+ * 单元测试没有额外的依赖且能在任何机器上运行。
+ * 单元测试不覆盖端到端的场景，如构建，合并和查询。
+ * 单元测试只需几分钟即可完成。
+* `mvn clean test -DskipRunIt=false` 运行集成测试，有很好的覆盖率。。
+ * 集成测试从生成随机数据开始，然后构建 Cube、合并 Cube，最后查询结果并与 Spark 引擎进行比较。
+ * 集成测试需要一个小时左右才能完成。
+
+如果您的代码改动很小那么只需要运行 UT，使用：
+`mvn test`
+如果您的代码改动涉及代码较多，那么需要运行 UT 和 IT，使用：
+`mvn clean test -DskipRunIt=false`
+
diff --git a/website/_dev40/howto_test.md b/website/_dev40/howto_test.md
new file mode 100644
index 0000000..1fcb259
--- /dev/null
+++ b/website/_dev40/howto_test.md
@@ -0,0 +1,23 @@
+---
+layout: dev40
+title: "How to Test"
+categories: development
+permalink: /development40/howto_test.html
+---
+
+In general, there should be unit tests to cover individual classes; there must be integration test to cover end-to-end scenarios like build, merge, and query. Unit test must run independently (does not require an external sandbox).
+
+## Test v4.x
+
+* `mvn clean test` runs unit tests, which has a limited test coverage.
+ * Unit tests has no external dependency and can run on any machine.
+ * The unit tests do not cover end-to-end scenarios like build, merge, and query.
+ * The unit tests take a few minutes to complete.
+* `mvn clean test -DskipRunIt=false` runs integration tests, which has the best test coverage.
+ * The integration tests start from generate random data, then build cube, merge cube, and finally query the result and compare to Spark.
+ * The integration tests take about one hour to complete.
+
+If your code changes is minor and it merely requires running UT, use:
+`mvn test`
+If your code changes involve more code, you need to run UT and IT, use:
+`mvn clean test -DskipRunIt=false`
diff --git a/website/_dev40/index.cn.md b/website/_dev40/index.cn.md
new file mode 100644
index 0000000..061c4f9
--- /dev/null
+++ b/website/_dev40/index.cn.md
@@ -0,0 +1,37 @@
+---
+layout: dev40-cn
+title: 开发快速指南
+permalink: /cn/development40/index.html
+---
+
+Apache Kylin 一直寻求的不只是代码的贡献，还寻求使用文档，性能报告，问答等方面的贡献。所有类型的贡献都为成为 Kylin Committer 铺平了道路。每个人都有机会，尤其是那些有分析和解决方案背景的，因为缺少来自于用户和解决方案视角的内容。
+
+这里是适用于 Apache Kylin4.x 版本的开发文档，查看其他版本开发文档：
+* [v3.x 及以前版本开发文档](/cn/development/)
+
+### 如何贡献
+查看[如何贡献](/cn/development40/howto_contribute.html)文档。
+
+### 源仓库
+Apache Kylin™ 源码使用 Git version control 进行版本控制：
+Commits [总结](https://github.com/apache/kylin/commits/master)
+源仓库：[https://github.com/apache/kylin ](https://github.com/apache/kylin )
+Gitbox 的镜像：[https://gitbox.apache.org/repos/asf?p=kylin.git ](https://gitbox.apache.org/repos/asf?p=kylin.git )
+
+### Issue 追踪
+在 Apache JIRA 上的 "Kylin" 项目追踪 issues（[浏览](http://issues.apache.org/jira/browse/KYLIN))。
+
+### 路线图
+- 支持 Hadoop 3.0（纠偏编码）：完成（v2.5）
+- 完全使用 Spark 的 Cube 引擎：完成（v2.5）
+- 支持实时数据分析的 Lambda 架构：完成（v3.0）
+- 接入更多的源（MySQL，Spark SQL 等）：完成（v2.6）
+- Flink 引擎：完成（v3.1）
+- 云原生的存储引擎（Parquet）：开发中（v4.0）
+- 分布式查询执行引擎：与 Parquet 存储一起进行中（v4.0）
+- 容器化/Kubernetes：完成（v3.1）
+- 查询下压 SDK（Presto，Clickhouse 等）：进行中（v3.1 支持查询下压 Presto）
+- 即席查询支持，无需构建 Cube
+
+
+
diff --git a/website/_dev40/index.md b/website/_dev40/index.md
new file mode 100644
index 0000000..3ae3358
--- /dev/null
+++ b/website/_dev40/index.md
@@ -0,0 +1,34 @@
+---
+layout: dev40
+title: Development Quick Guide
+permalink: /development40/index.html
+---
+
+Apache Kylin is always looking for contributions of not only code, but also usage document, performance report, Q&A etc. All kinds of contributions pave the way towards a Kylin Committer. There is opportunity for everyone, especially for those come from analysis and solution background, due to the lacking of content from user and solution perspective.
+
+Here is the development document for Apache kylin 4.x. heck the development documents of other versions:
+* [v3.x and earlier development documents](/development/)
+
+### How to Contribute
+Check out the [How to Contribute](/development40/howto_contribute.html) document.
+
+### Source Repository
+Apache Kylin™ source code is version controlled using Git version control:
+Commits [Summary](https://github.com/apache/kylin/commits/master)
+Source Repo: [https://github.com/apache/kylin ](https://github.com/apache/kylin )
+Mirrored to Gitbox: [https://gitbox.apache.org/repos/asf?p=kylin.git ](https://gitbox.apache.org/repos/asf?p=kylin.git )
+
+### Issue Tracking
+Track issues on the "Kylin" Project on the Apache JIRA ([browse](http://issues.apache.org/jira/browse/KYLIN)).
+
+### Roadmap
+- Hadoop 3.0 support (Erasure Coding) : DONE (v2.5)
+- Fully on Spark Cube engine : DONE (v2.5)
+- Real-time analytics with Lambda Architecture : DONE (v3.0)
+- Connect more data sources (MySQL, SparkSQL, etc) : DONE (v2.6)
+- Flink engine : Done (v3.1)
+- Cloud-native storage (Parquet) : In-progress (v4.0)
+- Distributed query execution engine (Spark) : In-progress, together with Parquet storage (v4.0)
+- Containerization/Kubernetes support : Done (v3.1)
+- Pushdown SDK with more engines (Presto, Clickhouse, etc) : In progress (Presto support in v3.1)
+- Ad-hoc queries without Cubing
diff --git a/website/_dev40/plugin_arch.cn.md b/website/_dev40/plugin_arch.cn.md
new file mode 100644
index 0000000..23b1ff4
--- /dev/null
+++ b/website/_dev40/plugin_arch.cn.md
@@ -0,0 +1,42 @@
+---
+layout: dev40-cn
+title: 插件架构
+categories: development
+permalink: /cn/development40/plugin_arch.html
+---
+
+插件架构旨在使 Kylin 在计算框架，数据源和 cube 存储方面具有可扩展性。从 v1 开始，Kylin 与作为计算框架的 Hadoop MapReduce，作为数据源的 Hive，作为存储的 HBase 紧密结合。这样的问题出现了：Kylin 可以使用 Spark 作为 cube 引擎，或者可以使用像 Cassandra 那样不同的存储。我们希望对不同的选择持开放态度，并确保 Kylin 用最好的技术堆栈进化。这就是 Kylin v2 中引入插件架构的原因。
+
+
+
+## 如何运行
+
+cube 元数据定义了 cube 所依赖的引擎，源和存储的类型。工厂模式用于构造每个依赖项的实例。适配器模式用于将部件连接在一起。
+
+例如一个 cube 描述可能包含:
+
+- fact_table: `SOME_HIVE_TABLE`
+- engine_type: `2` (MR Engine v2)
+- storage_type: `2` (HBase Storage v2)
+
+基于元数据，工厂创建 MR 引擎，Hive 数据源和 HBase 存储。
+
+
+
+引擎就像一个主板，源和存储必须由输入和输出接口定义。数据源和存储必须适应接口，以便连接到引擎主板。
+
+
+
+一旦上面的对象图被创建和连接，引擎就可以驱动 cube 构建过程。
+
+## 插件架构的好处
+
+- 自由
+ - Zoo 打破了，不再与 Hadoop 绑定
+ - 免费使用更好的引擎或存储
+- 可扩展性
+ - 接受任意输入，例如 Kafka
+ - 拥抱下一代分布式平台，例如 Spark
+- 灵活性
+ - 为不同的数据集选择不同的引擎
+
diff --git a/website/_dev40/plugin_arch.md b/website/_dev40/plugin_arch.md
new file mode 100644
index 0000000..a0eab55
--- /dev/null
+++ b/website/_dev40/plugin_arch.md
@@ -0,0 +1,42 @@
+---
+layout: dev40
+title: Plugin Architecture
+categories: development
+permalink: /development40/plugin_arch.html
+---
+
+The plugin architecture aims to make Kylin extensible regarding computation framework, data source, and cube storage. As of v1, Kylin tightly couples with Hadoop MapReduce as computation framework, Hive as data source, and HBase as storage. Questions came like: could Kylin use Spark as cube engine, or how about a different storage like Cassandra. We want to be open to different options, and to make sure Kylin evolve with the best tech stacks. That is why the plugin architecture is introd [...]
+
+
+
+## How it Works
+
+The cube metadata defines the type of engine, source, and storage that a cube depends on. Factory pattern is used to construct instances of each dependency. Adaptor pattern is used to connect the parts together.
+
+For example a cube descriptor may contains:
+
+- fact_table: `SOME_HIVE_TABLE`
+- engine_type: `2` (MR Engine v2)
+- storage_type: `2` (HBase Storage v2)
+
+Based on the metadata, factories creates MR engine, Hive data source, and HBase storage.
+
+
+
+The engine is like a motherboard, on which source and storage must be plugged as defined by the IN and OUT interfaces. Data source and storage must adapt to the interfaces in order to be connected to engine motherboard.
+
+
+
+Once the above object graph is created and connected, engine can drive the cube build process.
+
+## The Benefits of Plugin Architecture
+
+- Freedom
+ - Zoo break, not bound to Hadoop any more
+ - Free to go to a better engine or storage
+- Extensibility
+ - Accept any input, e.g. Kafka
+ - Embrace next-gen distributed platform, e.g. Spark
+- Flexibility
+ - Choose different engine for different data set
+
diff --git a/website/_dev40/web_tech.cn.md b/website/_dev40/web_tech.cn.md
new file mode 100644
index 0000000..df46912
--- /dev/null
+++ b/website/_dev40/web_tech.cn.md
@@ -0,0 +1,46 @@
+---
+layout: dev40-cn
+title: "Kylin Web 摘要"
+categories: development
+permalink: /cn/development40/web_tech.html
+---
+
+### 项目依赖
+* npm：用于开发阶段安装 grunt 和 bower
+* grunt：构建并安装 kylin web
+* bower：管理 kylin 技术依赖
+
+### 技术依赖
+* Angular JS：kylin web 的基础支持
+* ACE：sql 和 json 编辑器
+* D3 JS：绘制报表图表和 cube 图表
+* Bootstrap：css 库
+
+### 支持的用例:
+
+###### Kylin web 支持 BI 工作流中各种角色的需求
+
+* 分析师：运行查询和检出结果
+* Modeler：cube 设计，cube/job 操作和监视器
+* 管理员：系统操作
+
+### 技术概览
+Kylin web 是一个基于 restful 服务构建的单页应用程序。Kylin web 使用 nodejs 中的工具来管理项目，并使用 AngularJS 来启用单页 Web 应用程序。Kylin web 使用来自 js 开源社区的流行技术，使其易于追赶和贡献。
+
+### 强调:
+* 查询实用功能：
+ * 表和列名称的 SQL 自动建议
+ * 远程/本地查询保存
+ * 数据网格通过简单的 BI 操作能支持百万级数据
+ * 数据导出
+ * 简单的数据可视化（折线图，柱状图，饼图）
+* Cube 管理：
+ * 精心设计的 Cube 创建流程
+ * Cube 关系结构的可视化
+ * 精心设计的 Cube 访问管理
+* Job 管理：
+ * Job 步骤和日志监视器
+ * 杀死
+ * 恢复
+* 有用的管理工具
+* 精致的外观和感觉
diff --git a/website/_dev40/web_tech.md b/website/_dev40/web_tech.md
new file mode 100644
index 0000000..9676315
--- /dev/null
+++ b/website/_dev40/web_tech.md
@@ -0,0 +1,46 @@
+---
+layout: dev40
+title: "Kylin Web Summary"
+categories: development
+permalink: /development40/web_tech.html
+---
+
+### Project Dependencies
+* npm: used in development phase to install grunt and bower
+* grunt: build and set up kylin web
+* bower: manage kylin tech dependencies
+
+### Tech Dependencies
+* Angular JS: fundamental support of kylin web
+* ACE: sql and json editor
+* D3 JS: draw report chart and cube graph
+* Bootstrap: css lib
+
+### Supported Use Cases:
+
+###### Kylin web supports needs of various of roles in BI workflow.
+
+* Analyst: Run query and checkout results
+* Modeler: cube design, cube/job operation and monitor
+* Admin: system operation.
+
+### Tech Overview
+Kylin web is a one-page application build on top of restful services. Kylin web uses tools from nodejs to manage project and use AngularJS to enable one-page web app. Kylin web uses popular techs from js opensource community making it easy to catch up and contribute.
+
+### Highlights:
+* Query utility functions:
+ * SQL auto-suggestions on table and column name
+ * Query remote/local save.
+ * Data grid supporting million level data with easy BI operations
+ * Data export
+ * Simple data visualization(line, bar, pie)
+* Cube management:
+ * Well-designed cube creation flow
+ * Visualization of cube relational structure
+ * Well-designed cube access management
+* Job management:
+ * Job steps and log monitor
+ * Kill
+ * Resume
+* Useful admin tools
+* Refined look & feel
diff --git a/website/_includes/dev40_nav.cn.html b/website/_includes/dev40_nav.cn.html
new file mode 100644
index 0000000..6ab346d
--- /dev/null
+++ b/website/_includes/dev40_nav.cn.html
@@ -0,0 +1,33 @@
+<!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<div class="col-md-4 col-lg-4 col-xs-12 col-sm-12 aside1 visible-sm visible-xs visible-md visible-lg" id="nside1" style=" margin-top: 5em;float:right;">
+ <ul class="nav nav-pills nav-stacked">
+ {% for section in site.data.development40-cn %}
+ <li><a href="#{{ section | first }}" data-toggle="collapse" class="navtitle">{{ section.title }}</a></li>
+ <div class="collapse in">
+ <div class="list-group" id="list1">
+ <ul>
+ {% include dev40_ul.cn.html items=section.dev %}
+ <ul>
+ </div>
+</div>
+ {% endfor %}
+
+ </ul>
+</div>
\ No newline at end of file
diff --git a/website/_includes/dev40_nav.html b/website/_includes/dev40_nav.html
new file mode 100644
index 0000000..ecf6cbc
--- /dev/null
+++ b/website/_includes/dev40_nav.html
@@ -0,0 +1,33 @@
+<!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<div class="col-md-4 col-lg-4 col-xs-12 col-sm-12 aside1 visible-sm visible-xs visible-md visible-lg" id="nside1" style=" margin-top: 5em;float:right;">
+ <ul class="nav nav-pills nav-stacked">
+ {% for section in site.data.development40 %}
+ <li><a href="#{{ section | first }}" data-toggle="collapse" class="navtitle">{{ section.title }}</a></li>
+ <div class="collapse in">
+ <div class="list-group" id="list1">
+ <ul>
+ {% include dev40_ul.html items=section.dev %}
+ <ul>
+ </div>
+</div>
+ {% endfor %}
+
+ </ul>
+</div>
\ No newline at end of file
diff --git a/website/_includes/dev40_ul.cn.html b/website/_includes/dev40_ul.cn.html
new file mode 100644
index 0000000..ced686c
--- /dev/null
+++ b/website/_includes/dev40_ul.cn.html
@@ -0,0 +1,27 @@
+{% assign items = include.items %}
+
+
+
+{% for item in items %}
+
+ {% assign item_url = item | prepend:"/cn/development40/" | append:".html" %}
+
+ {% if item_url == page.url %}
+ {% assign c = "current" %}
+ {% else %}
+ {% assign c = "" %}
+ {% endif %}
+
+
+
+ {% for p in site.dev40 %}
+ {% if p.url == item_url %}
+ <li class="navlist">
+ <a href="{{ p.url }}" class="list-group-item-lay pjaxlink">{{p.title}}</a></li>
+ {% break %}
+ {% endif %}
+ {% endfor %}
+
+{% endfor %}
+
+
diff --git a/website/_includes/dev40_ul.html b/website/_includes/dev40_ul.html
new file mode 100644
index 0000000..8a7b471
--- /dev/null
+++ b/website/_includes/dev40_ul.html
@@ -0,0 +1,27 @@
+{% assign items = include.items %}
+
+
+
+{% for item in items %}
+
+ {% assign item_url = item | prepend:"/development40/" | append:".html" %}
+
+ {% if item_url == page.url %}
+ {% assign c = "current" %}
+ {% else %}
+ {% assign c = "" %}
+ {% endif %}
+
+
+
+ {% for p in site.dev40 %}
+ {% if p.url == item_url %}
+ <li class="navlist">
+ <a href="{{ p.url }}" class="list-group-item-lay pjaxlink">{{p.title}}</a></li>
+ {% break %}
+ {% endif %}
+ {% endfor %}
+
+{% endfor %}
+
+
diff --git a/website/_layouts/dev40-cn.html b/website/_layouts/dev40-cn.html
new file mode 100644
index 0000000..fb88093
--- /dev/null
+++ b/website/_layouts/dev40-cn.html
@@ -0,0 +1,47 @@
+<!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<!doctype html>
+<html>
+ {% include head.cn.html %}
+ <body>
+ {% include header.cn.html %}
+
+ <div class="container">
+ <div class="row" style="clear:both;">
+
+ <div class="col-md-8 col-lg-8 col-xs-12 col-sm-12 aside2" style="float: right;">
+ <div id="container" class="docs">
+ <div id="pjax">
+ <h1 class="post-title">{{ page.title }}</h1>
+ <article class="post-content" >
+ {{ content }}
+ </article>
+ </div>
+ </div>
+ </div>
+ {% include dev40_nav.cn.html %}
+ </div>
+ </div>
+ {% include footer.html %}
+
+ <script src="/assets/js/jquery-1.9.1.min.js"></script>
+ <script src="/assets/js/bootstrap.min.js"></script>
+ <script src="/assets/js/main.js"></script>
+ </body>
+</html>
diff --git a/website/_layouts/dev40.html b/website/_layouts/dev40.html
new file mode 100644
index 0000000..a187068
--- /dev/null
+++ b/website/_layouts/dev40.html
@@ -0,0 +1,47 @@
+<!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+<!doctype html>
+<html>
+ {% include head.html %}
+ <body>
+ {% include header.html %}
+
+ <div class="container">
+ <div class="row" style="clear:both;">
+
+ <div class="col-md-8 col-lg-8 col-xs-12 col-sm-12 aside2" style="float: right;">
+ <div id="container" class="docs">
+ <div id="pjax">
+ <h1 class="post-title">{{ page.title }}</h1>
+ <article class="post-content" >
+ {{ content }}
+ </article>
+ </div>
+ </div>
+ </div>
+ {% include dev40_nav.html %}
+ </div>
+ </div>
+ {% include footer.html %}
+
+ <script src="/assets/js/jquery-1.9.1.min.js"></script>
+ <script src="/assets/js/bootstrap.min.js"></script>
+ <script src="/assets/js/main.js"></script>
+ </body>
+</html>
diff --git a/website/images/develop40/debug_tomcat_config.png b/website/images/develop40/debug_tomcat_config.png
new file mode 100644
index 0000000..49202b2
Binary files /dev/null and b/website/images/develop40/debug_tomcat_config.png differ
diff --git a/website/website.iml b/website/website.iml
new file mode 100644
index 0000000..8021953
--- /dev/null
+++ b/website/website.iml
@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<module type="WEB_MODULE" version="4">
+ <component name="NewModuleRootManager" inherit-compiler-output="true">
+ <exclude-output />
+ <content url="file://$MODULE_DIR$" />
+ <orderEntry type="inheritedJdk" />
+ <orderEntry type="sourceFolder" forTests="false" />
+ </component>
+</module>
\ No newline at end of file
[kylin] 03/03: Add kylin4 blog
Posted by xx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a commit to branch document
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit 8b9dfbdcbc2432fa0d312f033d24d9f93c5da9b0
Author: yaqian.zhang <59...@qq.com>
AuthorDate: Fri Jul 2 18:39:36 2021 +0800
Add kylin4 blog
---
...ylin4-A-new-storage-and-compute-architecture.md | 92 +++++++++++++++++++++
.../blog/kylin4/1 apache_kylin_introduction.png | Bin 0 -> 45730 bytes
.../blog/kylin4/10 spark_parquet_solution.png | Bin 0 -> 365968 bytes
.../blog/kylin4/11 spark_parquet_architecture.png | Bin 0 -> 79197 bytes
website/images/blog/kylin4/2 cube_vs_cuboid.png | Bin 0 -> 122901 bytes
.../blog/kylin4/3 cuboid_selected_for_query.png | Bin 0 -> 131322 bytes
.../blog/kylin4/4 apache_kylin_query_process.png | Bin 0 -> 87024 bytes
.../blog/kylin4/5 query_using_precomputed_data.png | Bin 0 -> 289496 bytes
.../images/blog/kylin4/6 on-site-computation.png | Bin 0 -> 278849 bytes
.../7 precomputation_using_aggregated_data.png | Bin 0 -> 323673 bytes
.../kylin4/8 diagram_of_calcite_executions.png | Bin 0 -> 219636 bytes
.../kylin4/9 diagram_of_bottleneck_on_HBase.png | Bin 0 -> 265462 bytes
12 files changed, 92 insertions(+)
diff --git a/website/_posts/blog/2021-07-02-Apache-Kylin4-A-new-storage-and-compute-architecture.md b/website/_posts/blog/2021-07-02-Apache-Kylin4-A-new-storage-and-compute-architecture.md
new file mode 100644
index 0000000..fbb9f1c
--- /dev/null
+++ b/website/_posts/blog/2021-07-02-Apache-Kylin4-A-new-storage-and-compute-architecture.md
@@ -0,0 +1,92 @@
+---
+layout: post-blog
+title: Apache Kylin4 — A new storage and compute architecture
+date: 2021-07-02 15:00:00
+author: Jiatao Tao
+categories: blog
+---
+
+This article will discuss three aspects of Apache Kylin: First, we will briefly introduce query principles of Apache Kylin. Next, we will introduce Apache Parquet Storage, a project our team has been involved in that Kyligence is contributing back to the open source software community by the end of this year (2020). Finally, we will introduce the extensive use of precision count distinct by community users as well as its implementation in Kylin and some extensions.
+
+## 01 Introduction to Apache Kylin
+Apache Kylin is an open source distributed analysis engine that provides SQL query interfaces above Hadoop/Spark and OLAP capabilities to support extremely large data. It was initially developed at eBay Inc. and contributed to the open source software community. It can query massive relational tables with sub-second response times.
+
+
+As a SQL acceleration layer, Kylin can connect with various data sources such as Hive and Kafka, and can connect with commonly used BI systems such as Tableau and Power BI. It can also be queried directly (ad hoc) using standard SQL tools.
+
+If you find yourself confronted by unhappy BI users for any of the following reasons, you should consider using Apache Kylin:
+- Their batch of queries are too slow
+- Query or user concurrency should be higher
+- Resources usage should be lower
+- The system doesn’t fully support SQL syntax
+- The system doesn’t seamlessly integrate with their favorite BI tools\
+
+## 02 Apache Kylin Rationale
+Kylin's core idea is the precomputation of result sets, meaning it calculates all possible query results in advance according to the specified dimensions and indicators and uses space for time to speed up OLAP queries with fixed query patterns.
+
+
+Kylin’s design is based on cube theory. Each combination of dimensions is called a cuboid and the set of all cuboids is a cube. The cuboid composed of all dimensions is called the base cuboid, and the time, item, location, and supplier shown in the figure is an example of this. All cuboids can be calculated from the base cuboid. A cuboid can be understood as a wide table after precomputation. During the query, Kylin will automatically select the most suitable cuboid that meets the query [...]
+
+
+For example, the query in the above figure will look for the cuboid (time, item, location). Compared with the calculation from the user's original table, the calculation from the cuboid can greatly reduce the amount of scanned data and calculation.
+
+## 03 Apache Kylin Basic Query Process
+Let’s look briefly at the rationale of Kylin queries. The first three steps are the routine operations of all query engines. We use the Apache Calcite framework to complete this operation. We will not go into great detail here but, should you wish to learn more, there is plenty of related material online.
+
+
+The introduction here focuses on the last two steps: Kylin adaptation and query execution. Why do we need to do Kylin adaptation? Because the query plan we obtained earlier is directly converted according to the user's query, and so this query plan cannot directly query the precomputed data. Here, a rewrite is needed to create an execution plan so that it can query the precomputed data (i.e. cube data). Let's look at the following example:
+
+
+The user has a stock of goods. Item and user_id indicate which item has been accessed and the user wants to analyze the Page View (PV) of the goods. The user defines a cube where the dimension is item and the measure is COUNT (user_id). If the user wants to analyze the PV of the goods, he will issue the following SQL:
+
+```
+SELECT item, COUNT (user_id) FROM stock GROUP BY item;
+```
+
+After this SQL is sent to Kylin, Kylin cannot directly use its original semantics to query our cube data. This is because after the data is precomputed, there will only be one row of data in the key of each item. The rows of the same item key in the original table have been aggregated in advance, generating a new measure column to store how many user_id accesses each item key has, so the rewritten SQL will be similar to this:
+
+```
+ SELECT item, SUM (M_C) FROM stock GROUP BY item;
+```
+
+Why is there another SUM/GROUP BY operation here instead of directly fetching the data and returning it? Because the cuboid that may be hit by the query is more than one dimension of item, meaning it is not the most accurate cuboid. It needs to be aggregated again from these dimensions, but the amount of partially aggregated data still significantly reduces the amount of data and calculation compared with the data in the user's original table. If the query hits the cuboid accurately, we [...]
+
+
+The above figure is a scenario without precomputation, which requires on-site calculation. Agg and Join will involve shuffle, so the performance will be poor and more resources will be occupied with large amounts of data, which will affect the concurrency of queries.
+
+
+After the precomputation, the previously most time-consuming two-step operation (Agg/Join) disappeared from the rewritten execution plan, showing a cuboid precise match. Additionally, when defining the cube we can choose to order by column so the Sort operation does not need to be calculated. The whole calculation is a single stage without the expense of a shuffle. The calculation can be completed with only a few tasks therefore improving the concurrency of the query.
+
+## 04 Apache Kylin on HBase
+In the current open source version, the built data is stored in HBase, we've got a logical execution plan that can query cube data from the above section. Calcite framework will generate the corresponding physical execution plan according to this logical execution plan and, finally, each operator will generate its own executable code through code generation.
+
+
+This process is an iterator model. Data flows from the lowest TableScan operator to the upstream operator. The whole process is like a volcanic eruption, so it is also called Volcano Iterator Mode. The code generated by this TableScan will fetch cube data from HBase, and when the data is returned to Kylin Query Server, it will be consumed layer by layer by the upper operator.
+
+## 05 Bottlenecks with Kylin on HBase
+This scenario is not a big problem with simple SQL because, in the case of a precise matching cuboid, minimal computing will be done on Kylin Query Server after retrieving the data from HBase. However, for some more complex queries, Kylin Query Server will not only pull back a large amount of data from HBase but also compute very resource-intensive operations such as Joins and Aggregates.
+
+For example, a query joins two subqueries, each subquery hits its own cube and then does some more complicated aggregate operations at the outermost layer such as COUNT DISTINCT. When the amount of data becomes large, Kylin Query Server may be out of memory (OOM). The solution is to simply increase the memory of the Kylin Query Server.
+
+However, this is a vertical expansion process that becomes a bottleneck. We know from experience that bottlenecks in big data can be difficult to diagnose and can lead to the abandonment of a critical technology when selecting an architecture. In addition, there are many other limitations when using this system. For example, the operation and maintenance of HBase is notoriously difficult. It is safe to assume that once the performance of HBase is not good, the performance of Kylin will a [...]
+
+The resource isolation capabilities of HBase are also relatively weak. When there is a large load at a given moment, other applications using HBase will also be affected. This may cause Kylin to have unstable query performance which can be difficult to troubleshoot. All data stored in HBase are encoded Byte Array types and the overhead of serialization and deserialization cannot be ignored.
+
+## 06 Apache Kylin with Spark + Parquet
+Due to the limitations of the Kylin-on-HBase solution mentioned above, Kyligence has developed a new generation of Spark + Parquet-based solutions for the commercial version of Kylin. This was done early on to update and enhance the open source software solution for enterprise use.
+
+The following is an introduction to the overall framework of this new system.
+
+
+In fact, the new design is very simple. The visitor mode is used to traverse the previously generated logical execution plan tree that can query cube data. The nodes of the execution plan tree represent an operator, which actually stores nothing more than some information such as which table to scan, which columns to filter/project, etc. Each operator will be translated into a Spark operation on Dataframe on the original tree, each upstream node asks its downstream node for a DF up to th [...]
+
+## 07 Advantages of the Spark/Parquet Architecture
+This Kylin on Parquet plan relies on Spark. All calculations are distributed and there is no single point where performance can bottleneck. The computing power of the system can be improved through horizontal expansion (scale-out). There are various schemes for resource scheduling such as Yarn, K8S, or Mesos to meet the needs of enterprises for resource isolation. Spark's performance efforts can be naturally enjoyed. The overhead of serialization and deserialization of Kylin on HBase men [...]
+
+
+Reducing the dependence upon HBase simplifies operation and maintenance. All upstream and downstream dependencies can be handled by Spark for us, reducing our dependence and facilitating cloud access.
+
+For developers, the DF generated by each operator can be collected directly to observe whether there is any problem with the data at this level, and Spark + Parquet is currently a very popular SQL on Hadoop scheme. The open source committers at Kyligence are also familiar with these two projects and maintain their own Spark and Parquet branch. A lot of performance optimization and stability improvements have been done in this area for our specific scenarios.
+
+## 08 Summary
+Apache Kylin has over 1,000 users worldwide. But, in order for the project to ensure its future position as a vital, Cloud-Native technology for enterprise analytics, the Kylin community must periodically evaluate and update the key architectural assumptions being made to accomplish that goal. The removal of legacy connections to the Hadoop ecosystem in favor of Spark and Parquet is an important next step to realizing the dream of pervasive analytics based on open source technology for o [...]
diff --git a/website/images/blog/kylin4/1 apache_kylin_introduction.png b/website/images/blog/kylin4/1 apache_kylin_introduction.png
new file mode 100644
index 0000000..ec70cfc
Binary files /dev/null and b/website/images/blog/kylin4/1 apache_kylin_introduction.png differ
diff --git a/website/images/blog/kylin4/10 spark_parquet_solution.png b/website/images/blog/kylin4/10 spark_parquet_solution.png
new file mode 100644
index 0000000..34a5dbf
Binary files /dev/null and b/website/images/blog/kylin4/10 spark_parquet_solution.png differ
diff --git a/website/images/blog/kylin4/11 spark_parquet_architecture.png b/website/images/blog/kylin4/11 spark_parquet_architecture.png
new file mode 100644
index 0000000..59b143e
Binary files /dev/null and b/website/images/blog/kylin4/11 spark_parquet_architecture.png differ
diff --git a/website/images/blog/kylin4/2 cube_vs_cuboid.png b/website/images/blog/kylin4/2 cube_vs_cuboid.png
new file mode 100644
index 0000000..928207a
Binary files /dev/null and b/website/images/blog/kylin4/2 cube_vs_cuboid.png differ
diff --git a/website/images/blog/kylin4/3 cuboid_selected_for_query.png b/website/images/blog/kylin4/3 cuboid_selected_for_query.png
new file mode 100644
index 0000000..2ee7a9f
Binary files /dev/null and b/website/images/blog/kylin4/3 cuboid_selected_for_query.png differ
diff --git a/website/images/blog/kylin4/4 apache_kylin_query_process.png b/website/images/blog/kylin4/4 apache_kylin_query_process.png
new file mode 100644
index 0000000..aae7822
Binary files /dev/null and b/website/images/blog/kylin4/4 apache_kylin_query_process.png differ
diff --git a/website/images/blog/kylin4/5 query_using_precomputed_data.png b/website/images/blog/kylin4/5 query_using_precomputed_data.png
new file mode 100644
index 0000000..19c93a1
Binary files /dev/null and b/website/images/blog/kylin4/5 query_using_precomputed_data.png differ
diff --git a/website/images/blog/kylin4/6 on-site-computation.png b/website/images/blog/kylin4/6 on-site-computation.png
new file mode 100644
index 0000000..97e94e2
Binary files /dev/null and b/website/images/blog/kylin4/6 on-site-computation.png differ
diff --git a/website/images/blog/kylin4/7 precomputation_using_aggregated_data.png b/website/images/blog/kylin4/7 precomputation_using_aggregated_data.png
new file mode 100644
index 0000000..f47340b
Binary files /dev/null and b/website/images/blog/kylin4/7 precomputation_using_aggregated_data.png differ
diff --git a/website/images/blog/kylin4/8 diagram_of_calcite_executions.png b/website/images/blog/kylin4/8 diagram_of_calcite_executions.png
new file mode 100644
index 0000000..c8cb454
Binary files /dev/null and b/website/images/blog/kylin4/8 diagram_of_calcite_executions.png differ
diff --git a/website/images/blog/kylin4/9 diagram_of_bottleneck_on_HBase.png b/website/images/blog/kylin4/9 diagram_of_bottleneck_on_HBase.png
new file mode 100644
index 0000000..4b061dc
Binary files /dev/null and b/website/images/blog/kylin4/9 diagram_of_bottleneck_on_HBase.png differ