You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2016/02/13 13:19:37 UTC
kylin git commit: KYLIN-875 add document for arch design
Repository: kylin
Updated Branches:
refs/heads/document 88f1da47c -> 0752828a2
KYLIN-875 add document for arch design
Project: http://git-wip-us.apache.org/repos/asf/kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/kylin/commit/0752828a
Tree: http://git-wip-us.apache.org/repos/asf/kylin/tree/0752828a
Diff: http://git-wip-us.apache.org/repos/asf/kylin/diff/0752828a
Branch: refs/heads/document
Commit: 0752828a28133cadec698ffae6c50eddc0aed67e
Parents: 88f1da4
Author: Yang Li <li...@apache.org>
Authored: Sat Feb 13 20:19:14 2016 +0800
Committer: Yang Li <li...@apache.org>
Committed: Sat Feb 13 20:19:14 2016 +0800
----------------------------------------------------------------------
website/_data/development.yml | 1 +
website/_dev/plugin_arch.md | 42 +++++++++++++++++++
.../develop/plugin_arch_adaptor_pattern.png | Bin 0 -> 50424 bytes
.../develop/plugin_arch_factory_pattern.png | Bin 0 -> 62389 bytes
website/images/develop/plugin_arch_overview.png | Bin 0 -> 151459 bytes
5 files changed, 43 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/_data/development.yml
----------------------------------------------------------------------
diff --git a/website/_data/development.yml b/website/_data/development.yml
index 8b03f76..d3be519 100644
--- a/website/_data/development.yml
+++ b/website/_data/development.yml
@@ -25,6 +25,7 @@
- howto_docs
- howto_package
- howto_release
+ - plugin_arch
- new_metadata
- web_tech
- about_temp_files
http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/_dev/plugin_arch.md
----------------------------------------------------------------------
diff --git a/website/_dev/plugin_arch.md b/website/_dev/plugin_arch.md
new file mode 100644
index 0000000..35fc5e1
--- /dev/null
+++ b/website/_dev/plugin_arch.md
@@ -0,0 +1,42 @@
+---
+layout: dev
+title: Plugin Architecture
+categories: development
+permalink: /development/plugin_arch.html
+---
+
+The plugin architecture aims to make Kylin extensible regarding computation framework, data source, and cube storage. As of v1, Kylin tightly couples with Hadoop MapReduce as computation framework, Hive as data source, and HBase as storage. Questions came like: could Kylin use Spark as cube engine, or how about a different storage like Cassandra. We want to be open to different options, and to make sure Kylin evolve with the best tech stacks. That is why the plugin architecture is introduced in Kylin v2.
+
+
+
+## How it Works
+
+The cube metadata defines the type of engine, source, and storage that a cube depends on. Factory pattern is used to construct instances of each dependency. Adaptor pattern is used to connect the parts together.
+
+For example a cube descriptor may contains:
+
+- fact_table: `SOME_HIVE_TABLE`
+- engine_type: `2` (MR Engine v2)
+- storage_type: `2` (HBase Storage v2)
+
+Based on the metadata, factories creates MR engine, Hive data source, and HBase storage.
+
+
+
+The engine is like a motherboard, on which source and storage must be plugged as defined by the IN and OUT interfaces. Data source and storage must adapt to the interfaces in order to be connected to engine motherboard.
+
+
+
+Once the above object graph is created and connected, engine can drive the cube build process.
+
+## The Benefits of Plugin Architecture
+
+- Freedom
+ - Zoo break, not bound to Hadoop any more
+ - Free to go to a better engine or storage
+- Extensibility
+ - Accept any input, e.g. Kafka
+ - Embrace next-gen distributed platform, e.g. Spark
+- Flexibility
+ - Choose different engine for different data set
+
http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/images/develop/plugin_arch_adaptor_pattern.png
----------------------------------------------------------------------
diff --git a/website/images/develop/plugin_arch_adaptor_pattern.png b/website/images/develop/plugin_arch_adaptor_pattern.png
new file mode 100644
index 0000000..af49430
Binary files /dev/null and b/website/images/develop/plugin_arch_adaptor_pattern.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/images/develop/plugin_arch_factory_pattern.png
----------------------------------------------------------------------
diff --git a/website/images/develop/plugin_arch_factory_pattern.png b/website/images/develop/plugin_arch_factory_pattern.png
new file mode 100644
index 0000000..473270b
Binary files /dev/null and b/website/images/develop/plugin_arch_factory_pattern.png differ
http://git-wip-us.apache.org/repos/asf/kylin/blob/0752828a/website/images/develop/plugin_arch_overview.png
----------------------------------------------------------------------
diff --git a/website/images/develop/plugin_arch_overview.png b/website/images/develop/plugin_arch_overview.png
new file mode 100644
index 0000000..52af951
Binary files /dev/null and b/website/images/develop/plugin_arch_overview.png differ