You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by li...@apache.org on 2015/03/21 01:28:05 UTC
[01/29] incubator-kylin git commit: KYLIN-639 Add localhost:7070 as
kylin.rest.servers in kylin.properties
Repository: incubator-kylin
Updated Branches:
refs/heads/staging 37f4132b6 -> 3655b338f
KYLIN-639 Add localhost:7070 as kylin.rest.servers in kylin.properties
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/39797666
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/39797666
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/39797666
Branch: refs/heads/staging
Commit: 3979766683df58fdfb47ea89da9f0839333c911e
Parents: 37f4132
Author: Shao Feng, Shi <sh...@ebay.com>
Authored: Sat Mar 14 11:31:33 2015 +0800
Committer: Shao Feng, Shi <sh...@ebay.com>
Committed: Sat Mar 14 11:31:33 2015 +0800
----------------------------------------------------------------------
bin/sample.sh | 5 ++++-
conf/kylin.properties | 2 +-
2 files changed, 5 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/39797666/bin/sample.sh
----------------------------------------------------------------------
diff --git a/bin/sample.sh b/bin/sample.sh
index 84629e9..02930a9 100644
--- a/bin/sample.sh
+++ b/bin/sample.sh
@@ -2,7 +2,10 @@
dir=$(dirname ${0})
source ${dir}/check-env.sh
job_jar=`find ${KYLIN_HOME}/lib/ -name kylin-job*.jar`
+echo "Going to create sample tables in hive..."
cd ${KYLIN_HOME}/sample_cube/data
hive -f ${KYLIN_HOME}/sample_cube/create_sample_tables.sql || { exit 1; }
+echo "Sample hive tables are created successfully; Going to create sample cube..."
cd ${KYLIN_HOME}
-hbase org.apache.hadoop.util.RunJar ${job_jar} org.apache.kylin.common.persistence.ResourceTool upload ${KYLIN_HOME}/sample_cube/metadata || { exit 1; }
\ No newline at end of file
+hbase org.apache.hadoop.util.RunJar ${job_jar} org.apache.kylin.common.persistence.ResourceTool upload ${KYLIN_HOME}/sample_cube/metadata || { exit 1; }
+echo "Sample cube is created successfully in project 'learn_kylin'; Restart Kylin server or reload the metadata from web UI to see the change."
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/39797666/conf/kylin.properties
----------------------------------------------------------------------
diff --git a/conf/kylin.properties b/conf/kylin.properties
index f607dca..00bc91a 100644
--- a/conf/kylin.properties
+++ b/conf/kylin.properties
@@ -1,7 +1,7 @@
## Config for Kylin Engine ##
# List of web servers in use, this enables one web server instance to sync up with other servers.
-kylin.rest.servers=
+kylin.rest.servers=localhost:7070
# The metadata store in hbase
kylin.metadata.url=kylin_metadata@hbase
[18/29] incubator-kylin git commit: KYLIN-650 minor change
Posted by li...@apache.org.
KYLIN-650 minor change
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/4bd54078
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/4bd54078
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/4bd54078
Branch: refs/heads/staging
Commit: 4bd54078af6dd7c60c4bb5eb8aab4a99e2a6d5f3
Parents: eff1393
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 22:27:23 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 22:27:23 2015 -0700
----------------------------------------------------------------------
README.md | 12 ++++--
...requently Asked Questions on Installation.md | 26 ------------
.../MISC/FAQ on Kylin Installation and Usage.md | 40 +++++++++++++++++++
docs/MISC/How to Contribute.md | 42 ++++++++++++++++++++
4 files changed, 90 insertions(+), 30 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/4bd54078/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 232758c..2489576 100644
--- a/README.md
+++ b/README.md
@@ -17,6 +17,10 @@ Tutorial
Please follow this installation tutorial to start with Kylin: [Installation Tutorial](docs/Installation/Installation.md)
+Advance Usage
+-------
+
+
Get Help
------------
@@ -26,13 +30,13 @@ The fastest way to get response from our developers is to send email to our mail
Resources
------------
-* Web Site: <http://kylin.io>
+* [FAQ](docs/MISC/FAQ on Kylin Installation and Usage.md)
-* Developer Mail: <de...@kylin.incubator.apache.org>
+* Web Site: <http://kylin.incubator.apache.org/>
-* How To Contribute: See [wiki](https://github.com/KylinOLAP/Kylin/wiki/How-to-Contribute)
+* Developer Mail: <de...@kylin.incubator.apache.org>
-* Presentation: [Kylin Hadoop OLAP Engine v1.0](https://github.com/KylinOLAP/Kylin/blob/master/docs/Kylin_Hadoop_OLAP_Engine_v1.0.pdf?raw=true)
+* How To Contribute: See [this](docs/MISC/How to Contribute.md)
* Apache Proposal: [Apache Kylin](https://wiki.apache.org/incubator/KylinProposal)
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/4bd54078/docs/Installation/Frequently Asked Questions on Installation.md
----------------------------------------------------------------------
diff --git a/docs/Installation/Frequently Asked Questions on Installation.md b/docs/Installation/Frequently Asked Questions on Installation.md
deleted file mode 100644
index c160403..0000000
--- a/docs/Installation/Frequently Asked Questions on Installation.md
+++ /dev/null
@@ -1,26 +0,0 @@
-Frequently Asked Questions on Installation
----
-* Some NPM error causes ERROR exit (中国大陆地区用户请特别注意此问题)?
-> Check out https://github.com/KylinOLAP/Kylin/issues/35
-
-* Can't get master address from ZooKeeper" when installing Kylin on Hortonworks Sandbox
-> Check out https://github.com/KylinOLAP/Kylin/issues/9.
-
-* Install scripted finished in my virtual machine, but cannot visit via http://localhost:9080
-> Check out https://github.com/KylinOLAP/Kylin/issues/12.
-
-* Map Reduce Job information can't display on sandbox deployment
-> Check out https://github.com/KylinOLAP/Kylin/issues/40
-
-* Install Kylin on CDH 5.2 or Hadoop 2.5.x
-> Check out discussion: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kylin-olap/X0GZfsX1jLc/nzs6xAhNpLkJ
->
-```
-I was able to deploy Kylin with following option in POM.
-<hadoop2.version>2.5.0</hadoop2.version>
-<yarn.version>2.5.0</yarn.version>
-<hbase-hadoop2.version>0.98.6-hadoop2</hbase-hadoop2.version>
-<zookeeper.version>3.4.5</zookeeper.version>
-<hive.version>0.13.1</hive.version>
-My Cluster is running on Cloudera Distribution CDH 5.2.0.
-```
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/4bd54078/docs/MISC/FAQ on Kylin Installation and Usage.md
----------------------------------------------------------------------
diff --git a/docs/MISC/FAQ on Kylin Installation and Usage.md b/docs/MISC/FAQ on Kylin Installation and Usage.md
new file mode 100644
index 0000000..1c1291e
--- /dev/null
+++ b/docs/MISC/FAQ on Kylin Installation and Usage.md
@@ -0,0 +1,40 @@
+FAQ on Kylin Installation and Usage
+---
+#### Some NPM error causes ERROR exit (中国大陆地区用户请特别注意此问题)?
+Check out https://github.com/KylinOLAP/Kylin/issues/35
+
+#### Can't get master address from ZooKeeper" when installing Kylin on Hortonworks Sandbox
+Check out https://github.com/KylinOLAP/Kylin/issues/9.
+
+#### Install scripted finished in my virtual machine, but cannot visit via http://localhost:9080
+Check out https://github.com/KylinOLAP/Kylin/issues/12.
+
+#### Map Reduce Job information can't display on sandbox deployment
+Check out https://github.com/KylinOLAP/Kylin/issues/40
+
+#### Install Kylin on CDH 5.2 or Hadoop 2.5.x
+Check out discussion: https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kylin-olap/X0GZfsX1jLc/nzs6xAhNpLkJ
+```
+I was able to deploy Kylin with following option in POM.
+<hadoop2.version>2.5.0</hadoop2.version>
+<yarn.version>2.5.0</yarn.version>
+<hbase-hadoop2.version>0.98.6-hadoop2</hbase-hadoop2.version>
+<zookeeper.version>3.4.5</zookeeper.version>
+<hive.version>0.13.1</hive.version>
+My Cluster is running on Cloudera Distribution CDH 5.2.0.
+```
+
+#### Unable to load a big cube as HTable, with java.lang.OutOfMemoryError: unable to create new native thread
+HBase (as of writing) allocates one thread per region when bulk loading a HTable. Try reduce the number of regions of your cube by setting its "capacity" to "MEDIUM" or "LARGE". Also tweaks OS & JVM can allow more threads, for example see [this article](http://blog.egilh.com/2006/06/2811aspx.html).
+
+#### Failed to run BuildCubeWithEngineTest, saying failed to connect to hbase while hbase is active
+User may get this error when first time run hbase client, please check the error trace to see whether there is an error saying couldn't access a folder like "/hadoop/hbase/local/jars"; If that folder doesn't exist, create it.
+
+#### SUM(field) returns a negtive result while all the numbers in this field are > 0
+If a column is declared as integer in Hive, the SQL engine (calcite) will use column's type (integer) as the data type for "SUM(field)", while the aggregated value on this field may exceed the scope of integer; in that case the cast will cause a negtive value be returned; The workround is, alter that column's type to BIGINT in hive, and then sync the table schema to Kylin (the cube doesn't need rebuild); Keep in mind that, always declare as BIGINT in hive for an integer column which would be used as a measure in Kylin; See hive number types: https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-NumericTypes
+
+#### Why Kylin need extract the distinct columns from Fact Table before building cube?
+Kylin uses dictionary to encode the values in each column, this greatly reduce the cube's storage size. To build the dictionary, Kylin need fetch the distinct values for each column.
+
+#### Why Kylin calculate the HIVE table cardinality?
+The cardinality of dimensions is an important measure of cube complexity. The higher the cardinality, the bigger the cube, and thus the longer to build and the slower to query. Cardinality > 1,000 is worth attention and > 1,000,000 should be avoided at best effort. For optimal cube performance, try reduce high cardinality by categorize values or derive features.
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/4bd54078/docs/MISC/How to Contribute.md
----------------------------------------------------------------------
diff --git a/docs/MISC/How to Contribute.md b/docs/MISC/How to Contribute.md
new file mode 100644
index 0000000..e270606
--- /dev/null
+++ b/docs/MISC/How to Contribute.md
@@ -0,0 +1,42 @@
+#### Setup Dev Env
+* subscribe our developers' mail list via <de...@kylin.incubator.apache.org>
+* Fork on [GitHub](https://github.com/KylinOLAP)
+* ...
+
+#### Making Changes
+* Raise an issue on GitHub, describe the feature/enhancement/bug
+* Discuss with others in google group or issue comments, make sure the proposed changes fit in with what others are doing and have planned for the project
+* Make changes in your fork
+* Write unit test if no existing cover your change
+* Push to GitHub under your fork
+
+
+#### Contribute The Work
+* Raise a pull request on GitHub, include both **code** and **test**, link with **related issue**
+* Committer will review in terms of correctness, performance, design, coding style, test coverage
+* Discuss and revise if necessary
+* Finally committer merge code into main branch
+
+
+#### Wish List
+Some potential work items
+* Query Engine
+ * Cache generated class, reduce delay into ms level and avoid full GC triggered by perm generation
+ * [Issue #14](https://github.com/KylinOLAP/Kylin/issues/14) Derive meaningful cost in OLAP relational operator
+ * [Issue #15](https://github.com/KylinOLAP/Kylin/issues/15) Implement multi-column distinct count
+* Metadata
+ * [Issue #7](https://github.com/KylinOLAP/Kylin/issues/7) Merge multiple hbase tables
+* Job Engine
+ * [Issue #16](https://github.com/KylinOLAP/Kylin/issues/16) Tune HDFS & HBase parameters
+ * [Issue #17](https://github.com/KylinOLAP/Kylin/issues/17) Increase HDFS block size 1GB or close
+ * Shell command to support kill operation
+ * Use DoubleDouble instead of BigDecimal during cube build
+ * Drop quartz dependency, assess the cost/benefit first
+ * Cardinality run as one step job, allows progress tracking
+* ODBC/JDBC
+ * Test Kylin remote JDBC with java report tools
+ * Implement ODBC async mode, fetching from Kylin and feeding to client in parallel
+* Benchmark
+ * [Issue #18](https://github.com/KylinOLAP/Kylin/issues/18) Benchmark on standard dataset
+
+
[17/29] incubator-kylin git commit: KYLIN-650 add maillist
Posted by li...@apache.org.
KYLIN-650 add maillist
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/eff13931
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/eff13931
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/eff13931
Branch: refs/heads/staging
Commit: eff1393143a8795b7b08c1ba05d564c914cb09f1
Parents: 7faa216
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 19:30:17 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 19:30:17 2015 -0700
----------------------------------------------------------------------
README.md | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/eff13931/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 28c2df6..232758c 100644
--- a/README.md
+++ b/README.md
@@ -17,15 +17,18 @@ Tutorial
Please follow this installation tutorial to start with Kylin: [Installation Tutorial](docs/Installation/Installation.md)
+Get Help
+------------
+
+The fastest way to get response from our developers is to send email to our mail list <de...@kylin.incubator.apache.org>, and remember to subscribe our mail list via <de...@kylin.incubator.apache.org>
+
Resources
------------
* Web Site: <http://kylin.io>
-* Google Group: [Kylin OLAP Group](https://groups.google.com/forum/#!forum/kylin-olap)
-
-* Developer Mail: <ky...@googlegroups.com>
+* Developer Mail: <de...@kylin.incubator.apache.org>
* How To Contribute: See [wiki](https://github.com/KylinOLAP/Kylin/wiki/How-to-Contribute)
[11/29] incubator-kylin git commit: KYLIN-650 moving wikis to code
repository
Posted by li...@apache.org.
KYLIN-650 moving wikis to code repository
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/592e96b5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/592e96b5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/592e96b5
Branch: refs/heads/staging
Commit: 592e96b519171ae9906cd1774b2a3e88e9dc3d91
Parents: a540f40
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 18:56:04 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 18:56:21 2015 -0700
----------------------------------------------------------------------
README.md | 71 ++-----------
...equently Asked Questions on Installation.pdf | Bin 52261 -> 0 bytes
docs/Installation/Hadoop Environment.md | 21 ++++
.../How to generate install binaries.md | 6 ++
docs/Installation/Installation.md | 38 +++++++
.../Installation/Multiple Kylin REST servers.md | 9 ++
...f Hadoop CLI Installation (Dev Env Setup).md | 76 -------------
... Hadoop CLI Installation (Dev Env Setup).pdf | Bin 44909 -> 0 bytes
.../Installation/Off Hadoop CLI Installation.md | 67 ++++++++++++
docs/Installation/On Hadoop CLI installation.md | 106 +++----------------
.../Installation/On Hadoop CLI installation.pdf | Bin 704398 -> 0 bytes
...On Hadoop Kylin installation using Docker.md | 2 +-
...n Hadoop Kylin installation using Docker.pdf | Bin 40098 -> 0 bytes
docs/Installation/install_and_run.md | 83 ---------------
docs/Installation/off_cli_install_scene.png | Bin 0 -> 72567 bytes
docs/Installation/on_cli_install_scene.png | Bin 0 -> 53722 bytes
docs/README.md | 0
...n Cube Build and Job Monitoring Tutorial.pdf | Bin 1054988 -> 0 bytes
docs/Tutorial/Kylin Cube Creation Tutorial.md | 9 +-
docs/Tutorial/Kylin Cube Creation Tutorial.pdf | Bin 1488946 -> 0 bytes
.../Kylin Cube Permission Grant Tutorial.pdf | Bin 358047 -> 0 bytes
docs/Tutorial/Kylin ODBC Driver Tutorial.md | 7 ++
docs/Tutorial/Kylin ODBC Driver Tutorial.pdf | Bin 34429 -> 0 bytes
docs/Tutorial/Kylin Web Tutorial.md | 8 +-
docs/Tutorial/Kylin Web Tutorial.pdf | Bin 1714951 -> 0 bytes
docs/Tutorial/Kylin and Excel Tutorial.md | 3 +
docs/Tutorial/Kylin and Tableau Tutorial.pdf | Bin 1246553 -> 0 bytes
docs/Tutorial/Quick play with a sample cube.md | 10 ++
old_README.md | 94 ++++++++++++++++
29 files changed, 294 insertions(+), 316 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 44f6859..28c2df6 100644
--- a/README.md
+++ b/README.md
@@ -2,83 +2,24 @@ Kylin OLAP
===========
-Overview
+What is Kylin
------------
> Extreme OLAP Engine for Big Data
An open source distributed Analytical Engine from eBay to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop to support TB to PB size analysis.
+This [slice](docs/Apache Kylin 2014 Dec.pdf) might help you better understand what kylin is and how it works.
-Getting Started
----------------
-### Prerequisites ###
-
-#### Hadoop ####
-- Hadoop: 2.2.0.2.0.6.0-61 or above
-- Hive: 0.12.0.2.0.6.0-61 or above
-- HBase: 0.96.0.2.0.6.0-61-hadoop2
-
-
-_Tested with Hortonworks HDP 2.1.3 and Cloudera Quickstart VM 5.1._
-
-#### Misc ####
-- Command hadoop, hive, hbase is workable on your hadoop cluster
-- JDK Runtime: JDK7 (OpenJDK or Oracle JDK)
-- Maven
-- Git
-- Tomcat (CATALINA_HOME being set)
-- Npm
-
-#### Before Install ####
-- For the quick tutorial, we assume that your hadoop has priviledges disabled which means any user could run hadoop and hive command
-
-
-Installation
+Tutorial
------------
-It is very easy to install Kylin for exploration/development. There are 3 supported options:
-
-
-1. Sandbox (HDP or CDH sandbox)
-2. Dev Environment (IDE + Sandbox)
-3. Docker container (Docker)
-
-#### Sandbox (On-Hadoop-CLI installation) ####
-
-If you are free to install Kylin on your hadoop CLI machine or Hadoop sandbox, this is the most convenient scenario, for it puts everything in a single machine.
-
-
-
+Please follow this installation tutorial to start with Kylin: [Installation Tutorial](docs/Installation/Installation.md)
-For normal users, the application in the above picture means `Kylin Web`, which contains a web interface for cube building, querying and all sorts of management. Kylin Web launches a query engine for querying and a cube build engine for building cubes. These two engines interact with the components in Hadoop CLI, like hive and hbase.
-For a hands-on tutorial please visit [On-Hadoop-CLI installation](https://github.com/KylinOLAP/Kylin/wiki/On-Hadoop-CLI-installation).
-#### Dev Environment (Off-Hadoop-CLI Installation) ####
-
-This is typically for development environment setup.
-
-Applications does not necessarily mean Kylin Web, it can also be any test cases or java applications running at your local IDE(query engine and cube build engine can be launched programmatically in your code). In this case, it is no longer reasonable to assume that all the Kylin components reside in the same machine as your Hadoop CLI. Fortunately, Kylin still works under such condition with proper settings on your CLI machine.
-
-
-
-For a hands-on tutorial please visit [Off-Hadoop-CLI installation](https://github.com/KylinOLAP/Kylin/wiki/Off-Hadoop-CLI-Installation-(Dev-Env-Setup))
-
-#### Docker Container ####
-With help from [SequenceIQ](http://sequenceiq.com/), there's docker container for Kylin (along with Hadoop, HBase and Hive) available now:[sequenceiq/docker-kylin](https://github.com/sequenceiq/docker-kylin). The only thing you will need to do is to pull the container from the official Docker repository to be up and running in few minutes.
-
-Features:
-
-- Size - Light weight compared to downloading and setting up HDP or CDH sandbox.
-- Cluster support - Supports multi node installation.
-- Fully Automated - No manual steps. One command does it all
-
-For a hands-on tutorial please visit [Kylin Docker installation](https://github.com/KylinOLAP/Kylin/wiki/On-Hadoop-Kylin-installation-using-Docker).
-
-
-
-### Resources ###
+Resources
+------------
* Web Site: <http://kylin.io>
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/Frequently Asked Questions on Installation.pdf
----------------------------------------------------------------------
diff --git a/docs/Installation/Frequently Asked Questions on Installation.pdf b/docs/Installation/Frequently Asked Questions on Installation.pdf
deleted file mode 100644
index ff340ab..0000000
Binary files a/docs/Installation/Frequently Asked Questions on Installation.pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/Hadoop Environment.md
----------------------------------------------------------------------
diff --git a/docs/Installation/Hadoop Environment.md b/docs/Installation/Hadoop Environment.md
new file mode 100644
index 0000000..2e72527
--- /dev/null
+++ b/docs/Installation/Hadoop Environment.md
@@ -0,0 +1,21 @@
+## Hadoop Environment
+
+Kylin requires you having access to a hadoop CLI, where you have full permissions to hdfs, hive, hbase and map-reduce. To make things easier we strongly recommend you starting with running Kylin on a hadoop sandbox, like <http://hortonworks.com/products/hortonworks-sandbox/>. In the following tutorial we'll go with **Hortonworks Sandbox 2.1** and **Cloudera QuickStart VM 5.1**.
+
+To avoid permission issue, we suggest you using `root` account. The password for **Hortonworks Sandbox 2.1** is `hadoop` , for **Cloudera QuickStart VM 5.1** is `cloudera`.
+
+We also suggest you using bridged mode instead of NAT mode in your virtual box settings. Bridged mode will assign your sandbox an independent IP so that you can avoid issues like https://github.com/KylinOLAP/Kylin/issues/12
+
+### Start Hadoop
+
+Please make sure Hive, HDFS and HBase are available on our CLI machine.
+If you don't know how, here's a simple tutorial for hortonworks sanbox:
+
+Use ambari helps to launch hadoop:
+
+ ambari-agent start
+ ambari-server start
+
+With both command successfully run you can go to ambari homepage at <http://your_sandbox_ip:8080> (user:admin,password:admin) to check everything's status. **By default hortonworks ambari disables Hbase, you'll need manually start the `Hbase` service at ambari homepage.**
+
+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/How to generate install binaries.md
----------------------------------------------------------------------
diff --git a/docs/Installation/How to generate install binaries.md b/docs/Installation/How to generate install binaries.md
new file mode 100644
index 0000000..d7946e0
--- /dev/null
+++ b/docs/Installation/How to generate install binaries.md
@@ -0,0 +1,6 @@
+###Build from source
+> git clone https://github.com/KylinOLAP/Kylin.git
+> cd KylinOLAP/Kylin
+> sh script/package.sh
+
+In order to generate binary package, **maven** and **npm** are pre-requisites.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/Installation.md
----------------------------------------------------------------------
diff --git a/docs/Installation/Installation.md b/docs/Installation/Installation.md
new file mode 100644
index 0000000..5b33a56
--- /dev/null
+++ b/docs/Installation/Installation.md
@@ -0,0 +1,38 @@
+Installation
+------------
+
+### Prerequisites on hadoop###
+
+- Hadoop: 2.2.0.2.0.6.0-61 or above
+- Hive: 0.12.0.2.0.6.0-61 or above
+- HBase: 0.96.0.2.0.6.0-61-hadoop2
+
+_Tested with Hortonworks HDP 2.1.3 and Cloudera Quickstart VM 5.1._
+
+
+It is very easy to install Kylin for exploration/development. There are 3 installation scenarios:
+
+#### On-Hadoop-CLI installation ####
+
+If you are free to install Kylin on your hadoop CLI machine or Hadoop sandbox, this is the most convenient scenario, for it puts everything in a single machine.
+
+For a hands-on tutorial please visit [On-Hadoop-CLI installation](On Hadoop CLI installation.md).
+
+#### Dev Environment (Off-Hadoop-CLI Installation) ####
+
+This is typically for development environment setup.
+
+For a hands-on tutorial please visit [Off Hadoop CLI Installation (Dev Env Setup)](Off Hadoop CLI Installation.md).
+#### Docker Container ####
+With help from [SequenceIQ](http://sequenceiq.com/), there's docker container for Kylin (along with Hadoop, HBase and Hive) available now:[sequenceiq/docker-kylin](https://github.com/sequenceiq/docker-kylin). The only thing you will need to do is to pull the container from the official Docker repository to be up and running in few minutes.
+
+Features:
+
+- Size - Light weight compared to downloading and setting up HDP or CDH sandbox.
+- Cluster support - Supports multi node installation.
+- Fully Automated - No manual steps. One command does it all
+
+For a hands-on tutorial please visit [Kylin Docker installation](https://github.com/KylinOLAP/Kylin/wiki/On-Hadoop-Kylin-installation-using-Docker).
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/Multiple Kylin REST servers.md
----------------------------------------------------------------------
diff --git a/docs/Installation/Multiple Kylin REST servers.md b/docs/Installation/Multiple Kylin REST servers.md
new file mode 100644
index 0000000..2ee6832
--- /dev/null
+++ b/docs/Installation/Multiple Kylin REST servers.md
@@ -0,0 +1,9 @@
+### Multiple Kylin REST servers
+
+If you are running Kylin in a cluster or you have multiple Kylin REST server instances, please make sure you have the following property correctly configured in ${KYLIN_HOME}/conf/kylin.properties
+
+1. kylin.rest.servers
+ List of web servers in use, this enables one web server instance to sync up with other servers.
+
+2. kylin.server.mode
+ Make sure there is only one instance whose "kylin.server.mode" is set to "all" if there are multiple instances.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/Off Hadoop CLI Installation (Dev Env Setup).md
----------------------------------------------------------------------
diff --git a/docs/Installation/Off Hadoop CLI Installation (Dev Env Setup).md b/docs/Installation/Off Hadoop CLI Installation (Dev Env Setup).md
deleted file mode 100644
index ef7b205..0000000
--- a/docs/Installation/Off Hadoop CLI Installation (Dev Env Setup).md
+++ /dev/null
@@ -1,76 +0,0 @@
-Off Hadoop CLI Installation (Dev Env Setup)
-===
-Off-Hadoop-CLI installation is usually for **development use**.
-
-Developers want to run kylin test cases or applications at their development machine. The scenario is depicted at https://github.com/KylinOLAP/Kylin#off-hadoop-cli-installation.
-
-
-By following this tutorial, you will be able to build kylin test cubes by running a specific test case, and you can further run other test cases against the cubes having been built.
-
-
-## Environment on the Hadoop CLI
-
-Off-Hadoop-CLI installation requires you having a hadoop CLI machine (or a hadoop sandbox) as well as your local develop machine. To make things easier we strongly recommend you starting with running Kylin on a hadoop sandbox, like <http://hortonworks.com/products/hortonworks-sandbox/>. In the following tutorial we'll go with **Hortonworks Sandbox 2.1**.
-
-### Start Hadoop
-
-In Hortonworks sandbox, ambari helps to launch hadoop:
-
- ambari-agent start
- ambari-server start
-
-With both command successfully run you can go to ambari home page at <http://yoursandboxip:8080> (user:admin,password:admin) to check everything's status. By default ambari disables Hbase, you'll need manually start the `Hbase` service.
-
-For other hadoop distribution, basically start the hadoop cluster, make sure HDFS, YARN, Hive, HBase are running.
-
-
-## Environment on the dev machine
-
-### Install maven
-
-The latest maven can be found at <http://maven.apache.org/download.cgi>, we create a symbolic so that `mvn` can be run anywhere.
-
- cd ~
- wget http://apache.proserve.nl/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz
- tar -xzvf apache-maven-3.2.3-bin.tar.gz
- ln -s /root/apache-maven-3.2.3/bin/mvn /usr/bin/mvn
-
-### Compile
-
-First clone the Kylin project to your local:
-
- git clone https://github.com/KylinOLAP/Kylin.git
-
-Install Kylin artifacts to the maven repo
-
- mvn clean install -DskipTests
-
-### Modify local configuration
-
-Local configuration must be modified to point to your hadoop sandbox (or CLI) machine. If you are using a Hortonworks sandbox, this section may be skipped.
-
-* In **examples/test_case_data/sandbox/kylin.properties**
- * Find `sandbox` and replace with your hadoop hosts
- * Find `kylin.job.remote.cli.username` and `kylin.job.remote.cli.password`, fill in the user name and password used to login hadoop cluster for hadoop command execution
-
-* In **examples/test_case_data/sandbox**
- * For each configuration xml file, find all occurrence of `sandbox` and replace with your hadoop hosts
-
-An alternative to the host replacement is updating your `hosts` file to resolve `sandbox` and `sandbox.hortonworks.com` to the IP of your sandbox machine.
-
-### Run unit tests
-
-Run a end-to-end cube building test
-
- mvn test -Dtest=org.apache.kylin.job.BuildCubeWithEngineTest -DfailIfNoTests=false
-
-Run other tests, the end-to-end cube building test is exclueded
-
- mvn test
-
-### Launch Kylin Web Server
-
-In your Eclipse IDE, launch `org.apache.kylin.rest.DebugTomcat` with specifying VM arguments "-Dspring.profiles.active=sandbox". (By default Kylin server will listen on 7070 port; If you want to use another port, please specify it as a parameter when run `DebugTomcat)
-
-Check Kylin Web available at http://localhost:7070 (user:ADMIN,password:KYLIN)
-
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/Off Hadoop CLI Installation (Dev Env Setup).pdf
----------------------------------------------------------------------
diff --git a/docs/Installation/Off Hadoop CLI Installation (Dev Env Setup).pdf b/docs/Installation/Off Hadoop CLI Installation (Dev Env Setup).pdf
deleted file mode 100644
index 1cdf8da..0000000
Binary files a/docs/Installation/Off Hadoop CLI Installation (Dev Env Setup).pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/Off Hadoop CLI Installation.md
----------------------------------------------------------------------
diff --git a/docs/Installation/Off Hadoop CLI Installation.md b/docs/Installation/Off Hadoop CLI Installation.md
new file mode 100644
index 0000000..d63bf13
--- /dev/null
+++ b/docs/Installation/Off Hadoop CLI Installation.md
@@ -0,0 +1,67 @@
+Off Hadoop CLI Installation (Dev Env Setup)
+===
+Off-Hadoop-CLI installation is usually for **development use**.
+
+Developers want to run kylin test cases or applications at their development machine. The scenario is depicted as:
+
+
+
+The "Applications" here can be any unit cases running on your dev machine's IDE. By following this tutorial, you will be able to build kylin test cubes by running a specific test case, and you can further run other test cases against the cubes having been built.
+
+
+## Environment on the Hadoop CLI
+
+### Environment
+
+Kylin Rquires a properly setup hadoop environment to run. Please take a look [this](Hadoop Environment.md).
+
+## Environment on the dev machine
+
+### Install maven
+
+The latest maven can be found at <http://maven.apache.org/download.cgi>, we create a symbolic so that `mvn` can be run anywhere.
+
+ cd ~
+ wget http://apache.proserve.nl/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz
+ tar -xzvf apache-maven-3.2.3-bin.tar.gz
+ ln -s /root/apache-maven-3.2.3/bin/mvn /usr/bin/mvn
+
+### Compile
+
+First clone the Kylin project to your local:
+
+ git clone https://github.com/KylinOLAP/Kylin.git
+
+Install Kylin artifacts to the maven repo
+
+ mvn clean install -DskipTests
+
+### Modify local configuration
+
+Local configuration must be modified to point to your hadoop sandbox (or CLI) machine. If you are using a Hortonworks sandbox, this section may be skipped.
+
+* In **examples/test_case_data/sandbox/kylin.properties**
+ * Find `sandbox` and replace with your hadoop hosts
+ * Find `kylin.job.remote.cli.username` and `kylin.job.remote.cli.password`, fill in the user name and password used to login hadoop cluster for hadoop command execution
+
+* In **examples/test_case_data/sandbox**
+ * For each configuration xml file, find all occurrence of `sandbox` and replace with your hadoop hosts
+
+An alternative to the host replacement is updating your `hosts` file to resolve `sandbox` and `sandbox.hortonworks.com` to the IP of your sandbox machine.
+
+### Run unit tests
+
+Run a end-to-end cube building test
+
+ mvn test -Dtest=org.apache.kylin.job.BuildCubeWithEngineTest -DfailIfNoTests=false
+
+Run other tests, the end-to-end cube building test is exclueded
+
+ mvn test
+
+### Launch Kylin Web Server
+
+In your Eclipse IDE, launch `org.apache.kylin.rest.DebugTomcat` with specifying VM arguments "-Dspring.profiles.active=sandbox". (By default Kylin server will listen on 7070 port; If you want to use another port, please specify it as a parameter when run `DebugTomcat)
+
+Check Kylin Web available at http://localhost:7070 (user:ADMIN,password:KYLIN)
+
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/On Hadoop CLI installation.md
----------------------------------------------------------------------
diff --git a/docs/Installation/On Hadoop CLI installation.md b/docs/Installation/On Hadoop CLI installation.md
index a718722..db2409b 100644
--- a/docs/Installation/On Hadoop CLI installation.md
+++ b/docs/Installation/On Hadoop CLI installation.md
@@ -1,102 +1,30 @@
On Hadoop CLI installation
===
-On-Hadoop-CLI installation is for demo use, or for those who want to host their own web site to provide Kylin service. The scenario is depicted at https://github.com/KylinOLAP/Kylin#on-hadoop-cli-installation.
+On-Hadoop-CLI installation is the most common way of installing Kylin. It can be used for demo use, or for those who want to host their own web site to provide Kylin service. The scenario is depicted as:
-Except for some prerequisite software installations, the core of Kylin installation is accomplished by running a single script. After running the script, you will be able to build sample cube and query the tables behind the cubes via a unified web interface.
-
-## Environment
-
-Running On-Hadoop-CLI installation requires you having access to a hadoop CLI, where you have full permissions to hdfs, hive, hbase and map-reduce. To make things easier we strongly recommend you starting with running Kylin on a hadoop sandbox, like <http://hortonworks.com/products/hortonworks-sandbox/>. In the following tutorial we'll go with **Hortonworks Sandbox 2.1** and **Cloudera QuickStart VM 5.1**.
-
-To avoid permission issue, we suggest you using `root` account. The password for **Hortonworks Sandbox 2.1** is `hadoop` , for **Cloudera QuickStart VM 5.1** is `cloudera`.
-
-We also suggest you using bridged mode instead of NAT mode in your virtual box settings. Bridged mode will assign your sandbox an independent IP so that you can avoid issues like https://github.com/KylinOLAP/Kylin/issues/12
-
-### Start Hadoop
-
-For Hortonworks, ambari helps to launch hadoop:
-
- ambari-agent start
- ambari-server start
-
-With both command successfully run you can go to ambari homepage at <http://your_sandbox_ip:8080> (user:admin,password:admin) to check everything's status. **By default hortonworks ambari disables Hbase, you'll need manually start the `Hbase` service at ambari homepage.**
-
-
-
-For Cloudera, you can skip this step since they're by default activated.
-
-### Install tomcat
-
-The latest tomcat can be found at <http://tomcat.apache.org/download-70.cgi>, we need the variable `CATALINA_HOME` exported.
+
- cd ~
- wget http://apache.cs.uu.nl/dist/tomcat/tomcat-7/v7.0.57/bin/apache-tomcat-7.0.57.tar.gz
- tar -xzvf apache-tomcat-7.0.57.tar.gz
- export CATALINA_HOME=~/apache-tomcat-7.0.57
+For normal use cases, the application in the above picture means Kylin Web, which contains a web interface for cube building, querying and all sorts of management. Kylin Web launches a query engine for querying and a cube build engine for building cubes. These two engines interact with the components in Hadoop CLI, like hive and hbase.
+Except for some prerequisite software installations, the core of Kylin installation is accomplished by running a single script. After running the script, you will be able to build sample cube and query the tables behind the cubes via a unified web interface.
-### Install maven:
-
-For Cloudera, you can skip this step since maven is by default installed.
-
-The latest maven can be found at <http://maven.apache.org/download.cgi>, we create a symbolic so that `mvn` can be run anywhere.
-
- cd ~
- wget http://apache.proserve.nl/maven/maven-3/3.2.3/binaries/apache-maven-3.2.3-bin.tar.gz
- tar -xzvf apache-maven-3.2.3-bin.tar.gz
- ln -s ~/apache-maven-3.2.3/bin/mvn /usr/bin/mvn
-
-
-### Install npm
-
-Npm comes with latest versions of nodejs, just donwload nodejs from <http://nodejs.org/download/>, `npm` is located at the `/bin` folder of `nodejs`, we append the folder to `PATH` so that `npm` can be run anywhere.
-
- cd ~
- wget http://nodejs.org/dist/v0.10.32/node-v0.10.32-linux-x64.tar.gz
- tar -xzvf node-v0.10.32-linux-x64.tar.gz
- export PATH=~/node-v0.10.32-linux-x64/bin:$PATH
-
-
-## Build cubes & Query tables
-
-First clone the Kylin project to your local:
-
- git clone https://github.com/KylinOLAP/Kylin.git
-
-Go into the folder and run deploy script:
-
- cd Kylin/
- ./deploy.sh
-
-If you meet any problems, please check [FAQ](https://github.com/KylinOLAP/Kylin/wiki/Frequently-Asked-Questions-on-Installation) first.
-This script will help to:
-
-1. Check your environment
-2. Build Kylin artifacts with Maven
-3. Generate some sample tables
-3. Create empty cubes for these tables
-4. Lauch a one-stop website to build cubes & query tables
+### Environment
-After successfully running the script, please navigate to <http://your_sandbox_ip:7070> to build your cube and query it. The username/password is ADMIN/KYLIN
+Kylin Rquires a properly setup hadoop environment to run. Please take a look [this](Hadoop Environment.md).
-### Build Cube in Cubes Tab
+### Install Kylin
-
-### Check Building Progress in Job Tab
-
-### Query Tables in Query Tab
-
+1. Download latest Kylin binaries at http://kylin.incubator.apache.org/download/
+2. export KYLIN_HOME pointing to the extracted Kylin folder
+3. Make sure the user has the privilege to run hadoop, hive and hbase cmd in shell. If you are not so sure, you can just run **bin/check-env.sh**, it will print out the detail information if you have some environment issues.
+4. To start Kylin, simply run **bin/kylin.sh start**
+5. To stop Kylin, simply run **bin/kylin.sh stop**
-## What if I closed my VM?
+> If you want to have multiple Kylin instances please refer to [this](Multiple Kylin REST servers.md)
-If you shut down the VM and restarted it, Kylin will not automatically start. Depending on whether you succeed in running the deploy.sh, you should:
+After Kylin started you can visit <http://your_sandbox_ip:7070/kylin>. The username/password is ADMIN/KYLIN. It's a clean Kylin homepage with nothing in there. To start with you can:
-### If running deploy.sh failed last time
-1. Kill it if any tomcat instance exist
-2. Start over again
+1. [Quick play with a sample cube](../Tutorial/Quick play with a sample cube.md)
+2. [Create and Build your own cube](../Tutorial/Kylin Cube Creation Tutorial.md)
-### If running deploy.sh succeeded last time
-1. Kill it if any tomcat instance exist
-2. Make sure Hbase is running
-3. run `export CATALINA_HOME=~/apache-tomcat-7.0.56`
-4. run `sudo -i "${CATALINA_HOME}/bin/startup.sh"`
\ No newline at end of file
+Here's also a overview introduction on the website [Kylin Web Tutorial](../Tutorial/Kylin Web Tutorial.md)
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/On Hadoop CLI installation.pdf
----------------------------------------------------------------------
diff --git a/docs/Installation/On Hadoop CLI installation.pdf b/docs/Installation/On Hadoop CLI installation.pdf
deleted file mode 100644
index 636555a..0000000
Binary files a/docs/Installation/On Hadoop CLI installation.pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/On Hadoop Kylin installation using Docker.md
----------------------------------------------------------------------
diff --git a/docs/Installation/On Hadoop Kylin installation using Docker.md b/docs/Installation/On Hadoop Kylin installation using Docker.md
index 81b2d7f..694b738 100644
--- a/docs/Installation/On Hadoop Kylin installation using Docker.md
+++ b/docs/Installation/On Hadoop Kylin installation using Docker.md
@@ -36,5 +36,5 @@ Use `admin/admin` to login. Make sure HBase is running.
#Kylin Dashboard
http://<container_ip>:7070
```
-The default credentials to login are: `ADMIN:KYLIN`. The cluster is pre-populated with sample data and is ready to build cubes as shown [here](https://github.com/KylinOLAP/Kylin/wiki/Kylin-Cube-Creation-Tutorial).
+The default credentials to login are: `ADMIN:KYLIN`. The cluster is pre-populated with sample data and is ready to build cubes as shown [here](../Tutorial/Kylin Cube Build and Job Monitoring Tutorial.md).
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/On Hadoop Kylin installation using Docker.pdf
----------------------------------------------------------------------
diff --git a/docs/Installation/On Hadoop Kylin installation using Docker.pdf b/docs/Installation/On Hadoop Kylin installation using Docker.pdf
deleted file mode 100644
index a30fe15..0000000
Binary files a/docs/Installation/On Hadoop Kylin installation using Docker.pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/install_and_run.md
----------------------------------------------------------------------
diff --git a/docs/Installation/install_and_run.md b/docs/Installation/install_and_run.md
deleted file mode 100644
index 16179fa..0000000
--- a/docs/Installation/install_and_run.md
+++ /dev/null
@@ -1,83 +0,0 @@
-##Install and Run
-
-###How to run
-
-1. Download the release version(according to the hadoop distribution)
-2. Setup a KYLIN_HOME pointing to the corresponding directory where you extract the release tar
-3. Make sure the user has the privilege to run hadoop, hive and hbase cmd in shell. If you are not so sure, you can just run **bin/check-env.sh**, it will print out the detail information if you have some environment issues.
-4. To start Kylin, simply run **bin/kylin.sh start**
-5. To stop Kylin, simply run **bin/kylin.sh stop**
-
-
-If you are running Kylin in a cluster or you have multiple Kylin instances, please make sure you have the following property correctly configured.
-
-1. kylin.rest.servers
-
- List of web servers in use, this enables one web server instance to sync up with other servers.
-
-
-2. kylin.server.mode
-
- Make sure there is only one instance whose "kylin.server.mode" is set to "all" if there are multiple instances.
-
-
-###The directory structure of the installation
-
-> ├── bin
-> │ ├── check-env.sh
-> │ ├── find-hive-dependency.sh
-> │ ├── health-check.sh
-> │ ├── kylin.sh
-> │ └── sample.sh
-> │
-> ├── conf
-> │ ├── kylin_job_conf.xml
-> │ └── kylin.properties
-> │
-> ├── sample_cube
-> │ ├── data
-> │ └── metadata
-> │
-> ├── tomcat
-> │ ├── webapps
-> │ │ └── kylin.war
-> │ …….
-> │
-> └── lib
-> ├── kylin-coprocessor-${version}.jar
-> ├── kylin-job-${version}.jar
-> └── kylin-jdbc-${version}.jar
-
-###Create sample cube
-Kylin provides a script for you to create a sample Cube; the script will also create three sample hive tables:
-
-1. Run ${KYLIN_HOME}/bin/sample.sh
-2. Logon Kylin web, select project "learn_kylin";
-3. Select the sample cube "kylin_sales_cube", click "Actions" -> "Build", pick up a date later than 2014-01-01 (to cover all 10000 sample records);
-4. Check the build progress in "Jobs" tab, until 100%;
-5. Execute SQLs in the "Query" tab, for example:
- select cal_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by cal_dt order by cal_dt
-6. You can verify the query result and compare the response time with hive;
-
-###Build from source
-> git clone https://github.com/KylinOLAP/Kylin.git
-> cd KylinOLAP/Kylin
-> sh script/package.sh
-
-In order to generate binary package, **maven** and **npm** are pre-requisites.
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/off_cli_install_scene.png
----------------------------------------------------------------------
diff --git a/docs/Installation/off_cli_install_scene.png b/docs/Installation/off_cli_install_scene.png
new file mode 100644
index 0000000..26e9e4f
Binary files /dev/null and b/docs/Installation/off_cli_install_scene.png differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Installation/on_cli_install_scene.png
----------------------------------------------------------------------
diff --git a/docs/Installation/on_cli_install_scene.png b/docs/Installation/on_cli_install_scene.png
new file mode 100644
index 0000000..5b07d25
Binary files /dev/null and b/docs/Installation/on_cli_install_scene.png differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/README.md
----------------------------------------------------------------------
diff --git a/docs/README.md b/docs/README.md
deleted file mode 100644
index e69de29..0000000
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin Cube Build and Job Monitoring Tutorial.pdf
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin Cube Build and Job Monitoring Tutorial.pdf b/docs/Tutorial/Kylin Cube Build and Job Monitoring Tutorial.pdf
deleted file mode 100644
index b4253d4..0000000
Binary files a/docs/Tutorial/Kylin Cube Build and Job Monitoring Tutorial.pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin Cube Creation Tutorial.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin Cube Creation Tutorial.md b/docs/Tutorial/Kylin Cube Creation Tutorial.md
index cc85075..294f665 100644
--- a/docs/Tutorial/Kylin Cube Creation Tutorial.md
+++ b/docs/Tutorial/Kylin Cube Creation Tutorial.md
@@ -119,4 +119,11 @@ Choose partition type, partition column and start date.
You can overview your cube and go back to previous step to modify it. Click the `Save` button to complete the cube creation.
-
\ No newline at end of file
+
+
+### IV. What's next
+
+After cube being created, you might need to:
+
+1. [Build the cube so that it can be queried](Kylin Cube Build and Job Monitoring Tutorial.md)
+2. [Grant permission to cubes](Kylin Cube Permission Grant Tutorial.md)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin Cube Creation Tutorial.pdf
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin Cube Creation Tutorial.pdf b/docs/Tutorial/Kylin Cube Creation Tutorial.pdf
deleted file mode 100644
index cd6e934..0000000
Binary files a/docs/Tutorial/Kylin Cube Creation Tutorial.pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin Cube Permission Grant Tutorial.pdf
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin Cube Permission Grant Tutorial.pdf b/docs/Tutorial/Kylin Cube Permission Grant Tutorial.pdf
deleted file mode 100644
index adfb6d7..0000000
Binary files a/docs/Tutorial/Kylin Cube Permission Grant Tutorial.pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin ODBC Driver Tutorial.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin ODBC Driver Tutorial.md b/docs/Tutorial/Kylin ODBC Driver Tutorial.md
index 43439fc..7f84213 100644
--- a/docs/Tutorial/Kylin ODBC Driver Tutorial.md
+++ b/docs/Tutorial/Kylin ODBC Driver Tutorial.md
@@ -26,5 +26,12 @@ Kylin ODBC Driver Tutorial
3. Both drivers already be installed on Tableau Server, you properly should be able to publish to there without issues
+## Use ODBC compatible applications
+
+1. [Kylin and Tableau Tutorial](Kylin and Tableau Tutorial.md)
+2. [Kylin and Excel Tutorial](Kylin and Excel Tutorial.md)
+
+
+
## Bug Report
Open github issue here
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin ODBC Driver Tutorial.pdf
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin ODBC Driver Tutorial.pdf b/docs/Tutorial/Kylin ODBC Driver Tutorial.pdf
deleted file mode 100644
index e2d4986..0000000
Binary files a/docs/Tutorial/Kylin ODBC Driver Tutorial.pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin Web Tutorial.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin Web Tutorial.md b/docs/Tutorial/Kylin Web Tutorial.md
index 5328370..af49d9b 100644
--- a/docs/Tutorial/Kylin Web Tutorial.md
+++ b/docs/Tutorial/Kylin Web Tutorial.md
@@ -8,7 +8,7 @@ Kylin Web Tutorial
> Mac: Google Chrome, FireFox, Safari
## 1. Access & Login
-Host to access: http://your_sandbox_ip:9080
+Host to access: http://your_sandbox_ip:7070/kylin
Login with username/password: ADMIN/KYLIN
@@ -19,6 +19,9 @@ Although Kylin will using SQL as query interface and leverage Hive metadata, kyl
## 3. Kylin OLAP Cube
+
+> To make cubes availabe you'll have to create them first [Kylin Cube Creation Tutorial](Kylin Cube Creation Tutorial.md)
+
Kylin's OLAP Cubes are pre-calculation datasets from Star Schema Hive tables, Here's the web management interface for user to explorer, manage all cubes.Go to `Cubes` Menu, it will list all cubes available in system:
@@ -42,6 +45,9 @@ To explore more detail about the Cube
## 4. Write and Execute SQL on web
+
+>> To make queries availabe you'll have to create are build cubes first [Kylin Cube Creation Tutorial](Kylin Cube Creation Tutorial.md) and [Kylin Cube Build and Job Monitoring Tutorial](Kylin Cube Build and Job Monitoring Tutorial.md)
+
Kylin's web offer a simple query tool for user to run SQL to explorer existing cube, verify result and explorer the result set using #5's Pivot analysis and visualization
> **Query Limit**
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin Web Tutorial.pdf
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin Web Tutorial.pdf b/docs/Tutorial/Kylin Web Tutorial.pdf
deleted file mode 100644
index 6f27a4b..0000000
Binary files a/docs/Tutorial/Kylin Web Tutorial.pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin and Excel Tutorial.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin and Excel Tutorial.md b/docs/Tutorial/Kylin and Excel Tutorial.md
new file mode 100644
index 0000000..49961e0
--- /dev/null
+++ b/docs/Tutorial/Kylin and Excel Tutorial.md
@@ -0,0 +1,3 @@
+Kylin and Excel Tutorial
+===
+For now our ODBC driver has not yet finished supporting Excel, watch the mail list for updates.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Kylin and Tableau Tutorial.pdf
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin and Tableau Tutorial.pdf b/docs/Tutorial/Kylin and Tableau Tutorial.pdf
deleted file mode 100644
index 14edbe2..0000000
Binary files a/docs/Tutorial/Kylin and Tableau Tutorial.pdf and /dev/null differ
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/docs/Tutorial/Quick play with a sample cube.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Quick play with a sample cube.md b/docs/Tutorial/Quick play with a sample cube.md
new file mode 100644
index 0000000..cd767b3
--- /dev/null
+++ b/docs/Tutorial/Quick play with a sample cube.md
@@ -0,0 +1,10 @@
+### Quick start with sample cube
+Kylin provides a script for you to create a sample Cube; the script will also create three sample hive tables:
+
+1. Run ${KYLIN_HOME}/bin/sample.sh
+2. Logon Kylin web, select project "learn_kylin";
+3. Select the sample cube "kylin_sales_cube", click "Actions" -> "Build", pick up a date later than 2014-01-01 (to cover all 10000 sample records);
+4. Check the build progress in "Jobs" tab, until 100%;
+5. Execute SQLs in the "Query" tab, for example:
+ select cal_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by cal_dt order by cal_dt
+6. You can verify the query result and compare the response time with hive;
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/592e96b5/old_README.md
----------------------------------------------------------------------
diff --git a/old_README.md b/old_README.md
new file mode 100644
index 0000000..44f6859
--- /dev/null
+++ b/old_README.md
@@ -0,0 +1,94 @@
+Kylin OLAP
+===========
+
+
+Overview
+------------
+> Extreme OLAP Engine for Big Data
+
+An open source distributed Analytical Engine from eBay to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop to support TB to PB size analysis.
+
+
+Getting Started
+---------------
+
+### Prerequisites ###
+
+#### Hadoop ####
+- Hadoop: 2.2.0.2.0.6.0-61 or above
+- Hive: 0.12.0.2.0.6.0-61 or above
+- HBase: 0.96.0.2.0.6.0-61-hadoop2
+
+
+_Tested with Hortonworks HDP 2.1.3 and Cloudera Quickstart VM 5.1._
+
+#### Misc ####
+- Command hadoop, hive, hbase is workable on your hadoop cluster
+- JDK Runtime: JDK7 (OpenJDK or Oracle JDK)
+- Maven
+- Git
+- Tomcat (CATALINA_HOME being set)
+- Npm
+
+#### Before Install ####
+- For the quick tutorial, we assume that your hadoop has priviledges disabled which means any user could run hadoop and hive command
+
+
+Installation
+------------
+It is very easy to install Kylin for exploration/development. There are 3 supported options:
+
+
+1. Sandbox (HDP or CDH sandbox)
+2. Dev Environment (IDE + Sandbox)
+3. Docker container (Docker)
+
+#### Sandbox (On-Hadoop-CLI installation) ####
+
+If you are free to install Kylin on your hadoop CLI machine or Hadoop sandbox, this is the most convenient scenario, for it puts everything in a single machine.
+
+
+
+
+
+For normal users, the application in the above picture means `Kylin Web`, which contains a web interface for cube building, querying and all sorts of management. Kylin Web launches a query engine for querying and a cube build engine for building cubes. These two engines interact with the components in Hadoop CLI, like hive and hbase.
+
+For a hands-on tutorial please visit [On-Hadoop-CLI installation](https://github.com/KylinOLAP/Kylin/wiki/On-Hadoop-CLI-installation).
+
+#### Dev Environment (Off-Hadoop-CLI Installation) ####
+
+This is typically for development environment setup.
+
+Applications does not necessarily mean Kylin Web, it can also be any test cases or java applications running at your local IDE(query engine and cube build engine can be launched programmatically in your code). In this case, it is no longer reasonable to assume that all the Kylin components reside in the same machine as your Hadoop CLI. Fortunately, Kylin still works under such condition with proper settings on your CLI machine.
+
+
+
+For a hands-on tutorial please visit [Off-Hadoop-CLI installation](https://github.com/KylinOLAP/Kylin/wiki/Off-Hadoop-CLI-Installation-(Dev-Env-Setup))
+
+#### Docker Container ####
+With help from [SequenceIQ](http://sequenceiq.com/), there's docker container for Kylin (along with Hadoop, HBase and Hive) available now:[sequenceiq/docker-kylin](https://github.com/sequenceiq/docker-kylin). The only thing you will need to do is to pull the container from the official Docker repository to be up and running in few minutes.
+
+Features:
+
+- Size - Light weight compared to downloading and setting up HDP or CDH sandbox.
+- Cluster support - Supports multi node installation.
+- Fully Automated - No manual steps. One command does it all
+
+For a hands-on tutorial please visit [Kylin Docker installation](https://github.com/KylinOLAP/Kylin/wiki/On-Hadoop-Kylin-installation-using-Docker).
+
+
+
+### Resources ###
+
+* Web Site: <http://kylin.io>
+
+* Google Group: [Kylin OLAP Group](https://groups.google.com/forum/#!forum/kylin-olap)
+
+* Developer Mail: <ky...@googlegroups.com>
+
+* How To Contribute: See [wiki](https://github.com/KylinOLAP/Kylin/wiki/How-to-Contribute)
+
+* Presentation: [Kylin Hadoop OLAP Engine v1.0](https://github.com/KylinOLAP/Kylin/blob/master/docs/Kylin_Hadoop_OLAP_Engine_v1.0.pdf?raw=true)
+
+* Apache Proposal: [Apache Kylin](https://wiki.apache.org/incubator/KylinProposal)
+
[21/29] incubator-kylin git commit: KYLIN-650 category resting pages
to different topics
Posted by li...@apache.org.
KYLIN-650 category resting pages to different topics
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/5acdcd23
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/5acdcd23
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/5acdcd23
Branch: refs/heads/staging
Commit: 5acdcd23098dda31338ebdb687bc7fc86076b160
Parents: 284ef26
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 23:23:11 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 23:23:11 2015 -0700
----------------------------------------------------------------------
README.md | 15 +
docs/REST/Build Cube with Restful API.md | 51 +
...w to consume Kylin REST API in javascript.md | 42 +
docs/REST/Kylin Restful API List.md | 2020 ++++++++++++++++++
.../TechInside/New meta data model structure.md | 18 +
...n Kylin test case with HBase Mini Cluster.md | 15 +
.../How to use kylin remote jdbc driver.md | 83 +
7 files changed, 2244 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/5acdcd23/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index fa180b0..f3d383f 100644
--- a/README.md
+++ b/README.md
@@ -19,8 +19,19 @@ Please follow this installation tutorial to start with Kylin: [Installation Tuto
Advance Topics
-------
+####Connectivity
+1.[How to use kylin remote jdbc driver](docs/Tutorial/How to use kylin remote jdbc driver.md)
+
+2.[Kylin ODBC Driver Tutorial](docs/Tutorial/Kylin ODBC Driver Tutorial.md)
+
####REST
+1.[Kylin Restful API List](docs/REST/Kylin Restful API List.md)
+
+2.[Build Cube with Restful API](docs/REST/Build Cube with Restful API.md)
+
+3.[How to consume Kylin REST API in javascript](docs/REST/How to consume Kylin REST API in javascript.md)
+
####Operations
1.[Kylin Metadata Store](docs/Operations/Kylin Metadata Store.md)
@@ -30,8 +41,12 @@ Advance Topics
####Test
+1.[Run Kylin test case with HBase Mini Cluster](docs/Test/Run Kylin test case with HBase Mini Cluster.md)
+
+
####Technial Details
+1.[New meta data model structure](docs/TechInside/New meta data model structure.md)
Get Help
------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/5acdcd23/docs/REST/Build Cube with Restful API.md
----------------------------------------------------------------------
diff --git a/docs/REST/Build Cube with Restful API.md b/docs/REST/Build Cube with Restful API.md
new file mode 100644
index 0000000..4eb5381
--- /dev/null
+++ b/docs/REST/Build Cube with Restful API.md
@@ -0,0 +1,51 @@
+
+### 1. Authentication
+* Currently, Kylin uses [basic authentication](http://en.wikipedia.org/wiki/Basic_access_authentication).
+* Add `Authorization` header to first request for authentication
+* Or you can do a specific request by `POST http://localhost:7070/kylin/api/user/authentication`
+* Once authenticated, client can go subsequent requests with cookies.
+* Example
+```
+ POST http://localhost:7070/kylin/api/user/authentication
+
+ Authorization:Basic xxxxJD124xxxGFxxxSDF
+ Content-Type: application/json;charset=UTF-8
+```
+
+### 2. Get details of cube.
+* `GET http://localhost:7070/kylin/api/cubes?cubeName={cube_name}&limit=15&offset=0`
+* Client can find cube segment date ranges in returned cube detail.
+* Example
+```
+ GET http://localhost:7070/kylin/api/cubes?cubeName=test_kylin_cube_with_slr&limit=15&offset=0
+
+ Authorization:Basic xxxxJD124xxxGFxxxSDF
+ Content-Type: application/json;charset=UTF-8
+```
+
+### 3. Then submit a build job of the cube.
+* `PUT http://localhost:7070/kylin/api/cubes/{cube_name}/rebuild`
+* For put request body detail please refer to [service doc](https://github.com/KylinOLAP/Kylin/wiki/Restful-Service-Doc).
+ * `startTime` and `endTime` should be utc timestamp.
+ * `buildType` can be `BUILD` or `MERGE`. `BUILD` is for building a new segment or refreshing an existing segment. `MERGE` is for merging multiple existing segments into one bigger segment.
+* This method will return a newly created job instance, in which the uuid is the identity of job to track job status.
+* Example
+```
+ PUT http://localhost:7070/kylin/api/cubes/test_kylin_cube_with_slr/rebuild
+
+ Authorization:Basic xxxxJD124xxxGFxxxSDF
+ Content-Type: application/json;charset=UTF-8
+
+ {
+ "startTime": 0,
+ "endTime": 1388563200000,
+ "buildType": "BUILD"
+ }
+```
+
+### 4. Track job status.
+* `GET http://localhost:7070/kylin/api/jobs/{job_uuid}`
+* Returned `job_status` represents current status of job.
+
+### 5. If the job failed, you can resume the job.
+* `PUT http://localhost:7070/kylin/api/jobs/{job_uuid}/resume`
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/5acdcd23/docs/REST/How to consume Kylin REST API in javascript.md
----------------------------------------------------------------------
diff --git a/docs/REST/How to consume Kylin REST API in javascript.md b/docs/REST/How to consume Kylin REST API in javascript.md
new file mode 100644
index 0000000..a0ea1b6
--- /dev/null
+++ b/docs/REST/How to consume Kylin REST API in javascript.md
@@ -0,0 +1,42 @@
+Kylin securicty authorization is based on basic access authorization, so when you want to use API in your javascript,you need to involve the authorization info in http headers.
+
+
+## Here is a demo show how to use Kylin Query API.
+```
+$.ajaxSetup({
+ headers: { 'Authorization': "Basic eWFu**********X***ZA==", 'Content-Type': 'application/json;charset=utf-8' } // use your own authorization code here
+ });
+ var request = $.ajax({
+ url: "http://hostname/kylin/api/query",
+ type: "POST",
+ data: '{"sql":"select count(*) from SUMMARY;","offset":0,"limit":50000,"partial":true,"project":"test"}',
+ dataType: "json"
+ });
+ request.done(function( msg ) {
+ alert(msg);
+ });
+ request.fail(function( jqXHR, textStatus ) {
+ alert( "Request failed: " + textStatus );
+ });
+
+```
+
+
+For what is basic access authorization ,click here http://en.wikipedia.org/wiki/Basic_access_authentication
+
+## Keypoint:
+1. add basic access authorization info in heders.
+2. use right ajax type and right data synax
+
+How to generate your authorization code (download and import "jquery.base64.js" from https://github.com/yckart/jquery.base64.js)
+
+```
+var authorizationCode = $.base64('encode', 'NT_USERNAME' + ":" + 'NT_PASSWORD');
+
+$.ajaxSetup({
+ headers: {
+ 'Authorization': "Basic " + authorizationCode,
+ 'Content-Type': 'application/json;charset=utf-8'
+ }
+});
+```
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/5acdcd23/docs/REST/Kylin Restful API List.md
----------------------------------------------------------------------
diff --git a/docs/REST/Kylin Restful API List.md b/docs/REST/Kylin Restful API List.md
new file mode 100644
index 0000000..f1398e9
--- /dev/null
+++ b/docs/REST/Kylin Restful API List.md
@@ -0,0 +1,2020 @@
+## Kylin Restful API List
+
+* QUERY
+ * Login
+ * Query
+ * List queryable tables
+* CUBE
+ * List cubes
+ * Rebuild cube
+ * Disable cube
+ * Purge cube
+ * Enable cube
+* JOB
+ * Resume job
+ * Discard job
+ * Get job step log
+* Metadata
+ * Get Hive Table
+ * Get Hive Table (Extend Info)
+ * Get Hive Tables
+ * Load Hive Table
+* Wipe cache
+
+***
+
+### Login
+`POST /user/authentication`
+
+for example:
+```sh
+curl -c /path/to/cookiefile.txt -X POST -H "Authorization: Basic XXXXXXXXX" -H 'Content-Type: application/json' http://<host>:<port>/kylin/api/user/authentication
+```
+
+If login successfully, the JSESSIONID will be saved into the cookie file; In the subsequent http requests, attach the cookie, for example:
+```sh
+curl -b /path/to/cookiefile.txt -X PUT -H 'Content-Type: application/json' -d '{"startTime":'1423526400000', "endTime":'1423526400', "buildType":"BUILD"}' http://<host>:<port>/kylin/api/cubes/your_cube/rebuild
+```
+
+#### Header
+Authorization data encoded by basic auth. Header sample:
+Authorization:Basic {data}
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ "userDetails":{
+ "password":null,
+ "username":"sample",
+ "authorities":[
+ {
+ "authority":"ROLE_ANALYST"
+ },
+ {
+ "authority":"ROLE_MODELER"
+ }
+ ],
+ "accountNonExpired":true,
+ "accountNonLocked":true,
+ "credentialsNonExpired":true,
+ "enabled":true
+ }
+}
+```
+
+***
+
+### Query
+`POST /query`
+#### Request Body
+* sql - `required` `string` The sql query string.
+* offset - `optional` `int` Query offset. If offset is set in sql, curIndex will be ignored.
+* limit - `optional` `int` Query limit. If limit is set in sql, perPage will be ignored.
+* acceptPartial - `optional` `bool` Flag to switch partial query.
+* project - `optional` `string` Project to perform query. Default value is 'DEFAULT'
+```sh
+{
+ "sql":"select * from TEST_KYLIN_FACT",
+ "offset":0,
+ "limit":50000,
+ "acceptPartial":true,
+ "project":"DEFAULT"
+}
+```
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ "columnMetas":[
+ {
+ "isNullable":1,
+ "displaySize":0,
+ "label":"CAL_DT",
+ "name":"CAL_DT",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":0,
+ "scale":0,
+ "columnType":91,
+ "columnTypeName":"DATE",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ },
+ {
+ "isNullable":1,
+ "displaySize":10,
+ "label":"LEAF_CATEG_ID",
+ "name":"LEAF_CATEG_ID",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":10,
+ "scale":0,
+ "columnType":4,
+ "columnTypeName":"INTEGER",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ },
+ {
+ "isNullable":1,
+ "displaySize":10,
+ "label":"LSTG_SITE_ID",
+ "name":"LSTG_SITE_ID",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":10,
+ "scale":0,
+ "columnType":4,
+ "columnTypeName":"INTEGER",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ },
+ {
+ "isNullable":1,
+ "displaySize":5,
+ "label":"SLR_SEGMENT_CD",
+ "name":"SLR_SEGMENT_CD",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":5,
+ "scale":0,
+ "columnType":5,
+ "columnTypeName":"SMALLINT",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ },
+ {
+ "isNullable":1,
+ "displaySize":256,
+ "label":"LSTG_FORMAT_NAME",
+ "name":"LSTG_FORMAT_NAME",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":256,
+ "scale":0,
+ "columnType":12,
+ "columnTypeName":"VARCHAR(256)",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ },
+ {
+ "isNullable":1,
+ "displaySize":19,
+ "label":"SELLER_ID",
+ "name":"SELLER_ID",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":19,
+ "scale":0,
+ "columnType":-5,
+ "columnTypeName":"BIGINT",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ },
+ {
+ "isNullable":1,
+ "displaySize":19,
+ "label":"PRICE",
+ "name":"PRICE",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":19,
+ "scale":4,
+ "columnType":3,
+ "columnTypeName":"DECIMAL(19, 4)",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ },
+ {
+ "isNullable":1,
+ "displaySize":19,
+ "label":"MIN_PRICE_",
+ "name":"MIN_PRICE_",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":19,
+ "scale":4,
+ "columnType":3,
+ "columnTypeName":"DECIMAL(19, 4)",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ },
+ {
+ "isNullable":1,
+ "displaySize":19,
+ "label":"MAX_PRICE_",
+ "name":"MAX_PRICE_",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":19,
+ "scale":4,
+ "columnType":3,
+ "columnTypeName":"DECIMAL(19, 4)",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ },
+ {
+ "isNullable":1,
+ "displaySize":19,
+ "label":"COUNT__",
+ "name":"COUNT__",
+ "schemaName":null,
+ "catelogName":null,
+ "tableName":null,
+ "precision":19,
+ "scale":0,
+ "columnType":-5,
+ "columnTypeName":"BIGINT",
+ "readOnly":true,
+ "writable":false,
+ "caseSensitive":true,
+ "searchable":false,
+ "currency":false,
+ "signed":true,
+ "autoIncrement":false,
+ "definitelyWritable":false
+ }
+ ],
+ "results":[
+ [
+ "2013-08-07",
+ "32996",
+ "15",
+ "15",
+ "Auction",
+ "10000000",
+ "49.048952730908745",
+ "49.048952730908745",
+ "49.048952730908745",
+ "1"
+ ],
+ [
+ "2013-08-07",
+ "43398",
+ "0",
+ "14",
+ "ABIN",
+ "10000633",
+ "85.78317064220418",
+ "85.78317064220418",
+ "85.78317064220418",
+ "1"
+ ],
+ [
+ "2013-08-07",
+ "170083",
+ "3",
+ "12",
+ "FP-non GTC",
+ "10000688",
+ "49.4518822857833",
+ "49.4518822857833",
+ "49.4518822857833",
+ "1"
+ ],
+ [
+ "2013-08-07",
+ "73506",
+ "0",
+ "14",
+ "Others",
+ "10000858",
+ "35.7556514660872",
+ "35.7556514660872",
+ "35.7556514660872",
+ "1"
+ ],
+ [
+ "2013-08-07",
+ "26262",
+ "0",
+ "5",
+ "FP-GTC",
+ "10000914",
+ "69.63202173718798",
+ "69.63202173718798",
+ "69.63202173718798",
+ "1"
+ ],
+ [
+ "2013-08-07",
+ "1161",
+ "3",
+ "12",
+ "Others",
+ "10000959",
+ "76.72392839984632",
+ "76.72392839984632",
+ "76.72392839984632",
+ "1"
+ ]
+ ],
+ "cube":"test_kylin_cube_with_slr_desc",
+ "affectedRowCount":0,
+ "isException":false,
+ "exceptionMessage":null,
+ "duration":3451,
+ "partial":false
+}
+```
+
+***
+
+### List queryable tables
+`GET /tables_and_columns`
+#### Request Parameters
+* project - `required` `string` The project to load tables
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+[
+ {
+ "columns":[
+ {
+ "table_NAME":"TEST_CAL_DT",
+ "table_SCHEM":"EDW",
+ "column_NAME":"CAL_DT",
+ "data_TYPE":91,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":1,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"DATE"
+ },
+ {
+ "table_NAME":"TEST_CAL_DT",
+ "table_SCHEM":"EDW",
+ "column_NAME":"WEEK_BEG_DT",
+ "data_TYPE":91,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":2,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"DATE"
+ }
+ ],
+ "table_NAME":"TEST_CAL_DT",
+ "table_SCHEM":"EDW",
+ "ref_GENERATION":null,
+ "self_REFERENCING_COL_NAME":null,
+ "type_SCHEM":null,
+ "table_TYPE":"TABLE",
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_CAT":null,
+ "type_NAME":null
+ },
+ {
+ "columns":[
+ {
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "column_NAME":"LEAF_CATEG_ID",
+ "data_TYPE":4,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":1,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"INTEGER"
+ },
+ {
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "column_NAME":"SITE_ID",
+ "data_TYPE":4,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":2,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"INTEGER"
+ },
+ {
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "column_NAME":"META_CATEG_NAME",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":3,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ },
+ {
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "column_NAME":"CATEG_LVL2_NAME",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":4,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ },
+ {
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "column_NAME":"CATEG_LVL3_NAME",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":5,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ },
+ {
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "column_NAME":"USER_DEFINED_FIELD1",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":6,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ },
+ {
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "column_NAME":"USER_DEFINED_FIELD3",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":7,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ },
+ {
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "column_NAME":"UPD_DATE",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":8,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ },
+ {
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "column_NAME":"UPD_USER",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":9,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ }
+ ],
+ "table_NAME":"TEST_CATEGORY_GROUPINGS",
+ "table_SCHEM":"EDW",
+ "ref_GENERATION":null,
+ "self_REFERENCING_COL_NAME":null,
+ "type_SCHEM":null,
+ "table_TYPE":"TABLE",
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_CAT":null,
+ "type_NAME":null
+ },
+ {
+ "columns":[
+ {
+ "table_NAME":"TEST_KYLIN_FACT",
+ "table_SCHEM":"EDW",
+ "column_NAME":"CAL_DT",
+ "data_TYPE":91,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":1,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"DATE"
+ },
+ {
+ "table_NAME":"TEST_KYLIN_FACT",
+ "table_SCHEM":"EDW",
+ "column_NAME":"LEAF_CATEG_ID",
+ "data_TYPE":4,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":2,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"INTEGER"
+ },
+ {
+ "table_NAME":"TEST_KYLIN_FACT",
+ "table_SCHEM":"EDW",
+ "column_NAME":"LSTG_SITE_ID",
+ "data_TYPE":4,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":3,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"INTEGER"
+ },
+ {
+ "table_NAME":"TEST_KYLIN_FACT",
+ "table_SCHEM":"EDW",
+ "column_NAME":"SLR_SEGMENT_CD",
+ "data_TYPE":5,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":4,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"SMALLINT"
+ },
+ {
+ "table_NAME":"TEST_KYLIN_FACT",
+ "table_SCHEM":"EDW",
+ "column_NAME":"LSTG_FORMAT_NAME",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":5,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ },
+ {
+ "table_NAME":"TEST_KYLIN_FACT",
+ "table_SCHEM":"EDW",
+ "column_NAME":"SELLER_ID",
+ "data_TYPE":-5,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":6,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"BIGINT"
+ },
+ {
+ "table_NAME":"TEST_KYLIN_FACT",
+ "table_SCHEM":"EDW",
+ "column_NAME":"PRICE",
+ "data_TYPE":3,
+ "nullable":1,
+ "column_SIZE":19,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":4,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":19,
+ "ordinal_POSITION":7,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"DECIMAL(19, 4)"
+ }
+ ],
+ "table_NAME":"TEST_KYLIN_FACT",
+ "table_SCHEM":"EDW",
+ "ref_GENERATION":null,
+ "self_REFERENCING_COL_NAME":null,
+ "type_SCHEM":null,
+ "table_TYPE":"TABLE",
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_CAT":null,
+ "type_NAME":null
+ },
+ {
+ "columns":[
+ {
+ "table_NAME":"TEST_SELLER_TYPE_DIM",
+ "table_SCHEM":"EDW",
+ "column_NAME":"SELLER_TYPE_CD",
+ "data_TYPE":5,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":1,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"SMALLINT"
+ },
+ {
+ "table_NAME":"TEST_SELLER_TYPE_DIM",
+ "table_SCHEM":"EDW",
+ "column_NAME":"SELLER_TYPE_DESC",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":2,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ }
+ ],
+ "table_NAME":"TEST_SELLER_TYPE_DIM",
+ "table_SCHEM":"EDW",
+ "ref_GENERATION":null,
+ "self_REFERENCING_COL_NAME":null,
+ "type_SCHEM":null,
+ "table_TYPE":"TABLE",
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_CAT":null,
+ "type_NAME":null
+ },
+ {
+ "columns":[
+ {
+ "table_NAME":"TEST_SITES",
+ "table_SCHEM":"EDW",
+ "column_NAME":"SITE_ID",
+ "data_TYPE":4,
+ "nullable":1,
+ "column_SIZE":-1,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":-1,
+ "ordinal_POSITION":1,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"INTEGER"
+ },
+ {
+ "table_NAME":"TEST_SITES",
+ "table_SCHEM":"EDW",
+ "column_NAME":"SITE_NAME",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":2,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ },
+ {
+ "table_NAME":"TEST_SITES",
+ "table_SCHEM":"EDW",
+ "column_NAME":"CRE_USER",
+ "data_TYPE":12,
+ "nullable":1,
+ "column_SIZE":256,
+ "buffer_LENGTH":-1,
+ "decimal_DIGITS":0,
+ "num_PREC_RADIX":10,
+ "column_DEF":null,
+ "sql_DATA_TYPE":-1,
+ "sql_DATETIME_SUB":-1,
+ "char_OCTET_LENGTH":256,
+ "ordinal_POSITION":3,
+ "is_NULLABLE":"YES",
+ "scope_CATLOG":null,
+ "scope_SCHEMA":null,
+ "scope_TABLE":null,
+ "source_DATA_TYPE":-1,
+ "iS_AUTOINCREMENT":null,
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_NAME":"VARCHAR(256) CHARACTER SET \"UTF-16LE\" COLLATE \"UTF-16LE$en_US$primary\""
+ }
+ ],
+ "table_NAME":"TEST_SITES",
+ "table_SCHEM":"EDW",
+ "ref_GENERATION":null,
+ "self_REFERENCING_COL_NAME":null,
+ "type_SCHEM":null,
+ "table_TYPE":"TABLE",
+ "table_CAT":"defaultCatalog",
+ "remarks":null,
+ "type_CAT":null,
+ "type_NAME":null
+ }
+]
+```
+
+
+***
+
+### List cubes
+`GET /cubes`
+#### Request Parameters
+* cubeName - `optional` `string` Cube name to find.
+* offset - `required` `int` Offset used by pagination
+* limit - `required` `int ` Cubes per page.
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+[
+ {
+ "uuid":"1eaca32a-a33e-4b69-83dd-0bb8b1f8c53b",
+ "last_modified":1407831634847,
+ "name":"test_kylin_cube_with_slr_empty",
+ "owner":null,
+ "version":null,
+ "descriptor":"test_kylin_cube_with_slr_desc",
+ "cost":50,
+ "status":"DISABLED",
+ "segments":[
+ ],
+ "create_time":null,
+ "source_records_count":0,
+ "source_records_size":0,
+ "size_kb":0
+ }
+]
+```
+
+### Rebuild cube
+`PUT /cubes/{cubeName}/rebuild`
+#### Path Variable
+* cubeName - `required` `string` Cube name.
+
+#### Request Body
+* startTime - `required` `long` Start timestamp of data to build, e.g. 1388563200000 for 2014-1-1
+* endTime - `required` `long` End timestamp of data to build
+* buildType - `required` `string` Build type: 'BUILD' OR 'MERGE'
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ "uuid":"c143e0e4-ac5f-434d-acf3-46b0d15e3dc6",
+ "last_modified":1407908916705,
+ "name":"test_kylin_cube_with_slr_empty - 19700101000000_20140731160000 - BUILD - PDT 2014-08-12 22:48:36",
+ "type":"BUILD",
+ "duration":0,
+ "related_cube":"test_kylin_cube_with_slr_empty",
+ "related_segment":"19700101000000_20140731160000",
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "mr_waiting":0,
+ "steps":[
+ {
+ "interruptCmd":null,
+ "name":"Create Intermediate Flat Hive Table",
+ "sequence_id":0,
+ "exec_cmd":"hive -e \"DROP TABLE IF EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6;\nCREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\n(\nCAL_DT date\n,LEAF_CATEG_ID int\n,LSTG_SITE_ID int\n,META_CATEG_NAME string\n,CATEG_LVL2_NAME string\n,CATEG_LVL3_NAME string\n,LSTG_FORMAT_NAME string\n,SLR_SEGMENT_CD smallint\n,SELLER_ID bigint\n,PRICE decimal\n)\nROW FORMAT DELIMITED FIELDS TERMINATED BY '\\177'\nSTORED AS SEQUENCEFILE\nLOCATION '/tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6';\nSET mapreduce.job.split.metainfo.maxsize=-1;\nSET mapred.compress.map.output=true;\nSET mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;\nSET mapred.output.compress=true;\nSET ma
pred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;\nSET mapred.output.compression.type=BLOCK;\nSET mapreduce.job.max.split.locations=2000;\nSET hive.exec.compress.output=true;\nSET hive.auto.convert.join.noconditionaltask = true;\nSET hive.auto.convert.join.noconditionaltask.size = 300000000;\nINSERT OVERWRITE TABLE kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\nSELECT\nTEST_KYLIN_FACT.CAL_DT\n,TEST_KYLIN_FACT.LEAF_CATEG_ID\n,TEST_KYLIN_FACT.LSTG_SITE_ID\n,TEST_CATEGORY_GROUPINGS.META_CATEG_NAME\n,TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME\n,TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME\n,TEST_KYLIN_FACT.LSTG_FORMAT_NAME\n,TEST_KYLIN_FACT.SLR_SEGMENT_CD\n,TEST_KYLIN_FACT.SELLER_ID\n,TEST_KYLIN_FACT.PRICE\nFROM TEST_KYLIN_FACT\nINNER JOIN TEST_CAL_DT\nON TEST_KYLIN_FACT.CAL_DT = TEST_CAL_DT.CAL_DT\nINNER JOIN TEST_CATEGORY_GROUPINGS\nON TEST_KYLIN_FACT.LEAF_CATEG_ID = TEST_CATEGORY_GROUPINGS.LEAF_CATEG_ID AN
D TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_CATEGORY_GROUPINGS.SITE_ID\nINNER JOIN TEST_SITES\nON TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_SITES.SITE_ID\nINNER JOIN TEST_SELLER_TYPE_DIM\nON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD\nWHERE (test_kylin_fact.cal_dt < '2014-07-31 16:00:00')\n;\n\"",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"SHELL_CMD_HADOOP",
+ "info":null,
+ "run_async":false
+ },
+ {
+ "interruptCmd":null,
+ "name":"Extract Fact Table Distinct Columns",
+ "sequence_id":1,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns -jobname Kylin_Fact_Distinct_Columns_test_kylin_cube_with_slr_empty_Step_1",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_FACTDISTINCT",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build Dimension Dictionary",
+ "sequence_id":2,
+ "exec_cmd":" -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NO_MR_DICTIONARY",
+ "info":null,
+ "run_async":false
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build Base Cuboid Data",
+ "sequence_id":3,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/base_cuboid -jobname Kylin_Base_Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_3 -level 0",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_BASECUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 8-Dimension",
+ "sequence_id":4,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/base_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/8d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_4 -level 1",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 7-Dimension",
+ "sequence_id":5,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/8d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/7d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_5 -level 2",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 6-Dimension",
+ "sequence_id":6,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/7d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/6d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_6 -level 3",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 5-Dimension",
+ "sequence_id":7,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/6d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/5d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_7 -level 4",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 4-Dimension",
+ "sequence_id":8,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/5d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/4d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_8 -level 5",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Calculate HTable Region Splits",
+ "sequence_id":9,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/* -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/rowkey_stats -jobname Kylin_Region_Splits_Calculator_test_kylin_cube_with_slr_empty_Step_9 -cubename test_kylin_cube_with_slr_empty",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_RANGEKEYDISTRIBUTION",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Create HTable",
+ "sequence_id":10,
+ "exec_cmd":" -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/rowkey_stats/part-r-00000 -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADDOP_NO_MR_CREATEHTABLE",
+ "info":null,
+ "run_async":false
+ },
+ {
+ "interruptCmd":null,
+ "name":"Convert Cuboid Data to HFile",
+ "sequence_id":11,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/* -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -jobname Kylin_HFile_Generator_test_kylin_cube_with_slr_empty_Step_11",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_CONVERTHFILE",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Load HFile to HBase Table",
+ "sequence_id":12,
+ "exec_cmd":" -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile/ -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -cubename test_kylin_cube_with_slr_empty",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NO_MR_BULKLOAD",
+ "info":null,
+ "run_async":false
+ }
+ ],
+ "job_status":"PENDING",
+ "progress":0.0
+}
+```
+
+
+### Disable Cube
+`PUT /cubes/{cubeName}/disable`
+#### Path variable
+* cubeName - `required` `string` Cube name.
+
+#### Response Sample
+(Same as "Enable Cube")
+
+
+### Purge Cube
+`PUT /cubes/{cubeName}/purge`
+#### Path variable
+* cubeName - `required` `string` Cube name.
+
+#### Response Sample
+(Same as "Enable Cube")
+
+
+### Enable Cube
+`PUT /cubes/{cubeName}/enable`
+#### Path variable
+* cubeName - `required` `string` Cube name.
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ "uuid":"1eaca32a-a33e-4b69-83dd-0bb8b1f8c53b",
+ "last_modified":1407909046305,
+ "name":"test_kylin_cube_with_slr_ready",
+ "owner":null,
+ "version":null,
+ "descriptor":"test_kylin_cube_with_slr_desc",
+ "cost":50,
+ "status":"ACTIVE",
+ "segments":[
+ {
+ "name":"19700101000000_20140531160000",
+ "storage_location_identifier":"KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_READY-19700101000000_20140531160000_BF043D2D-9A4A-45E9-AA59-5A17D3F34A50",
+ "date_range_start":0,
+ "date_range_end":1401552000000,
+ "status":"READY",
+ "size_kb":4758,
+ "source_records":6000,
+ "source_records_size":620356,
+ "last_build_time":1407832663227,
+ "last_build_job_id":"2c7a2b63-b052-4a51-8b09-0c24b5792cda",
+ "binary_signature":null,
+ "dictionaries":{
+ "TEST_CATEGORY_GROUPINGS/CATEG_LVL2_NAME":"/dict/TEST_CATEGORY_GROUPINGS/CATEG_LVL2_NAME/16d8185c-ee6b-4f8c-a919-756d9809f937.dict",
+ "TEST_KYLIN_FACT/LSTG_SITE_ID":"/dict/TEST_SITES/SITE_ID/0bec6bb3-1b0d-469c-8289-b8c4ca5d5001.dict",
+ "TEST_KYLIN_FACT/SLR_SEGMENT_CD":"/dict/TEST_SELLER_TYPE_DIM/SELLER_TYPE_CD/0c5d77ec-316b-47e0-ba9a-0616be890ad6.dict",
+ "TEST_KYLIN_FACT/CAL_DT":"/dict/PREDEFINED/date(yyyy-mm-dd)/64ac4f82-f2af-476e-85b9-f0805001014e.dict",
+ "TEST_CATEGORY_GROUPINGS/CATEG_LVL3_NAME":"/dict/TEST_CATEGORY_GROUPINGS/CATEG_LVL3_NAME/270fbfb0-281c-4602-8413-2970a7439c47.dict",
+ "TEST_KYLIN_FACT/LEAF_CATEG_ID":"/dict/TEST_CATEGORY_GROUPINGS/LEAF_CATEG_ID/2602386c-debb-4968-8d2f-b52b8215e385.dict",
+ "TEST_CATEGORY_GROUPINGS/META_CATEG_NAME":"/dict/TEST_CATEGORY_GROUPINGS/META_CATEG_NAME/0410d2c4-4686-40bc-ba14-170042a2de94.dict"
+ },
+ "snapshots":{
+ "TEST_CAL_DT":"/table_snapshot/TEST_CAL_DT.csv/8f7cfc8a-020d-4019-b419-3c6deb0ffaa0.snapshot",
+ "TEST_SELLER_TYPE_DIM":"/table_snapshot/TEST_SELLER_TYPE_DIM.csv/c60fd05e-ac94-4016-9255-96521b273b81.snapshot",
+ "TEST_CATEGORY_GROUPINGS":"/table_snapshot/TEST_CATEGORY_GROUPINGS.csv/363f4a59-b725-4459-826d-3188bde6a971.snapshot",
+ "TEST_SITES":"/table_snapshot/TEST_SITES.csv/78e0aecc-3ec6-4406-b86e-bac4b10ea63b.snapshot"
+ }
+ }
+ ],
+ "create_time":null,
+ "source_records_count":6000,
+ "source_records_size":0,
+ "size_kb":4758
+}
+```
+
+### Resume Job
+`PUT /jobs/{jobId}/resume`
+
+#### Path variable
+* jobId- `required` `string` Job id.
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ "uuid":"c143e0e4-ac5f-434d-acf3-46b0d15e3dc6",
+ "last_modified":1407908916705,
+ "name":"test_kylin_cube_with_slr_empty - 19700101000000_20140731160000 - BUILD - PDT 2014-08-12 22:48:36",
+ "type":"BUILD",
+ "duration":0,
+ "related_cube":"test_kylin_cube_with_slr_empty",
+ "related_segment":"19700101000000_20140731160000",
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "mr_waiting":0,
+ "steps":[
+ {
+ "interruptCmd":null,
+ "name":"Create Intermediate Flat Hive Table",
+ "sequence_id":0,
+ "exec_cmd":"hive -e \"DROP TABLE IF EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6;\nCREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\n(\nCAL_DT date\n,LEAF_CATEG_ID int\n,LSTG_SITE_ID int\n,META_CATEG_NAME string\n,CATEG_LVL2_NAME string\n,CATEG_LVL3_NAME string\n,LSTG_FORMAT_NAME string\n,SLR_SEGMENT_CD smallint\n,SELLER_ID bigint\n,PRICE decimal\n)\nROW FORMAT DELIMITED FIELDS TERMINATED BY '\\177'\nSTORED AS SEQUENCEFILE\nLOCATION '/tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6';\nSET mapreduce.job.split.metainfo.maxsize=-1;\nSET mapred.compress.map.output=true;\nSET mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;\nSET mapred.output.compress=true;\nSET ma
pred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;\nSET mapred.output.compression.type=BLOCK;\nSET mapreduce.job.max.split.locations=2000;\nSET hive.exec.compress.output=true;\nSET hive.auto.convert.join.noconditionaltask = true;\nSET hive.auto.convert.join.noconditionaltask.size = 300000000;\nINSERT OVERWRITE TABLE kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\nSELECT\nTEST_KYLIN_FACT.CAL_DT\n,TEST_KYLIN_FACT.LEAF_CATEG_ID\n,TEST_KYLIN_FACT.LSTG_SITE_ID\n,TEST_CATEGORY_GROUPINGS.META_CATEG_NAME\n,TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME\n,TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME\n,TEST_KYLIN_FACT.LSTG_FORMAT_NAME\n,TEST_KYLIN_FACT.SLR_SEGMENT_CD\n,TEST_KYLIN_FACT.SELLER_ID\n,TEST_KYLIN_FACT.PRICE\nFROM TEST_KYLIN_FACT\nINNER JOIN TEST_CAL_DT\nON TEST_KYLIN_FACT.CAL_DT = TEST_CAL_DT.CAL_DT\nINNER JOIN TEST_CATEGORY_GROUPINGS\nON TEST_KYLIN_FACT.LEAF_CATEG_ID = TEST_CATEGORY_GROUPINGS.LEAF_CATEG_ID AN
D TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_CATEGORY_GROUPINGS.SITE_ID\nINNER JOIN TEST_SITES\nON TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_SITES.SITE_ID\nINNER JOIN TEST_SELLER_TYPE_DIM\nON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD\nWHERE (test_kylin_fact.cal_dt < '2014-07-31 16:00:00')\n;\n\"",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"SHELL_CMD_HADOOP",
+ "info":null,
+ "run_async":false
+ },
+ {
+ "interruptCmd":null,
+ "name":"Extract Fact Table Distinct Columns",
+ "sequence_id":1,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns -jobname Kylin_Fact_Distinct_Columns_test_kylin_cube_with_slr_empty_Step_1",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_FACTDISTINCT",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build Dimension Dictionary",
+ "sequence_id":2,
+ "exec_cmd":" -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NO_MR_DICTIONARY",
+ "info":null,
+ "run_async":false
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build Base Cuboid Data",
+ "sequence_id":3,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/base_cuboid -jobname Kylin_Base_Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_3 -level 0",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_BASECUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 8-Dimension",
+ "sequence_id":4,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/base_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/8d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_4 -level 1",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 7-Dimension",
+ "sequence_id":5,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/8d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/7d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_5 -level 2",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 6-Dimension",
+ "sequence_id":6,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/7d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/6d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_6 -level 3",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 5-Dimension",
+ "sequence_id":7,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/6d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/5d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_7 -level 4",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 4-Dimension",
+ "sequence_id":8,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/5d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/4d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_8 -level 5",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Calculate HTable Region Splits",
+ "sequence_id":9,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/* -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/rowkey_stats -jobname Kylin_Region_Splits_Calculator_test_kylin_cube_with_slr_empty_Step_9 -cubename test_kylin_cube_with_slr_empty",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_RANGEKEYDISTRIBUTION",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Create HTable",
+ "sequence_id":10,
+ "exec_cmd":" -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/rowkey_stats/part-r-00000 -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADDOP_NO_MR_CREATEHTABLE",
+ "info":null,
+ "run_async":false
+ },
+ {
+ "interruptCmd":null,
+ "name":"Convert Cuboid Data to HFile",
+ "sequence_id":11,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/* -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -jobname Kylin_HFile_Generator_test_kylin_cube_with_slr_empty_Step_11",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_CONVERTHFILE",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Load HFile to HBase Table",
+ "sequence_id":12,
+ "exec_cmd":" -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile/ -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -cubename test_kylin_cube_with_slr_empty",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NO_MR_BULKLOAD",
+ "info":null,
+ "run_async":false
+ }
+ ],
+ "job_status":"PENDING",
+ "progress":0.0
+}
+```
+
+### Discard Job
+`PUT /jobs/{jobId}/cancel`
+#### Path variable
+* jobId- `required` `string` Job id.
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ "uuid":"c143e0e4-ac5f-434d-acf3-46b0d15e3dc6",
+ "last_modified":1407908916705,
+ "name":"test_kylin_cube_with_slr_empty - 19700101000000_20140731160000 - BUILD - PDT 2014-08-12 22:48:36",
+ "type":"BUILD",
+ "duration":0,
+ "related_cube":"test_kylin_cube_with_slr_empty",
+ "related_segment":"19700101000000_20140731160000",
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "mr_waiting":0,
+ "steps":[
+ {
+ "interruptCmd":null,
+ "name":"Create Intermediate Flat Hive Table",
+ "sequence_id":0,
+ "exec_cmd":"hive -e \"DROP TABLE IF EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6;\nCREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\n(\nCAL_DT date\n,LEAF_CATEG_ID int\n,LSTG_SITE_ID int\n,META_CATEG_NAME string\n,CATEG_LVL2_NAME string\n,CATEG_LVL3_NAME string\n,LSTG_FORMAT_NAME string\n,SLR_SEGMENT_CD smallint\n,SELLER_ID bigint\n,PRICE decimal\n)\nROW FORMAT DELIMITED FIELDS TERMINATED BY '\\177'\nSTORED AS SEQUENCEFILE\nLOCATION '/tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6';\nSET mapreduce.job.split.metainfo.maxsize=-1;\nSET mapred.compress.map.output=true;\nSET mapred.map.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;\nSET mapred.output.compress=true;\nSET ma
pred.output.compression.codec=com.hadoop.compression.lzo.LzoCodec;\nSET mapred.output.compression.type=BLOCK;\nSET mapreduce.job.max.split.locations=2000;\nSET hive.exec.compress.output=true;\nSET hive.auto.convert.join.noconditionaltask = true;\nSET hive.auto.convert.join.noconditionaltask.size = 300000000;\nINSERT OVERWRITE TABLE kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6\nSELECT\nTEST_KYLIN_FACT.CAL_DT\n,TEST_KYLIN_FACT.LEAF_CATEG_ID\n,TEST_KYLIN_FACT.LSTG_SITE_ID\n,TEST_CATEGORY_GROUPINGS.META_CATEG_NAME\n,TEST_CATEGORY_GROUPINGS.CATEG_LVL2_NAME\n,TEST_CATEGORY_GROUPINGS.CATEG_LVL3_NAME\n,TEST_KYLIN_FACT.LSTG_FORMAT_NAME\n,TEST_KYLIN_FACT.SLR_SEGMENT_CD\n,TEST_KYLIN_FACT.SELLER_ID\n,TEST_KYLIN_FACT.PRICE\nFROM TEST_KYLIN_FACT\nINNER JOIN TEST_CAL_DT\nON TEST_KYLIN_FACT.CAL_DT = TEST_CAL_DT.CAL_DT\nINNER JOIN TEST_CATEGORY_GROUPINGS\nON TEST_KYLIN_FACT.LEAF_CATEG_ID = TEST_CATEGORY_GROUPINGS.LEAF_CATEG_ID AN
D TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_CATEGORY_GROUPINGS.SITE_ID\nINNER JOIN TEST_SITES\nON TEST_KYLIN_FACT.LSTG_SITE_ID = TEST_SITES.SITE_ID\nINNER JOIN TEST_SELLER_TYPE_DIM\nON TEST_KYLIN_FACT.SLR_SEGMENT_CD = TEST_SELLER_TYPE_DIM.SELLER_TYPE_CD\nWHERE (test_kylin_fact.cal_dt < '2014-07-31 16:00:00')\n;\n\"",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"SHELL_CMD_HADOOP",
+ "info":null,
+ "run_async":false
+ },
+ {
+ "interruptCmd":null,
+ "name":"Extract Fact Table Distinct Columns",
+ "sequence_id":1,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns -jobname Kylin_Fact_Distinct_Columns_test_kylin_cube_with_slr_empty_Step_1",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_FACTDISTINCT",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build Dimension Dictionary",
+ "sequence_id":2,
+ "exec_cmd":" -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/fact_distinct_columns",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NO_MR_DICTIONARY",
+ "info":null,
+ "run_async":false
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build Base Cuboid Data",
+ "sequence_id":3,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/kylin_intermediate_test_kylin_cube_with_slr_desc_19700101000000_20140731160000_c143e0e4_ac5f_434d_acf3_46b0d15e3dc6 -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/base_cuboid -jobname Kylin_Base_Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_3 -level 0",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_BASECUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 8-Dimension",
+ "sequence_id":4,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/base_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/8d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_4 -level 1",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 7-Dimension",
+ "sequence_id":5,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/8d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/7d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_5 -level 2",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 6-Dimension",
+ "sequence_id":6,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/7d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/6d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_6 -level 3",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 5-Dimension",
+ "sequence_id":7,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/6d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/5d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_7 -level 4",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Build N-Dimension Cuboid Data : 4-Dimension",
+ "sequence_id":8,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -segmentname 19700101000000_20140731160000 -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/5d_cuboid -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/4d_cuboid -jobname Kylin_ND-Cuboid_Builder_test_kylin_cube_with_slr_empty_Step_8 -level 5",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NDCUBOID",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Calculate HTable Region Splits",
+ "sequence_id":9,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/* -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/rowkey_stats -jobname Kylin_Region_Splits_Calculator_test_kylin_cube_with_slr_empty_Step_9 -cubename test_kylin_cube_with_slr_empty",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_RANGEKEYDISTRIBUTION",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Create HTable",
+ "sequence_id":10,
+ "exec_cmd":" -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/rowkey_stats/part-r-00000 -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADDOP_NO_MR_CREATEHTABLE",
+ "info":null,
+ "run_async":false
+ },
+ {
+ "interruptCmd":null,
+ "name":"Convert Cuboid Data to HFile",
+ "sequence_id":11,
+ "exec_cmd":" -conf C:/kylin/Kylin/server/src/main/resources/hadoop_job_conf_medium.xml -cubename test_kylin_cube_with_slr_empty -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/cuboid/* -output /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -jobname Kylin_HFile_Generator_test_kylin_cube_with_slr_empty_Step_11",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_CONVERTHFILE",
+ "info":null,
+ "run_async":true
+ },
+ {
+ "interruptCmd":null,
+ "name":"Load HFile to HBase Table",
+ "sequence_id":12,
+ "exec_cmd":" -input /tmp/kylin-c143e0e4-ac5f-434d-acf3-46b0d15e3dc6/test_kylin_cube_with_slr_empty/hfile/ -htablename KYLIN-CUBE-TEST_KYLIN_CUBE_WITH_SLR_EMPTY-19700101000000_20140731160000_11BB4326-5975-4358-804C-70D53642E03A -cubename test_kylin_cube_with_slr_empty",
+ "interrupt_cmd":null,
+ "exec_start_time":0,
+ "exec_end_time":0,
+ "exec_wait_time":0,
+ "step_status":"PENDING",
+ "cmd_type":"JAVA_CMD_HADOOP_NO_MR_BULKLOAD",
+ "info":null,
+ "run_async":false
+ }
+ ],
+ "job_status":"PENDING",
+ "progress":0.0
+}
+```
+
+### Get job step output
+`GET /{jobId}/steps/{stepId}/output`
+#### Path Variable
+* jobId - `required` `string` Job id.
+* stepId - `required` `string` Step id; the step id is composed by jobId with step sequence id; for example, the jobId is "fb479e54-837f-49a2-b457-651fc50be110", its 3rd step id is "fb479e54-837f-49a2-b457-651fc50be110-3",
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ "cmd_output":"log string"
+}
+```
+
+
+### Get Hive Table
+`GET /tables/{tableName}`
+#### Request Parameters
+* tableName - `required` `string` table name to find.
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ uuid: "69cc92c0-fc42-4bb9-893f-bd1141c91dbe",
+ name: "SAMPLE_07",
+ columns: [{
+ id: "1",
+ name: "CODE",
+ datatype: "string"
+ }, {
+ id: "2",
+ name: "DESCRIPTION",
+ datatype: "string"
+ }, {
+ id: "3",
+ name: "TOTAL_EMP",
+ datatype: "int"
+ }, {
+ id: "4",
+ name: "SALARY",
+ datatype: "int"
+ }],
+ database: "DEFAULT",
+ last_modified: 1419330476755
+}
+```
+
+### Get Hive Table (Extend Info)
+`GET /tables/{tableName}/exd-map`
+#### Request Parameters
+* tableName - `optional` `string` table name to find.
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ "minFileSize": "46055",
+ "totalNumberFiles": "1",
+ "location": "hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/sample_07",
+ "lastAccessTime": "1418374103365",
+ "lastUpdateTime": "1398176493340",
+ "columns": "struct columns { string code, string description, i32 total_emp, i32 salary}",
+ "partitionColumns": "",
+ "EXD_STATUS": "true",
+ "maxFileSize": "46055",
+ "inputformat": "org.apache.hadoop.mapred.TextInputFormat",
+ "partitioned": "false",
+ "tableName": "sample_07",
+ "owner": "hue",
+ "totalFileSize": "46055",
+ "outputformat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
+}
+```
+
+### Get Hive Tables
+`GET /tables`
+#### Request Parameters
+* project- `required` `string` will list all tables in the project.
+* ext- `optional` `boolean` set true to get extend info of table.
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+[
+ {
+ uuid: "53856c96-fe4d-459e-a9dc-c339b1bc3310",
+ name: "SAMPLE_08",
+ columns: [{
+ id: "1",
+ name: "CODE",
+ datatype: "string"
+ }, {
+ id: "2",
+ name: "DESCRIPTION",
+ datatype: "string"
+ }, {
+ id: "3",
+ name: "TOTAL_EMP",
+ datatype: "int"
+ }, {
+ id: "4",
+ name: "SALARY",
+ datatype: "int"
+ }],
+ database: "DEFAULT",
+ cardinality: {},
+ last_modified: 0,
+ exd: {
+ minFileSize: "46069",
+ totalNumberFiles: "1",
+ location: "hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/sample_08",
+ lastAccessTime: "1398176495945",
+ lastUpdateTime: "1398176495981",
+ columns: "struct columns { string code, string description, i32 total_emp, i32 salary}",
+ partitionColumns: "",
+ EXD_STATUS: "true",
+ maxFileSize: "46069",
+ inputformat: "org.apache.hadoop.mapred.TextInputFormat",
+ partitioned: "false",
+ tableName: "sample_08",
+ owner: "hue",
+ totalFileSize: "46069",
+ outputformat: "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
+ }
+}, {
+ uuid: "82d72d9c-705c-4706-aaf5-358ac2661cf9",
+ name: "TEST_KYLIN_FACT",
+ columns: [{
+ id: "1",
+ name: "TRANS_ID",
+ datatype: "bigint"
+ }, {
+ id: "2",
+ name: "CAL_DT",
+ datatype: "date"
+ }, {
+ id: "3",
+ name: "LSTG_FORMAT_NAME",
+ datatype: "string"
+ }, {
+ id: "4",
+ name: "LEAF_CATEG_ID",
+ datatype: "int"
+ }, {
+ id: "5",
+ name: "LSTG_SITE_ID",
+ datatype: "int"
+ }, {
+ id: "6",
+ name: "SLR_SEGMENT_CD",
+ datatype: "smallint"
+ }, {
+ id: "7",
+ name: "PRICE",
+ datatype: "decimal(38,16)"
+ }, {
+ id: "8",
+ name: "ITEM_COUNT",
+ datatype: "bigint"
+ }, {
+ id: "9",
+ name: "SELLER_ID",
+ datatype: "bigint"
+ }],
+ database: "DEFAULT",
+ cardinality: {},
+ last_modified: 0,
+ exd: {
+ minFileSize: "607328",
+ totalNumberFiles: "1",
+ location: "hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/test_kylin_fact",
+ lastAccessTime: "1418899958520",
+ lastUpdateTime: "1414051037179",
+ columns: "struct columns { i64 trans_id, date cal_dt, string lstg_format_name, i32 leaf_categ_id, i32 lstg_site_id, i16 slr_segment_cd, decimal(38,16) price, i64 item_count, i64 seller_id}",
+ partitionColumns: "",
+ EXD_STATUS: "true",
+ maxFileSize: "607328",
+ inputformat: "org.apache.hadoop.mapred.TextInputFormat",
+ partitioned: "false",
+ tableName: "test_kylin_fact",
+ owner: "root",
+ totalFileSize: "607328",
+ outputformat: "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"
+ }
+ }
+]
+```
+
+### Load Hive Tables
+`POST /tables/{tables}/{project}`
+
+* tables- `required` `string` tables you want to load from hive,split with comma.
+* project- `required` `String` specif.
+
+
+#### Response Sample
+```sh
+Status: 200OK
+```
+```sh
+{
+ "result.loaded": ["DEFAULT.SAMPLE_07"],
+ "result.unloaded": ["sapmle_08"]
+}
+```
+
+
+### Wipe cache
+`GET /cache/{type}/{name}/{action}`
+#### Path variable
+* type - `required` `string` 'METADATA' or 'CUBE'
+* name - `required` `string` Cache key.
+* action - `required` `string` 'create', 'update' or 'drop'
+
+#### Response Sample
+```sh
+Status: 200OK
+```
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/5acdcd23/docs/TechInside/New meta data model structure.md
----------------------------------------------------------------------
diff --git a/docs/TechInside/New meta data model structure.md b/docs/TechInside/New meta data model structure.md
new file mode 100644
index 0000000..74fc74b
--- /dev/null
+++ b/docs/TechInside/New meta data model structure.md
@@ -0,0 +1,18 @@
+#New meta data model structure
+Kylin is doing a round code refactor which will introduce the following two changes on meta data:
+
+* Abstract a "model_desc" layer from "cube_desc"
+
+Before define a cube, user will need firstly define a model ("model_desc"); The model defines which is the fact table, which are lookup tables and how they be joined;
+
+With the model be defined, when user define a cube ("cube_desc"), he/she only need to specify the table/column name for a dimension, as the join conditions have already been defined;
+
+This abstraction is to extend the meta data to fulfill non-cube queries (coming soon);
+
+* Support data tables from multiple hive databases;
+
+User has the case that tables are from multiple hive database, and the table name might be the same; To support this case Kylin will use the database name + table name as the unique name for tables; And user need to specify the database name (if it is not "default") in SQL when query Kylin.
+
+Here is a sample; the fact table "test_kylin_fact" is from default hive database, you don't need to specify the db name; while lookup table is from "edw", you need use "edw.test_cal_dt" in the query:
+
+ select test_cal_dt.Week_Beg_Dt, sum(price) as c1, count(1) as c2 from test_kylin_fact inner JOIN edw.test_cal_dt as test_cal_dt ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt where test_kylin_fact.lstg_format_name='ABIN'
\ No newline at end of file
[25/29] incubator-kylin git commit: KYLIN-650 typo
Posted by li...@apache.org.
KYLIN-650 typo
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/269eac01
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/269eac01
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/269eac01
Branch: refs/heads/staging
Commit: 269eac01c35489727ba6ec509b5449d9d4c8dcde
Parents: 0be8930
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 23:53:26 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 23:53:26 2015 -0700
----------------------------------------------------------------------
docs/MISC/Kylin Document Convensions.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/269eac01/docs/MISC/Kylin Document Convensions.md
----------------------------------------------------------------------
diff --git a/docs/MISC/Kylin Document Convensions.md b/docs/MISC/Kylin Document Convensions.md
index 52fa6f7..a5b4fa0 100644
--- a/docs/MISC/Kylin Document Convensions.md
+++ b/docs/MISC/Kylin Document Convensions.md
@@ -1,4 +1,4 @@
-Kylin Document Convernsions
+Kylin Document Convensions
-----
1. Keep documentation in markdown files
[07/29] incubator-kylin git commit: KYLIN-644 join table as null
error when build the cube
Posted by li...@apache.org.
KYLIN-644 join table as null error when build the cube
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/4a3278c0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/4a3278c0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/4a3278c0
Branch: refs/heads/staging
Commit: 4a3278c085fe03485528d8b7b32916737d74877b
Parents: 53eb20f
Author: Shao Feng, Shi <sh...@ebay.com>
Authored: Tue Mar 17 16:47:16 2015 +0800
Committer: Shao Feng, Shi <sh...@ebay.com>
Committed: Tue Mar 17 16:47:16 2015 +0800
----------------------------------------------------------------------
cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java | 4 ++++
.../main/java/org/apache/kylin/job/hadoop/AbstractHadoopJob.java | 3 ++-
.../main/java/org/apache/kylin/job/tools/CubeMigrationCLI.java | 4 ++--
3 files changed, 8 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/4a3278c0/cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
----------------------------------------------------------------------
diff --git a/cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java b/cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
index 0c9cf7a..499dd63 100644
--- a/cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
+++ b/cube/src/main/java/org/apache/kylin/cube/model/CubeDesc.java
@@ -212,6 +212,10 @@ public class CubeDesc extends RootPersistentEntity {
return functions;
}
+ /**
+ * @deprecated use getModel().getAllTables() instead
+ * @return
+ */
public List<TableDesc> listTables() {
MetadataManager metaMgr = MetadataManager.getInstance(config);
HashSet<String> tableNames = new HashSet<String>();
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/4a3278c0/job/src/main/java/org/apache/kylin/job/hadoop/AbstractHadoopJob.java
----------------------------------------------------------------------
diff --git a/job/src/main/java/org/apache/kylin/job/hadoop/AbstractHadoopJob.java b/job/src/main/java/org/apache/kylin/job/hadoop/AbstractHadoopJob.java
index 087dd04..9cd01a4 100644
--- a/job/src/main/java/org/apache/kylin/job/hadoop/AbstractHadoopJob.java
+++ b/job/src/main/java/org/apache/kylin/job/hadoop/AbstractHadoopJob.java
@@ -231,7 +231,8 @@ public abstract class AbstractHadoopJob extends Configured implements Tool {
dumpList.add(cube.getResourcePath());
dumpList.add(cube.getDescriptor().getModel().getResourcePath());
dumpList.add(cube.getDescriptor().getResourcePath());
- for (TableDesc table : cube.getDescriptor().listTables()) {
+ for (String tableName : cube.getDescriptor().getModel().getAllTables()) {
+ TableDesc table = MetadataManager.getInstance(kylinConfig).getTableDesc(tableName);
dumpList.add(table.getResourcePath());
}
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/4a3278c0/job/src/main/java/org/apache/kylin/job/tools/CubeMigrationCLI.java
----------------------------------------------------------------------
diff --git a/job/src/main/java/org/apache/kylin/job/tools/CubeMigrationCLI.java b/job/src/main/java/org/apache/kylin/job/tools/CubeMigrationCLI.java
index 6fe4a68..a5904fc 100644
--- a/job/src/main/java/org/apache/kylin/job/tools/CubeMigrationCLI.java
+++ b/job/src/main/java/org/apache/kylin/job/tools/CubeMigrationCLI.java
@@ -210,8 +210,8 @@ public class CubeMigrationCLI {
metaResource.add(cube.getResourcePath());
metaResource.add(cubeDesc.getResourcePath());
- for (TableDesc tableDesc : cubeDesc.listTables()) {
- metaResource.add(tableDesc.getResourcePath());
+ for (String table : cubeDesc.getModel().getAllTables()) {
+ metaResource.add(TableDesc.concatResourcePath(table.toUpperCase()));
}
for (CubeSegment segment : cube.getSegments()) {
[02/29] incubator-kylin git commit: KYLIN-630,
define upper bound for date dictionary "9999-12-31". Make "10000-1-1"
invalid.
Posted by li...@apache.org.
KYLIN-630, define upper bound for date dictionary "9999-12-31". Make "10000-1-1" invalid.
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/0bc0f167
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/0bc0f167
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/0bc0f167
Branch: refs/heads/staging
Commit: 0bc0f1679e8c3f08ab6588c43cce1e3e425a7991
Parents: 3979766
Author: Li, Yang <ya...@ebay.com>
Authored: Mon Mar 16 13:41:58 2015 +0800
Committer: Li, Yang <ya...@ebay.com>
Committed: Mon Mar 16 13:41:58 2015 +0800
----------------------------------------------------------------------
.../apache/kylin/dict/DateStrDictionary.java | 18 ++++++++-----
.../kylin/dict/DateStrDictionaryTest.java | 27 ++++++++++++++++++++
2 files changed, 38 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/0bc0f167/dictionary/src/main/java/org/apache/kylin/dict/DateStrDictionary.java
----------------------------------------------------------------------
diff --git a/dictionary/src/main/java/org/apache/kylin/dict/DateStrDictionary.java b/dictionary/src/main/java/org/apache/kylin/dict/DateStrDictionary.java
index 0081751..7cace15 100644
--- a/dictionary/src/main/java/org/apache/kylin/dict/DateStrDictionary.java
+++ b/dictionary/src/main/java/org/apache/kylin/dict/DateStrDictionary.java
@@ -35,8 +35,8 @@ import org.apache.commons.lang.StringUtils;
/**
* A dictionary for date string (date only, no time).
*
- * Dates are numbered from 1970-1-1 -- 0 for 1970-1-1, 1 for 1-2, 2 for 1-3 and
- * so on. With 2 bytes, 65536 states, can express dates up to the year of 2149.
+ * Dates are numbered from 0000-1-1 -- 0 for "0000-1-1", 1 for "0000-1-2", 2 for "0000-1-3" and
+ * up to 3652426 for "9999-12-31".
*
* Note the implementation is not thread-safe.
*
@@ -48,6 +48,8 @@ public class DateStrDictionary extends Dictionary<String> {
static final String DEFAULT_DATETIME_PATTERN_WITHOUT_MILLISECONDS = "yyyy-MM-dd HH:mm:ss";
static final String DEFAULT_DATETIME_PATTERN_WITH_MILLISECONDS = "yyyy-MM-dd HH:mm:ss.SSS";
+ static final int ID_9999_12_31 = 3652426; // assume 0 based
+
static final private Map<String, ThreadLocal<SimpleDateFormat>> threadLocalMap = new ConcurrentHashMap<String, ThreadLocal<SimpleDateFormat>>();
static SimpleDateFormat getDateFormat(String datePattern) {
@@ -102,6 +104,7 @@ public class DateStrDictionary extends Dictionary<String> {
private String pattern;
private int baseId;
+ private int maxId;
public DateStrDictionary() {
init(DEFAULT_DATE_PATTERN, 0);
@@ -114,6 +117,7 @@ public class DateStrDictionary extends Dictionary<String> {
private void init(String datePattern, int baseId) {
this.pattern = datePattern;
this.baseId = baseId;
+ this.maxId = baseId + ID_9999_12_31;
}
@Override
@@ -123,7 +127,7 @@ public class DateStrDictionary extends Dictionary<String> {
@Override
public int getMaxId() {
- return Integer.MAX_VALUE;
+ return maxId;
}
@Override
@@ -145,16 +149,16 @@ public class DateStrDictionary extends Dictionary<String> {
final protected int getIdFromValueImpl(String value, int roundFlag) {
Date date = stringToDate(value, pattern);
int id = calcIdFromSeqNo(getNumOfDaysSince0000(date));
- if (id < 0 || id >= 16777216)
- throw new IllegalArgumentException("'" + value + "' encodes to '" + id + "' which is out of range of 3 bytes");
+ if (id < baseId || id > maxId)
+ throw new IllegalArgumentException("'" + value + "' encodes to '" + id + "' which is out of range [" + baseId + "," + maxId + "]");
return id;
}
@Override
final protected String getValueFromIdImpl(int id) {
- if (id < baseId)
- throw new IllegalArgumentException("ID '" + id + "' must not be less than base ID " + baseId);
+ if (id < baseId || id > maxId)
+ throw new IllegalArgumentException("ID '" + id + "' is out of range [" + baseId + "," + maxId + "]");
Date d = getDateFromNumOfDaysSince0000(calcSeqNoFromId(id));
return dateToString(d, pattern);
}
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/0bc0f167/dictionary/src/test/java/org/apache/kylin/dict/DateStrDictionaryTest.java
----------------------------------------------------------------------
diff --git a/dictionary/src/test/java/org/apache/kylin/dict/DateStrDictionaryTest.java b/dictionary/src/test/java/org/apache/kylin/dict/DateStrDictionaryTest.java
index fa2ef67..f393ead 100644
--- a/dictionary/src/test/java/org/apache/kylin/dict/DateStrDictionaryTest.java
+++ b/dictionary/src/test/java/org/apache/kylin/dict/DateStrDictionaryTest.java
@@ -33,6 +33,33 @@ public class DateStrDictionaryTest {
}
@Test
+ public void testMinMaxId() {
+ assertEquals(0, dict.getIdFromValue("0000-01-01"));
+ assertEquals(DateStrDictionary.ID_9999_12_31, dict.getIdFromValue("9999-12-31"));
+
+ try {
+ dict.getValueFromId(-2); // -1 is id for NULL
+ fail("IllegalArgumentException expected");
+ } catch (IllegalArgumentException e) {
+ // good
+ }
+
+ try {
+ dict.getValueFromId(DateStrDictionary.ID_9999_12_31 + 1);
+ fail("IllegalArgumentException expected");
+ } catch (IllegalArgumentException e) {
+ // good
+ }
+
+ try {
+ dict.getIdFromValue("10000-1-1");
+ fail("IllegalArgumentException expected");
+ } catch (IllegalArgumentException e) {
+ // good
+ }
+ }
+
+ @Test
public void testNull() {
int nullId = dict.getIdFromValue(null);
assertNull(dict.getValueFromId(nullId));
[03/29] incubator-kylin git commit: KYLIN-640 fix sum of float bug
Posted by li...@apache.org.
KYLIN-640 fix sum of float bug
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/4f6869e1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/4f6869e1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/4f6869e1
Branch: refs/heads/staging
Commit: 4f6869e1f1f6b2ed22e6d3b638a4048905d27e49
Parents: 0bc0f16
Author: honma <ho...@ebay.com>
Authored: Fri Mar 13 11:24:29 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Mon Mar 16 11:27:54 2015 -0700
----------------------------------------------------------------------
storage/src/main/java/org/apache/kylin/storage/tuple/Tuple.java | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/4f6869e1/storage/src/main/java/org/apache/kylin/storage/tuple/Tuple.java
----------------------------------------------------------------------
diff --git a/storage/src/main/java/org/apache/kylin/storage/tuple/Tuple.java b/storage/src/main/java/org/apache/kylin/storage/tuple/Tuple.java
index dd19e0c..dd904d4 100644
--- a/storage/src/main/java/org/apache/kylin/storage/tuple/Tuple.java
+++ b/storage/src/main/java/org/apache/kylin/storage/tuple/Tuple.java
@@ -100,7 +100,10 @@ public class Tuple implements ITuple {
fieldValue = ((BigDecimal) fieldValue).doubleValue();
} else if ("integer".equals(dataType) && !(fieldValue instanceof Integer)) {
fieldValue = ((Number) fieldValue).intValue();
+ } else if ("float".equals(dataType) && fieldValue instanceof BigDecimal) {
+ fieldValue = ((BigDecimal) fieldValue).floatValue();
}
+
setFieldObjectValue(fieldName, fieldValue);
}
@@ -120,7 +123,6 @@ public class Tuple implements ITuple {
return sb.toString();
}
-
public static Object convertOptiqCellValue(String strValue, String dataType) {
if (strValue == null)
return null;
[19/29] incubator-kylin git commit: KYLIN-650 add operations sections
Posted by li...@apache.org.
KYLIN-650 add operations sections
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/284ef260
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/284ef260
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/284ef260
Branch: refs/heads/staging
Commit: 284ef260140d0cae845dfb6075a1e679f9e9300c
Parents: 4bd5407
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 23:07:09 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 23:07:09 2015 -0700
----------------------------------------------------------------------
README.md | 14 ++++++++-
.../Advanced settings of Kylin environment.md | 28 ++++++++++++++++++
docs/Operations/Export Kylin HBase data.md | 19 ++++++++++++
docs/Operations/Kylin Metadata Store.md | 31 ++++++++++++++++++++
4 files changed, 91 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/284ef260/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 2489576..fa180b0 100644
--- a/README.md
+++ b/README.md
@@ -17,8 +17,20 @@ Tutorial
Please follow this installation tutorial to start with Kylin: [Installation Tutorial](docs/Installation/Installation.md)
-Advance Usage
+Advance Topics
-------
+####REST
+
+####Operations
+1.[Kylin Metadata Store](docs/Operations/Kylin Metadata Store.md)
+
+2.[Export Kylin HBase data](docs/Operations/Export Kylin HBase data.md)
+
+3.[Advanced settings of Kylin environment](docs/Operations/Advanced settings of Kylin environment.md)
+
+####Test
+
+####Technial Details
Get Help
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/284ef260/docs/Operations/Advanced settings of Kylin environment.md
----------------------------------------------------------------------
diff --git a/docs/Operations/Advanced settings of Kylin environment.md b/docs/Operations/Advanced settings of Kylin environment.md
new file mode 100644
index 0000000..eabfa43
--- /dev/null
+++ b/docs/Operations/Advanced settings of Kylin environment.md
@@ -0,0 +1,28 @@
+Advanced settings of Kylin environment
+----
+
+### A. Enabling LZO compression
+
+LZO compression can be leveraged to compress the output of MR jobs, as well as hbase table storage, reducing the storage overhead. By default we do not enable LZO compression in Kylin because hadoop sandbox venders tend to not include LZO in their distributions due to license(GPL) issues.
+
+To enable LZO in Kylin, there are three steps:
+
+#### Make sure LZO is working in your environment
+
+we have a simple tool to test whether LZO is well installed and configured in your environment(we only test it on the hadoop CLI that you deployed Kylin), Just run
+
+`hbase org.apache.hadoop.util.RunJar kylin-job-latest.jar com.kylinolap.job.tools.LZOSupportnessChecker`
+
+If the program prints "LZO supported by current env? true", you're good to go. Otherwise you'll need to first install LZO properly.
+
+#### Modify kylin_job_conf.xml
+
+You'll need to stop Kylin first by running `${KYLIN_HOME}/bin/kylin.sh stop`, and then modify /etc ${KYLIN_HOME}/conf/kylin_job_conf.xml by uncommenting some configuration entries related to LZO compression.
+
+#### export KYLIN_LD_LIBRARY_PATH to carry your native library paths
+
+Before starting Kylin again, you'll need to uncomment KYLIN_LD_LIBRARY_PATH in ${KYLIN_HOME}/bin/setenv.sh to carry your LZO native library paths. Here's an example for hdp 2.1:
+
+ export KYLIN_LD_LIBRARY_PATH=::/usr/lib/hadoop/lib/native/Linux-amd64-64:/usr/lib/hadoop/lib/native
+
+After exporting, you need to run `${KYLIN_HOME}/bin/kylin.sh start` to start Kylin again. Now Kylin will use LZO to compress MR outputs and hbase tables.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/284ef260/docs/Operations/Export Kylin HBase data.md
----------------------------------------------------------------------
diff --git a/docs/Operations/Export Kylin HBase data.md b/docs/Operations/Export Kylin HBase data.md
new file mode 100644
index 0000000..90c4778
--- /dev/null
+++ b/docs/Operations/Export Kylin HBase data.md
@@ -0,0 +1,19 @@
+# Export Kylin HBase data
+
+Kylin persists all data (meta data and cube) in HBase; You may want to export the data sometimes for whatever purposes (backup, migration, troubleshotting etc); This page describes the steps to do this and also there is a Java app for you to do this easily;
+
+Steps:
+
+1. Cleanup unused cubes to save storage space (be cautious on production!): run the following command in hbase CLI:
+```
+hbase org.apache.hadoop.util.RunJar /${KYLIN_HOME}/lib/kylin-job-(version).jar org.apache.kylin.job.hadoop.cube.StorageCleanupJob --delete true
+```
+2. List all HBase tables, iterate and then export each Kylin table to HDFS; See https://hbase.apache.org/book/ops_mgt.html#export
+
+3. Copy the export folder from HDFS to local file system, and then archive it;
+
+4. (optional) Download the archive from Hadoop CLI to local;
+
+5. Cleanup the export folder from CLI HDFS and local file system;
+
+Kylin provide the "ExportHBaseData.java" (currently only exist in "minicluster" branch) for you to do the step 2-5 in one run; Please ensure the correct path of "kylin.properties" has been set in the sys env; This Java uses the sandbox config by default;
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/284ef260/docs/Operations/Kylin Metadata Store.md
----------------------------------------------------------------------
diff --git a/docs/Operations/Kylin Metadata Store.md b/docs/Operations/Kylin Metadata Store.md
new file mode 100644
index 0000000..33c1afe
--- /dev/null
+++ b/docs/Operations/Kylin Metadata Store.md
@@ -0,0 +1,31 @@
+Kylin organizes all of its metadata(including cube descriptions and instances, projects, inverted index description and instances, jobs, tables and dictionaries) as a hierarchy file system. However, Kylin uses hbase to store it, rather than normal file system. If you check your kylin configuration file(kylin.properties) you will find such a line:
+
+`# The metadata store in hbase`
+`kylin.metadata.url=kylin_metadata_qa@hbase:sandbox.hortonworks.com:2181:/hbase-unsecure`
+
+This indicates that the metadata will be saved as a htable called `kylin_metadata_qa`. You can scan the htable in hbase shell to check it out.
+
+# Backup Metadata Store
+
+Sometimes you need to backup the Kylin's Metadata Store from hbase to your disk file system.
+In such cases, assuming you're on the hadoop CLI(or sandbox) where you deployed Kylin, you can use:
+
+ mkdir ~/meta_dump
+
+ hbase org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-SNAPSHOT-job.jar com.kylinolap.common.persistence.ResourceTool copy ${KYLIN_HOME}/conf/kylin.properties ~/meta_dump
+
+to dump your metadata to your local folder ~/meta_dump.
+
+# Restore Metadata Store
+
+In case you find your metadata store messed up, and you want to restore to a previous backup:
+
+first clean up the metadata store:
+
+ hbase org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-SNAPSHOT-job.jar com.kylinolap.common.persistence.ResourceTool reset
+
+then upload the backup metadata in ~/meta_dump to Kylin's metadata store:
+
+ hbase org.apache.hadoop.util.RunJar ${KYLIN_HOME}/lib/kylin-job-x.x.x-SNAPSHOT-job.jar com.kylinolap.common.persistence.ResourceTool copy ~/meta_dump ${KYLIN_HOME}/conf/kylin.properties
+
+
[04/29] incubator-kylin git commit: KYLIN-642 Couldn't refresh cube
segment
Posted by li...@apache.org.
KYLIN-642 Couldn't refresh cube segment
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/79fe5a36
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/79fe5a36
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/79fe5a36
Branch: refs/heads/staging
Commit: 79fe5a365f480cc712a8ab7a2fbf3566fa3fcbc1
Parents: 4f6869e
Author: Shao Feng, Shi <sh...@ebay.com>
Authored: Tue Mar 17 11:15:04 2015 +0800
Committer: Shao Feng, Shi <sh...@ebay.com>
Committed: Tue Mar 17 11:15:04 2015 +0800
----------------------------------------------------------------------
.../java/org/apache/kylin/cube/CubeManager.java | 50 +++++----
.../apache/kylin/rest/service/JobService.java | 110 +++++++++----------
webapp/app/js/controllers/cubes.js | 2 +-
3 files changed, 81 insertions(+), 81 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/79fe5a36/cube/src/main/java/org/apache/kylin/cube/CubeManager.java
----------------------------------------------------------------------
diff --git a/cube/src/main/java/org/apache/kylin/cube/CubeManager.java b/cube/src/main/java/org/apache/kylin/cube/CubeManager.java
index 8493c59..75f984c 100644
--- a/cube/src/main/java/org/apache/kylin/cube/CubeManager.java
+++ b/cube/src/main/java/org/apache/kylin/cube/CubeManager.java
@@ -18,36 +18,19 @@
package org.apache.kylin.cube;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.Date;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Random;
-import java.util.UUID;
-import java.util.concurrent.ConcurrentHashMap;
-
-import org.apache.commons.lang3.StringUtils;
-import org.apache.hadoop.hbase.util.Pair;
-import org.apache.kylin.cube.model.CubeBuildTypeEnum;
-import org.apache.kylin.cube.model.CubeDesc;
-import org.apache.kylin.cube.model.DimensionDesc;
-import org.apache.kylin.metadata.realization.*;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import com.google.common.base.Preconditions;
import com.google.common.collect.HashMultimap;
import com.google.common.collect.Lists;
import com.google.common.collect.Multimap;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.hadoop.hbase.util.Pair;
import org.apache.kylin.common.KylinConfig;
import org.apache.kylin.common.persistence.JsonSerializer;
import org.apache.kylin.common.persistence.ResourceStore;
import org.apache.kylin.common.persistence.Serializer;
import org.apache.kylin.common.restclient.Broadcaster;
import org.apache.kylin.common.restclient.CaseInsensitiveStringCache;
+import org.apache.kylin.cube.model.CubeDesc;
+import org.apache.kylin.cube.model.DimensionDesc;
import org.apache.kylin.dict.Dictionary;
import org.apache.kylin.dict.DictionaryInfo;
import org.apache.kylin.dict.DictionaryManager;
@@ -61,6 +44,13 @@ import org.apache.kylin.metadata.model.SegmentStatusEnum;
import org.apache.kylin.metadata.model.TableDesc;
import org.apache.kylin.metadata.model.TblColRef;
import org.apache.kylin.metadata.project.ProjectManager;
+import org.apache.kylin.metadata.realization.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
/**
@@ -254,7 +244,7 @@ public class CubeManager implements IRealizationProvider {
public Pair<CubeSegment, CubeSegment> appendAndMergeSegments(CubeInstance cube, long endDate) throws IOException {
checkNoBuildingSegment(cube);
checkCubeIsPartitioned(cube);
-
+
if (cube.getSegments().size() == 0)
throw new IllegalStateException("expect at least one existing segment");
@@ -292,6 +282,18 @@ public class CubeManager implements IRealizationProvider {
return newSegment;
}
+
+ public CubeSegment refreshSegment(CubeInstance cube, long startDate, long endDate) throws IOException {
+ checkNoBuildingSegment(cube);
+
+ CubeSegment newSegment = newSegment(cube, startDate, endDate);
+ cube.getSegments().add(newSegment);
+ Collections.sort(cube.getSegments());
+ updateCube(cube);
+
+ return newSegment;
+ }
+
public CubeSegment mergeSegments(CubeInstance cube, final long startDate, final long endDate) throws IOException {
checkNoBuildingSegment(cube);
checkCubeIsPartitioned(cube);
@@ -509,7 +511,7 @@ public class CubeManager implements IRealizationProvider {
/**
* Smartly figure out the TOBE segments once all new segments are built.
- * - Ensures no gap, no overlap
+ * - Ensures no gap, no overlap
* - Favors new segments over the old
* - Favors big segments over the small
*/
@@ -533,7 +535,7 @@ public class CubeManager implements IRealizationProvider {
}
firstSeg.validate();
- for (int i = 0, j = 1; j < tobe.size();) {
+ for (int i = 0, j = 1; j < tobe.size(); ) {
CubeSegment is = tobe.get(i);
CubeSegment js = tobe.get(j);
js.validate();
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/79fe5a36/server/src/main/java/org/apache/kylin/rest/service/JobService.java
----------------------------------------------------------------------
diff --git a/server/src/main/java/org/apache/kylin/rest/service/JobService.java b/server/src/main/java/org/apache/kylin/rest/service/JobService.java
index 1b89f87..05031b9 100644
--- a/server/src/main/java/org/apache/kylin/rest/service/JobService.java
+++ b/server/src/main/java/org/apache/kylin/rest/service/JobService.java
@@ -18,25 +18,12 @@
package org.apache.kylin.rest.service;
-import java.io.IOException;
-import java.util.Collections;
-import java.util.EnumSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import org.apache.hadoop.hbase.util.Pair;
-import org.apache.kylin.rest.constant.Constant;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.security.access.prepost.PreAuthorize;
-import org.springframework.stereotype.Component;
-
import com.google.common.base.Function;
import com.google.common.base.Preconditions;
import com.google.common.collect.FluentIterable;
import com.google.common.collect.Lists;
import com.google.common.collect.Sets;
+import org.apache.hadoop.hbase.util.Pair;
import org.apache.kylin.cube.CubeInstance;
import org.apache.kylin.cube.CubeSegment;
import org.apache.kylin.cube.model.CubeBuildTypeEnum;
@@ -50,10 +37,18 @@ import org.apache.kylin.job.cube.CubingJob;
import org.apache.kylin.job.cube.CubingJobBuilder;
import org.apache.kylin.job.engine.JobEngineConfig;
import org.apache.kylin.job.exception.JobException;
+import org.apache.kylin.job.execution.AbstractExecutable;
import org.apache.kylin.job.execution.ExecutableState;
import org.apache.kylin.job.execution.Output;
-import org.apache.kylin.job.execution.AbstractExecutable;
import org.apache.kylin.metadata.model.SegmentStatusEnum;
+import org.apache.kylin.rest.constant.Constant;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.security.access.prepost.PreAuthorize;
+import org.springframework.stereotype.Component;
+
+import java.io.IOException;
+import java.util.*;
/**
* @author ysong1
@@ -105,20 +100,20 @@ public class JobService extends BasicService {
private ExecutableState parseToExecutableState(JobStatusEnum status) {
switch (status) {
- case DISCARDED:
- return ExecutableState.DISCARDED;
- case ERROR:
- return ExecutableState.ERROR;
- case FINISHED:
- return ExecutableState.SUCCEED;
- case NEW:
- return ExecutableState.READY;
- case PENDING:
- return ExecutableState.READY;
- case RUNNING:
- return ExecutableState.RUNNING;
- default:
- throw new RuntimeException("illegal status:" + status);
+ case DISCARDED:
+ return ExecutableState.DISCARDED;
+ case ERROR:
+ return ExecutableState.ERROR;
+ case FINISHED:
+ return ExecutableState.SUCCEED;
+ case NEW:
+ return ExecutableState.READY;
+ case PENDING:
+ return ExecutableState.READY;
+ case RUNNING:
+ return ExecutableState.RUNNING;
+ default:
+ throw new RuntimeException("illegal status:" + status);
}
}
@@ -135,7 +130,7 @@ public class JobService extends BasicService {
CubingJob job;
CubingJobBuilder builder = new CubingJobBuilder(new JobEngineConfig(getConfig()));
builder.setSubmitter(submitter);
-
+
if (buildType == CubeBuildTypeEnum.BUILD) {
if (cube.getDescriptor().hasHolisticCountDistinctMeasures() && cube.getSegments().size() > 0) {
Pair<CubeSegment, CubeSegment> segs = getCubeManager().appendAndMergeSegments(cube, endDate);
@@ -147,6 +142,9 @@ public class JobService extends BasicService {
} else if (buildType == CubeBuildTypeEnum.MERGE) {
CubeSegment newSeg = getCubeManager().mergeSegments(cube, startDate, endDate);
job = builder.mergeJob(newSeg);
+ } else if (buildType == CubeBuildTypeEnum.REFRESH) {
+ CubeSegment refreshSeg = getCubeManager().refreshSegment(cube, startDate, endDate);
+ job = builder.buildJob(refreshSeg);
} else {
throw new JobException("invalid build type:" + buildType);
}
@@ -215,37 +213,37 @@ public class JobService extends BasicService {
private JobStatusEnum parseToJobStatus(ExecutableState state) {
switch (state) {
- case READY:
- return JobStatusEnum.PENDING;
- case RUNNING:
- return JobStatusEnum.RUNNING;
- case ERROR:
- return JobStatusEnum.ERROR;
- case DISCARDED:
- return JobStatusEnum.DISCARDED;
- case SUCCEED:
- return JobStatusEnum.FINISHED;
- case STOPPED:
- default:
- throw new RuntimeException("invalid state:" + state);
+ case READY:
+ return JobStatusEnum.PENDING;
+ case RUNNING:
+ return JobStatusEnum.RUNNING;
+ case ERROR:
+ return JobStatusEnum.ERROR;
+ case DISCARDED:
+ return JobStatusEnum.DISCARDED;
+ case SUCCEED:
+ return JobStatusEnum.FINISHED;
+ case STOPPED:
+ default:
+ throw new RuntimeException("invalid state:" + state);
}
}
private JobStepStatusEnum parseToJobStepStatus(ExecutableState state) {
switch (state) {
- case READY:
- return JobStepStatusEnum.PENDING;
- case RUNNING:
- return JobStepStatusEnum.RUNNING;
- case ERROR:
- return JobStepStatusEnum.ERROR;
- case DISCARDED:
- return JobStepStatusEnum.DISCARDED;
- case SUCCEED:
- return JobStepStatusEnum.FINISHED;
- case STOPPED:
- default:
- throw new RuntimeException("invalid state:" + state);
+ case READY:
+ return JobStepStatusEnum.PENDING;
+ case RUNNING:
+ return JobStepStatusEnum.RUNNING;
+ case ERROR:
+ return JobStepStatusEnum.ERROR;
+ case DISCARDED:
+ return JobStepStatusEnum.DISCARDED;
+ case SUCCEED:
+ return JobStepStatusEnum.FINISHED;
+ case STOPPED:
+ default:
+ throw new RuntimeException("invalid state:" + state);
}
}
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/79fe5a36/webapp/app/js/controllers/cubes.js
----------------------------------------------------------------------
diff --git a/webapp/app/js/controllers/cubes.js b/webapp/app/js/controllers/cubes.js
index df86816..a08422a 100755
--- a/webapp/app/js/controllers/cubes.js
+++ b/webapp/app/js/controllers/cubes.js
@@ -339,7 +339,7 @@ KylinApp
return cube;
},
buildType: function () {
- return 'BUILD';
+ return 'REFRESH';
}
}
});
[20/29] incubator-kylin git commit: KYLIN-650 category resting pages
to different topics
Posted by li...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/5acdcd23/docs/Test/Run Kylin test case with HBase Mini Cluster.md
----------------------------------------------------------------------
diff --git a/docs/Test/Run Kylin test case with HBase Mini Cluster.md b/docs/Test/Run Kylin test case with HBase Mini Cluster.md
new file mode 100644
index 0000000..6088eb9
--- /dev/null
+++ b/docs/Test/Run Kylin test case with HBase Mini Cluster.md
@@ -0,0 +1,15 @@
+## Run test case with HBase Mini Cluster
+
+Kylin is moving as many as possible unit test cases from sandbox to HBase mini cluster, so that user can run tests easily in local without a hadoop sandbox; Two maven profiles are created in the root pom.xml, "default" and "sandbox"; The default profile will startup a HBase Mini Cluster to prepare the test data and run the unit tests (the test cases that are not supported by Mini cluster have been added in the "exclude" list); If you want to keep using Sandbox to run test, just run:
+mvn test -P sandbox
+
+
+### When use the "default" profile, Kylin will:
+
+* Startup a HBase minicluster and update KylinConfig with the dynamic HBase configurations;
+* Create Kylin metadata tables and import six example cube tables;
+* Import the hbase data from a tar ball from local: examples/test_case_data/minicluster/hbase-export.tar.gz (the hbase-export.tar.gz will be updated on complete of running BuildCubeWithEngineTest）
+* After all test cases be completed, shutdown minicluster and cleanup KylinConfig cache;
+
+### To ensure Mini cluster can run successfully, you need:
+* Make sure JAVA_HOME is properly set;
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/5acdcd23/docs/Tutorial/How to use kylin remote jdbc driver.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/How to use kylin remote jdbc driver.md b/docs/Tutorial/How to use kylin remote jdbc driver.md
new file mode 100644
index 0000000..45de447
--- /dev/null
+++ b/docs/Tutorial/How to use kylin remote jdbc driver.md
@@ -0,0 +1,83 @@
+### Authentication
+Build on kylin authentication restful service. Supported parameters:
+* user : username
+* password : password
+* ssl: true/false. Default be false; If true, all the services call will use https.
+
+### Connection URL format:
+```
+jdbc:kylin://<hostname>:<port>/<kylin_project_name>
+```
+* If "ssl" = true, the "port" should be Kylin server's HTTPS port;
+* If "port" is not specified, the driver will use default port: HTTP 80, HTTPS 443;
+* The "kylin_project_name" must be specified and user need ensure it exists in Kylin server;
+
+### 1. Query with Statement
+```
+ Driver driver = (Driver) Class.forName("org.apache.kylin.jdbc.Driver").newInstance();
+
+ Properties info = new Properties();
+ info.put("user", "ADMIN");
+ info.put("password", "KYLIN");
+ Connection conn = driver.connect("jdbc:kylin://localhost:7070/kylin_project_name", info);
+ Statement state = conn.createStatement();
+ ResultSet resultSet = state.executeQuery("select * from test_table");
+
+ while (resultSet.next()) {
+ assertEquals("foo", resultSet.getString(1));
+ assertEquals("bar", resultSet.getString(2));
+ assertEquals("tool", resultSet.getString(3));
+ }
+```
+
+### 2. Query with PreparedStatement
+Supported prepared statement parameters:
+* setString
+* setInt
+* setShort
+* setLong
+* setFloat
+* setDouble
+* setBoolean
+* setByte
+* setDate
+* setTime
+* setTimestamp
+
+```
+ Driver driver = (Driver) Class.forName("org.apache.kylin.jdbc.Driver").newInstance();
+ Properties info = new Properties();
+ info.put("user", "ADMIN");
+ info.put("password", "KYLIN");
+ Connection conn = driver.connect("jdbc:kylin://localhost:7070/kylin_project_name", info);
+ PreparedStatement state = conn.prepareStatement("select * from test_table where id=?");
+ state.setInt(1, 10);
+ ResultSet resultSet = state.executeQuery();
+
+ while (resultSet.next()) {
+ assertEquals("foo", resultSet.getString(1));
+ assertEquals("bar", resultSet.getString(2));
+ assertEquals("tool", resultSet.getString(3));
+ }
+```
+
+### 3. Get query result metadata
+Kylin jdbc driver supports metadata list methods:
+List catalog, schema, table and column with sql pattern filters(such as %).
+
+```
+ Driver driver = (Driver) Class.forName("org.apache.kylin.jdbc.Driver").newInstance();
+ Properties info = new Properties();
+ info.put("user", "ADMIN");
+ info.put("password", "KYLIN");
+ Connection conn = driver.connect("jdbc:kylin://localhost:7070/kylin_project_name", info);
+ Statement state = conn.createStatement();
+ ResultSet resultSet = state.executeQuery("select * from test_table");
+
+ ResultSet tables = conn.getMetaData().getTables(null, null, "dummy", null);
+ while (tables.next()) {
+ for (int i = 0; i < 10; i++) {
+ assertEquals("dummy", tables.getString(i + 1));
+ }
+ }
+```
[22/29] incubator-kylin git commit: KYLIN-650 cleaning
Posted by li...@apache.org.
KYLIN-650 cleaning
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/af42057a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/af42057a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/af42057a
Branch: refs/heads/staging
Commit: af42057ad2ff01a7ffca32b69091177af9ee80fa
Parents: 5acdcd2
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 23:27:00 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 23:27:00 2015 -0700
----------------------------------------------------------------------
README.md | 2 +
docs/JobEngine/Design.md | 2 +-
old_README.md | 94 -------------------------------------------
3 files changed, 3 insertions(+), 95 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/af42057a/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index f3d383f..fba7c0a 100644
--- a/README.md
+++ b/README.md
@@ -48,6 +48,8 @@ Advance Topics
1.[New meta data model structure](docs/TechInside/New meta data model structure.md)
+2.[Job Engine Design](docs/JobEngine/Design.md)
+
Get Help
------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/af42057a/docs/JobEngine/Design.md
----------------------------------------------------------------------
diff --git a/docs/JobEngine/Design.md b/docs/JobEngine/Design.md
index 9c1a94d..59b9ebe 100644
--- a/docs/JobEngine/Design.md
+++ b/docs/JobEngine/Design.md
@@ -4,7 +4,7 @@
### Executable
-
+
**Executable** is a top-level interface for all kinds of jobs or tasks.
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/af42057a/old_README.md
----------------------------------------------------------------------
diff --git a/old_README.md b/old_README.md
deleted file mode 100644
index 44f6859..0000000
--- a/old_README.md
+++ /dev/null
@@ -1,94 +0,0 @@
-Kylin OLAP
-===========
-
-
-Overview
-------------
-> Extreme OLAP Engine for Big Data
-
-An open source distributed Analytical Engine from eBay to provide SQL interface and multi-dimensional analysis (OLAP) on Hadoop to support TB to PB size analysis.
-
-
-Getting Started
----------------
-
-### Prerequisites ###
-
-#### Hadoop ####
-- Hadoop: 2.2.0.2.0.6.0-61 or above
-- Hive: 0.12.0.2.0.6.0-61 or above
-- HBase: 0.96.0.2.0.6.0-61-hadoop2
-
-
-_Tested with Hortonworks HDP 2.1.3 and Cloudera Quickstart VM 5.1._
-
-#### Misc ####
-- Command hadoop, hive, hbase is workable on your hadoop cluster
-- JDK Runtime: JDK7 (OpenJDK or Oracle JDK)
-- Maven
-- Git
-- Tomcat (CATALINA_HOME being set)
-- Npm
-
-#### Before Install ####
-- For the quick tutorial, we assume that your hadoop has priviledges disabled which means any user could run hadoop and hive command
-
-
-Installation
-------------
-It is very easy to install Kylin for exploration/development. There are 3 supported options:
-
-
-1. Sandbox (HDP or CDH sandbox)
-2. Dev Environment (IDE + Sandbox)
-3. Docker container (Docker)
-
-#### Sandbox (On-Hadoop-CLI installation) ####
-
-If you are free to install Kylin on your hadoop CLI machine or Hadoop sandbox, this is the most convenient scenario, for it puts everything in a single machine.
-
-
-
-
-
-For normal users, the application in the above picture means `Kylin Web`, which contains a web interface for cube building, querying and all sorts of management. Kylin Web launches a query engine for querying and a cube build engine for building cubes. These two engines interact with the components in Hadoop CLI, like hive and hbase.
-
-For a hands-on tutorial please visit [On-Hadoop-CLI installation](https://github.com/KylinOLAP/Kylin/wiki/On-Hadoop-CLI-installation).
-
-#### Dev Environment (Off-Hadoop-CLI Installation) ####
-
-This is typically for development environment setup.
-
-Applications does not necessarily mean Kylin Web, it can also be any test cases or java applications running at your local IDE(query engine and cube build engine can be launched programmatically in your code). In this case, it is no longer reasonable to assume that all the Kylin components reside in the same machine as your Hadoop CLI. Fortunately, Kylin still works under such condition with proper settings on your CLI machine.
-
-
-
-For a hands-on tutorial please visit [Off-Hadoop-CLI installation](https://github.com/KylinOLAP/Kylin/wiki/Off-Hadoop-CLI-Installation-(Dev-Env-Setup))
-
-#### Docker Container ####
-With help from [SequenceIQ](http://sequenceiq.com/), there's docker container for Kylin (along with Hadoop, HBase and Hive) available now:[sequenceiq/docker-kylin](https://github.com/sequenceiq/docker-kylin). The only thing you will need to do is to pull the container from the official Docker repository to be up and running in few minutes.
-
-Features:
-
-- Size - Light weight compared to downloading and setting up HDP or CDH sandbox.
-- Cluster support - Supports multi node installation.
-- Fully Automated - No manual steps. One command does it all
-
-For a hands-on tutorial please visit [Kylin Docker installation](https://github.com/KylinOLAP/Kylin/wiki/On-Hadoop-Kylin-installation-using-Docker).
-
-
-
-### Resources ###
-
-* Web Site: <http://kylin.io>
-
-* Google Group: [Kylin OLAP Group](https://groups.google.com/forum/#!forum/kylin-olap)
-
-* Developer Mail: <ky...@googlegroups.com>
-
-* How To Contribute: See [wiki](https://github.com/KylinOLAP/Kylin/wiki/How-to-Contribute)
-
-* Presentation: [Kylin Hadoop OLAP Engine v1.0](https://github.com/KylinOLAP/Kylin/blob/master/docs/Kylin_Hadoop_OLAP_Engine_v1.0.pdf?raw=true)
-
-* Apache Proposal: [Apache Kylin](https://wiki.apache.org/incubator/KylinProposal)
-
[14/29] incubator-kylin git commit: KYLIN-650 add "what's next" part
Posted by li...@apache.org.
KYLIN-650 add "what's next" part
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/ae75f5f4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/ae75f5f4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/ae75f5f4
Branch: refs/heads/staging
Commit: ae75f5f408c123b9b140f0334e185cd475159724
Parents: da16316
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 19:10:03 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 19:10:03 2015 -0700
----------------------------------------------------------------------
.../Kylin Cube Build and Job Monitoring Tutorial.md | 10 +++++++++-
docs/Tutorial/Kylin ODBC Driver Tutorial.md | 1 -
docs/Tutorial/Quick play with a sample cube.md | 8 +++++---
3 files changed, 14 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/ae75f5f4/docs/Tutorial/Kylin Cube Build and Job Monitoring Tutorial.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin Cube Build and Job Monitoring Tutorial.md b/docs/Tutorial/Kylin Cube Build and Job Monitoring Tutorial.md
index 827cd94..452f24b 100644
--- a/docs/Tutorial/Kylin Cube Build and Job Monitoring Tutorial.md
+++ b/docs/Tutorial/Kylin Cube Build and Job Monitoring Tutorial.md
@@ -57,4 +57,12 @@ Click the icon button show in each step to see the details: `Parameters`, `Log`,
- 
\ No newline at end of file
+ 
+
+#### What's next
+
+After cube being built, you might need to:
+
+1. [Query the cube via web interface](Kylin Web Tutorial.md)
+2. [Query the cube via ODBC](Kylin ODBC Driver Tutorial.md)
+3. [Grant permission to cubes](Kylin Cube Permission Grant Tutorial.md)
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/ae75f5f4/docs/Tutorial/Kylin ODBC Driver Tutorial.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin ODBC Driver Tutorial.md b/docs/Tutorial/Kylin ODBC Driver Tutorial.md
index 7f84213..17653fd 100644
--- a/docs/Tutorial/Kylin ODBC Driver Tutorial.md
+++ b/docs/Tutorial/Kylin ODBC Driver Tutorial.md
@@ -32,6 +32,5 @@ Kylin ODBC Driver Tutorial
2. [Kylin and Excel Tutorial](Kylin and Excel Tutorial.md)
-
## Bug Report
Open github issue here
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/ae75f5f4/docs/Tutorial/Quick play with a sample cube.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Quick play with a sample cube.md b/docs/Tutorial/Quick play with a sample cube.md
index 37c44ff..9495b4a 100644
--- a/docs/Tutorial/Quick play with a sample cube.md
+++ b/docs/Tutorial/Quick play with a sample cube.md
@@ -10,9 +10,11 @@ Kylin provides a script for you to create a sample Cube; the script will also cr
select cal_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by cal_dt order by cal_dt
6. You can verify the query result and compare the response time with hive;
+
#### What's next
-After cube being created, you might need to:
+After cube being built, you might need to:
-1. [Build the cube so that it can be queried](Kylin Cube Build and Job Monitoring Tutorial.md)
-2. [Grant permission to cubes](Kylin Cube Permission Grant Tutorial.md)
\ No newline at end of file
+1. [Query the cube via web interface](Kylin Web Tutorial.md)
+2. [Query the cube via ODBC](Kylin ODBC Driver Tutorial.md)
+3. [Grant permission to cubes](Kylin Cube Permission Grant Tutorial.md)
\ No newline at end of file
[16/29] incubator-kylin git commit: KYLIN-650 remove reference to wiki
Posted by li...@apache.org.
KYLIN-650 remove reference to wiki
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/7faa216c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/7faa216c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/7faa216c
Branch: refs/heads/staging
Commit: 7faa216c056d3076414ce5b350bf6c432b162bbb
Parents: f22582d
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 19:20:55 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 19:20:55 2015 -0700
----------------------------------------------------------------------
docs/Tutorial/Kylin and Tableau Tutorial.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/7faa216c/docs/Tutorial/Kylin and Tableau Tutorial.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin and Tableau Tutorial.md b/docs/Tutorial/Kylin and Tableau Tutorial.md
index a2b28b2..dad90a8 100644
--- a/docs/Tutorial/Kylin and Tableau Tutorial.md
+++ b/docs/Tutorial/Kylin and Tableau Tutorial.md
@@ -25,7 +25,7 @@ Enter your Sever location and credentials: server host, port, username and passw
-Click "Connect" to get the list of projects that you have permission to access. See details about permission in [Kylin Cube Permission Grant Tutorial](https://github.com/KylinOLAP/Kylin/wiki/Kylin-Cube-Permission-Grant-Tutorial). Then choose the project you want to connect in the drop down list.
+Click "Connect" to get the list of projects that you have permission to access. See details about permission in [Kylin Cube Permission Grant Tutorial](Kylin Cube Permission Grant Tutorial.md). Then choose the project you want to connect in the drop down list.
[15/29] incubator-kylin git commit: KYLIN-650 reiterate port issue in
ODBC tutorial
Posted by li...@apache.org.
KYLIN-650 reiterate port issue in ODBC tutorial
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/f22582da
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/f22582da
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/f22582da
Branch: refs/heads/staging
Commit: f22582da3b7f77b0bab405b2b5d3396322626813
Parents: ae75f5f
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 19:16:12 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 19:16:12 2015 -0700
----------------------------------------------------------------------
docs/Tutorial/Kylin and Tableau Tutorial.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/f22582da/docs/Tutorial/Kylin and Tableau Tutorial.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Kylin and Tableau Tutorial.md b/docs/Tutorial/Kylin and Tableau Tutorial.md
index 8630dab..a2b28b2 100644
--- a/docs/Tutorial/Kylin and Tableau Tutorial.md
+++ b/docs/Tutorial/Kylin and Tableau Tutorial.md
@@ -10,11 +10,13 @@ Kylin and Tableau Tutorial
> More detail information or any issue, please contact Kylin Team: `kylinolap@gmail.com`
### Step 1. Install ODBC Driver
-Refer to wiki page [Kylin ODBC Driver Tutorial](https://github.com/KylinOLAP/Kylin/wiki/Kylin-ODBC-Driver-Tutorial).
+Refer to wiki page [Kylin ODBC Driver Tutorial](Kylin ODBC Driver Tutorial.md).
### Step 2. Connect to Kylin Server
> We recommended to use Connect Using Driver instead of Using DSN since Tableau team will not manage your DSN on Tableau Servers.
+**The snapshots are a little bit outdated because at that time Kylin ues port 9080 as default port, for new comers please use the new default port 7070**
+
Connect Using Driver: Select "Other Database(ODBC)" in the left panel and choose KylinODBCDriver in the pop-up window.
[27/29] incubator-kylin git commit: KYLIN-648, refine test sql
Posted by li...@apache.org.
KYLIN-648, refine test sql
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/3563a914
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/3563a914
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/3563a914
Branch: refs/heads/staging
Commit: 3563a91469619dc1ca22816175e26e22557694ce
Parents: c66a74b
Author: Li, Yang <ya...@ebay.com>
Authored: Fri Mar 20 15:31:37 2015 +0800
Committer: Li, Yang <ya...@ebay.com>
Committed: Fri Mar 20 15:31:37 2015 +0800
----------------------------------------------------------------------
query/src/test/resources/query/sql/query78.sql | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/3563a914/query/src/test/resources/query/sql/query78.sql
----------------------------------------------------------------------
diff --git a/query/src/test/resources/query/sql/query78.sql b/query/src/test/resources/query/sql/query78.sql
index f1d8fb2..0da9611 100644
--- a/query/src/test/resources/query/sql/query78.sql
+++ b/query/src/test/resources/query/sql/query78.sql
@@ -1,4 +1,4 @@
-select sum(PRICE), LSTG_FORMAT_NAME
+select sum(PRICE) as GMV, LSTG_FORMAT_NAME
from test_kylin_fact
where (LSTG_FORMAT_NAME in ('ABIN')) or (LSTG_FORMAT_NAME>='FP-GTC' and LSTG_FORMAT_NAME<='Others')
group by LSTG_FORMAT_NAME
[24/29] incubator-kylin git commit: KYLIN-650 add more convensions
Posted by li...@apache.org.
KYLIN-650 add more convensions
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/0be89306
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/0be89306
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/0be89306
Branch: refs/heads/staging
Commit: 0be893069d7e4fae7dcaf4b85ce839f930ce93d7
Parents: a8757d5
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 23:38:56 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 23:38:56 2015 -0700
----------------------------------------------------------------------
docs/MISC/Kylin Document Convensions.md | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/0be89306/docs/MISC/Kylin Document Convensions.md
----------------------------------------------------------------------
diff --git a/docs/MISC/Kylin Document Convensions.md b/docs/MISC/Kylin Document Convensions.md
index 0d45dd9..52fa6f7 100644
--- a/docs/MISC/Kylin Document Convensions.md
+++ b/docs/MISC/Kylin Document Convensions.md
@@ -5,4 +5,5 @@ Kylin Document Convernsions
2. In markdown files, try to use relative URI for document/image reference.
3. Never link to github wiki pages
4. Try not to simply list all docuemnts in one page. Instead, try to organize different documents into a "story", so that readers are presented deeper materials as they proceed.
-5. ...
\ No newline at end of file
+5. Always keep documents updated
+6. ...
\ No newline at end of file
[29/29] incubator-kylin git commit: Merge branch 'staging' of
https://github.com/KylinOLAP/Kylin into staging
Posted by li...@apache.org.
Merge branch 'staging' of https://github.com/KylinOLAP/Kylin into staging
Conflicts:
query/src/test/resources/query/sql/query78.sql
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/3655b338
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/3655b338
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/3655b338
Branch: refs/heads/staging
Commit: 3655b338f9833a894fc873a393e294dcb7a3cf78
Parents: 222b144 3563a91
Author: Li, Yang <ya...@ebay.com>
Authored: Fri Mar 20 15:45:37 2015 +0800
Committer: Li, Yang <ya...@ebay.com>
Committed: Fri Mar 20 15:45:37 2015 +0800
----------------------------------------------------------------------
----------------------------------------------------------------------
[26/29] incubator-kylin git commit: KYLIN-650 typo
Posted by li...@apache.org.
KYLIN-650 typo
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/c66a74b3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/c66a74b3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/c66a74b3
Branch: refs/heads/staging
Commit: c66a74b37dc03f6115bc69c5de64b747d893719f
Parents: 269eac0
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 23:55:15 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 23:55:15 2015 -0700
----------------------------------------------------------------------
README.md | 2 +-
docs/MISC/Kylin Document Convensions.md | 9 ---------
docs/MISC/Kylin Document Conventions.md | 9 +++++++++
3 files changed, 10 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/c66a74b3/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index 39bb3e1..144e01b 100644
--- a/README.md
+++ b/README.md
@@ -69,6 +69,6 @@ Resources
* Apache Proposal: [Apache Kylin](https://wiki.apache.org/incubator/KylinProposal)
-* [Kylin Document Convensions](docs/MISC/Kylin Document Convensions.md)
+* [Kylin Document Conventions](docs/MISC/Kylin Document Conventions.md)
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/c66a74b3/docs/MISC/Kylin Document Convensions.md
----------------------------------------------------------------------
diff --git a/docs/MISC/Kylin Document Convensions.md b/docs/MISC/Kylin Document Convensions.md
deleted file mode 100644
index a5b4fa0..0000000
--- a/docs/MISC/Kylin Document Convensions.md
+++ /dev/null
@@ -1,9 +0,0 @@
-Kylin Document Convensions
------
-
-1. Keep documentation in markdown files
-2. In markdown files, try to use relative URI for document/image reference.
-3. Never link to github wiki pages
-4. Try not to simply list all docuemnts in one page. Instead, try to organize different documents into a "story", so that readers are presented deeper materials as they proceed.
-5. Always keep documents updated
-6. ...
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/c66a74b3/docs/MISC/Kylin Document Conventions.md
----------------------------------------------------------------------
diff --git a/docs/MISC/Kylin Document Conventions.md b/docs/MISC/Kylin Document Conventions.md
new file mode 100644
index 0000000..5e42852
--- /dev/null
+++ b/docs/MISC/Kylin Document Conventions.md
@@ -0,0 +1,9 @@
+Kylin Document Conventions
+-----
+
+1. Keep documentation in markdown files
+2. In markdown files, try to use relative URI for document/image reference.
+3. Never link to github wiki pages
+4. Try not to simply list all docuemnts in one page. Instead, try to organize different documents into a "story", so that readers are presented deeper materials as they proceed.
+5. Always keep documents updated
+6. ...
\ No newline at end of file
[13/29] incubator-kylin git commit: KYLIN-650 updates
Posted by li...@apache.org.
KYLIN-650 updates
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/da16316a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/da16316a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/da16316a
Branch: refs/heads/staging
Commit: da16316a54f6f04c91ad62d104bea6844c533a49
Parents: 592e96b
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 19:04:45 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 19:04:45 2015 -0700
----------------------------------------------------------------------
docs/Installation/Installation.md | 2 +-
docs/Installation/Off Hadoop CLI Installation.md | 2 +-
docs/Tutorial/Quick play with a sample cube.md | 10 +++++++++-
3 files changed, 11 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/da16316a/docs/Installation/Installation.md
----------------------------------------------------------------------
diff --git a/docs/Installation/Installation.md b/docs/Installation/Installation.md
index 5b33a56..2fa345c 100644
--- a/docs/Installation/Installation.md
+++ b/docs/Installation/Installation.md
@@ -32,7 +32,7 @@ Features:
- Cluster support - Supports multi node installation.
- Fully Automated - No manual steps. One command does it all
-For a hands-on tutorial please visit [Kylin Docker installation](https://github.com/KylinOLAP/Kylin/wiki/On-Hadoop-Kylin-installation-using-Docker).
+For a hands-on tutorial please visit [Kylin Docker installation](On Hadoop Kylin installation using Docker.md).
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/da16316a/docs/Installation/Off Hadoop CLI Installation.md
----------------------------------------------------------------------
diff --git a/docs/Installation/Off Hadoop CLI Installation.md b/docs/Installation/Off Hadoop CLI Installation.md
index d63bf13..f83ed5d 100644
--- a/docs/Installation/Off Hadoop CLI Installation.md
+++ b/docs/Installation/Off Hadoop CLI Installation.md
@@ -63,5 +63,5 @@ Run other tests, the end-to-end cube building test is exclueded
In your Eclipse IDE, launch `org.apache.kylin.rest.DebugTomcat` with specifying VM arguments "-Dspring.profiles.active=sandbox". (By default Kylin server will listen on 7070 port; If you want to use another port, please specify it as a parameter when run `DebugTomcat)
-Check Kylin Web available at http://localhost:7070 (user:ADMIN,password:KYLIN)
+Check Kylin Web available at http://localhost:7070/kylin (user:ADMIN,password:KYLIN)
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/da16316a/docs/Tutorial/Quick play with a sample cube.md
----------------------------------------------------------------------
diff --git a/docs/Tutorial/Quick play with a sample cube.md b/docs/Tutorial/Quick play with a sample cube.md
index cd767b3..37c44ff 100644
--- a/docs/Tutorial/Quick play with a sample cube.md
+++ b/docs/Tutorial/Quick play with a sample cube.md
@@ -1,4 +1,5 @@
### Quick start with sample cube
+
Kylin provides a script for you to create a sample Cube; the script will also create three sample hive tables:
1. Run ${KYLIN_HOME}/bin/sample.sh
@@ -7,4 +8,11 @@ Kylin provides a script for you to create a sample Cube; the script will also cr
4. Check the build progress in "Jobs" tab, until 100%;
5. Execute SQLs in the "Query" tab, for example:
select cal_dt, sum(price) as total_selled, count(distinct seller_id) as sellers from kylin_sales group by cal_dt order by cal_dt
-6. You can verify the query result and compare the response time with hive;
\ No newline at end of file
+6. You can verify the query result and compare the response time with hive;
+
+#### What's next
+
+After cube being created, you might need to:
+
+1. [Build the cube so that it can be queried](Kylin Cube Build and Job Monitoring Tutorial.md)
+2. [Grant permission to cubes](Kylin Cube Permission Grant Tutorial.md)
\ No newline at end of file
[23/29] incubator-kylin git commit: KYLIN-650 Kylin Document
Convensions
Posted by li...@apache.org.
KYLIN-650 Kylin Document Convensions
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/a8757d57
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/a8757d57
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/a8757d57
Branch: refs/heads/staging
Commit: a8757d576083d69ee996ae75c63bfae813aedbb4
Parents: af42057
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 23:36:55 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 23:36:55 2015 -0700
----------------------------------------------------------------------
README.md | 7 +++++--
docs/MISC/Kylin Document Convensions.md | 8 ++++++++
2 files changed, 13 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a8757d57/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index fba7c0a..39bb3e1 100644
--- a/README.md
+++ b/README.md
@@ -17,7 +17,7 @@ Tutorial
Please follow this installation tutorial to start with Kylin: [Installation Tutorial](docs/Installation/Installation.md)
-Advance Topics
+Advanced Topics
-------
####Connectivity
1.[How to use kylin remote jdbc driver](docs/Tutorial/How to use kylin remote jdbc driver.md)
@@ -67,5 +67,8 @@ Resources
* How To Contribute: See [this](docs/MISC/How to Contribute.md)
-* Apache Proposal: [Apache Kylin](https://wiki.apache.org/incubator/KylinProposal)
+* Apache Proposal: [Apache Kylin](https://wiki.apache.org/incubator/KylinProposal)
+
+* [Kylin Document Convensions](docs/MISC/Kylin Document Convensions.md)
+
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a8757d57/docs/MISC/Kylin Document Convensions.md
----------------------------------------------------------------------
diff --git a/docs/MISC/Kylin Document Convensions.md b/docs/MISC/Kylin Document Convensions.md
new file mode 100644
index 0000000..0d45dd9
--- /dev/null
+++ b/docs/MISC/Kylin Document Convensions.md
@@ -0,0 +1,8 @@
+Kylin Document Convernsions
+-----
+
+1. Keep documentation in markdown files
+2. In markdown files, try to use relative URI for document/image reference.
+3. Never link to github wiki pages
+4. Try not to simply list all docuemnts in one page. Instead, try to organize different documents into a "story", so that readers are presented deeper materials as they proceed.
+5. ...
\ No newline at end of file
[05/29] incubator-kylin git commit: KYLIN-643 JDBC couldn't connect
to Kylin: "java.sql.SQLException: Authentication Failed"
Posted by li...@apache.org.
KYLIN-643 JDBC couldn't connect to Kylin: "java.sql.SQLException: Authentication Failed"
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/933aa9ea
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/933aa9ea
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/933aa9ea
Branch: refs/heads/staging
Commit: 933aa9ea95fd0a3dd60cfe0138a61370d2d1ac1c
Parents: 79fe5a3
Author: Shao Feng, Shi <sh...@ebay.com>
Authored: Tue Mar 17 13:54:06 2015 +0800
Committer: Shao Feng, Shi <sh...@ebay.com>
Committed: Tue Mar 17 13:54:06 2015 +0800
----------------------------------------------------------------------
.../org/apache/kylin/jdbc/stub/KylinClient.java | 8 +++---
.../kylin/metadata/project/ProjectL2Cache.java | 27 ++++++++++----------
2 files changed, 17 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/933aa9ea/jdbc/src/main/java/org/apache/kylin/jdbc/stub/KylinClient.java
----------------------------------------------------------------------
diff --git a/jdbc/src/main/java/org/apache/kylin/jdbc/stub/KylinClient.java b/jdbc/src/main/java/org/apache/kylin/jdbc/stub/KylinClient.java
index c46f0fb..ccbfe5d 100644
--- a/jdbc/src/main/java/org/apache/kylin/jdbc/stub/KylinClient.java
+++ b/jdbc/src/main/java/org/apache/kylin/jdbc/stub/KylinClient.java
@@ -80,9 +80,9 @@ public class KylinClient implements RemoteClient {
httpClient.executeMethod(post);
if (post.getStatusCode() != 200 && post.getStatusCode() != 201) {
- logger.error("Authentication Failed with error code " + post.getStatusCode() + " and message:\n" + post.getResponseBodyAsString());
+ logger.error("Connect failed with error code " + post.getStatusCode() + " and message:\n" + post.getResponseBodyAsString());
- throw new ConnectionException("Authentication Failed.");
+ throw new ConnectionException("Connect failed, error code " + post.getStatusCode() + " and message: " + post.getResponseBodyAsString());
}
} catch (HttpException e) {
logger.error(e.getLocalizedMessage(), e);
@@ -108,9 +108,9 @@ public class KylinClient implements RemoteClient {
httpClient.executeMethod(get);
if (get.getStatusCode() != 200 && get.getStatusCode() != 201) {
- logger.error("Authentication Failed with error code " + get.getStatusCode() + " and message:\n" + get.getResponseBodyAsString());
+ logger.error("Connect failed with error code " + get.getStatusCode() + " and message:\n" + get.getResponseBodyAsString());
- throw new ConnectionException("Authentication Failed.");
+ throw new ConnectionException("Connect failed, error code " + get.getStatusCode() + " and message: " + get.getResponseBodyAsString());
}
tableMetaStubs = new ObjectMapper().readValue(get.getResponseBodyAsString(), new TypeReference<List<TableMetaStub>>() {
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/933aa9ea/metadata/src/main/java/org/apache/kylin/metadata/project/ProjectL2Cache.java
----------------------------------------------------------------------
diff --git a/metadata/src/main/java/org/apache/kylin/metadata/project/ProjectL2Cache.java b/metadata/src/main/java/org/apache/kylin/metadata/project/ProjectL2Cache.java
index 50843de..d936d08 100644
--- a/metadata/src/main/java/org/apache/kylin/metadata/project/ProjectL2Cache.java
+++ b/metadata/src/main/java/org/apache/kylin/metadata/project/ProjectL2Cache.java
@@ -18,30 +18,25 @@
package org.apache.kylin.metadata.project;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import org.apache.kylin.metadata.model.TableDesc;
-import org.apache.kylin.metadata.model.TblColRef;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
import org.apache.kylin.metadata.MetadataManager;
-import org.apache.kylin.metadata.model.ColumnDesc;
-import org.apache.kylin.metadata.model.FunctionDesc;
-import org.apache.kylin.metadata.model.MeasureDesc;
+import org.apache.kylin.metadata.model.*;
import org.apache.kylin.metadata.realization.IRealization;
import org.apache.kylin.metadata.realization.RealizationRegistry;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
/**
* This is a second level cache that is built on top of first level cached objects,
* including Realization, TableDesc, ColumnDesc etc, to speed up query time metadata lookup.
- *
+ * <p/>
* On any object update, the L2 cache simply gets wiped out because it's cheap to rebuild.
*/
class ProjectL2Cache {
@@ -158,6 +153,10 @@ class ProjectL2Cache {
ProjectCache result = new ProjectCache(project);
ProjectInstance pi = mgr.getProject(project);
+
+ if (pi == null)
+ throw new IllegalArgumentException("Project '" + project + "' does not exist;");
+
MetadataManager metaMgr = mgr.getMetadataManager();
for (String tableName : pi.getTables()) {
[06/29] incubator-kylin git commit: KYLIN-644 join table as null
error when build the cube
Posted by li...@apache.org.
KYLIN-644 join table as null error when build the cube
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/53eb20fa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/53eb20fa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/53eb20fa
Branch: refs/heads/staging
Commit: 53eb20faecb2fa071cd8f361f06e0cc99bab32b8
Parents: 933aa9e
Author: Shao Feng, Shi <sh...@ebay.com>
Authored: Tue Mar 17 14:57:16 2015 +0800
Committer: Shao Feng, Shi <sh...@ebay.com>
Committed: Tue Mar 17 14:57:16 2015 +0800
----------------------------------------------------------------------
.../org/apache/kylin/job/JoinedFlatTable.java | 46 ++++++++++++++------
.../hadoop/hive/CubeJoinedFlatTableDesc.java | 29 ------------
.../job/hadoop/hive/IIJoinedFlatTableDesc.java | 20 ---------
.../job/hadoop/hive/IJoinedFlatTableDesc.java | 5 ---
4 files changed, 33 insertions(+), 67 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/53eb20fa/job/src/main/java/org/apache/kylin/job/JoinedFlatTable.java
----------------------------------------------------------------------
diff --git a/job/src/main/java/org/apache/kylin/job/JoinedFlatTable.java b/job/src/main/java/org/apache/kylin/job/JoinedFlatTable.java
index d9b1f66..7e714d0 100644
--- a/job/src/main/java/org/apache/kylin/job/JoinedFlatTable.java
+++ b/job/src/main/java/org/apache/kylin/job/JoinedFlatTable.java
@@ -21,15 +21,13 @@ package org.apache.kylin.job;
import java.io.File;
import java.io.IOException;
import java.text.SimpleDateFormat;
-import java.util.Date;
-import java.util.HashSet;
-import java.util.Set;
-import java.util.TimeZone;
+import java.util.*;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
+import org.apache.kylin.cube.model.DimensionDesc;
import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;
@@ -54,6 +52,10 @@ import org.apache.kylin.metadata.model.TblColRef;
public class JoinedFlatTable {
+ public static final String FACT_TABLE_ALIAS = "FACT_TABLE";
+
+ public static final String LOOKUP_TABLE_ALAIS_PREFIX = "LOOKUP_";
+
public static String getTableDir(IJoinedFlatTableDesc intermediateTableDesc, String storageDfsDir, String jobUUID) {
return storageDfsDir + "/" + intermediateTableDesc.getTableName(jobUUID);
}
@@ -128,25 +130,43 @@ public class JoinedFlatTable {
StringBuilder sql = new StringBuilder();
sql.append("SELECT" + "\n");
String tableAlias;
+ Map<String, String> tableAliasMap = buildTableAliasMap(intermediateTableDesc.getDataModel());
for (int i = 0; i < intermediateTableDesc.getColumnList().size(); i++) {
IntermediateColumnDesc col = intermediateTableDesc.getColumnList().get(i);
if (i > 0) {
sql.append(",");
}
- tableAlias = intermediateTableDesc.getTableAlias(col.getTableName());
+ tableAlias = tableAliasMap.get(col.getTableName());
sql.append(tableAlias + "." + col.getColumnName() + "\n");
}
- appendJoinStatement(intermediateTableDesc, sql);
- appendWhereStatement(intermediateTableDesc, sql);
+ appendJoinStatement(intermediateTableDesc, sql, tableAliasMap);
+ appendWhereStatement(intermediateTableDesc, sql, tableAliasMap);
return sql.toString();
}
- private static void appendJoinStatement(IJoinedFlatTableDesc intermediateTableDesc, StringBuilder sql) {
+ private static Map buildTableAliasMap(DataModelDesc dataModelDesc) {
+ Map<String, String> tableAliasMap = new HashMap<String, String>();
+
+ tableAliasMap.put(dataModelDesc.getFactTable().toUpperCase(), FACT_TABLE_ALIAS);
+
+ int i = 1;
+ for (LookupDesc lookupDesc: dataModelDesc.getLookups()) {
+ JoinDesc join = lookupDesc.getJoin();
+ if (join != null) {
+ tableAliasMap.put(lookupDesc.getTable().toUpperCase(), LOOKUP_TABLE_ALAIS_PREFIX + i);
+ i++;
+ }
+
+ }
+ return tableAliasMap;
+ }
+
+ private static void appendJoinStatement(IJoinedFlatTableDesc intermediateTableDesc, StringBuilder sql, Map<String, String> tableAliasMap) {
Set<String> dimTableCache = new HashSet<String>();
DataModelDesc dataModelDesc = intermediateTableDesc.getDataModel();
String factTableName = dataModelDesc.getFactTable();
- String factTableAlias = intermediateTableDesc.getTableAlias(factTableName);
+ String factTableAlias = tableAliasMap.get(factTableName);
sql.append("FROM " + factTableName + " as " + factTableAlias + " \n");
for (LookupDesc lookupDesc : dataModelDesc.getLookups()) {
@@ -160,13 +180,13 @@ public class JoinedFlatTable {
if (pk.length != fk.length) {
throw new RuntimeException("Invalid join condition of lookup table:" + lookupDesc);
}
- sql.append(joinType + " JOIN " + dimTableName + " as " + intermediateTableDesc.getTableAlias(dimTableName) + "\n");
+ sql.append(joinType + " JOIN " + dimTableName + " as " + tableAliasMap.get(dimTableName) + "\n");
sql.append("ON ");
for (int i = 0; i < pk.length; i++) {
if (i > 0) {
sql.append(" AND ");
}
- sql.append(factTableAlias + "." + fk[i].getName() + " = " + intermediateTableDesc.getTableAlias(dimTableName) + "." + pk[i].getName());
+ sql.append(factTableAlias + "." + fk[i].getName() + " = " + tableAliasMap.get(dimTableName) + "." + pk[i].getName());
}
sql.append("\n");
@@ -176,7 +196,7 @@ public class JoinedFlatTable {
}
}
- private static void appendWhereStatement(IJoinedFlatTableDesc intermediateTableDesc, StringBuilder sql) {
+ private static void appendWhereStatement(IJoinedFlatTableDesc intermediateTableDesc, StringBuilder sql, Map<String, String> tableAliasMap) {
if (!(intermediateTableDesc instanceof CubeJoinedFlatTableDesc)) {
return;//TODO: for now only cube segments support filter and partition
}
@@ -207,7 +227,7 @@ public class JoinedFlatTable {
if (indexOfDot > 0) {
String partitionTableName = partitionColumnName.substring(0, indexOfDot);
String columeOnly = partitionColumnName.substring(indexOfDot);
- String partitionTableAlias = desc.getTableAlias(partitionTableName);
+ String partitionTableAlias = tableAliasMap.get(partitionTableName);
partitionColumnName = partitionTableAlias + columeOnly;
}
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/53eb20fa/job/src/main/java/org/apache/kylin/job/hadoop/hive/CubeJoinedFlatTableDesc.java
----------------------------------------------------------------------
diff --git a/job/src/main/java/org/apache/kylin/job/hadoop/hive/CubeJoinedFlatTableDesc.java b/job/src/main/java/org/apache/kylin/job/hadoop/hive/CubeJoinedFlatTableDesc.java
index 77475ec..a972692 100644
--- a/job/src/main/java/org/apache/kylin/job/hadoop/hive/CubeJoinedFlatTableDesc.java
+++ b/job/src/main/java/org/apache/kylin/job/hadoop/hive/CubeJoinedFlatTableDesc.java
@@ -44,14 +44,8 @@ public class CubeJoinedFlatTableDesc implements IJoinedFlatTableDesc {
private int[] rowKeyColumnIndexes; // the column index on flat table
private int[][] measureColumnIndexes; // [i] is the i.th measure related column index on flat table
- // Map for table alias:
- // key -> table name;
- // value -> alias;
- private Map<String, String> tableAliasMap;
private List<IntermediateColumnDesc> columnList = Lists.newArrayList();
-
-
public CubeJoinedFlatTableDesc(CubeDesc cubeDesc, CubeSegment cubeSegment) {
this.cubeDesc = cubeDesc;
this.cubeSegment = cubeSegment;
@@ -120,24 +114,6 @@ public class CubeJoinedFlatTableDesc implements IJoinedFlatTableDesc {
}
columnCount = columnIndex;
-
- buildTableAliasMap();
- }
-
- private void buildTableAliasMap() {
- tableAliasMap = new HashMap<String, String>();
-
- tableAliasMap.put(cubeDesc.getFactTable().toUpperCase(), FACT_TABLE_ALIAS);
-
- int i = 1;
- for (DimensionDesc dim : cubeDesc.getDimensions()) {
- JoinDesc join = dim.getJoin();
- if (join != null) {
- tableAliasMap.put(dim.getTable().toUpperCase(), LOOKUP_TABLE_ALAIS_PREFIX + i);
- i++;
- }
-
- }
}
private int contains(List<IntermediateColumnDesc> columnList, TblColRef c) {
@@ -191,11 +167,6 @@ public class CubeJoinedFlatTableDesc implements IJoinedFlatTableDesc {
return cubeDesc.getModel().getCapacity();
}
- @Override
- public String getTableAlias(String tableName) {
- return tableAliasMap.get(tableName.toUpperCase());
- }
-
private static String colName(String canonicalColName) {
return canonicalColName.replace(".", "_");
}
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/53eb20fa/job/src/main/java/org/apache/kylin/job/hadoop/hive/IIJoinedFlatTableDesc.java
----------------------------------------------------------------------
diff --git a/job/src/main/java/org/apache/kylin/job/hadoop/hive/IIJoinedFlatTableDesc.java b/job/src/main/java/org/apache/kylin/job/hadoop/hive/IIJoinedFlatTableDesc.java
index efc8cce..187b3da 100644
--- a/job/src/main/java/org/apache/kylin/job/hadoop/hive/IIJoinedFlatTableDesc.java
+++ b/job/src/main/java/org/apache/kylin/job/hadoop/hive/IIJoinedFlatTableDesc.java
@@ -53,22 +53,6 @@ public class IIJoinedFlatTableDesc implements IJoinedFlatTableDesc {
columnList.add(new IntermediateColumnDesc(String.valueOf(columnIndex), col));
columnIndex++;
}
- buildTableAliasMap();
- }
-
- private void buildTableAliasMap() {
- tableAliasMap = new TreeMap<>(String.CASE_INSENSITIVE_ORDER);
-
- tableAliasMap.put(iiDesc.getFactTableName(), FACT_TABLE_ALIAS);
-
- int i = 1;
- for (LookupDesc lookupDesc : iiDesc.getModel().getLookups()) {
- JoinDesc join = lookupDesc.getJoin();
- if (join != null) {
- tableAliasMap.put(lookupDesc.getTable(), LOOKUP_TABLE_ALAIS_PREFIX + i);
- i++;
- }
- }
}
@Override
@@ -90,8 +74,4 @@ public class IIJoinedFlatTableDesc implements IJoinedFlatTableDesc {
return DataModelDesc.RealizationCapacity.SMALL;
}
- @Override
- public String getTableAlias(String tableName) {
- return tableAliasMap.get(tableName);
- }
}
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/53eb20fa/job/src/main/java/org/apache/kylin/job/hadoop/hive/IJoinedFlatTableDesc.java
----------------------------------------------------------------------
diff --git a/job/src/main/java/org/apache/kylin/job/hadoop/hive/IJoinedFlatTableDesc.java b/job/src/main/java/org/apache/kylin/job/hadoop/hive/IJoinedFlatTableDesc.java
index 031f872..0841350 100644
--- a/job/src/main/java/org/apache/kylin/job/hadoop/hive/IJoinedFlatTableDesc.java
+++ b/job/src/main/java/org/apache/kylin/job/hadoop/hive/IJoinedFlatTableDesc.java
@@ -27,10 +27,6 @@ import org.apache.kylin.metadata.model.DataModelDesc;
*/
public interface IJoinedFlatTableDesc {
- public static final String FACT_TABLE_ALIAS = "FACT_TABLE";
-
- public static final String LOOKUP_TABLE_ALAIS_PREFIX = "LOOKUP_";
-
public String getTableName(String jobUUID);
public List<IntermediateColumnDesc> getColumnList();
@@ -39,5 +35,4 @@ public interface IJoinedFlatTableDesc {
public DataModelDesc.RealizationCapacity getCapacity();
- public String getTableAlias(String tableName);
}
[28/29] incubator-kylin git commit: KYLIN-648, refine test sql
Posted by li...@apache.org.
KYLIN-648, refine test sql
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/222b1440
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/222b1440
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/222b1440
Branch: refs/heads/staging
Commit: 222b1440a83a1990c56b2441b0fb3a905204510c
Parents: c66a74b
Author: Li, Yang <ya...@ebay.com>
Authored: Fri Mar 20 15:31:37 2015 +0800
Committer: Li, Yang <ya...@ebay.com>
Committed: Fri Mar 20 15:44:28 2015 +0800
----------------------------------------------------------------------
query/src/test/resources/query/sql/query78.sql | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/222b1440/query/src/test/resources/query/sql/query78.sql
----------------------------------------------------------------------
diff --git a/query/src/test/resources/query/sql/query78.sql b/query/src/test/resources/query/sql/query78.sql
index f1d8fb2..d055b8e 100644
--- a/query/src/test/resources/query/sql/query78.sql
+++ b/query/src/test/resources/query/sql/query78.sql
@@ -1,4 +1,4 @@
-select sum(PRICE), LSTG_FORMAT_NAME
+select sum(PRICE) as GMV, LSTG_FORMAT_NAME as FORMAT_NAME
from test_kylin_fact
where (LSTG_FORMAT_NAME in ('ABIN')) or (LSTG_FORMAT_NAME>='FP-GTC' and LSTG_FORMAT_NAME<='Others')
group by LSTG_FORMAT_NAME
[08/29] incubator-kylin git commit: Update unit tests
Posted by li...@apache.org.
Update unit tests
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/0170668f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/0170668f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/0170668f
Branch: refs/heads/staging
Commit: 0170668f0e35fdc9d4826af08e30323c692f4144
Parents: 4a3278c
Author: Shao Feng, Shi <sh...@ebay.com>
Authored: Wed Mar 18 11:34:06 2015 +0800
Committer: Shao Feng, Shi <sh...@ebay.com>
Committed: Wed Mar 18 11:34:06 2015 +0800
----------------------------------------------------------------------
.../java/org/apache/kylin/job/hadoop/cube/BaseCuboidJobTest.java | 3 +++
.../org/apache/kylin/job/hadoop/cube/MergeCuboidJobTest.java | 4 ++--
.../java/org/apache/kylin/job/hadoop/cube/NDCuboidJobTest.java | 4 ++--
.../kylin/job/hadoop/cube/RangeKeyDistributionJobTest.java | 4 ++--
pom.xml | 4 ++++
5 files changed, 13 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/0170668f/job/src/test/java/org/apache/kylin/job/hadoop/cube/BaseCuboidJobTest.java
----------------------------------------------------------------------
diff --git a/job/src/test/java/org/apache/kylin/job/hadoop/cube/BaseCuboidJobTest.java b/job/src/test/java/org/apache/kylin/job/hadoop/cube/BaseCuboidJobTest.java
index 15db7da..a1d3569 100644
--- a/job/src/test/java/org/apache/kylin/job/hadoop/cube/BaseCuboidJobTest.java
+++ b/job/src/test/java/org/apache/kylin/job/hadoop/cube/BaseCuboidJobTest.java
@@ -42,6 +42,9 @@ public class BaseCuboidJobTest extends LocalFileMetadataTestCase {
@Before
public void setup() throws Exception {
conf = new Configuration();
+ conf.set("fs.default.name", "file:///");
+ conf.set("mapred.job.tracker", "local");
+
// for local runner out-of-memory issue
conf.set("mapreduce.task.io.sort.mb", "10");
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/0170668f/job/src/test/java/org/apache/kylin/job/hadoop/cube/MergeCuboidJobTest.java
----------------------------------------------------------------------
diff --git a/job/src/test/java/org/apache/kylin/job/hadoop/cube/MergeCuboidJobTest.java b/job/src/test/java/org/apache/kylin/job/hadoop/cube/MergeCuboidJobTest.java
index 640c4d2..aae54b4 100644
--- a/job/src/test/java/org/apache/kylin/job/hadoop/cube/MergeCuboidJobTest.java
+++ b/job/src/test/java/org/apache/kylin/job/hadoop/cube/MergeCuboidJobTest.java
@@ -42,8 +42,8 @@ public class MergeCuboidJobTest extends LocalFileMetadataTestCase {
@Before
public void setup() throws Exception {
conf = new Configuration();
- // conf.set("fs.default.name", "file:///");
- // conf.set("mapred.job.tracker", "local");
+ conf.set("fs.default.name", "file:///");
+ conf.set("mapred.job.tracker", "local");
// for local runner out-of-memory issue
conf.set("mapreduce.task.io.sort.mb", "10");
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/0170668f/job/src/test/java/org/apache/kylin/job/hadoop/cube/NDCuboidJobTest.java
----------------------------------------------------------------------
diff --git a/job/src/test/java/org/apache/kylin/job/hadoop/cube/NDCuboidJobTest.java b/job/src/test/java/org/apache/kylin/job/hadoop/cube/NDCuboidJobTest.java
index 0e6fc02..c8a3d51 100644
--- a/job/src/test/java/org/apache/kylin/job/hadoop/cube/NDCuboidJobTest.java
+++ b/job/src/test/java/org/apache/kylin/job/hadoop/cube/NDCuboidJobTest.java
@@ -42,8 +42,8 @@ public class NDCuboidJobTest extends LocalFileMetadataTestCase {
@Before
public void setup() throws Exception {
conf = new Configuration();
- // conf.set("fs.default.name", "file:///");
- // conf.set("mapred.job.tracker", "local");
+ conf.set("fs.default.name", "file:///");
+ conf.set("mapred.job.tracker", "local");
// for local runner out-of-memory issue
conf.set("mapreduce.task.io.sort.mb", "10");
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/0170668f/job/src/test/java/org/apache/kylin/job/hadoop/cube/RangeKeyDistributionJobTest.java
----------------------------------------------------------------------
diff --git a/job/src/test/java/org/apache/kylin/job/hadoop/cube/RangeKeyDistributionJobTest.java b/job/src/test/java/org/apache/kylin/job/hadoop/cube/RangeKeyDistributionJobTest.java
index 833a8b3..b3bcc30 100644
--- a/job/src/test/java/org/apache/kylin/job/hadoop/cube/RangeKeyDistributionJobTest.java
+++ b/job/src/test/java/org/apache/kylin/job/hadoop/cube/RangeKeyDistributionJobTest.java
@@ -42,8 +42,8 @@ public class RangeKeyDistributionJobTest extends LocalFileMetadataTestCase {
@Before
public void setup() throws Exception {
conf = new Configuration();
- // conf.set("fs.default.name", "file:///");
- // conf.set("mapred.job.tracker", "local");
+ conf.set("fs.default.name", "file:///");
+ conf.set("mapred.job.tracker", "local");
// for local runner out-of-memory issue
conf.set("mapreduce.task.io.sort.mb", "10");
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/0170668f/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 841d75b..7d3fa1b 100644
--- a/pom.xml
+++ b/pom.xml
@@ -621,6 +621,10 @@
<value>true</value>
</property>
<property>
+ <name>hdp.version</name>
+ <value>2.2.0.0-2041</value>
+ </property>
+ <property>
<name>buildCubeUsingProvidedData</name>
<value>false</value>
</property>
[10/29] incubator-kylin git commit: KYLIN-648
Posted by li...@apache.org.
KYLIN-648
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/014ef833
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/014ef833
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/014ef833
Branch: refs/heads/staging
Commit: 014ef833b8f0eff75ca5240d23adf1973419bd0f
Parents: cf99527
Author: Li, Yang <ya...@ebay.com>
Authored: Thu Mar 19 22:30:19 2015 +0800
Committer: Li, Yang <ya...@ebay.com>
Committed: Thu Mar 19 22:30:19 2015 +0800
----------------------------------------------------------------------
query/src/test/resources/query/sql/query78.sql | 4 ++++
.../kylin/storage/hbase/CubeStorageEngine.java | 15 ++++++++++-----
2 files changed, 14 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/014ef833/query/src/test/resources/query/sql/query78.sql
----------------------------------------------------------------------
diff --git a/query/src/test/resources/query/sql/query78.sql b/query/src/test/resources/query/sql/query78.sql
new file mode 100644
index 0000000..f1d8fb2
--- /dev/null
+++ b/query/src/test/resources/query/sql/query78.sql
@@ -0,0 +1,4 @@
+select sum(PRICE), LSTG_FORMAT_NAME
+from test_kylin_fact
+where (LSTG_FORMAT_NAME in ('ABIN')) or (LSTG_FORMAT_NAME>='FP-GTC' and LSTG_FORMAT_NAME<='Others')
+group by LSTG_FORMAT_NAME
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/014ef833/storage/src/main/java/org/apache/kylin/storage/hbase/CubeStorageEngine.java
----------------------------------------------------------------------
diff --git a/storage/src/main/java/org/apache/kylin/storage/hbase/CubeStorageEngine.java b/storage/src/main/java/org/apache/kylin/storage/hbase/CubeStorageEngine.java
index a4ecc2a..8737e8c 100644
--- a/storage/src/main/java/org/apache/kylin/storage/hbase/CubeStorageEngine.java
+++ b/storage/src/main/java/org/apache/kylin/storage/hbase/CubeStorageEngine.java
@@ -109,7 +109,7 @@ public class CubeStorageEngine implements IStorageEngine {
context.setExactAggregation(isExactAggregation);
// translate filter for scan range and compose returning groups for coprocessor, note:
- // - columns on evaluate-able filter have to return
+ // - columns on non-evaluatable filter have to return
// - columns on loosened filter (due to derived translation) have to return
Set<TblColRef> groupsCopD = Sets.newHashSet(groupsD);
groupsCopD.addAll(context.getOtherMandatoryColumns()); // TODO: this is tricky, to generalize
@@ -538,8 +538,10 @@ public class CubeStorageEngine implements IStorageEngine {
List<Pair<byte[], byte[]>> newFuzzyKeys = new ArrayList<Pair<byte[], byte[]>>(mergeSize);
List<Collection<ColumnValueRange>> newFlatOrAndFilter = Lists.newLinkedList();
+ boolean hasNonFuzzyRange = false;
for (int k = from; k <= to; k++) {
HBaseKeyRange nextRange = keyRanges.get(k);
+ hasNonFuzzyRange = hasNonFuzzyRange || nextRange.getFuzzyKeys().isEmpty();
newFuzzyKeys.addAll(nextRange.getFuzzyKeys());
newFlatOrAndFilter.addAll(nextRange.getFlatOrAndFilter());
if (Bytes.compareTo(stopKey, nextRange.getStopKey()) < 0) {
@@ -553,11 +555,14 @@ public class CubeStorageEngine implements IStorageEngine {
}
}
+ // if any range is non-fuzzy, then all fuzzy keys must be cleared
+ if (hasNonFuzzyRange) {
+ newFuzzyKeys.clear();
+ }
+
partitionColumnStartDate = (partitionColumnStartDate == Long.MAX_VALUE) ? 0 : partitionColumnStartDate;
partitionColumnEndDate = (partitionColumnEndDate == 0) ? Long.MAX_VALUE : partitionColumnEndDate;
- keyRange =
-
- new HBaseKeyRange(cubeSegment, cuboid, startKey, stopKey, newFuzzyKeys, newFlatOrAndFilter, partitionColumnStartDate, partitionColumnEndDate);
+ keyRange = new HBaseKeyRange(cubeSegment, cuboid, startKey, stopKey, newFuzzyKeys, newFlatOrAndFilter, partitionColumnStartDate, partitionColumnEndDate);
}
return keyRange;
}
@@ -605,7 +610,7 @@ public class CubeStorageEngine implements IStorageEngine {
}
private void setLimit(TupleFilter filter, StorageContext context) {
- boolean goodAggr = context.isExactAggregation() ;
+ boolean goodAggr = context.isExactAggregation();
boolean goodFilter = filter == null || (TupleFilter.isEvaluableRecursively(filter) && context.isCoprocessorEnabled());
boolean goodSort = context.hasSort() == false;
if (goodAggr && goodFilter && goodSort) {
[12/29] incubator-kylin git commit: add a separate setenv.sh for jvm
settings, debug settings and native libarary
Posted by li...@apache.org.
add a separate setenv.sh for jvm settings, debug settings and native libarary
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/a540f402
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/a540f402
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/a540f402
Branch: refs/heads/staging
Commit: a540f4024134f2198c52c0a8dfede10da2a08f39
Parents: 014ef83
Author: honma <ho...@ebay.com>
Authored: Thu Mar 19 16:10:11 2015 -0700
Committer: honma <ho...@ebay.com>
Committed: Thu Mar 19 18:56:21 2015 -0700
----------------------------------------------------------------------
bin/kylin.sh | 16 ++++++++--------
bin/setenv.sh | 36 ++++++++++++++++++++++++++++++++++++
2 files changed, 44 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a540f402/bin/kylin.sh
----------------------------------------------------------------------
diff --git a/bin/kylin.sh b/bin/kylin.sh
index a6983df..232feef 100644
--- a/bin/kylin.sh
+++ b/bin/kylin.sh
@@ -7,12 +7,6 @@ then
tomcat_root=${dir}/../tomcat
export tomcat_root
- #if [ ! -z "$KYLIN_LD_LIBRARY_PATH" ]
- #then
- # echo "KYLIN_LD_LIBRARY_PATH is set to $KYLIN_LD_LIBRARY_PATH"
- #else
- # exit 1
- #fi
#The location of all hadoop/hbase configurations are difficult to get.
#Plus, some of the system properties are secretly set in hadoop/hbase shell command.
@@ -23,7 +17,6 @@ then
#In this way we no longer need to explicitly configure hadoop/hbase related classpath for tomcat,
#hbase command will do all the dirty tasks for us:
- #-Djava.library.path=${KYLIN_LD_LIBRARY_PATH} \
useSandbox=`cat ${KYLIN_HOME}/conf/kylin.properties | grep 'kylin.sandbox' | awk -F '=' '{print $2}'`
@@ -31,12 +24,19 @@ then
if [ "$useSandbox" = "true" ]
then spring_profile="sandbox"
fi
+
+ #retrive $hive_dependency
source ${dir}/find-hive-dependency.sh
+ #retrive $KYLIN_EXTRA_START_OPTS
+ if [ -f "${dir}/setenv.sh" ]
+ then source ${dir}/setenv.sh
+ fi
export HBASE_CLASSPATH_PREFIX=${tomcat_root}/bin/bootstrap.jar:${tomcat_root}/bin/tomcat-juli.jar:${tomcat_root}/lib/*:$HBASE_CLASSPATH_PREFIX
export HBASE_CLASSPATH=$hive_dependency:${HBASE_CLASSPATH}
- hbase -Djava.util.logging.config.file=${tomcat_root}/conf/logging.properties \
+ hbase ${KYLIN_EXTRA_START_OPTS} \
+ -Djava.util.logging.config.file=${tomcat_root}/conf/logging.properties \
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
-Dorg.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true \
-Dorg.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH=true \
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/a540f402/bin/setenv.sh
----------------------------------------------------------------------
diff --git a/bin/setenv.sh b/bin/setenv.sh
new file mode 100755
index 0000000..53f614d
--- /dev/null
+++ b/bin/setenv.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+# uncomment following to for it to take effect
+# export KYLIN_JVM_SETTINGS="-Xms16g -Xmx16g -XX:MaxPermSize=512m -XX:NewSize=3g -XX:MaxNewSize=3g -XX:SurvivorRatio=4 -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:CMSInitiatingOccupancyFraction=70 -XX:+DisableExplicitGC"
+
+# uncomment following to for it to take effect
+# export KYLIN_DEBUG_SETTINGS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:${KYLIN_HOME}/lib/CustomAgent.jar -Dcom.ebay.lnp.rmiagent.port=8080"
+
+# uncomment following to for it to take effect
+# export KYLIN_LD_LIBRARY_SETTINGS="-Djava.library.path=/apache/hadoop/lib/native/Linux-amd64-64"
+
+export KYLIN_EXTRA_START_OPTS=""
+
+if [ ! -z "${KYLIN_JVM_SETTINGS}" ]
+then
+ echo "KYLIN_JVM_SETTINGS is ${KYLIN_JVM_SETTINGS}"
+ KYLIN_EXTRA_START_OPTS="${KYLIN_JVM_SETTINGS} ${KYLIN_EXTRA_START_OPTS}"
+else
+ echo "KYLIN_JVM_SETTINGS is not set, using default jvm settings"
+fi
+
+if [ ! -z "${KYLIN_DEBUG_SETTINGS}" ]
+then
+ echo "KYLIN_DEBUG_SETTINGS is ${KYLIN_DEBUG_SETTINGS}"
+ KYLIN_EXTRA_START_OPTS="${KYLIN_DEBUG_SETTINGS} ${KYLIN_EXTRA_START_OPTS}"
+else
+ echo "KYLIN_DEBUG_SETTINGS is not set, will not enable remote debuging"
+fi
+
+if [ ! -z "${KYLIN_LD_LIBRARY_SETTINGS}" ]
+then
+ echo "KYLIN_LD_LIBRARY_SETTINGS is ${KYLIN_LD_LIBRARY_SETTINGS}"
+ KYLIN_EXTRA_START_OPTS="${KYLIN_LD_LIBRARY_SETTINGS} ${KYLIN_EXTRA_START_OPTS}"
+else
+ echo "KYLIN_LD_LIBRARY_SETTINGS is not set, lzo compression at MR and hbase might not work"
+fi
[09/29] incubator-kylin git commit: fix export result bug
Posted by li...@apache.org.
fix export result bug
Project: http://git-wip-us.apache.org/repos/asf/incubator-kylin/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kylin/commit/cf99527b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kylin/tree/cf99527b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kylin/diff/cf99527b
Branch: refs/heads/staging
Commit: cf99527bd5de7329e6526646921a29f51653a0a3
Parents: 0170668
Author: jiazhong <ji...@ebay.com>
Authored: Thu Mar 19 17:43:53 2015 +0800
Committer: jiazhong <ji...@ebay.com>
Committed: Thu Mar 19 17:43:53 2015 +0800
----------------------------------------------------------------------
webapp/app/partials/query/query_detail.html | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kylin/blob/cf99527b/webapp/app/partials/query/query_detail.html
----------------------------------------------------------------------
diff --git a/webapp/app/partials/query/query_detail.html b/webapp/app/partials/query/query_detail.html
index a4c11bf..7157e9c 100644
--- a/webapp/app/partials/query/query_detail.html
+++ b/webapp/app/partials/query/query_detail.html
@@ -88,7 +88,7 @@
<span ng-if="curQuery.graph.show"><i class="fa fa-list-ul"></i> Grid</span>
</button>
<a class="btn btn-default btn-xs"
- href="{{config.service.url}}query/format/csv?sql={{sanitate(curQuery.sql)}}&project={{state.selectedProject}}" target="_blank"><i
+ href="{{config.service.url}}query/format/csv?sql={{sanitate(curQuery.sql)}}&project={{curQuery.project}}" target="_blank"><i
class="fa fa-cloud-download"></i> Export</a>
<a class="btn btn-default btn-xs"
ng-click="refreshUi();"><i