You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kylin.apache.org by xx...@apache.org on 2020/11/08 08:13:05 UTC
[kylin] 05/13: KYLIN-4801 Kylin continuous integration testing
This is an automated email from the ASF dual-hosted git repository.
xxyu pushed a commit to branch kylin-on-parquet-v2
in repository https://gitbox.apache.org/repos/asf/kylin.git
commit e6f579ef05cf60eaad77662431b9d120a7d77133
Author: yaqian.zhang <59...@qq.com>
AuthorDate: Tue Oct 20 20:32:52 2020 +0800
KYLIN-4801 Kylin continuous integration testing
---
KylinTesting.zip | Bin 0 -> 116536 bytes
build/CI/testing/README.md | 95 +++
.../generic_desc_data/generic_desc_data_3x.json | 682 +++++++++++++++++
.../generic_desc_data/generic_desc_data_4x.json | 625 ++++++++++++++++
build/CI/testing/data/release_test_0001.json | 626 ++++++++++++++++
build/CI/testing/env/default/default.properties | 25 +
build/CI/testing/env/default/python.properties | 4 +
.../specs/authentication/authentication_0001.spec | 18 +
.../read_write_separation.spec | 5 +
build/CI/testing/features/specs/generic_test.spec | 63 ++
build/CI/testing/features/specs/sample.spec | 5 +
.../step_impl/authentication/authentication.py | 37 +
.../CI/testing/features/step_impl/before_suite.py | 44 ++
.../features/step_impl/generic_test_step.py | 98 +++
.../read_write_separation/read_write_separation.py | 0
build/CI/testing/features/step_impl/sample.py | 14 +
.../CI/testing/kylin_instances/kylin_instance.yml | 7 +
build/CI/testing/kylin_utils/basic.py | 90 +++
build/CI/testing/kylin_utils/equals.py | 204 +++++
build/CI/testing/kylin_utils/kylin.py | 826 +++++++++++++++++++++
build/CI/testing/kylin_utils/shell.py | 125 ++++
build/CI/testing/kylin_utils/util.py | 64 ++
build/CI/testing/manifest.json | 6 +
build/CI/testing/requirements.txt | 6 +
24 files changed, 3669 insertions(+)
diff --git a/KylinTesting.zip b/KylinTesting.zip
new file mode 100644
index 0000000..b3c46a4
Binary files /dev/null and b/KylinTesting.zip differ
diff --git a/build/CI/testing/README.md b/build/CI/testing/README.md
new file mode 100644
index 0000000..c2936dc
--- /dev/null
+++ b/build/CI/testing/README.md
@@ -0,0 +1,95 @@
+# kylin-test
+Automated test code repo based on [gauge](https://docs.gauge.org/?os=macos&language=python&ide=vscode) for [Apache Kylin](https://github.com/apache/kylin).
+
+### IDE
+Gauge support IntelliJ IDEA and VSCode as development IDE.
+However, IDEA cannot detect the step implementation method of Python language, just support java.
+VSCode is recommended as the development IDE.
+
+### Clone repo
+```
+git clone https://github.com/zhangayqian/kylin-test
+```
+
+### Prepare environment
+ * Install python3 compiler and version 3.6 recommended
+ * Install gauge
+ ```
+ brew install gauge
+ ```
+ If you encounter the below error:
+ ```
+ Download failed: https://homebrew.bintray.com/bottles/gauge- 1.1.1.mojave.bottle.1.tar.gz
+ ```
+ You can try to download the compressed package manually, put it in the downloads directory of homebrew cache directory, and execute the installation command of gauge again.
+
+* Install required dependencies
+```
+pip install -r requirements.txt
+```
+
+## Directory structure
+* features/specs: Directory of specification file.
+ A specification is a business test case which describes a particular feature of the application that needs testing. Gauge specifications support a .spec or .md file format and these specifications are written in a syntax similar to Markdown.
+
+* features/step_impl: Directory of Step implementations methods.
+ Every step implementation has an equivalent code as per the language plugin used while installing Gauge. The code is run when the steps inside a spec are executed. The code must have the same number of parameters as mentioned in the step.
+ Steps can be implemented in different ways such as simple step, step with table, step alias, and enum data type used as step parameters.
+
+* data: Directory of data files needed to execute test cases. Such as cube_desc.json.
+
+* env/default: Gauge configuration file directory.
+
+* kylin_instance: Kylin instance configuration file directory.
+
+* kylin_utils: Tools method directory.
+
+## Run Gauge specifications
+* Run all specification
+```
+gauge run
+```
+* Run specification or step or spec according tags, such as:
+```
+gauge run --tags 3.x
+```
+* Please refer to https://docs.gauge.org/execution.html?os=macos&language=python&ide=vscode learn more.
+
+## Tips
+
+A specification consists of different sections; some of which are mandatory and few are optional. The components of a specification are listed as follows:
+
+- Specification heading
+- Scenario
+- Step
+- Parameters
+- Tags
+- Comments
+
+#### Note
+
+Tags - optional, executable component when the specification is run
+Comments - optional, non-executable component when the specification is run
+
+### About tags
+
+Here, we stipulate that all test scenarios should have tags. Mandatory tags include 3.x and 4.x to indicate which versions are supported by the test scenario. Such as:
+```
+# Flink Engine
+Tags:3.x
+```
+```
+# Cube management
+Tags:3.x,4.x
+```
+You can put the tag in the specification heading, so that all scenarios in this specification will have this tag.
+You can also tag your own test spec to make it easier for you to run your own test cases.
+
+### About Project
+There are two project names already occupied, they are `generic_test_project` and `pushdown_test_project`.
+
+Every time you run this test, @befroe_suit method will be execute in advance to create `generic_test_project`. And the model and cube in this project are universal, and the cube has been fully built. They include dimensions and measures as much as possible. When you need to use a built cube to perform tests, you may use it.
+
+`pushdown_test_project` used to compare sql query result. This is a empty project.
+
+Please refer to https://docs.gauge.org/writing-specifications.html?os=macos&language=python&ide=vscode learn more.
diff --git a/build/CI/testing/data/generic_desc_data/generic_desc_data_3x.json b/build/CI/testing/data/generic_desc_data/generic_desc_data_3x.json
new file mode 100644
index 0000000..72ca5a4
--- /dev/null
+++ b/build/CI/testing/data/generic_desc_data/generic_desc_data_3x.json
@@ -0,0 +1,682 @@
+{
+ "load_table_list":
+ "DEFAULT.KYLIN_SALES,DEFAULT.KYLIN_CAL_DT,DEFAULT.KYLIN_CATEGORY_GROUPINGS,DEFAULT.KYLIN_ACCOUNT,DEFAULT.KYLIN_COUNTRY",
+
+ "model_desc_data":
+ {
+ "uuid": "0928468a-9fab-4185-9a14-6f2e7c74823f",
+ "last_modified": 0,
+ "version": "3.0.0.20500",
+ "name": "generic_test_model",
+ "owner": null,
+ "is_draft": false,
+ "description": "",
+ "fact_table": "DEFAULT.KYLIN_SALES",
+ "lookups": [
+ {
+ "table": "DEFAULT.KYLIN_CAL_DT",
+ "kind": "LOOKUP",
+ "alias": "KYLIN_CAL_DT",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "KYLIN_CAL_DT.CAL_DT"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.PART_DT"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_CATEGORY_GROUPINGS",
+ "kind": "LOOKUP",
+ "alias": "KYLIN_CATEGORY_GROUPINGS",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID",
+ "KYLIN_CATEGORY_GROUPINGS.SITE_ID"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.LEAF_CATEG_ID",
+ "KYLIN_SALES.LSTG_SITE_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_ACCOUNT",
+ "kind": "LOOKUP",
+ "alias": "BUYER_ACCOUNT",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "BUYER_ACCOUNT.ACCOUNT_ID"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.BUYER_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_ACCOUNT",
+ "kind": "LOOKUP",
+ "alias": "SELLER_ACCOUNT",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "SELLER_ACCOUNT.ACCOUNT_ID"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.SELLER_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_COUNTRY",
+ "kind": "LOOKUP",
+ "alias": "BUYER_COUNTRY",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "BUYER_COUNTRY.COUNTRY"
+ ],
+ "foreign_key": [
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_COUNTRY",
+ "kind": "LOOKUP",
+ "alias": "SELLER_COUNTRY",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "SELLER_COUNTRY.COUNTRY"
+ ],
+ "foreign_key": [
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY"
+ ]
+ }
+ }
+ ],
+ "dimensions": [
+ {
+ "table": "KYLIN_SALES",
+ "columns": [
+ "TRANS_ID",
+ "SELLER_ID",
+ "BUYER_ID",
+ "PART_DT",
+ "LEAF_CATEG_ID",
+ "LSTG_FORMAT_NAME",
+ "LSTG_SITE_ID",
+ "OPS_USER_ID",
+ "OPS_REGION"
+ ]
+ },
+ {
+ "table": "KYLIN_CAL_DT",
+ "columns": [
+ "CAL_DT",
+ "WEEK_BEG_DT",
+ "MONTH_BEG_DT",
+ "YEAR_BEG_DT"
+ ]
+ },
+ {
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "columns": [
+ "USER_DEFINED_FIELD1",
+ "USER_DEFINED_FIELD3",
+ "META_CATEG_NAME",
+ "CATEG_LVL2_NAME",
+ "CATEG_LVL3_NAME",
+ "LEAF_CATEG_ID",
+ "SITE_ID"
+ ]
+ },
+ {
+ "table": "BUYER_ACCOUNT",
+ "columns": [
+ "ACCOUNT_ID",
+ "ACCOUNT_BUYER_LEVEL",
+ "ACCOUNT_SELLER_LEVEL",
+ "ACCOUNT_COUNTRY",
+ "ACCOUNT_CONTACT"
+ ]
+ },
+ {
+ "table": "SELLER_ACCOUNT",
+ "columns": [
+ "ACCOUNT_ID",
+ "ACCOUNT_BUYER_LEVEL",
+ "ACCOUNT_SELLER_LEVEL",
+ "ACCOUNT_COUNTRY",
+ "ACCOUNT_CONTACT"
+ ]
+ },
+ {
+ "table": "BUYER_COUNTRY",
+ "columns": [
+ "COUNTRY",
+ "NAME"
+ ]
+ },
+ {
+ "table": "SELLER_COUNTRY",
+ "columns": [
+ "COUNTRY",
+ "NAME"
+ ]
+ }
+ ],
+ "metrics": [
+ "KYLIN_SALES.PRICE",
+ "KYLIN_SALES.ITEM_COUNT"
+ ],
+ "filter_condition": "",
+ "partition_desc": {
+ "partition_date_column": "KYLIN_SALES.PART_DT",
+ "partition_time_column": null,
+ "partition_date_start": 0,
+ "partition_date_format": "yyyy-MM-dd HH:mm:ss",
+ "partition_time_format": "HH:mm:ss",
+ "partition_type": "APPEND",
+ "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
+ },
+ "capacity": "MEDIUM",
+ "projectName": "generic_test_project"
+ },
+ "cube_desc_data":
+ {
+ "uuid": "02669388-8b98-591a-9fb7-9addcdb2da57",
+ "last_modified": 0,
+ "version": "3.0.0.20500",
+ "name": "generic_test_cube",
+ "is_draft": false,
+ "model_name": "generic_test_model",
+ "description": "",
+ "null_string": null,
+ "dimensions": [
+ {
+ "name": "TRANS_ID",
+ "table": "KYLIN_SALES",
+ "column": "TRANS_ID",
+ "derived": null
+ },
+ {
+ "name": "YEAR_BEG_DT",
+ "table": "KYLIN_CAL_DT",
+ "column": null,
+ "derived": [
+ "YEAR_BEG_DT"
+ ]
+ },
+ {
+ "name": "MONTH_BEG_DT",
+ "table": "KYLIN_CAL_DT",
+ "column": null,
+ "derived": [
+ "MONTH_BEG_DT"
+ ]
+ },
+ {
+ "name": "WEEK_BEG_DT",
+ "table": "KYLIN_CAL_DT",
+ "column": null,
+ "derived": [
+ "WEEK_BEG_DT"
+ ]
+ },
+ {
+ "name": "USER_DEFINED_FIELD1",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": null,
+ "derived": [
+ "USER_DEFINED_FIELD1"
+ ]
+ },
+ {
+ "name": "USER_DEFINED_FIELD3",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": null,
+ "derived": [
+ "USER_DEFINED_FIELD3"
+ ]
+ },
+ {
+ "name": "META_CATEG_NAME",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": "META_CATEG_NAME",
+ "derived": null
+ },
+ {
+ "name": "CATEG_LVL2_NAME",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": "CATEG_LVL2_NAME",
+ "derived": null
+ },
+ {
+ "name": "CATEG_LVL3_NAME",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": "CATEG_LVL3_NAME",
+ "derived": null
+ },
+ {
+ "name": "LSTG_FORMAT_NAME",
+ "table": "KYLIN_SALES",
+ "column": "LSTG_FORMAT_NAME",
+ "derived": null
+ },
+ {
+ "name": "SELLER_ID",
+ "table": "KYLIN_SALES",
+ "column": "SELLER_ID",
+ "derived": null
+ },
+ {
+ "name": "BUYER_ID",
+ "table": "KYLIN_SALES",
+ "column": "BUYER_ID",
+ "derived": null
+ },
+ {
+ "name": "ACCOUNT_BUYER_LEVEL",
+ "table": "BUYER_ACCOUNT",
+ "column": "ACCOUNT_BUYER_LEVEL",
+ "derived": null
+ },
+ {
+ "name": "ACCOUNT_SELLER_LEVEL",
+ "table": "SELLER_ACCOUNT",
+ "column": "ACCOUNT_SELLER_LEVEL",
+ "derived": null
+ },
+ {
+ "name": "BUYER_COUNTRY",
+ "table": "BUYER_ACCOUNT",
+ "column": "ACCOUNT_COUNTRY",
+ "derived": null
+ },
+ {
+ "name": "SELLER_COUNTRY",
+ "table": "SELLER_ACCOUNT",
+ "column": "ACCOUNT_COUNTRY",
+ "derived": null
+ },
+ {
+ "name": "BUYER_COUNTRY_NAME",
+ "table": "BUYER_COUNTRY",
+ "column": "NAME",
+ "derived": null
+ },
+ {
+ "name": "SELLER_COUNTRY_NAME",
+ "table": "SELLER_COUNTRY",
+ "column": "NAME",
+ "derived": null
+ },
+ {
+ "name": "OPS_USER_ID",
+ "table": "KYLIN_SALES",
+ "column": "OPS_USER_ID",
+ "derived": null
+ },
+ {
+ "name": "OPS_REGION",
+ "table": "KYLIN_SALES",
+ "column": "OPS_REGION",
+ "derived": null
+ }
+ ],
+ "measures": [
+ {
+ "name": "GMV_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.PRICE"
+ },
+ "returntype": "decimal(19,4)"
+ }
+ },
+ {
+ "name": "BUYER_LEVEL_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "SELLER_LEVEL_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "TRANS_CNT",
+ "function": {
+ "expression": "COUNT",
+ "parameter": {
+ "type": "constant",
+ "value": "1"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "SELLER_CNT_HLL",
+ "function": {
+ "expression": "COUNT_DISTINCT",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.SELLER_ID"
+ },
+ "returntype": "hllc(10)"
+ }
+ },
+ {
+ "name": "TOP_SELLER",
+ "function": {
+ "expression": "TOP_N",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.PRICE",
+ "next_parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.SELLER_ID"
+ }
+ },
+ "returntype": "topn(100,4)",
+ "configuration": {
+ "topn.encoding.KYLIN_SALES.SELLER_ID": "dict",
+ "topn.encoding_version.KYLIN_SALES.SELLER_ID": "1"
+ }
+ }
+ },
+ {
+ "name": "BUYER_CNT_BITMAP",
+ "function": {
+ "expression": "COUNT_DISTINCT",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.BUYER_ID"
+ },
+ "returntype": "bitmap"
+ }
+ },
+ {
+ "name": "MIN_PRICE",
+ "function": {
+ "expression": "MIN",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.PRICE"
+ },
+ "returntype": "decimal(19,4)"
+ }
+ },
+ {
+ "name": "MAX_PRICE",
+ "function": {
+ "expression": "MAX",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.PRICE"
+ },
+ "returntype": "decimal(19,4)"
+ }
+ },
+ {
+ "name": "PERCENTILE_PRICE",
+ "function": {
+ "expression": "PERCENTILE_APPROX",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.PRICE"
+ },
+ "returntype": "percentile(100)"
+ }
+ }
+ ],
+ "dictionaries": [
+ {
+ "column": "KYLIN_SALES.BUYER_ID",
+ "builder": "org.apache.kylin.dict.GlobalDictionaryBuilder",
+ "cube": null,
+ "model": null
+ }
+ ],
+ "rowkey": {
+ "rowkey_columns": [
+ {
+ "column": "KYLIN_SALES.BUYER_ID",
+ "encoding": "integer:4",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.SELLER_ID",
+ "encoding": "integer:4",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.TRANS_ID",
+ "encoding": "integer:4",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.PART_DT",
+ "encoding": "date",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.LEAF_CATEG_ID",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "BUYER_COUNTRY.NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "SELLER_COUNTRY.NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.LSTG_FORMAT_NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.LSTG_SITE_ID",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.OPS_USER_ID",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.OPS_REGION",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ }
+ ]
+ },
+ "hbase_mapping": {
+ "column_family": [
+ {
+ "name": "F1",
+ "columns": [
+ {
+ "qualifier": "M",
+ "measure_refs": [
+ "GMV_SUM",
+ "BUYER_LEVEL_SUM",
+ "SELLER_LEVEL_SUM",
+ "TRANS_CNT",
+ "TOP_SELLER",
+ "MIN_PRICE",
+ "MAX_PRICE",
+ "PERCENTILE_PRICE"
+ ]
+ }
+ ]
+ },
+ {
+ "name": "F2",
+ "columns": [
+ {
+ "qualifier": "M",
+ "measure_refs": [
+ "SELLER_CNT_HLL",
+ "BUYER_CNT_BITMAP"
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "aggregation_groups": [
+ {
+ "includes": [
+ "KYLIN_SALES.PART_DT",
+ "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "KYLIN_SALES.LEAF_CATEG_ID",
+ "KYLIN_SALES.LSTG_FORMAT_NAME",
+ "KYLIN_SALES.LSTG_SITE_ID",
+ "KYLIN_SALES.OPS_USER_ID",
+ "KYLIN_SALES.OPS_REGION",
+ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "BUYER_COUNTRY.NAME",
+ "SELLER_COUNTRY.NAME"
+ ],
+ "select_rule": {
+ "hierarchy_dims": [
+ [
+ "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "KYLIN_SALES.LEAF_CATEG_ID"
+ ]
+ ],
+ "mandatory_dims": [
+ "KYLIN_SALES.PART_DT"
+ ],
+ "joint_dims": [
+ [
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "BUYER_COUNTRY.NAME"
+ ],
+ [
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "SELLER_COUNTRY.NAME"
+ ],
+ [
+ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
+ ],
+ [
+ "KYLIN_SALES.LSTG_FORMAT_NAME",
+ "KYLIN_SALES.LSTG_SITE_ID"
+ ],
+ [
+ "KYLIN_SALES.OPS_USER_ID",
+ "KYLIN_SALES.OPS_REGION"
+ ]
+ ]
+ }
+ }
+ ],
+ "signature": "HpTV7qyxn6IDFiRbFdsb5g==",
+ "notify_list": [],
+ "status_need_notify": [],
+ "partition_date_start": 0,
+ "partition_date_end": 3153600000000,
+ "auto_merge_time_ranges": [],
+ "volatile_range": 0,
+ "retention_range": 0,
+ "engine_type": 2,
+ "storage_type": 2,
+ "override_kylin_properties": {
+ "kylin.cube.aggrgroup.is-mandatory-only-valid": "true",
+ "kylin.engine.spark.rdd-partition-cut-mb": "500"
+ },
+ "cuboid_black_list": [],
+ "parent_forward": 3,
+ "mandatory_dimension_set_list": [],
+ "snapshot_table_desc_list": []
+ }
+}
\ No newline at end of file
diff --git a/build/CI/testing/data/generic_desc_data/generic_desc_data_4x.json b/build/CI/testing/data/generic_desc_data/generic_desc_data_4x.json
new file mode 100644
index 0000000..8d533b5
--- /dev/null
+++ b/build/CI/testing/data/generic_desc_data/generic_desc_data_4x.json
@@ -0,0 +1,625 @@
+{
+ "load_table_list":
+ "DEFAULT.KYLIN_SALES,DEFAULT.KYLIN_CAL_DT,DEFAULT.KYLIN_CATEGORY_GROUPINGS,DEFAULT.KYLIN_ACCOUNT,DEFAULT.KYLIN_COUNTRY",
+
+ "model_desc_data":
+ {
+ "uuid": "0928468a-9fab-4185-9a14-6f2e7c74823f",
+ "last_modified": 0,
+ "version": "3.0.0.20500",
+ "name": "generic_test_model",
+ "owner": null,
+ "is_draft": false,
+ "description": "",
+ "fact_table": "DEFAULT.KYLIN_SALES",
+ "lookups": [
+ {
+ "table": "DEFAULT.KYLIN_CAL_DT",
+ "kind": "LOOKUP",
+ "alias": "KYLIN_CAL_DT",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "KYLIN_CAL_DT.CAL_DT"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.PART_DT"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_CATEGORY_GROUPINGS",
+ "kind": "LOOKUP",
+ "alias": "KYLIN_CATEGORY_GROUPINGS",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID",
+ "KYLIN_CATEGORY_GROUPINGS.SITE_ID"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.LEAF_CATEG_ID",
+ "KYLIN_SALES.LSTG_SITE_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_ACCOUNT",
+ "kind": "LOOKUP",
+ "alias": "BUYER_ACCOUNT",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "BUYER_ACCOUNT.ACCOUNT_ID"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.BUYER_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_ACCOUNT",
+ "kind": "LOOKUP",
+ "alias": "SELLER_ACCOUNT",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "SELLER_ACCOUNT.ACCOUNT_ID"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.SELLER_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_COUNTRY",
+ "kind": "LOOKUP",
+ "alias": "BUYER_COUNTRY",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "BUYER_COUNTRY.COUNTRY"
+ ],
+ "foreign_key": [
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_COUNTRY",
+ "kind": "LOOKUP",
+ "alias": "SELLER_COUNTRY",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "SELLER_COUNTRY.COUNTRY"
+ ],
+ "foreign_key": [
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY"
+ ]
+ }
+ }
+ ],
+ "dimensions": [
+ {
+ "table": "KYLIN_SALES",
+ "columns": [
+ "TRANS_ID",
+ "SELLER_ID",
+ "BUYER_ID",
+ "PART_DT",
+ "LEAF_CATEG_ID",
+ "LSTG_FORMAT_NAME",
+ "LSTG_SITE_ID",
+ "OPS_USER_ID",
+ "OPS_REGION"
+ ]
+ },
+ {
+ "table": "KYLIN_CAL_DT",
+ "columns": [
+ "CAL_DT",
+ "WEEK_BEG_DT",
+ "MONTH_BEG_DT",
+ "YEAR_BEG_DT"
+ ]
+ },
+ {
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "columns": [
+ "USER_DEFINED_FIELD1",
+ "USER_DEFINED_FIELD3",
+ "META_CATEG_NAME",
+ "CATEG_LVL2_NAME",
+ "CATEG_LVL3_NAME",
+ "LEAF_CATEG_ID",
+ "SITE_ID"
+ ]
+ },
+ {
+ "table": "BUYER_ACCOUNT",
+ "columns": [
+ "ACCOUNT_ID",
+ "ACCOUNT_BUYER_LEVEL",
+ "ACCOUNT_SELLER_LEVEL",
+ "ACCOUNT_COUNTRY",
+ "ACCOUNT_CONTACT"
+ ]
+ },
+ {
+ "table": "SELLER_ACCOUNT",
+ "columns": [
+ "ACCOUNT_ID",
+ "ACCOUNT_BUYER_LEVEL",
+ "ACCOUNT_SELLER_LEVEL",
+ "ACCOUNT_COUNTRY",
+ "ACCOUNT_CONTACT"
+ ]
+ },
+ {
+ "table": "BUYER_COUNTRY",
+ "columns": [
+ "COUNTRY",
+ "NAME"
+ ]
+ },
+ {
+ "table": "SELLER_COUNTRY",
+ "columns": [
+ "COUNTRY",
+ "NAME"
+ ]
+ }
+ ],
+ "metrics": [
+ "KYLIN_SALES.PRICE",
+ "KYLIN_SALES.ITEM_COUNT"
+ ],
+ "filter_condition": "",
+ "partition_desc": {
+ "partition_date_column": "KYLIN_SALES.PART_DT",
+ "partition_time_column": null,
+ "partition_date_start": 0,
+ "partition_date_format": "yyyy-MM-dd HH:mm:ss",
+ "partition_time_format": "HH:mm:ss",
+ "partition_type": "APPEND",
+ "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
+ },
+ "capacity": "MEDIUM",
+ "projectName": "generic_test_project"
+ },
+ "cube_desc_data":
+ {
+ "uuid": "b1c89f5b-5346-05db-0b82-8851ccb72737",
+ "last_modified": 0,
+ "version": "3.0.0.20500",
+ "name": "generic_test_cube",
+ "is_draft": false,
+ "model_name": "generic_test_model",
+ "description": "",
+ "null_string": null,
+ "dimensions": [
+ {
+ "name": "TRANS_ID",
+ "table": "KYLIN_SALES",
+ "column": "TRANS_ID",
+ "derived": null
+ },
+ {
+ "name": "YEAR_BEG_DT",
+ "table": "KYLIN_CAL_DT",
+ "column": null,
+ "derived": [
+ "YEAR_BEG_DT"
+ ]
+ },
+ {
+ "name": "MONTH_BEG_DT",
+ "table": "KYLIN_CAL_DT",
+ "column": null,
+ "derived": [
+ "MONTH_BEG_DT"
+ ]
+ },
+ {
+ "name": "WEEK_BEG_DT",
+ "table": "KYLIN_CAL_DT",
+ "column": null,
+ "derived": [
+ "WEEK_BEG_DT"
+ ]
+ },
+ {
+ "name": "USER_DEFINED_FIELD1",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": null,
+ "derived": [
+ "USER_DEFINED_FIELD1"
+ ]
+ },
+ {
+ "name": "USER_DEFINED_FIELD3",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": null,
+ "derived": [
+ "USER_DEFINED_FIELD3"
+ ]
+ },
+ {
+ "name": "META_CATEG_NAME",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": "META_CATEG_NAME",
+ "derived": null
+ },
+ {
+ "name": "CATEG_LVL2_NAME",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": "CATEG_LVL2_NAME",
+ "derived": null
+ },
+ {
+ "name": "CATEG_LVL3_NAME",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": "CATEG_LVL3_NAME",
+ "derived": null
+ },
+ {
+ "name": "LSTG_FORMAT_NAME",
+ "table": "KYLIN_SALES",
+ "column": "LSTG_FORMAT_NAME",
+ "derived": null
+ },
+ {
+ "name": "SELLER_ID",
+ "table": "KYLIN_SALES",
+ "column": "SELLER_ID",
+ "derived": null
+ },
+ {
+ "name": "BUYER_ID",
+ "table": "KYLIN_SALES",
+ "column": "BUYER_ID",
+ "derived": null
+ },
+ {
+ "name": "ACCOUNT_BUYER_LEVEL",
+ "table": "BUYER_ACCOUNT",
+ "column": "ACCOUNT_BUYER_LEVEL",
+ "derived": null
+ },
+ {
+ "name": "ACCOUNT_SELLER_LEVEL",
+ "table": "SELLER_ACCOUNT",
+ "column": "ACCOUNT_SELLER_LEVEL",
+ "derived": null
+ },
+ {
+ "name": "BUYER_COUNTRY",
+ "table": "BUYER_ACCOUNT",
+ "column": "ACCOUNT_COUNTRY",
+ "derived": null
+ },
+ {
+ "name": "SELLER_COUNTRY",
+ "table": "SELLER_ACCOUNT",
+ "column": "ACCOUNT_COUNTRY",
+ "derived": null
+ },
+ {
+ "name": "BUYER_COUNTRY_NAME",
+ "table": "BUYER_COUNTRY",
+ "column": "NAME",
+ "derived": null
+ },
+ {
+ "name": "SELLER_COUNTRY_NAME",
+ "table": "SELLER_COUNTRY",
+ "column": "NAME",
+ "derived": null
+ },
+ {
+ "name": "OPS_USER_ID",
+ "table": "KYLIN_SALES",
+ "column": "OPS_USER_ID",
+ "derived": null
+ },
+ {
+ "name": "OPS_REGION",
+ "table": "KYLIN_SALES",
+ "column": "OPS_REGION",
+ "derived": null
+ }
+ ],
+ "measures": [
+ {
+ "name": "GMV_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.PRICE"
+ },
+ "returntype": "decimal(19,4)"
+ }
+ },
+ {
+ "name": "BUYER_LEVEL_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "SELLER_LEVEL_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "TRANS_CNT",
+ "function": {
+ "expression": "COUNT",
+ "parameter": {
+ "type": "constant",
+ "value": "1"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "TOP_SELLER",
+ "function": {
+ "expression": "TOP_N",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.PRICE",
+ "next_parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.SELLER_ID"
+ }
+ },
+ "returntype": "topn(100,4)",
+ "configuration": {
+ "topn.encoding.KYLIN_SALES.SELLER_ID": "dict",
+ "topn.encoding_version.KYLIN_SALES.SELLER_ID": "1"
+ }
+ }
+ },
+ {
+ "name": "SELLER_CNT_HLL",
+ "function": {
+ "expression": "COUNT_DISTINCT",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.SELLER_ID"
+ },
+ "returntype": "hllc(10)"
+ }
+ },
+ {
+ "name": "BUYER_CNT_BITMAP",
+ "function": {
+ "expression": "COUNT_DISTINCT",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.BUYER_ID"
+ },
+ "returntype": "bitmap"
+ }
+ }
+ ],
+ "dictionaries": [
+ {
+ "column": "KYLIN_SALES.BUYER_ID",
+ "builder": "org.apache.kylin.dict.GlobalDictionaryBuilder",
+ "cube": null,
+ "model": null
+ }
+ ],
+ "rowkey": {
+ "rowkey_columns": [
+ {
+ "column": "KYLIN_SALES.BUYER_ID",
+ "encoding": "integer:4",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.SELLER_ID",
+ "encoding": "integer:4",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.TRANS_ID",
+ "encoding": "integer:4",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.PART_DT",
+ "encoding": "date",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.LEAF_CATEG_ID",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "BUYER_COUNTRY.NAME",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "SELLER_COUNTRY.NAME",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.LSTG_FORMAT_NAME",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.LSTG_SITE_ID",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.OPS_USER_ID",
+ "encoding": "dict",
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.OPS_REGION",
+ "encoding": "dict",
+ "isShardBy": false
+ }
+ ]
+ },
+ "hbase_mapping": {
+ "column_family": [
+ {
+ "name": "F1",
+ "columns": [
+ {
+ "qualifier": "M",
+ "measure_refs": [
+ "GMV_SUM",
+ "BUYER_LEVEL_SUM",
+ "SELLER_LEVEL_SUM",
+ "TRANS_CNT",
+ "TOP_SELLER"
+ ]
+ }
+ ]
+ },
+ {
+ "name": "F2",
+ "columns": [
+ {
+ "qualifier": "M",
+ "measure_refs": [
+ "SELLER_CNT_HLL",
+ "BUYER_CNT_BITMAP"
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "aggregation_groups": [
+ {
+ "includes": [
+ "KYLIN_SALES.PART_DT",
+ "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "KYLIN_SALES.LEAF_CATEG_ID",
+ "KYLIN_SALES.LSTG_FORMAT_NAME",
+ "KYLIN_SALES.LSTG_SITE_ID",
+ "KYLIN_SALES.OPS_USER_ID",
+ "KYLIN_SALES.OPS_REGION",
+ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "BUYER_COUNTRY.NAME",
+ "SELLER_COUNTRY.NAME"
+ ],
+ "select_rule": {
+ "hierarchy_dims": [
+ [
+ "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "KYLIN_SALES.LEAF_CATEG_ID"
+ ]
+ ],
+ "mandatory_dims": [
+ "KYLIN_SALES.PART_DT"
+ ],
+ "joint_dims": [
+ [
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "BUYER_COUNTRY.NAME"
+ ],
+ [
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "SELLER_COUNTRY.NAME"
+ ],
+ [
+ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
+ ],
+ [
+ "KYLIN_SALES.LSTG_FORMAT_NAME",
+ "KYLIN_SALES.LSTG_SITE_ID"
+ ],
+ [
+ "KYLIN_SALES.OPS_USER_ID",
+ "KYLIN_SALES.OPS_REGION"
+ ]
+ ]
+ }
+ }
+ ],
+ "signature": "vbkxiXn2AOQm8zdkfY1kSw==",
+ "notify_list": [],
+ "status_need_notify": [],
+ "partition_date_start": 0,
+ "partition_date_end": 3153600000000,
+ "auto_merge_time_ranges": [],
+ "volatile_range": 0,
+ "retention_range": 0,
+ "engine_type": 6,
+ "storage_type": 4,
+ "override_kylin_properties": {},
+ "cuboid_black_list": [],
+ "parent_forward": 3,
+ "mandatory_dimension_set_list": [],
+ "snapshot_table_desc_list": []
+ }
+ }
\ No newline at end of file
diff --git a/build/CI/testing/data/release_test_0001.json b/build/CI/testing/data/release_test_0001.json
new file mode 100644
index 0000000..0b5558a
--- /dev/null
+++ b/build/CI/testing/data/release_test_0001.json
@@ -0,0 +1,626 @@
+{
+ "load_table_list":
+ "DEFAULT.KYLIN_SALES,DEFAULT.KYLIN_CAL_DT,DEFAULT.KYLIN_CATEGORY_GROUPINGS,DEFAULT.KYLIN_ACCOUNT,DEFAULT.KYLIN_COUNTRY",
+
+ "model_desc_data":
+ {
+ "uuid": "0928468a-9fab-4185-9a14-6f2e7c74823f",
+ "last_modified": 0,
+ "version": "3.0.0.20500",
+ "name": "release_test_0001_model",
+ "owner": null,
+ "is_draft": false,
+ "description": "",
+ "fact_table": "DEFAULT.KYLIN_SALES",
+ "lookups": [
+ {
+ "table": "DEFAULT.KYLIN_CAL_DT",
+ "kind": "LOOKUP",
+ "alias": "KYLIN_CAL_DT",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "KYLIN_CAL_DT.CAL_DT"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.PART_DT"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_CATEGORY_GROUPINGS",
+ "kind": "LOOKUP",
+ "alias": "KYLIN_CATEGORY_GROUPINGS",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID",
+ "KYLIN_CATEGORY_GROUPINGS.SITE_ID"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.LEAF_CATEG_ID",
+ "KYLIN_SALES.LSTG_SITE_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_ACCOUNT",
+ "kind": "LOOKUP",
+ "alias": "BUYER_ACCOUNT",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "BUYER_ACCOUNT.ACCOUNT_ID"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.BUYER_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_ACCOUNT",
+ "kind": "LOOKUP",
+ "alias": "SELLER_ACCOUNT",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "SELLER_ACCOUNT.ACCOUNT_ID"
+ ],
+ "foreign_key": [
+ "KYLIN_SALES.SELLER_ID"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_COUNTRY",
+ "kind": "LOOKUP",
+ "alias": "BUYER_COUNTRY",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "BUYER_COUNTRY.COUNTRY"
+ ],
+ "foreign_key": [
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY"
+ ]
+ }
+ },
+ {
+ "table": "DEFAULT.KYLIN_COUNTRY",
+ "kind": "LOOKUP",
+ "alias": "SELLER_COUNTRY",
+ "join": {
+ "type": "inner",
+ "primary_key": [
+ "SELLER_COUNTRY.COUNTRY"
+ ],
+ "foreign_key": [
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY"
+ ]
+ }
+ }
+ ],
+ "dimensions": [
+ {
+ "table": "KYLIN_SALES",
+ "columns": [
+ "TRANS_ID",
+ "SELLER_ID",
+ "BUYER_ID",
+ "PART_DT",
+ "LEAF_CATEG_ID",
+ "LSTG_FORMAT_NAME",
+ "LSTG_SITE_ID",
+ "OPS_USER_ID",
+ "OPS_REGION"
+ ]
+ },
+ {
+ "table": "KYLIN_CAL_DT",
+ "columns": [
+ "CAL_DT",
+ "WEEK_BEG_DT",
+ "MONTH_BEG_DT",
+ "YEAR_BEG_DT"
+ ]
+ },
+ {
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "columns": [
+ "USER_DEFINED_FIELD1",
+ "USER_DEFINED_FIELD3",
+ "META_CATEG_NAME",
+ "CATEG_LVL2_NAME",
+ "CATEG_LVL3_NAME",
+ "LEAF_CATEG_ID",
+ "SITE_ID"
+ ]
+ },
+ {
+ "table": "BUYER_ACCOUNT",
+ "columns": [
+ "ACCOUNT_ID",
+ "ACCOUNT_BUYER_LEVEL",
+ "ACCOUNT_SELLER_LEVEL",
+ "ACCOUNT_COUNTRY",
+ "ACCOUNT_CONTACT"
+ ]
+ },
+ {
+ "table": "SELLER_ACCOUNT",
+ "columns": [
+ "ACCOUNT_ID",
+ "ACCOUNT_BUYER_LEVEL",
+ "ACCOUNT_SELLER_LEVEL",
+ "ACCOUNT_COUNTRY",
+ "ACCOUNT_CONTACT"
+ ]
+ },
+ {
+ "table": "BUYER_COUNTRY",
+ "columns": [
+ "COUNTRY",
+ "NAME"
+ ]
+ },
+ {
+ "table": "SELLER_COUNTRY",
+ "columns": [
+ "COUNTRY",
+ "NAME"
+ ]
+ }
+ ],
+ "metrics": [
+ "KYLIN_SALES.PRICE",
+ "KYLIN_SALES.ITEM_COUNT"
+ ],
+ "filter_condition": "",
+ "partition_desc": {
+ "partition_date_column": "KYLIN_SALES.PART_DT",
+ "partition_time_column": null,
+ "partition_date_start": 0,
+ "partition_date_format": "yyyy-MM-dd HH:mm:ss",
+ "partition_time_format": "HH:mm:ss",
+ "partition_type": "APPEND",
+ "partition_condition_builder": "org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
+ },
+ "capacity": "MEDIUM",
+ "projectName": null
+ },
+ "cube_desc_data":
+ {
+ "uuid": "0ef9b7a8-3929-4dff-b59d-2100aadc8dbf",
+ "last_modified": 0,
+ "version": "3.0.0.20500",
+ "name": "release_test_0001_cube",
+ "is_draft": false,
+ "model_name": "release_test_0001_model",
+ "description": "",
+ "null_string": null,
+ "dimensions": [
+ {
+ "name": "TRANS_ID",
+ "table": "KYLIN_SALES",
+ "column": "TRANS_ID",
+ "derived": null
+ },
+ {
+ "name": "YEAR_BEG_DT",
+ "table": "KYLIN_CAL_DT",
+ "column": null,
+ "derived": [
+ "YEAR_BEG_DT"
+ ]
+ },
+ {
+ "name": "MONTH_BEG_DT",
+ "table": "KYLIN_CAL_DT",
+ "column": null,
+ "derived": [
+ "MONTH_BEG_DT"
+ ]
+ },
+ {
+ "name": "WEEK_BEG_DT",
+ "table": "KYLIN_CAL_DT",
+ "column": null,
+ "derived": [
+ "WEEK_BEG_DT"
+ ]
+ },
+ {
+ "name": "USER_DEFINED_FIELD1",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": null,
+ "derived": [
+ "USER_DEFINED_FIELD1"
+ ]
+ },
+ {
+ "name": "USER_DEFINED_FIELD3",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": null,
+ "derived": [
+ "USER_DEFINED_FIELD3"
+ ]
+ },
+ {
+ "name": "META_CATEG_NAME",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": "META_CATEG_NAME",
+ "derived": null
+ },
+ {
+ "name": "CATEG_LVL2_NAME",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": "CATEG_LVL2_NAME",
+ "derived": null
+ },
+ {
+ "name": "CATEG_LVL3_NAME",
+ "table": "KYLIN_CATEGORY_GROUPINGS",
+ "column": "CATEG_LVL3_NAME",
+ "derived": null
+ },
+ {
+ "name": "LSTG_FORMAT_NAME",
+ "table": "KYLIN_SALES",
+ "column": "LSTG_FORMAT_NAME",
+ "derived": null
+ },
+ {
+ "name": "SELLER_ID",
+ "table": "KYLIN_SALES",
+ "column": "SELLER_ID",
+ "derived": null
+ },
+ {
+ "name": "BUYER_ID",
+ "table": "KYLIN_SALES",
+ "column": "BUYER_ID",
+ "derived": null
+ },
+ {
+ "name": "ACCOUNT_BUYER_LEVEL",
+ "table": "BUYER_ACCOUNT",
+ "column": "ACCOUNT_BUYER_LEVEL",
+ "derived": null
+ },
+ {
+ "name": "ACCOUNT_SELLER_LEVEL",
+ "table": "SELLER_ACCOUNT",
+ "column": "ACCOUNT_SELLER_LEVEL",
+ "derived": null
+ },
+ {
+ "name": "BUYER_COUNTRY",
+ "table": "BUYER_ACCOUNT",
+ "column": "ACCOUNT_COUNTRY",
+ "derived": null
+ },
+ {
+ "name": "SELLER_COUNTRY",
+ "table": "SELLER_ACCOUNT",
+ "column": "ACCOUNT_COUNTRY",
+ "derived": null
+ },
+ {
+ "name": "BUYER_COUNTRY_NAME",
+ "table": "BUYER_COUNTRY",
+ "column": "NAME",
+ "derived": null
+ },
+ {
+ "name": "SELLER_COUNTRY_NAME",
+ "table": "SELLER_COUNTRY",
+ "column": "NAME",
+ "derived": null
+ },
+ {
+ "name": "OPS_USER_ID",
+ "table": "KYLIN_SALES",
+ "column": "OPS_USER_ID",
+ "derived": null
+ },
+ {
+ "name": "OPS_REGION",
+ "table": "KYLIN_SALES",
+ "column": "OPS_REGION",
+ "derived": null
+ }
+ ],
+ "measures": [
+ {
+ "name": "GMV_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.PRICE"
+ },
+ "returntype": "decimal(19,4)"
+ }
+ },
+ {
+ "name": "BUYER_LEVEL_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "SELLER_LEVEL_SUM",
+ "function": {
+ "expression": "SUM",
+ "parameter": {
+ "type": "column",
+ "value": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "TRANS_CNT",
+ "function": {
+ "expression": "COUNT",
+ "parameter": {
+ "type": "constant",
+ "value": "1"
+ },
+ "returntype": "bigint"
+ }
+ },
+ {
+ "name": "SELLER_CNT_HLL",
+ "function": {
+ "expression": "COUNT_DISTINCT",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.SELLER_ID"
+ },
+ "returntype": "hllc(10)"
+ }
+ },
+ {
+ "name": "TOP_SELLER",
+ "function": {
+ "expression": "TOP_N",
+ "parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.PRICE",
+ "next_parameter": {
+ "type": "column",
+ "value": "KYLIN_SALES.SELLER_ID"
+ }
+ },
+ "returntype": "topn(100)",
+ "configuration": {
+ "topn.encoding.KYLIN_SALES.SELLER_ID": "dict",
+ "topn.encoding_version.KYLIN_SALES.SELLER_ID": "1"
+ }
+ }
+ }
+ ],
+ "rowkey": {
+ "rowkey_columns": [
+ {
+ "column": "KYLIN_SALES.BUYER_ID",
+ "encoding": "integer:4",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.SELLER_ID",
+ "encoding": "integer:4",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.TRANS_ID",
+ "encoding": "integer:4",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.PART_DT",
+ "encoding": "date",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.LEAF_CATEG_ID",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "BUYER_COUNTRY.NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "SELLER_COUNTRY.NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.LSTG_FORMAT_NAME",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.LSTG_SITE_ID",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.OPS_USER_ID",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ },
+ {
+ "column": "KYLIN_SALES.OPS_REGION",
+ "encoding": "dict",
+ "encoding_version": 1,
+ "isShardBy": false
+ }
+ ]
+ },
+ "hbase_mapping": {
+ "column_family": [
+ {
+ "name": "F1",
+ "columns": [
+ {
+ "qualifier": "M",
+ "measure_refs": [
+ "GMV_SUM",
+ "BUYER_LEVEL_SUM",
+ "SELLER_LEVEL_SUM",
+ "TRANS_CNT"
+ ]
+ }
+ ]
+ },
+ {
+ "name": "F2",
+ "columns": [
+ {
+ "qualifier": "M",
+ "measure_refs": [
+ "SELLER_CNT_HLL",
+ "TOP_SELLER"
+ ]
+ }
+ ]
+ }
+ ]
+ },
+ "aggregation_groups": [
+ {
+ "includes": [
+ "KYLIN_SALES.PART_DT",
+ "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "KYLIN_SALES.LEAF_CATEG_ID",
+ "KYLIN_SALES.LSTG_FORMAT_NAME",
+ "KYLIN_SALES.LSTG_SITE_ID",
+ "KYLIN_SALES.OPS_USER_ID",
+ "KYLIN_SALES.OPS_REGION",
+ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL",
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "BUYER_COUNTRY.NAME",
+ "SELLER_COUNTRY.NAME"
+ ],
+ "select_rule": {
+ "hierarchy_dims": [
+ [
+ "KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME",
+ "KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME",
+ "KYLIN_SALES.LEAF_CATEG_ID"
+ ]
+ ],
+ "mandatory_dims": [
+ "KYLIN_SALES.PART_DT"
+ ],
+ "joint_dims": [
+ [
+ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
+ "BUYER_COUNTRY.NAME"
+ ],
+ [
+ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
+ "SELLER_COUNTRY.NAME"
+ ],
+ [
+ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
+ "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL"
+ ],
+ [
+ "KYLIN_SALES.LSTG_FORMAT_NAME",
+ "KYLIN_SALES.LSTG_SITE_ID"
+ ],
+ [
+ "KYLIN_SALES.OPS_USER_ID",
+ "KYLIN_SALES.OPS_REGION"
+ ]
+ ]
+ }
+ }
+ ],
+ "signature": null,
+ "notify_list": [],
+ "status_need_notify": [],
+ "partition_date_start": 0,
+ "partition_date_end": 3153600000000,
+ "auto_merge_time_ranges": [],
+ "volatile_range": 0,
+ "retention_range": 0,
+ "engine_type": 2,
+ "storage_type": 2,
+ "override_kylin_properties": {
+ "kylin.cube.aggrgroup.is-mandatory-only-valid": "true",
+ "kylin.engine.spark.rdd-partition-cut-mb": "500"
+ },
+ "cuboid_black_list": [],
+ "parent_forward": 3,
+ "mandatory_dimension_set_list": [],
+ "snapshot_table_desc_list": []
+ }
+}
\ No newline at end of file
diff --git a/build/CI/testing/env/default/default.properties b/build/CI/testing/env/default/default.properties
new file mode 100644
index 0000000..461ec37
--- /dev/null
+++ b/build/CI/testing/env/default/default.properties
@@ -0,0 +1,25 @@
+# default.properties
+# properties set here will be available to the test execution as environment variables
+
+# sample_key = sample_value
+
+#The path to the gauge reports directory. Should be either relative to the project directory or an absolute path
+gauge_reports_dir = reports
+
+#Set as false if gauge reports should not be overwritten on each execution. A new time-stamped directory will be created on each execution.
+overwrite_reports = true
+
+# Set to false to disable screenshots on failure in reports.
+screenshot_on_failure = false
+
+# The path to the gauge logs directory. Should be either relative to the project directory or an absolute path
+logs_directory = logs
+
+# The path the gauge specifications directory. Takes a comma separated list of specification files/directories.
+gauge_specs_dir = features/specs
+
+# The default delimiter used read csv files.
+csv_delimiter = ,
+
+# Allows steps to be written in multiline
+allow_multiline_step = false
\ No newline at end of file
diff --git a/build/CI/testing/env/default/python.properties b/build/CI/testing/env/default/python.properties
new file mode 100644
index 0000000..077d659
--- /dev/null
+++ b/build/CI/testing/env/default/python.properties
@@ -0,0 +1,4 @@
+GAUGE_PYTHON_COMMAND = python3
+
+# Comma seperated list of dirs. path should be relative to project root.
+STEP_IMPL_DIR = features/step_impl
diff --git a/build/CI/testing/features/specs/authentication/authentication_0001.spec b/build/CI/testing/features/specs/authentication/authentication_0001.spec
new file mode 100644
index 0000000..b915e26
--- /dev/null
+++ b/build/CI/testing/features/specs/authentication/authentication_0001.spec
@@ -0,0 +1,18 @@
+# Authentication Test
+Tags:front-end
+## Prepare browser
+
+* Initialize "chrome" browser and connect to "kylin_instance.yml"
+
+## Use the user name and password for user authentication
+
+* Authentication with user "test" and password "password".
+
+* Authentication with built-in user
+ |User |Password |
+ |-------|--------------|
+ |ADMIN |KYLIN |
+
+
+
+
diff --git a/build/CI/testing/features/specs/deploy_in_cluster_mode/read_write_separation.spec b/build/CI/testing/features/specs/deploy_in_cluster_mode/read_write_separation.spec
new file mode 100644
index 0000000..5ce3f08
--- /dev/null
+++ b/build/CI/testing/features/specs/deploy_in_cluster_mode/read_write_separation.spec
@@ -0,0 +1,5 @@
+# Read and write separation deployment
+Tags: 4.x
+
+## Prepare env
+* Get kylin instance
diff --git a/build/CI/testing/features/specs/generic_test.spec b/build/CI/testing/features/specs/generic_test.spec
new file mode 100644
index 0000000..95e68fa
--- /dev/null
+++ b/build/CI/testing/features/specs/generic_test.spec
@@ -0,0 +1,63 @@
+# Kylin Release Test
+Tags:3.x
+## Prepare env
+* Get kylin instance
+
+* prepare data file from "release_test_0001.json"
+
+* Create project "release_test_0001_project" and load table "load_table_list"
+
+
+## MR engine
+
+* Create model with "model_desc_data" in "release_test_0001_project"
+
+* Create cube with "cube_desc_data" in "release_test_0001_project", cube name is "release_test_0001_cube"
+
+* Build segment from "1325347200000" to "1356969600000" in "release_test_0001_cube"
+
+* Build segment from "1356969600000" to "1391011200000" in "release_test_0001_cube"
+
+* Merge cube "release_test_0001_cube" segment from "1325347200000" to "1391011200000"
+
+
+## SPARK engine
+
+* Clone cube "release_test_0001_cube" and name it "kylin_spark_cube" in "release_test_0001_project", modify build engine to "SPARK"
+
+* Build segment from "1325347200000" to "1356969600000" in "kylin_spark_cube"
+
+* Build segment from "1356969600000" to "1391011200000" in "kylin_spark_cube"
+
+* Merge cube "kylin_spark_cube" segment from "1325347200000" to "1391011200000"
+
+
+## FLINK engine
+
+* Clone cube "release_test_0001_cube" and name it "kylin_flink_cube" in "release_test_0001_project", modify build engine to "FLINK"
+
+* Build segment from "1325347200000" to "1356969600000" in "kylin_flink_cube"
+
+* Build segment from "1356969600000" to "1391011200000" in "kylin_flink_cube"
+
+* Merge cube "kylin_flink_cube" segment from "1325347200000" to "1391011200000"
+
+
+## Query cube and pushdown
+
+* Query SQL "select count(*) from kylin_sales" and specify "release_test_0001_cube" cube to query in "release_test_0001_project", compare result with "10000"
+
+* Query SQL "select count(*) from kylin_sales" and specify "kylin_spark_cube" cube to query in "release_test_0001_project", compare result with "10000"
+
+* Query SQL "select count(*) from kylin_sales" and specify "kylin_flink_cube" cube to query in "release_test_0001_project", compare result with "10000"
+
+* Disable cube "release_test_0001_cube"
+
+* Disable cube "kylin_spark_cube"
+
+* Disable cube "kylin_flink_cube"
+
+* Query SQL "select count(*) from kylin_sales" in "release_test_0001_project" and pushdown, compare result with "10000"
+
+
+
diff --git a/build/CI/testing/features/specs/sample.spec b/build/CI/testing/features/specs/sample.spec
new file mode 100644
index 0000000..bb9c9f5
--- /dev/null
+++ b/build/CI/testing/features/specs/sample.spec
@@ -0,0 +1,5 @@
+# test
+Tags:test,3.x,4.x
+## test
+* Get kylin instance
+* Query sql "select count(*) from kylin_sales" in "generic_test_project" and compare result with pushdown result
diff --git a/build/CI/testing/features/step_impl/authentication/authentication.py b/build/CI/testing/features/step_impl/authentication/authentication.py
new file mode 100644
index 0000000..044d1e2
--- /dev/null
+++ b/build/CI/testing/features/step_impl/authentication/authentication.py
@@ -0,0 +1,37 @@
+from time import sleep
+
+from getgauge.python import step
+from kylin_utils import util
+
+
+class LoginTest:
+
+ @step("Initialize <browser_type> browser and connect to <file_name>")
+ def setup_browser(self, browser_type, file_name):
+ global browser
+ browser = util.setup_browser(browser_type=browser_type)
+
+ browser.get(util.kylin_url(file_name))
+ sleep(3)
+
+ browser.refresh()
+ browser.set_window_size(1400, 800)
+
+ @step("Authentication with user <user> and password <password>.")
+ def assert_authentication_failed(self, user, password):
+ browser.find_element_by_id("username").clear()
+ browser.find_element_by_id("username").send_keys(user)
+ browser.find_element_by_id("password").clear()
+ browser.find_element_by_id("password").send_keys(password)
+
+ browser.find_element_by_class_name("bigger-110").click()
+
+ @step("Authentication with built-in user <table>")
+ def assert_authentication_success(self, table):
+ for i in range(1, 2):
+ user = table.get_row(i)
+ browser.find_element_by_id("username").clear()
+ browser.find_element_by_id("username").send_keys(user[0])
+ browser.find_element_by_id("password").clear()
+ browser.find_element_by_id("password").send_keys(user[1])
+ browser.find_element_by_class_name("bigger-110").click()
diff --git a/build/CI/testing/features/step_impl/before_suite.py b/build/CI/testing/features/step_impl/before_suite.py
new file mode 100644
index 0000000..4cce795
--- /dev/null
+++ b/build/CI/testing/features/step_impl/before_suite.py
@@ -0,0 +1,44 @@
+from getgauge.python import before_suite
+import os
+import json
+
+from kylin_utils import util
+
+
+@before_suite()
+def create_generic_model_and_cube():
+ client = util.setup_instance('kylin_instance.yml')
+ if client.version == '3.x':
+ with open(os.path.join('data/generic_desc_data', 'generic_desc_data_3x.json'), 'r') as f:
+ data = json.load(f)
+ elif client.version == '4.x':
+ with open(os.path.join('data/generic_desc_data', 'generic_desc_data_4x.json'), 'r') as f:
+ data = json.load(f)
+
+ project_name = client.generic_project
+ if not util.if_project_exists(kylin_client=client, project=project_name):
+ client.create_project(project_name)
+
+ tables = data.get('load_table_list')
+ resp = client.load_table(project_name=project_name, tables=tables)
+ assert ",".join(resp["result.loaded"]) == tables
+
+ model_desc_data = data.get('model_desc_data')
+ model_name = model_desc_data.get('name')
+
+ if not util.if_model_exists(kylin_client=client, model_name=model_name, project=project_name):
+ resp = client.create_model(project_name=project_name,
+ model_name=model_name,
+ model_desc_data=model_desc_data)
+ assert json.loads(resp['modelDescData'])['name'] == model_name
+
+ cube_desc_data = data.get('cube_desc_data')
+ cube_name = cube_desc_data.get('name')
+ if not util.if_cube_exists(kylin_client=client, cube_name=cube_name, project=project_name):
+ resp = client.create_cube(project_name=project_name,
+ cube_name=cube_name,
+ cube_desc_data=cube_desc_data)
+ assert json.loads(resp['cubeDescData'])['name'] == cube_name
+ if client.get_cube_instance(cube_name=cube_name).get('status') != 'READY':
+ resp = client.full_build_cube(cube_name=cube_name)
+ assert client.await_job_finished(job_id=resp['uuid'], waiting_time=20)
diff --git a/build/CI/testing/features/step_impl/generic_test_step.py b/build/CI/testing/features/step_impl/generic_test_step.py
new file mode 100644
index 0000000..cf04d55
--- /dev/null
+++ b/build/CI/testing/features/step_impl/generic_test_step.py
@@ -0,0 +1,98 @@
+from getgauge.python import step
+import os
+import json
+
+from kylin_utils import util
+
+
+@step("Get kylin instance")
+def get_kylin_instance_with_config_file():
+ global client
+ client = util.setup_instance('kylin_instance.yml')
+
+
+@step("prepare data file from <release_test_0001.json>")
+def prepare_data_file_from(file_name):
+ global data
+ with open(os.path.join('data', file_name), 'r') as f:
+ data = json.load(f)
+
+
+@step("Create project <project_name> and load table <tables>")
+def prepare_project_step(project_name, tables):
+ client.create_project(project_name=project_name)
+ tables = data.get(tables)
+ resp = client.load_table(project_name=project_name, tables=tables)
+ assert ",".join(resp["result.loaded"]) == tables
+
+
+@step("Create model with <model_desc> in <project>")
+def create_model_step(model_desc, project):
+ model_name = data.get(model_desc).get('name')
+ model_desc_data = data.get(model_desc)
+
+ resp = client.create_model(project_name=project,
+ model_name=model_name,
+ model_desc_data=model_desc_data)
+ assert json.loads(resp['modelDescData'])['name'] == model_name
+
+
+@step("Create cube with <cube_desc> in <prpject>, cube name is <cube_name>")
+def create_cube_step(cube_desc, project, cube_name):
+ resp = client.create_cube(project_name=project,
+ cube_name=cube_name,
+ cube_desc_data=data.get(cube_desc))
+ assert json.loads(resp['cubeDescData'])['name'] == cube_name
+
+
+@step("Build segment from <start_time> to <end_time> in <cube_name>")
+def build_first_segment_step(start_time, end_time, cube_name):
+ resp = client.build_segment(start_time=start_time,
+ end_time=end_time,
+ cube_name=cube_name)
+ assert client.await_job_finished(job_id=resp['uuid'], waiting_time=20)
+
+
+@step("Merge cube <cube_name> segment from <start_name> to <end_time>")
+def merge_segment_step(cube_name, start_time, end_time):
+ resp = client.merge_segment(cube_name=cube_name,
+ start_time=start_time,
+ end_time=end_time)
+ assert client.await_job_finished(job_id=resp['uuid'], waiting_time=20)
+
+
+@step("Clone cube <old_cube_name> and name it <new_cube_name> in <project>, modify build engine to <engine>")
+def clone_cube_step(old_cube_name, new_cube_name, project, build_engine):
+ resp = client.clone_cube(cube_name=old_cube_name,
+ new_cube_name=new_cube_name,
+ project_name=project)
+ assert resp.get('name') == new_cube_name
+ client.update_cube_engine(cube_name=new_cube_name,
+ engine_type=build_engine)
+
+
+@step("Query SQL <SQL> and specify <cube_name> cube to query in <project>, compare result with <result>")
+def query_cube_step(sql, cube_name, project, result):
+ resp = client.execute_query(cube_name=cube_name,
+ project_name=project,
+ sql=sql)
+ assert resp.get('isException') is False
+ assert resp.get('results')[0][0] == result
+ assert resp.get('cube') == 'CUBE[name=' + cube_name + ']'
+ assert resp.get('pushDown') is False
+
+
+@step("Disable cube <cube_name>")
+def disable_cube_step(cube_name):
+ resp = client.disable_cube(cube_name=cube_name)
+ assert resp.get('status') == 'DISABLED'
+
+
+@step("Query SQL <SQL> in <project> and pushdown, compare result with <result>")
+def query_pushdown_step(sql, project, result):
+ resp = client.execute_query(project_name=project, sql=sql)
+ assert resp.get('isException') is False
+ assert resp.get('results')[0][0] == result
+ assert resp.get('cube') == ''
+ assert resp.get('pushDown') is True
+
diff --git a/build/CI/testing/features/step_impl/read_write_separation/read_write_separation.py b/build/CI/testing/features/step_impl/read_write_separation/read_write_separation.py
new file mode 100644
index 0000000..e69de29
diff --git a/build/CI/testing/features/step_impl/sample.py b/build/CI/testing/features/step_impl/sample.py
new file mode 100644
index 0000000..d7ac9bb
--- /dev/null
+++ b/build/CI/testing/features/step_impl/sample.py
@@ -0,0 +1,14 @@
+from getgauge.python import step, before_spec
+from kylin_utils import util
+from kylin_utils import equals
+
+
+@before_spec()
+def before_spec_hook():
+ global client
+ client = util.setup_instance("kylin_instance.yml")
+
+
+@step("Query sql <select count(*) from kylin_sales> in <project> and compare result with pushdown result")
+def query_sql_and_compare_result_with_pushdown_result(sql, project):
+ equals.compare_sql_result(sql=sql, project=project, kylin_client=client)
diff --git a/build/CI/testing/kylin_instances/kylin_instance.yml b/build/CI/testing/kylin_instances/kylin_instance.yml
new file mode 100644
index 0000000..ca76d00
--- /dev/null
+++ b/build/CI/testing/kylin_instances/kylin_instance.yml
@@ -0,0 +1,7 @@
+---
+# All mode
+- host: kylin-all
+ port: 7070
+ version: 3.x
+ hadoop_platform: HDP2.4
+ deploy_mode: ALL
\ No newline at end of file
diff --git a/build/CI/testing/kylin_utils/basic.py b/build/CI/testing/kylin_utils/basic.py
new file mode 100644
index 0000000..cd8e416
--- /dev/null
+++ b/build/CI/testing/kylin_utils/basic.py
@@ -0,0 +1,90 @@
+import logging
+import requests
+
+
+class BasicHttpClient:
+ _host = None
+ _port = None
+
+ _headers = {}
+
+ _auth = ('ADMIN', 'KYLIN')
+
+ _inner_session = requests.Session()
+
+ def __init__(self, host, port):
+ if not host or not port:
+ raise ValueError('init http client failed')
+
+ self._host = host
+ self._port = port
+
+ def token(self, token):
+ self._headers['Authorization'] = 'Basic {token}'.format(token=token)
+
+ def auth(self, username, password):
+ self._auth = (username, password)
+
+ def header(self, name, value):
+ self._headers[name] = value
+
+ def headers(self, headers):
+ self._headers = headers
+
+ def _request(self, method, url, params=None, data=None, json=None, # pylint: disable=too-many-arguments
+ files=None, headers=None, stream=False, to_json=True, inner_session=False, timeout=60):
+ if inner_session:
+ return self._request_with_session(self._inner_session, method, url,
+ params=params,
+ data=data,
+ json=json,
+ files=files,
+ headers=headers,
+ stream=stream,
+ to_json=to_json,
+ timeout=timeout)
+ with requests.Session() as session:
+ session.auth = self._auth
+ return self._request_with_session(session, method, url,
+ params=params,
+ data=data,
+ json=json,
+ files=files,
+ headers=headers,
+ stream=stream,
+ to_json=to_json,
+ timeout=timeout)
+
+ def _request_with_session(self, session, method, url, params=None, data=None, # pylint: disable=too-many-arguments
+ json=None, files=None, headers=None, stream=False, to_json=True, timeout=60):
+ if headers is None:
+ headers = self._headers
+ resp = session.request(method, url,
+ params=params,
+ data=data,
+ json=json,
+ files=files,
+ headers=headers,
+ stream=stream,
+ timeout=timeout
+ )
+
+ try:
+ if stream:
+ return resp.raw
+ if not resp.content:
+ return None
+
+ if to_json:
+ data = resp.json()
+ resp.raise_for_status()
+ return data
+ return resp.text
+ except requests.HTTPError as http_error:
+ err_msg = f"{str(http_error)} [{data.get('msg', '')}]\n" \
+ f"{data.get('stacktrace', '')}"
+ logging.error(err_msg)
+ raise requests.HTTPError(err_msg, request=http_error.request, response=http_error.response, )
+ except Exception as error:
+ logging.error(str(error))
+ raise error
diff --git a/build/CI/testing/kylin_utils/equals.py b/build/CI/testing/kylin_utils/equals.py
new file mode 100644
index 0000000..f610f4e
--- /dev/null
+++ b/build/CI/testing/kylin_utils/equals.py
@@ -0,0 +1,204 @@
+import logging
+from kylin_utils import util
+
+_array_types = (list, tuple, set)
+_object_types = (dict, )
+
+
+def api_response_equals(actual, expected, ignore=None):
+ if ignore is None:
+ ignore = []
+
+ def _get_value(ignore):
+ def get_value(key, container):
+ if isinstance(container, _object_types):
+ return container.get(key)
+ if isinstance(container, _array_types):
+ errmsg = ''
+ for item in container:
+ try:
+ api_response_equals(item, key, ignore=ignore)
+ return item
+ except AssertionError as e:
+ errmsg += str(e) + '\n'
+ raise AssertionError(errmsg)
+
+ return None
+
+ return get_value
+
+ getvalue = _get_value(ignore)
+ assert_failed = AssertionError(
+ f'assert json failed, expected: [{expected}], actual: [{actual}]')
+
+ if isinstance(expected, _array_types):
+ if not isinstance(actual, _array_types):
+ raise assert_failed
+ for item in expected:
+ api_response_equals(getvalue(item, actual), item, ignore=ignore)
+
+ elif isinstance(expected, _object_types):
+ if not isinstance(actual, _object_types):
+ raise assert_failed
+ for key, value in expected.items():
+ if key not in ignore:
+ api_response_equals(getvalue(key, actual),
+ value,
+ ignore=ignore)
+ else:
+ if key not in actual:
+ raise assert_failed
+ else:
+ if actual != expected:
+ raise assert_failed
+
+
+INTEGER_FAMILY = ['TINYINT', 'SMALLINT', 'INTEGER', 'BIGINT']
+
+FRACTION_FAMILY = ['DECIMAL', 'DOUBLE', 'FLOAT']
+
+STRING_FAMILY = ['CHAR', 'VARCHAR', 'STRING']
+
+
+def _is_family(datatype1, datatype2):
+ if datatype1 in STRING_FAMILY and datatype2 in STRING_FAMILY:
+ return True
+ if datatype1 in FRACTION_FAMILY and datatype2 in FRACTION_FAMILY:
+ return True
+ if datatype1 in INTEGER_FAMILY and datatype2 in INTEGER_FAMILY:
+ return True
+ return datatype1 == datatype2
+
+
+class _Row(tuple):
+ def __init__(self, values, types): # pylint: disable=unused-argument
+ tuple.__init__(self)
+ if len(values) != len(types):
+ raise ValueError('???')
+
+ self._types = types
+
+ self._has_fraction = False
+ for datatype in self._types:
+ if datatype in FRACTION_FAMILY:
+ self._has_fraction = True
+
+ def __new__(cls, values, types): # pylint: disable=unused-argument
+ return tuple.__new__(cls, values)
+
+ def __eq__(self, other):
+ if not self._has_fraction or not other._has_fraction:
+ return tuple.__eq__(self, other)
+
+ if len(self._types) != len(other._types):
+ return False
+
+ for i in range(len(self._types)):
+ stype = self._types[i]
+ otype = other._types[i]
+
+ if not _is_family(stype, otype):
+ return False
+
+ svalue = self[i]
+ ovalue = other[i]
+
+ if stype in FRACTION_FAMILY:
+ fsvalue = float(svalue)
+ fovalue = float(ovalue)
+
+ diff = abs(fsvalue - fovalue)
+
+ rate = diff / min(fsvalue, fovalue
+ ) if fsvalue != 0 and fovalue != 0 else diff
+ if abs(rate) > 0.01:
+ return False
+
+ else:
+ if svalue != ovalue:
+ return False
+
+ return True
+
+ def __hash__(self):
+ # Always use __eq__ to compare
+ return 0
+
+
+def query_result_equals(expect_resp, actual_resp):
+ expect_column_types = [
+ x['columnTypeName'] for x in expect_resp['columnMetas']
+ ]
+ expect_result = [[y.strip() if y else y for y in x]
+ for x in expect_resp['results']]
+
+ actual_column_types = [
+ x['columnTypeName'] for x in actual_resp['columnMetas']
+ ]
+ actual_result = [[y.strip() if y else y for y in x]
+ for x in actual_resp['results']]
+
+ if len(expect_column_types) != len(actual_column_types):
+ logging.error('column count assert failed [%s,%s]',
+ len(expect_column_types), len(actual_column_types))
+ return False
+
+ return dataset_equals(expect_result, actual_result, expect_column_types,
+ actual_column_types)
+
+
+def dataset_equals(expect,
+ actual,
+ expect_col_types=None,
+ actual_col_types=None):
+ if len(expect) != len(actual):
+ logging.error('row count assert failed [%s,%s]', len(expect),
+ len(actual))
+ return False
+
+ if expect_col_types is None:
+ expect_col_types = ['VARCHAR'] * len(expect[0])
+ expect_set = set()
+ for values in expect:
+ expect_set.add(_Row(values, expect_col_types))
+
+ if actual_col_types is None:
+ actual_col_types = expect_col_types if expect_col_types else [
+ 'VARCHAR'
+ ] * len(actual[0])
+ actual_set = set()
+ for values in actual:
+ actual_set.add(_Row(values, actual_col_types))
+
+ assert_result = expect_set ^ actual_set
+ if assert_result:
+ logging.error('diff[%s]', len(assert_result))
+ if len(assert_result) < 20:
+ print(assert_result)
+ return False
+
+ return True
+
+
+def compare_sql_result(sql, project, kylin_client, cube=None):
+ pushdown_project = kylin_client.pushdown_project
+ if not util.if_project_exists(kylin_client=kylin_client, project=pushdown_project):
+ kylin_client.create_project(project_name=pushdown_project)
+
+ hive_tables = kylin_client.list_hive_tables(project_name=project)
+ if hive_tables is not None:
+ for table_info in kylin_client.list_hive_tables(project_name=project):
+ if table_info.get('source_type') == 0:
+ kylin_client.load_table(project_name=pushdown_project,
+ tables='{database}.{table}'.format(
+ database=table_info.get('database'),
+ table=table_info.get('name')))
+ kylin_resp = kylin_client.execute_query(cube_name=cube,
+ project_name=project,
+ sql=sql)
+ assert kylin_resp.get('isException') is False
+
+ pushdown_resp = kylin_client.execute_query(project_name=pushdown_project, sql=sql)
+ assert pushdown_resp.get('isException') is False
+
+ assert query_result_equals(kylin_resp, pushdown_resp)
\ No newline at end of file
diff --git a/build/CI/testing/kylin_utils/kylin.py b/build/CI/testing/kylin_utils/kylin.py
new file mode 100644
index 0000000..252ce21
--- /dev/null
+++ b/build/CI/testing/kylin_utils/kylin.py
@@ -0,0 +1,826 @@
+import json
+import logging
+import time
+import random
+
+import requests
+
+from .basic import BasicHttpClient
+
+
+class KylinHttpClient(BasicHttpClient): # pylint: disable=too-many-public-methods
+ _base_url = 'http://{host}:{port}/kylin/api'
+
+ def __init__(self, host, port, version):
+ super().__init__(host, port)
+
+ self._headers = {
+ 'Content-Type': 'application/json;charset=utf-8'
+ }
+
+ self._base_url = self._base_url.format(host=self._host, port=self._port)
+ self.generic_project = "generic_test_project"
+ self.pushdown_project = "pushdown_test_project"
+ self.version = version
+
+ def login(self, username, password):
+ self._inner_session.request('POST', self._base_url + '/user/authentication', auth=(username, password))
+ return self._request('GET', '/user/authentication', inner_session=True)
+
+ def check_login_state(self):
+ return self._request('GET', '/user/authentication', inner_session=True)
+
+ def get_session(self):
+ return self._inner_session
+
+ def logout(self):
+ self._inner_session = requests.Session()
+
+ def list_projects(self, limit=100, offset=0):
+ params = {'limit': limit, 'offset': offset}
+ resp = self._request('GET', '/projects', params=params)
+ return resp
+
+ def create_project(self, project_name, description=None, override_kylin_properties=None):
+ data = {'name': project_name,
+ 'description': description,
+ 'override_kylin_properties': override_kylin_properties,
+ }
+ payload = {
+ 'projectDescData': json.dumps(data),
+ }
+ resp = self._request('POST', '/projects', json=payload)
+ return resp
+
+ def update_project(self, project_name, description=None, override_kylin_properties=None):
+ """
+ :param project_name: project name
+ :param description: description of project
+ :param override_kylin_properties: the kylin properties that needs to be override
+ :return:
+ """
+ data = {'name': project_name,
+ 'description': description,
+ 'override_kylin_properties': override_kylin_properties,
+ }
+ payload = {
+ 'formerProjectName': project_name,
+ 'projectDescData': json.dumps(data),
+ }
+ resp = self._request('PUT', '/projects', json=payload)
+ return resp
+
+ def delete_project(self, project_name, force=False):
+ """
+ delete project API, before delete the project, make sure the project does not contain models and cubes.
+ If you want to force delete the project, make force=True
+ :param project_name: project name
+ :param force: if force, delete cubes and models before delete project
+ :return:
+ """
+ if force:
+ cubes = self.list_cubes(project_name)
+ logging.debug("Cubes to be deleted: %s", cubes)
+ while cubes:
+ for cube in cubes:
+ self.delete_cube(cube['name'])
+ cubes = self.list_cubes(project_name)
+ models = self.list_model_desc(project_name)
+ logging.debug("Models to be deleted: %s", models)
+ while models:
+ for model in models:
+ self.delete_model(model['name'])
+ models = self.list_model_desc(project_name)
+ url = '/projects/{project}'.format(project=project_name)
+ resp = self._request('DELETE', url)
+ return resp
+
+ def load_table(self, project_name, tables, calculate=True):
+ """
+ load or reload table api
+ :param calculate: Default is True
+ :param project_name: project name
+ :param tables: table list, for instance, ['default.kylin_fact', 'default.kylin_sales']
+ :return:
+ """
+ # workaround of #15337
+ # time.sleep(random.randint(5, 10))
+ url = '/tables/{tables}/{project}/'.format(tables=tables, project=project_name)
+ payload = {'calculate': calculate
+ }
+ resp = self._request('POST', url, json=payload)
+ return resp
+
+ def unload_table(self, project_name, tables):
+ url = '/tables/{tables}/{project}'.format(tables=tables, project=project_name)
+ resp = self._request('DELETE', url)
+ return resp
+
+ def list_hive_tables(self, project_name, extension=False, user_session=False):
+ """
+ :param project_name: project name
+ :param extension: specify whether the table's extension information is returned
+ :param user_session: boolean, true for using login session to execute
+ :return:
+ """
+ url = '/tables'
+ params = {'project': project_name, 'ext': extension}
+ resp = self._request('GET', url, params=params, inner_session=user_session)
+ return resp
+
+ def get_table_info(self, project_name, table_name):
+ """
+ :param project_name: project name
+ :param table_name: table name
+ :return: hive table information
+ """
+ url = '/tables/{project}/{table}'.format(project=project_name, table=table_name)
+ resp = self._request('GET', url)
+ return resp
+
+ def get_tables_info(self, project_name, ext='true'):
+ url = '/tables'
+ params = {'project': project_name, 'ext': ext}
+ resp = self._request('GET', url, params=params)
+ return resp
+
+ def get_table_streaming_config(self, project_name, table_name, limit=100, offset=0):
+ params = {'table': table_name, 'project': project_name, 'limit': limit, 'offset': offset}
+ resp = self._request('GET', '/streaming/getConfig', params=params)
+ return resp
+
+ def load_kafka_table(self, project_name, kafka_config, streaming_config, table_data, message=None):
+ url = '/streaming'
+ payload = {'project': project_name,
+ 'kafkaConfig': json.dumps(kafka_config),
+ 'streamingConfig': json.dumps(streaming_config),
+ 'tableData': json.dumps(table_data),
+ 'message': message}
+ resp = self._request('POST', url, json=payload)
+ return resp
+
+ def update_kafka_table(self, project_name, kafka_config, streaming_config, table_data, cluster_index=0):
+ url = '/streaming'
+ payload = {'project': project_name,
+ 'kafkaConfig': kafka_config,
+ 'streamingConfig': streaming_config,
+ 'tableData': table_data,
+ 'clusterIndex': cluster_index}
+ resp = self._request('PUT', url, json=payload)
+ return resp
+
+ def list_model_desc(self, project_name=None, model_name=None, limit=100, offset=0):
+
+ """
+ :param offset:
+ :param limit:
+ :param project_name: project name
+ :param model_name: model name
+ :return: model desc list
+ """
+ params = {'limit': limit,
+ 'offset': offset,
+ 'modelName': model_name,
+ 'projectName': project_name
+ }
+ resp = self._request('GET', '/models', params=params)
+ return resp
+
+ def create_model(self, project_name, model_name, model_desc_data, user_session=False):
+ url = '/models'
+ payload = {
+ 'project': project_name,
+ 'model': model_name,
+ 'modelDescData': json.dumps(model_desc_data)
+ }
+ logging.debug("Current payload for creating model is %s", payload)
+ resp = self._request('POST', url, json=payload, inner_session=user_session)
+ return resp
+
+ def update_model(self, project_name, model_name, model_desc_data, user_session=False):
+ url = '/models'
+ payload = {
+ 'project': project_name,
+ 'model': model_name,
+ 'modelDescData': json.dumps(model_desc_data)
+ }
+ resp = self._request('PUT', url, json=payload, inner_session=user_session)
+ return resp
+
+ def clone_model(self, project_name, model_name, new_model_name):
+ url = '/models/{model}/clone'.format(model=model_name)
+ payload = {'modelName': new_model_name, 'project': project_name}
+ resp = self._request('PUT', url, json=payload)
+ return resp
+
+ def delete_model(self, model_name):
+ url = '/models/{model}'.format(model=model_name)
+ # return value is None here
+ return self._request('DELETE', url)
+
+ def get_cube_desc(self, cube_name):
+ url = '/cube_desc/{cube}'.format(cube=cube_name)
+ resp = self._request('GET', url)
+ return resp
+
+ def list_cubes(self, project=None, offset=0, limit=10000, cube_name=None, model_name=None, user_session=False):
+ params = {'projectName': project, 'offset': offset, 'limit': limit,
+ 'cubeName': cube_name, 'modelName': model_name}
+ resp = self._request('GET', '/cubes/', params=params, inner_session=user_session)
+ return resp
+
+ def get_cube_instance(self, cube_name):
+ url = '/cubes/{cube}'.format(cube=cube_name)
+ resp = self._request('GET', url)
+ return resp
+
+ def create_cube(self, project_name, cube_name, cube_desc_data, user_session=False):
+ # workaround of #15337
+ time.sleep(random.randint(5, 10))
+ url = '/cubes'
+ payload = {
+ 'project': project_name,
+ 'cubeName': cube_name,
+ 'cubeDescData': json.dumps(cube_desc_data)
+ }
+ resp = self._request('POST', url, json=payload, inner_session=user_session)
+ return resp
+
+ def update_cube(self, project_name, cube_name, cube_desc_data, user_session=False):
+ # workaround of #15337
+ time.sleep(random.randint(5, 10))
+ url = '/cubes'
+ payload = {
+ 'project': project_name,
+ 'cubeName': cube_name,
+ 'cubeDescData': json.dumps(cube_desc_data)
+ }
+ resp = self._request('PUT', url, json=payload, inner_session=user_session)
+ return resp
+
+ def update_cube_engine(self, cube_name, engine_type):
+ url = '/cubes/{cube}/{engine}'.format(cube=cube_name, engine=engine_type)
+ resp = self._request('PUT', url)
+ return resp
+
+ def build_segment(self, cube_name, start_time, end_time, force=False):
+ """
+ :param cube_name: the name of the cube to be built
+ :param force: force submit mode
+ :param start_time: long, start time, corresponding to the timestamp in GMT format,
+ for instance, 1388534400000 corresponding to 2014-01-01 00:00:00
+ :param end_time: long, end time, corresponding to the timestamp in GMT format
+ :return:
+ """
+ url = '/cubes/{cube}/build'.format(cube=cube_name)
+ payload = {
+ 'buildType': 'BUILD',
+ 'startTime': start_time,
+ 'endTime': end_time,
+ 'force': force
+ }
+ resp = self._request('PUT', url, json=payload)
+ return resp
+
+ def full_build_cube(self, cube_name, force=False):
+ """
+ :param cube_name: the name of the cube to be built
+ :param force: force submit mode
+ :return:
+ """
+ return self.build_segment(cube_name, force=force, start_time=0, end_time=31556995200000)
+
+ def merge_segment(self, cube_name, start_time=0, end_time=31556995200000, force=True):
+ """
+ :param cube_name: the name of the cube to be built
+ :param force: force submit mode
+ :param start_time: long, start time, corresponding to the timestamp in GMT format,
+ for instance, 1388534400000 corresponding to 2014-01-01 00:00:00
+ :param end_time: long, end time, corresponding to the timestamp in GMT format
+ :return:
+ """
+ url = '/cubes/{cube}/build'.format(cube=cube_name)
+ payload = {
+ 'buildType': 'MERGE',
+ 'startTime': start_time,
+ 'endTime': end_time,
+ 'force': force
+ }
+ resp = self._request('PUT', url, json=payload)
+ return resp
+
+ def refresh_segment(self, cube_name, start_time, end_time, force=True):
+ """
+ :param cube_name: the name of the cube to be built
+ :param force: force submit mode
+ :param start_time: long, start time, corresponding to the timestamp in GMT format,
+ for instance, 1388534400000 corresponding to 2014-01-01 00:00:00
+ :param end_time: long, end time, corresponding to the timestamp in GMT format
+ :return:
+ """
+ url = '/cubes/{cube}/build'.format(cube=cube_name)
+ payload = {
+ 'buildType': 'REFRESH',
+ 'startTime': start_time,
+ 'endTime': end_time,
+ 'force': force
+ }
+ resp = self._request('PUT', url, json=payload)
+ return resp
+
+ def delete_segments(self, cube_name, segment_name):
+ url = '/cubes/{cube}/segs/{segment}'.format(cube=cube_name, segment=segment_name)
+ resp = self._request('DELETE', url)
+ return resp
+
+ def build_streaming_cube(self, project_name, cube_name, source_offset_start=0,
+ source_offset_end='9223372036854775807'):
+ """
+ :param cube_name: cube name
+ :param source_offset_start: long, the start offset where build begins. Here 0 means it is from the last position
+ :param source_offset_end: long, the end offset where build ends. 9223372036854775807 (Long.MAX_VALUE) means to
+ the end position on Kafka topic.
+ :param mp_values: string, multiple partition values of corresponding model
+ :param force: boolean, force submit mode
+ :return:
+ """
+ url = '/cubes/{cube}/segments/build_streaming'.format(cube=cube_name)
+ payload = {
+ 'buildType': 'BUILD',
+ 'project': project_name,
+ 'sourceOffsetStart': source_offset_start,
+ 'sourceOffsetEnd': source_offset_end,
+ }
+ resp = self._request('PUT', url, json=payload)
+ return resp
+
+ def build_cube_customized(self, cube_name, source_offset_start, source_offset_end=None, mp_values=None,
+ force=False):
+ """
+ :param cube_name: cube name
+ :param source_offset_start: long, the start offset where build begins
+ :param source_offset_end: long, the end offset where build ends
+ :param mp_values: string, multiple partition values of corresponding model
+ :param force: boolean, force submit mode
+ :return:
+ """
+ url = '/cubes/{cube}/segments/build_customized'.format(cube=cube_name)
+ payload = {
+ 'buildType': 'BUILD',
+ 'sourceOffsetStart': source_offset_start,
+ 'sourceOffsetEnd': source_offset_end,
+ 'mpValues': mp_values,
+ 'force': force
+ }
+ resp = self._request('PUT', url, json=payload)
+ return resp
+
+ def clone_cube(self, project_name, cube_name, new_cube_name):
+ """
+ :param project_name: project name
+ :param cube_name: cube name of being cloned
+ :param new_cube_name: cube name to be cloned to
+ :return:
+ """
+ url = '/cubes/{cube}/clone'.format(cube=cube_name)
+ payload = {
+ 'cubeName': new_cube_name,
+ 'project': project_name
+ }
+ resp = self._request('PUT', url, json=payload)
+ return resp
+
+ def enable_cube(self, cube_name):
+ url = '/cubes/{cube}/enable'.format(cube=cube_name)
+ resp = self._request('PUT', url)
+ return resp
+
+ def disable_cube(self, cube_name):
+ url = '/cubes/{cube}/disable'.format(cube=cube_name)
+ resp = self._request('PUT', url)
+ return resp
+
+ def purge_cube(self, cube_name):
+ url = '/cubes/{cube}/purge'.format(cube=cube_name)
+ resp = self._request('PUT', url)
+ return resp
+
+ def delete_cube(self, cube_name):
+ url = '/cubes/{cube}'.format(cube=cube_name)
+ return self._request('DELETE', url)
+
+ def list_holes(self, cube_name):
+ """
+ A healthy cube in production should not have holes in the meaning of inconsecutive segments.
+ :param cube_name: cube name
+ :return:
+ """
+ url = '/cubes/{cube}/holes'.format(cube=cube_name)
+ resp = self._request('GET', url)
+ return resp
+
+ def fill_holes(self, cube_name):
+ """
+ For non-streaming data based Cube, Kyligence Enterprise will submit normal build cube job(s) with
+ corresponding time partition value range(s); For streaming data based Cube, please make sure that
+ corresponding data is not expired or deleted in source before filling holes, otherwise the build job will fail.
+ :param cube_name: string, cube name
+ :return:
+ """
+ url = '/cubes/{cube}/holes'.format(cube=cube_name)
+ resp = self._request('PUT', url)
+ return resp
+
+ def export_cuboids(self, cube_name):
+ url = '/cubes/{cube}/cuboids/export'.fomat(cube=cube_name)
+ resp = self._request('PUT', url)
+ return resp
+
+ def refresh_lookup(self, cube_name, lookup_table):
+ """
+ Only lookup tables of SCD Type 1 are supported to refresh.
+ :param cube_name: cube name
+ :param lookup_table: the name of lookup table to be refreshed with the format DATABASE.TABLE
+ :return:
+ """
+ url = '/cubes/{cube}/refresh_lookup'.format(cube=cube_name)
+ payload = {
+ 'cubeName': cube_name,
+ 'lookupTableName': lookup_table
+ }
+ resp = self._request('PUT', url, json=payload)
+ return resp
+
+ def get_job_info(self, job_id):
+ url = '/jobs/{job_id}'.format(job_id=job_id)
+ resp = self._request('GET', url)
+ return resp
+
+ def get_job_status(self, job_id):
+ return self.get_job_info(job_id)['job_status']
+
+ def get_step_output(self, job_id, step_id):
+ url = '/jobs/{jobId}/steps/{stepId}/output'.format(jobId=job_id, stepId=step_id)
+ resp = self._request('GET', url)
+ return resp
+
+ def pause_job(self, job_id):
+ url = '/jobs/{jobId}/pause'.format(jobId=job_id)
+ resp = self._request('PUT', url)
+ return resp
+
+ def resume_job(self, job_id):
+ url = '/jobs/{jobId}/resume'.format(jobId=job_id)
+ resp = self._request('PUT', url)
+ return resp
+
+ def discard_job(self, job_id):
+ url = '/jobs/{jobId}/cancel'.format(jobId=job_id)
+ resp = self._request('PUT', url)
+ return resp
+
+ def delete_job(self, job_id):
+ url = '/jobs/{jobId}/drop'.format(jobId=job_id)
+ resp = self._request('DELETE', url)
+ return resp
+
+ def resubmit_job(self, job_id):
+ url = '/jobs/{jobId}/resubmit'.format(jobId=job_id)
+ resp = self._request('PUT', url)
+ return resp
+
+ def list_jobs(self, project_name, status=None, offset=0, limit=10000, time_filter=1, job_search_mode='ALL'):
+ """
+ list jobs in specific project
+ :param job_search_mode: CUBING_ONLY, CHECKPOINT_ONLY, ALL
+ :param project_name: project name
+ :param status: int, 0 -> NEW, 1 -> PENDING, 2 -> RUNNING,
+ 4 -> FINISHED, 8 -> ERROR, 16 -> DISCARDED, 32 -> STOPPED
+ :param offset: offset of returned result
+ :param limit: quantity of returned result per page
+ :param time_filter: int, 0 -> last one day, 1 -> last one week,
+ 2 -> last one month, 3 -> last one year, 4 -> all
+ :return:
+ """
+ url = '/jobs'
+ params = {
+ 'projectName': project_name,
+ 'status': status,
+ 'offset': offset,
+ 'limit': limit,
+ 'timeFilter': time_filter,
+ 'jobSearchMode': job_search_mode
+ }
+ resp = self._request('GET', url, params=params)
+ return resp
+
+ def await_all_jobs(self, project_name, waiting_time=30):
+ """
+ await all jobs to be finished, default timeout is 30 minutes
+ :param project_name: project name
+ :param waiting_time: timeout, in minutes
+ :return: boolean, timeout will return false
+ """
+ running_flag = ['PENDING', 'RUNNING']
+ try_time = 0
+ max_try_time = waiting_time * 2
+ # finish_flags = ['ERROR', 'FINISHED', 'DISCARDED']
+ while try_time < max_try_time:
+ jobs = self.list_jobs(project_name)
+ all_finished = True
+ for job in jobs:
+ if job['job_status'] in running_flag:
+ all_finished = False
+ break
+ if all_finished:
+ return True
+ time.sleep(30)
+ try_time += 1
+ return False
+
+ def await_job(self, job_id, waiting_time=20, interval=1, excepted_status=None):
+ """
+ Await specific job to be given status, default timeout is 20 minutes.
+ :param job_id: job id
+ :param waiting_time: timeout, in minutes.
+ :param interval: check interval, default value is 1 second
+ :param excepted_status: excepted job status list, default contains 'ERROR', 'FINISHED' and 'DISCARDED'
+ :return: boolean, if the job is in finish status, return true
+ """
+ finish_flags = ['ERROR', 'FINISHED', 'DISCARDED']
+ if excepted_status is None:
+ excepted_status = finish_flags
+ timeout = waiting_time * 60
+ start = time.time()
+ while time.time() - start < timeout:
+ job_status = self.get_job_status(job_id)
+ if job_status in excepted_status:
+ return True
+ if job_status in finish_flags:
+ return False
+ time.sleep(interval)
+ return False
+
+ def await_job_finished(self, job_id, waiting_time=20, interval=1):
+ """
+ Await specific job to be finished, default timeout is 20 minutes.
+ :param job_id: job id
+ :param waiting_time: timeout, in minutes.
+ :param interval: check interval, default value is 1 second
+ :return: boolean, if the job is in finish status, return true
+ """
+ return self.await_job(job_id, waiting_time, interval, excepted_status=['FINISHED'])
+
+ def await_job_error(self, job_id, waiting_time=20, interval=1):
+ """
+ Await specific job to be error, default timeout is 20 minutes.
+ :param job_id: job id
+ :param waiting_time: timeout, in minutes.
+ :param interval: check interval, default value is 1 second
+ :return: boolean, if the job is in finish status, return true
+ """
+ return self.await_job(job_id, waiting_time, interval, excepted_status=['ERROR'])
+
+ def await_job_discarded(self, job_id, waiting_time=20, interval=1):
+ """
+ Await specific job to be discarded, default timeout is 20 minutes.
+ :param job_id: job id
+ :param waiting_time: timeout, in minutes.
+ :param interval: check interval, default value is 1 second
+ :return: boolean, if the job is in finish status, return true
+ """
+ return self.await_job(job_id, waiting_time, interval, excepted_status=['DISCARDED'])
+
+ def await_job_step(self, job_id, step, excepted_status=None, waiting_time=20, interval=1):
+ """
+ Await specific job step to be given status, default timeout is 20 minutes.
+ :param job_id: job id
+ :param step: job step
+ :param waiting_time: timeout, in minutes.
+ :param interval: check interval, default value is 1 second
+ :param excepted_status: excepted job status list, default contains 'ERROR', 'FINISHED' and 'DISCARDED'
+ :return: boolean, if the job is in finish status, return true
+ """
+ finish_flags = ['ERROR', 'FINISHED', 'DISCARDED']
+ if excepted_status is None:
+ excepted_status = finish_flags
+ timeout = waiting_time * 60
+ start = time.time()
+ while time.time() - start < timeout:
+ job_info = self.get_job_info(job_id)
+ job_status = job_info['steps'][step]['step_status']
+ if job_status in excepted_status:
+ return True
+ if job_status in finish_flags:
+ return False
+ time.sleep(interval)
+ return False
+
+ def execute_query(self, project_name, sql, cube_name=None, offset=None, limit=None, backdoortoggles=None,
+ user_session=False,
+ timeout=60):
+ url = '/query'
+ payload = {
+ 'project': project_name,
+ 'sql': sql,
+ 'offset': offset,
+ 'limit': limit
+ }
+ if cube_name:
+ backdoortoggles = {"backdoorToggles": {"DEBUG_TOGGLE_HIT_CUBE": cube_name}}
+ if backdoortoggles:
+ payload.update(backdoortoggles)
+ resp = self._request('POST', url, json=payload, inner_session=user_session, timeout=timeout)
+ return resp
+
+ def save_query(self, sql_name, project_name, sql, description=None):
+ url = '/saved_queries'
+ payload = {
+ 'name': sql_name,
+ 'project': project_name,
+ 'sql': sql,
+ 'description': description
+ }
+ self._request('POST', url, json=payload)
+
+ def get_queries(self, project_name, user_session=False):
+ url = '/saved_queries'
+ params = {
+ 'project': project_name
+ }
+ response = self._request('GET', url, params=params, inner_session=user_session)
+ return response
+
+ def remove_query(self, sql_id):
+ url = '/saved_queries/{id}'.format(id=sql_id)
+ self._request('DELETE', url)
+
+ def list_queryable_tables(self, project_name):
+ url = '/tables_and_columns'
+ params = {'project': project_name}
+ resp = self._request('GET', url, params=params)
+ return resp
+
+ def get_all_system_prop(self, server=None):
+ url = '/admin/config'
+ if server is not None:
+ url = '/admin/config?server={serverName}'.format(serverName=server)
+ prop_resp = self._request('GET', url).get('config')
+ property_values = {}
+ if prop_resp is None:
+ return property_values
+ prop_lines = prop_resp.splitlines(False)
+ for prop_line in prop_lines:
+ splits = prop_line.split('=')
+ property_values[splits[0]] = splits[1]
+ return property_values
+
+ def create_user(self, user_name, password, authorities, disabled=False, user_session=False):
+ """
+ create a user
+ :param user_name: string, target user name
+ :param password: string, target password
+ :param authorities: array, user's authorities
+ :param disabled: boolean, true for disabled user false for enable user
+ :param user_session: boolean, true for using login session to execute
+ :return:
+ """
+ url = '/user/{username}'.format(username=user_name)
+ payload = {
+ 'username': user_name,
+ 'password': password,
+ 'authorities': authorities,
+ 'disabled': disabled,
+ }
+ resp = self._request('POST', url, json=payload, inner_session=user_session)
+ return resp
+
+ def delete_user(self, user_name, user_session=False):
+ """
+ delete user
+ :param user_name: string
+ :param user_session: boolean, true for using login session to execute
+ :return:
+ """
+ url = '/user/{username}'.format(username=user_name)
+ resp = self._request('DELETE', url, inner_session=user_session)
+ return resp
+
+ def update_user(self, user_name, authorities, password=None, disabled=False,
+ user_session=False, payload_user_name=None):
+ """
+ update user's info
+ :param user_name: string, target user name
+ :param password: string, target password
+ :param authorities: array, user's authorities
+ :param disabled: boolean, true for disabled user false for enable user
+ :param user_session: boolean, true for using login session to execute
+ :param payload_user_name: string, true for using login session to execute
+ :return:
+ """
+ url = '/user/{username}'.format(username=user_name)
+ username_in_payload = user_name if payload_user_name is None else payload_user_name
+ payload = {
+ 'username': username_in_payload,
+ 'password': password,
+ 'authorities': authorities,
+ 'disabled': disabled,
+ }
+ resp = self._request('PUT', url, json=payload, inner_session=user_session)
+ return resp
+
+ def update_user_password(self, user_name, new_password, password=None, user_session=False):
+ """
+ update user's password
+ :param user_name: string, target for username
+ :param new_password: string, user's new password
+ :param password: string, user's old password
+ :param user_session: boolean, true for using login session to execute
+ :return:
+ """
+ url = '/user/password'
+ payload = {
+ 'username': user_name,
+ 'password': password,
+ 'newPassword': new_password
+ }
+ resp = self._request('PUT', url, json=payload, inner_session=user_session)
+ return resp
+
+ def list_users(self, project_name=None, group_name=None, is_fuzz_match=False, name=None, offset=0, limit=10000
+ , user_session=False):
+ """
+ list users
+ :param group_name:string, group name
+ :param project_name: string, project's name
+ :param offset: offset of returned result
+ :param limit: quantity of returned result per page
+ :param is_fuzz_match: bool, true for param name fuzzy match
+ :param name: string, user's name
+ :param user_session: boolean, true for using login session to execute
+ :return:
+ """
+ url = '/user/users'
+ params = {
+ 'offset': offset,
+ 'limit': limit,
+ 'groupName': group_name,
+ 'project': project_name,
+ 'isFuzzMatch': is_fuzz_match,
+ 'name': name
+ }
+ resp = self._request('GET', url, params=params, inner_session=user_session)
+ return resp
+
+ def list_user_authorities(self, project_name, user_session=False):
+ """
+ list groups in a project
+ :param project_name: string, target project name
+ :param user_session: boolean, true for using login session to execute
+ :return:
+ """
+ url = '/user_group/groups'
+ params = {
+ 'project': project_name
+ }
+ resp = self._request('GET', url, params=params, inner_session=user_session)
+ return resp
+
+ def create_group(self, group_name, user_session=False):
+ """
+ create a group with group_name
+ :param group_name: string, target group name
+ :param user_session: boolean, true for using login session to execute
+ :return:
+ """
+ url = '/user_group/{group_name}'.format(group_name=group_name)
+ resp = self._request('POST', url, inner_session=user_session)
+ return resp
+
+ def delete_group(self, group_name, user_session=False):
+ """
+ delete group by group_name
+ :param group_name: string, target group name
+ :param user_session: boolean, true for using login session to execute
+ :return:
+ """
+ url = '/user_group/{group_name}'.format(group_name=group_name)
+ resp = self._request('DELETE', url, inner_session=user_session)
+ return resp
+
+ def add_or_del_users(self, group_name, users):
+ url = '/user_group/users/{group}'.format(group=group_name)
+ payload = {'users': users}
+ resp = self._request('POST', url, json=payload)
+ return resp
+
+ def _request(self, method, url, **kwargs): # pylint: disable=arguments-differ
+ return super()._request(method, self._base_url + url, **kwargs)
+
+
+def connect(**conf):
+ _host = conf.get('host')
+ _port = conf.get('port')
+ _version = conf.get('version')
+
+ return KylinHttpClient(_host, _port, _version)
diff --git a/build/CI/testing/kylin_utils/shell.py b/build/CI/testing/kylin_utils/shell.py
new file mode 100644
index 0000000..3263636
--- /dev/null
+++ b/build/CI/testing/kylin_utils/shell.py
@@ -0,0 +1,125 @@
+import logging
+from shlex import quote as shlex_quote
+
+import delegator
+import paramiko
+
+
+class SSHShellProcess:
+ def __init__(self, return_code, stdout, stderr):
+ self.return_code = return_code
+ self.output = stdout
+ self.err = stderr
+
+ @property
+ def ok(self):
+ return self.return_code == 0
+
+ def __repr__(self) -> str:
+ return f'SSHShellProcess {{return code: {self.return_code}, output: {self.output}, err: {self.err}}}'
+
+
+class SSHShell:
+ def __init__(self, host, username=None, password=None):
+ self.client = paramiko.SSHClient()
+ self.client.set_missing_host_key_policy(paramiko.AutoAddPolicy)
+ self.client.connect(host, username=username, password=password)
+
+ def command(self, script, timeout=120, get_pty=False):
+ logging.debug(f'ssh exec: {script}')
+ self.client.get_transport().set_keepalive(5)
+ chan = self.client.get_transport().open_session()
+
+ if get_pty:
+ chan.get_pty()
+
+ chan.settimeout(timeout)
+
+ chan.exec_command(f'bash --login -c {shlex_quote(script)}')
+
+ bufsize = 4096
+
+ stdout = ''.join(chan.makefile('r', bufsize))
+ stderr = ''.join(chan.makefile_stderr('r', bufsize))
+
+ return SSHShellProcess(chan.recv_exit_status(), stdout, stderr)
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, type, value, traceback): # pylint: disable=redefined-builtin
+ self.client.close()
+
+
+class BashProcess:
+ """bash process object"""
+
+ def __init__(self, script, blocking: bool = True, timeout: int = 60) -> None:
+ """constructor"""
+ # Environ inherits from parent.
+
+ # Remember passed-in arguments.
+ self.script = script
+
+ # Run the subprocess.
+ self.sub = delegator.run(
+ script, block=blocking, timeout=timeout
+ )
+
+ @property
+ def output(self) -> str:
+ """stdout of the running process"""
+ return str(self.sub.out)
+
+ @property
+ def err(self) -> str:
+ """stderr of the running process"""
+ return str(self.sub.err)
+
+ @property
+ def ok(self) -> bool:
+ """if the process exited with a 0 exit code"""
+ return self.sub.ok
+
+ @property
+ def return_code(self) -> int:
+ """the exit code of the process"""
+ return self.sub.return_code
+
+ def __repr__(self) -> str:
+ return f'BashProcess {{return code: {self.return_code}, output: {self.output}, err: {self.err}}}'
+
+
+class Bash:
+ """an instance of bash"""
+
+ def _exec(self, script, timeout=60) -> BashProcess: # pylint: disable=unused-argument
+ """execute the bash process as a child of this process"""
+ return BashProcess(script, timeout=timeout)
+
+ def command(self, script: str, timeout=60) -> BashProcess:
+ """form up the command with shlex and execute"""
+ logging.debug(f'bash exec: {script}')
+ return self._exec(f"bash -c {shlex_quote(script)}", timeout=timeout)
+
+
+def sshexec(script, host, username=None, password=None):
+ with sshshell(host, username=username, password=password) as ssh:
+ return ssh.command(script)
+
+
+def sshshell(host, username=None, password=None):
+ return SSHShell(host, username=username, password=password)
+
+
+def exec(script): # pylint: disable=redefined-builtin
+ return Bash().command(script)
+
+
+def shell():
+ return Bash()
+
+
+if __name__ == '__main__':
+ sh = sshshell('10.1.3.94', username='root', password='hadoop')
+ print(sh.command('pwd'))
\ No newline at end of file
diff --git a/build/CI/testing/kylin_utils/util.py b/build/CI/testing/kylin_utils/util.py
new file mode 100644
index 0000000..47ca11e
--- /dev/null
+++ b/build/CI/testing/kylin_utils/util.py
@@ -0,0 +1,64 @@
+from selenium import webdriver
+from yaml import load, loader
+import os
+
+from kylin_utils import kylin
+
+
+def setup_instance(file_name):
+ instances_file = os.path.join('kylin_instances/', file_name)
+ stream = open(instances_file, 'r')
+ for item in load(stream, Loader=loader.SafeLoader):
+ host = item['host']
+ port = item['port']
+ version = item['version']
+ return kylin.connect(host=host, port=port, version=version)
+
+
+def kylin_url(file_name):
+ instances_file = os.path.join('kylin_instances/', file_name)
+ stream = open(instances_file, 'r')
+ for item in load(stream, Loader=loader.SafeLoader):
+ host = item['host']
+ port = item['port']
+ return "http://{host}:{port}/kylin".format(host=host, port=port)
+
+
+def setup_browser(browser_type):
+ if browser_type == "chrome":
+ browser = webdriver.Chrome(executable_path="browser_driver/chromedriver")
+
+ if browser_type == "firefox":
+ browser = webdriver.Firefox(executable_path="browser_driver/geckodriver")
+
+ if browser_type == "safari":
+ browser = webdriver.Safari(executable_path="browser_driver/safaridriver")
+
+ return browser
+
+
+def if_project_exists(kylin_client, project):
+ exists = 0
+ resp = kylin_client.list_projects()
+ for project_info in resp:
+ if project_info.get('name') == project:
+ exists = 1
+ return exists
+
+
+def if_cube_exists(kylin_client, cube_name, project=None):
+ exists = 0
+ resp = kylin_client.list_cubes(project=project)
+ if resp is not None:
+ for cube_info in resp:
+ if cube_info.get('name') == cube_name:
+ exists = 1
+ return exists
+
+
+def if_model_exists(kylin_client, model_name, project):
+ exists = 0
+ resp = kylin_client.list_model_desc(project_name=project, model_name=model_name)
+ if len(resp) == 1:
+ exists = 1
+ return exists
diff --git a/build/CI/testing/manifest.json b/build/CI/testing/manifest.json
new file mode 100644
index 0000000..bc5c9c8
--- /dev/null
+++ b/build/CI/testing/manifest.json
@@ -0,0 +1,6 @@
+{
+ "Language": "python",
+ "Plugins": [
+ "html-report"
+ ]
+}
\ No newline at end of file
diff --git a/build/CI/testing/requirements.txt b/build/CI/testing/requirements.txt
new file mode 100644
index 0000000..8c47779
--- /dev/null
+++ b/build/CI/testing/requirements.txt
@@ -0,0 +1,6 @@
+flake8==3.8.3
+getgauge==0.3.11
+pylint==2.3.1
+requests==2.21.0
+selenium==3.141.0
+yapf==0.30.0