You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by "bingli3@iflytek.com" <bi...@iflytek.com> on 2017/03/31 07:49:31 UTC
How Kylin Cuboid Scheduler Work With Aggregation Groups ?
Hi，all
I have a Cube, the desc is :
{
"uuid": "bcf11be2-83e4-497e-9e35-a402460a6446",
"last_modified": 1490860973892,
"version": "1.6.0",
"name": "adx_flow_insight",
"model_name": "adx_operator",
"description": "",
"null_string": null,
"dimensions": [
{
"name": "GENDER",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "GENDER",
"derived": null
},
{
"name": "AGE",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "AGE",
"derived": null
},
{
"name": "BRAND",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "BRAND",
"derived": null
},
{
"name": "MODEL",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "MODEL",
"derived": null
},
{
"name": "RESOLUTION",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "RESOLUTION",
"derived": null
},
{
"name": "OS_VERSION",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "OS_VERSION",
"derived": null
},
{
"name": "NTT",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "NTT",
"derived": null
},
{
"name": "TS_MINUTE",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "TS_MINUTE",
"derived": null
},
{
"name": "TS_HOUR",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "TS_HOUR",
"derived": null
},
{
"name": "DAY_TIME",
"table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
"column": "DAY_TIME",
"derived": null
}
],
"measures": [
{
"name": "_COUNT_",
"function": {
"expression": "COUNT",
"parameter": {
"type": "constant",
"value": "1",
"next_parameter": null
},
"returntype": "bigint"
},
"dependent_measure_ref": null
},
{
"name": "REQUEST_PV",
"function": {
"expression": "SUM",
"parameter": {
"type": "column",
"value": "REQUEST",
"next_parameter": null
},
"returntype": "bigint"
},
"dependent_measure_ref": null
},
{
"name": "IMPRESS_PV",
"function": {
"expression": "SUM",
"parameter": {
"type": "column",
"value": "IMPRESS",
"next_parameter": null
},
"returntype": "bigint"
},
"dependent_measure_ref": null
},
{
"name": "CLICK_PV",
"function": {
"expression": "SUM",
"parameter": {
"type": "column",
"value": "CLICK",
"next_parameter": null
},
"returntype": "bigint"
},
"dependent_measure_ref": null
},
{
"name": "FILL_PV",
"function": {
"expression": "SUM",
"parameter": {
"type": "column",
"value": "FILL",
"next_parameter": null
},
"returntype": "bigint"
},
"dependent_measure_ref": null
},
{
"name": "UV_DID",
"function": {
"expression": "COUNT_DISTINCT",
"parameter": {
"type": "column",
"value": "DID",
"next_parameter": null
},
"returntype": "hllc(10)"
},
"dependent_measure_ref": null
}
],
"dictionaries": [],
"rowkey": {
"rowkey_columns": [
{
"column": "DAY_TIME",
"encoding": "date",
"isShardBy": false
},
{
"column": "TS_MINUTE",
"encoding": "integer:4",
"isShardBy": false
},
{
"column": "TS_HOUR",
"encoding": "integer:4",
"isShardBy": false
},
{
"column": "GENDER",
"encoding": "dict",
"isShardBy": false
},
{
"column": "AGE",
"encoding": "dict",
"isShardBy": false
},
{
"column": "BRAND",
"encoding": "dict",
"isShardBy": false
},
{
"column": "MODEL",
"encoding": "dict",
"isShardBy": false
},
{
"column": "RESOLUTION",
"encoding": "dict",
"isShardBy": false
},
{
"column": "OS_VERSION",
"encoding": "dict",
"isShardBy": false
},
{
"column": "NTT",
"encoding": "dict",
"isShardBy": false
}
]
},
"hbase_mapping": {
"column_family": [
{
"name": "F1",
"columns": [
{
"qualifier": "M",
"measure_refs": [
"_COUNT_",
"REQUEST_PV",
"IMPRESS_PV",
"CLICK_PV",
"FILL_PV"
]
}
]
},
{
"name": "F2",
"columns": [
{
"qualifier": "M",
"measure_refs": [
"UV_DID"
]
}
]
}
]
},
"aggregation_groups": [
{
"includes": [
"DAY_TIME",
"GENDER"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"DAY_TIME"
],
"joint_dims": []
}
},
{
"includes": [
"DAY_TIME",
"AGE"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"DAY_TIME"
],
"joint_dims": []
}
},
{
"includes": [
"DAY_TIME",
"BRAND"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"DAY_TIME"
],
"joint_dims": []
}
},
{
"includes": [
"DAY_TIME",
"MODEL"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"DAY_TIME"
],
"joint_dims": []
}
},
{
"includes": [
"DAY_TIME",
"RESOLUTION"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"RESOLUTION"
],
"joint_dims": []
}
},
{
"includes": [
"DAY_TIME",
"OS_VERSION"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"DAY_TIME"
],
"joint_dims": []
}
},
{
"includes": [
"DAY_TIME",
"NTT"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"DAY_TIME"
],
"joint_dims": []
}
},
{
"includes": [
"TS_MINUTE"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"TS_MINUTE"
],
"joint_dims": []
}
},
{
"includes": [
"TS_HOUR"
],
"select_rule": {
"hierarchy_dims": [],
"mandatory_dims": [
"TS_HOUR"
],
"joint_dims": []
}
}
],
"signature": "DSSmByHn2sATiETlBdjANQ==",
"notify_list": [],
"status_need_notify": [
"ERROR",
"DISCARDED",
"SUCCEED"
],
"partition_date_start": 1488326400000,
"partition_date_end": 3153600000000,
"auto_merge_time_ranges": [
604800000,
2419200000
],
"retention_range": 0,
"engine_type": 2,
"storage_type": 2,
"override_kylin_properties": {
"kylin.job.mr.config.override.mapreduce.job.queuename": "ad"
}
}
There have 10 dims, and use aggregation groups. I want Cube only contains 10 combs:
<day_time, gender> 576
<day_time, age> 544
<day_time, brand> 528
<day_time, model> 520
<day_time, resolution> 516
<day_time, os_version> 514
<day_time, ntt> 513
<ts_minute> 256
<ts_hour> 128
<day_time> 512
But the Cuboid Scheduler parse as follower:
2017-03-31 15:32:47,735 (main) [INFO - org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:908)] Loaded 4 cubes, fail on 0 cubes
1023
516
576
513
528
514
544
520
2017-03-31 15:32:47,742 (Thread-0) [INFO - org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.closeMasterService(ConnectionManager.java:2259)] Closing master protocol: MasterService
Question:
How Aggregation Groups Work? I can not set single dim in aggregation?
Thanks for you suggestion~~~
bingli3@iflytek.com
Re: How Kylin Cuboid Scheduler Work With Aggregation Groups ?
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Bing,
An aggregation group is a dimension group, or say a sub-cube; it is NOT a
cuboid.
I guess you want to precisely define the cuboids/combinations, that isn't
supported as in many cases user couldn't list all the combinations they
use. But you can describe them with the agg group / mandatory / joint as
close as possible.
2017-03-31 15:49 GMT+08:00 bingli3@iflytek.com <bi...@iflytek.com>:
> Hi，all
> I have a Cube, the desc is :
>
> {
> "uuid": "bcf11be2-83e4-497e-9e35-a402460a6446",
> "last_modified": 1490860973892,
> "version": "1.6.0",
> "name": "adx_flow_insight",
> "model_name": "adx_operator",
> "description": "",
> "null_string": null,
> "dimensions": [
> {
> "name": "GENDER",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "GENDER",
> "derived": null
> },
> {
> "name": "AGE",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "AGE",
> "derived": null
> },
> {
> "name": "BRAND",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "BRAND",
> "derived": null
> },
> {
> "name": "MODEL",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "MODEL",
> "derived": null
> },
> {
> "name": "RESOLUTION",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "RESOLUTION",
> "derived": null
> },
> {
> "name": "OS_VERSION",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "OS_VERSION",
> "derived": null
> },
> {
> "name": "NTT",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "NTT",
> "derived": null
> },
> {
> "name": "TS_MINUTE",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "TS_MINUTE",
> "derived": null
> },
> {
> "name": "TS_HOUR",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "TS_HOUR",
> "derived": null
> },
> {
> "name": "DAY_TIME",
> "table": "FLOW_INSIGHT.VIEW_FLOW_INSIGHT",
> "column": "DAY_TIME",
> "derived": null
> }
> ],
> "measures": [
> {
> "name": "_COUNT_",
> "function": {
> "expression": "COUNT",
> "parameter": {
> "type": "constant",
> "value": "1",
> "next_parameter": null
> },
> "returntype": "bigint"
> },
> "dependent_measure_ref": null
> },
> {
> "name": "REQUEST_PV",
> "function": {
> "expression": "SUM",
> "parameter": {
> "type": "column",
> "value": "REQUEST",
> "next_parameter": null
> },
> "returntype": "bigint"
> },
> "dependent_measure_ref": null
> },
> {
> "name": "IMPRESS_PV",
> "function": {
> "expression": "SUM",
> "parameter": {
> "type": "column",
> "value": "IMPRESS",
> "next_parameter": null
> },
> "returntype": "bigint"
> },
> "dependent_measure_ref": null
> },
> {
> "name": "CLICK_PV",
> "function": {
> "expression": "SUM",
> "parameter": {
> "type": "column",
> "value": "CLICK",
> "next_parameter": null
> },
> "returntype": "bigint"
> },
> "dependent_measure_ref": null
> },
> {
> "name": "FILL_PV",
> "function": {
> "expression": "SUM",
> "parameter": {
> "type": "column",
> "value": "FILL",
> "next_parameter": null
> },
> "returntype": "bigint"
> },
> "dependent_measure_ref": null
> },
> {
> "name": "UV_DID",
> "function": {
> "expression": "COUNT_DISTINCT",
> "parameter": {
> "type": "column",
> "value": "DID",
> "next_parameter": null
> },
> "returntype": "hllc(10)"
> },
> "dependent_measure_ref": null
> }
> ],
> "dictionaries": [],
> "rowkey": {
> "rowkey_columns": [
> {
> "column": "DAY_TIME",
> "encoding": "date",
> "isShardBy": false
> },
> {
> "column": "TS_MINUTE",
> "encoding": "integer:4",
> "isShardBy": false
> },
> {
> "column": "TS_HOUR",
> "encoding": "integer:4",
> "isShardBy": false
> },
> {
> "column": "GENDER",
> "encoding": "dict",
> "isShardBy": false
> },
> {
> "column": "AGE",
> "encoding": "dict",
> "isShardBy": false
> },
> {
> "column": "BRAND",
> "encoding": "dict",
> "isShardBy": false
> },
> {
> "column": "MODEL",
> "encoding": "dict",
> "isShardBy": false
> },
> {
> "column": "RESOLUTION",
> "encoding": "dict",
> "isShardBy": false
> },
> {
> "column": "OS_VERSION",
> "encoding": "dict",
> "isShardBy": false
> },
> {
> "column": "NTT",
> "encoding": "dict",
> "isShardBy": false
> }
> ]
> },
> "hbase_mapping": {
> "column_family": [
> {
> "name": "F1",
> "columns": [
> {
> "qualifier": "M",
> "measure_refs": [
> "_COUNT_",
> "REQUEST_PV",
> "IMPRESS_PV",
> "CLICK_PV",
> "FILL_PV"
> ]
> }
> ]
> },
> {
> "name": "F2",
> "columns": [
> {
> "qualifier": "M",
> "measure_refs": [
> "UV_DID"
> ]
> }
> ]
> }
> ]
> },
> "aggregation_groups": [
> {
> "includes": [
> "DAY_TIME",
> "GENDER"
> ],
> "select_rule": {
> "hierarchy_dims": [],
> "mandatory_dims": [
> "DAY_TIME"
> ],
> "joint_dims": []
> }
> },
> {
> "includes": [
> "DAY_TIME",
> "AGE"
> ],
> "select_rule": {
> "hierarchy_dims": [],
> "mandatory_dims": [
> "DAY_TIME"
> ],
> "joint_dims": []
> }
> },
> {
> "includes": [
> "DAY_TIME",
> "BRAND"
> ],
> "select_rule": {
> "hierarchy_dims": [],
> "mandatory_dims": [
> "DAY_TIME"
> ],
> "joint_dims": []
> }
> },
> {
> "includes": [
> "DAY_TIME",
> "MODEL"
> ],
> "select_rule": {
> "hierarchy_dims": [],
> "mandatory_dims": [
> "DAY_TIME"
> ],
> "joint_dims": []
> }
> },
> {
> "includes": [
> "DAY_TIME",
> "RESOLUTION"
> ],
> "select_rule": {
> "hierarchy_dims": [],
> "mandatory_dims": [
> "RESOLUTION"
> ],
> "joint_dims": []
> }
> },
> {
> "includes": [
> "DAY_TIME",
> "OS_VERSION"
> ],
> "select_rule": {
> "hierarchy_dims": [],
> "mandatory_dims": [
> "DAY_TIME"
> ],
> "joint_dims": []
> }
> },
> {
> "includes": [
> "DAY_TIME",
> "NTT"
> ],
> "select_rule": {
> "hierarchy_dims": [],
> "mandatory_dims": [
> "DAY_TIME"
> ],
> "joint_dims": []
> }
> },
> {
> "includes": [
> "TS_MINUTE"
> ],
> "select_rule": {
> "hierarchy_dims": [],
> "mandatory_dims": [
> "TS_MINUTE"
> ],
> "joint_dims": []
> }
> },
> {
> "includes": [
> "TS_HOUR"
> ],
> "select_rule": {
> "hierarchy_dims": [],
> "mandatory_dims": [
> "TS_HOUR"
> ],
> "joint_dims": []
> }
> }
> ],
> "signature": "DSSmByHn2sATiETlBdjANQ==",
> "notify_list": [],
> "status_need_notify": [
> "ERROR",
> "DISCARDED",
> "SUCCEED"
> ],
> "partition_date_start": 1488326400000,
> "partition_date_end": 3153600000000,
> "auto_merge_time_ranges": [
> 604800000,
> 2419200000
> ],
> "retention_range": 0,
> "engine_type": 2,
> "storage_type": 2,
> "override_kylin_properties": {
> "kylin.job.mr.config.override.mapreduce.job.queuename": "ad"
> }
>
> }
>
> There have 10 dims, and use aggregation groups. I want Cube only
> contains 10 combs:
> <day_time, gender> 576
> <day_time, age> 544
> <day_time, brand> 528
> <day_time, model> 520
> <day_time, resolution> 516
> <day_time, os_version> 514
> <day_time, ntt> 513
> <ts_minute> 256
> <ts_hour> 128
> <day_time> 512
>
> But the Cuboid Scheduler parse as follower:
>
> 2017-03-31 15:32:47,735 (main) [INFO - org..apache.kylin.cube.CubeManager.
> loadAllCubeInstance(CubeManager.java:908)] Loaded 4 cubes, fail on 0 cubes
> 1023
> 516
> 576
> 513
> 528
> 514
> 544
> 520
> 2017-03-31 15:32:47,742 (Thread-0) [INFO - org.apache.hadoop.hbase.client.
> ConnectionManager$HConnectionImplementation.closeMasterService(
> ConnectionManager.java:2259)] Closing master protocol: MasterService
>
> Question:
> How Aggregation Groups Work? I can not set single dim in
> aggregation?
>
> Thanks for you suggestion~~~
>
> ------------------------------
> bingli3@iflytek.com
>
--
Best regards,
Shaofeng Shi 史少锋