You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by op <52...@qq.com> on 2017/10/17 10:06:40 UTC
data format when building increment cube
hello everyone.
how can i define my own data format?
our hive tables are partitioned by month_id(format :yyyymm) and subpartitioned by day_id(format ：dd)
thanks!
Re: data format when building increment cube
Posted by ShaoFeng Shi <sh...@apache.org>.
Run "metastore.sh clean --delete true" will remove finished jobs earlier
than 30 days;
If you want to delete a specific job, you need to get the job uuid, then
run:
"metastore.sh remove /execute/<job_uuid>"
If you want to clean up all jobs, you can run (this is dangerous, you'd
better to take a backup):
"metastore.sh remove /execute"
2017-10-26 9:29 GMT+08:00 op <52...@qq.com>:
> hello shaofeng .
> i want to know how to clean the monitor list. could you help me?
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2017年10月23日(星期一) 晚上11:09
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: data format when building increment cube
>
>
>
> Oh, I'm using 2.2 (will release soon).
>
> Then you can run "bin/mestore.sh backup", and then find the model json file
> in "model_desc" folder, in it you will see. After modify the
> partition class, you can restore from that folder back to HBase, and then
> restart Kylin.
>
> 2017-10-23 18:00 GMT+08:00 op <52...@qq.com>:
>
> > on kylin web,there is no "Edit Json" ......... my version 2.0
> >
> >
> > ------------------ 原始邮件 ------------------
> > *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
> > *发送时间:* 2017年10月23日(星期一) 下午5:53
> > *收件人:* "dev"<de...@kylin.apache.org>;
> > *主题:* Re: data format when building increment cube
> >
> > You can backup the metastore to local, then you will see the model json
> > file.
> >
> > On Kylin web, move mouse to the model, click "Action" -> "Edit JSON",
> will
> > see the JSON content.
> >
> > 2017-10-23 17:23 GMT+08:00 op <52...@qq.com>:
> >
> > > thank you shaofeng，but i cannot find the json file
> > > or you mean change it on kylin web？
> > >
> > >
> > >
> > >
> > > ------------------ 原始邮件 ------------------
> > > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > > 发送时间: 2017年10月23日(星期一) 下午5:09
> > > 收件人: "dev"<de...@kylin.apache.org>;
> > >
> > > 主题: Re: data format when building increment cube
> > >
> > >
> > >
> > > If you create a new model from Kylin web, the model has a property
> > > "partition_condition_builder" which refers to the class that
> > > implements the PartitionConditionBuilder.
> > > You can manually change it to your customized implementation:
> > >
> > >
> > > {
> > > "uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
> > > "last_modified" : 1508748727953,
> > > "name" : "test_parition",
> > > "owner" : "admin",
> > > "is_draft" : false,
> > > "description" : "",
> > > "fact_table" : "DEFAULT.KYLIN_SALES",
> > > "lookups" : [ ],
> > > "dimensions" : [ {
> > > "table" : "KYLIN_SALES",
> > > "columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
> > > "LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID",
> > "BUYER_ID",
> > > "OPS_USER_ID", "OPS_REGION" ]
> > > } ],
> > > "metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
> > > "filter_condition" : "",
> > > "partition_desc" : {
> > > "partition_date_column" : "KYLIN_SALES.PART_DT",
> > > "partition_time_column" : null,
> > > "partition_date_start" : 0,
> > > "partition_date_format" : "yyyy-MM-dd",
> > > "partition_time_format" : "HH:mm:ss",
> > > "partition_type" : "APPEND",
> > > "partition_condition_builder" :
> > > "org.apache.kylin.metadata.model.PartitionDesc$
> > > DefaultPartitionConditionBuilder"
> > > },
> > > "capacity" : "MEDIUM"
> > > }
> > >
> > > 2017-10-23 16:19 GMT+08:00 op <52...@qq.com>:
> > >
> > > > hi shaofeng
> > > > then how to use a new PartitionConditionBuilder？
> > > > thanks！
> > > >
> > > >
> > > > ------------------ 原始邮件 ------------------
> > > > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > > > 发送时间: 2017年10月17日(星期二) 晚上6:16
> > > > 收件人: "dev"<de...@kylin.apache.org>;
> > > >
> > > > 主题: Re: data format when building increment cube
> > > >
> > > >
> > > >
> > > > It is not supported. You need to develop a new
> > PartitionConditionBuilder.
> > > > Please refer
> > > > to org.apache.kylin.metadata.model.PartitionDesc.
> > > > YearMonthDayPartitionConditionBuilder
> > > >
> > > > 2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
> > > >
> > > > >
> > > > > hello everyone.
> > > > > how can i define my own data format?
> > > > > our hive tables are partitioned by month_id(format :yyyymm) and
> > > > > subpartitioned by day_id(format ：dd)
> > > > >
> > > > > thanks!
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > >
> > > > Shaofeng Shi 史少锋
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
> >
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
回复： data format when building increment cube
Posted by op <52...@qq.com>.
hello shaofeng .
i want to know how to clean the monitor list. could you help me?
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年10月23日(星期一) 晚上11:09
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: data format when building increment cube
Oh, I'm using 2.2 (will release soon).
Then you can run "bin/mestore.sh backup", and then find the model json file
in "model_desc" folder, in it you will see. After modify the
partition class, you can restore from that folder back to HBase, and then
restart Kylin.
2017-10-23 18:00 GMT+08:00 op <52...@qq.com>:
> on kylin web,there is no "Edit Json" ......... my version 2.0
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
> *发送时间:* 2017年10月23日(星期一) 下午5:53
> *收件人:* "dev"<de...@kylin.apache.org>;
> *主题:* Re: data format when building increment cube
>
> You can backup the metastore to local, then you will see the model json
> file.
>
> On Kylin web, move mouse to the model, click "Action" -> "Edit JSON", will
> see the JSON content.
>
> 2017-10-23 17:23 GMT+08:00 op <52...@qq.com>:
>
> > thank you shaofeng，but i cannot find the json file
> > or you mean change it on kylin web？
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > 发送时间: 2017年10月23日(星期一) 下午5:09
> > 收件人: "dev"<de...@kylin.apache.org>;
> >
> > 主题: Re: data format when building increment cube
> >
> >
> >
> > If you create a new model from Kylin web, the model has a property
> > "partition_condition_builder" which refers to the class that
> > implements the PartitionConditionBuilder.
> > You can manually change it to your customized implementation:
> >
> >
> > {
> > "uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
> > "last_modified" : 1508748727953,
> > "name" : "test_parition",
> > "owner" : "admin",
> > "is_draft" : false,
> > "description" : "",
> > "fact_table" : "DEFAULT.KYLIN_SALES",
> > "lookups" : [ ],
> > "dimensions" : [ {
> > "table" : "KYLIN_SALES",
> > "columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
> > "LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID",
> "BUYER_ID",
> > "OPS_USER_ID", "OPS_REGION" ]
> > } ],
> > "metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
> > "filter_condition" : "",
> > "partition_desc" : {
> > "partition_date_column" : "KYLIN_SALES.PART_DT",
> > "partition_time_column" : null,
> > "partition_date_start" : 0,
> > "partition_date_format" : "yyyy-MM-dd",
> > "partition_time_format" : "HH:mm:ss",
> > "partition_type" : "APPEND",
> > "partition_condition_builder" :
> > "org.apache.kylin.metadata.model.PartitionDesc$
> > DefaultPartitionConditionBuilder"
> > },
> > "capacity" : "MEDIUM"
> > }
> >
> > 2017-10-23 16:19 GMT+08:00 op <52...@qq.com>:
> >
> > > hi shaofeng
> > > then how to use a new PartitionConditionBuilder？
> > > thanks！
> > >
> > >
> > > ------------------ 原始邮件 ------------------
> > > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > > 发送时间: 2017年10月17日(星期二) 晚上6:16
> > > 收件人: "dev"<de...@kylin.apache.org>;
> > >
> > > 主题: Re: data format when building increment cube
> > >
> > >
> > >
> > > It is not supported. You need to develop a new
> PartitionConditionBuilder.
> > > Please refer
> > > to org.apache.kylin.metadata.model.PartitionDesc.
> > > YearMonthDayPartitionConditionBuilder
> > >
> > > 2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
> > >
> > > >
> > > > hello everyone.
> > > > how can i define my own data format?
> > > > our hive tables are partitioned by month_id(format :yyyymm) and
> > > > subpartitioned by day_id(format ：dd)
> > > >
> > > > thanks!
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: data format when building increment cube
Posted by ShaoFeng Shi <sh...@apache.org>.
Oh, I'm using 2.2 (will release soon).
Then you can run "bin/mestore.sh backup", and then find the model json file
in "model_desc" folder, in it you will see. After modify the
partition class, you can restore from that folder back to HBase, and then
restart Kylin.
2017-10-23 18:00 GMT+08:00 op <52...@qq.com>:
> on kylin web,there is no "Edit Json" ......... my version 2.0
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
> *发送时间:* 2017年10月23日(星期一) 下午5:53
> *收件人:* "dev"<de...@kylin.apache.org>;
> *主题:* Re: data format when building increment cube
>
> You can backup the metastore to local, then you will see the model json
> file.
>
> On Kylin web, move mouse to the model, click "Action" -> "Edit JSON", will
> see the JSON content.
>
> 2017-10-23 17:23 GMT+08:00 op <52...@qq.com>:
>
> > thank you shaofeng，but i cannot find the json file
> > or you mean change it on kylin web？
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > 发送时间: 2017年10月23日(星期一) 下午5:09
> > 收件人: "dev"<de...@kylin.apache.org>;
> >
> > 主题: Re: data format when building increment cube
> >
> >
> >
> > If you create a new model from Kylin web, the model has a property
> > "partition_condition_builder" which refers to the class that
> > implements the PartitionConditionBuilder.
> > You can manually change it to your customized implementation:
> >
> >
> > {
> > "uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
> > "last_modified" : 1508748727953,
> > "name" : "test_parition",
> > "owner" : "admin",
> > "is_draft" : false,
> > "description" : "",
> > "fact_table" : "DEFAULT.KYLIN_SALES",
> > "lookups" : [ ],
> > "dimensions" : [ {
> > "table" : "KYLIN_SALES",
> > "columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
> > "LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID",
> "BUYER_ID",
> > "OPS_USER_ID", "OPS_REGION" ]
> > } ],
> > "metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
> > "filter_condition" : "",
> > "partition_desc" : {
> > "partition_date_column" : "KYLIN_SALES.PART_DT",
> > "partition_time_column" : null,
> > "partition_date_start" : 0,
> > "partition_date_format" : "yyyy-MM-dd",
> > "partition_time_format" : "HH:mm:ss",
> > "partition_type" : "APPEND",
> > "partition_condition_builder" :
> > "org.apache.kylin.metadata.model.PartitionDesc$
> > DefaultPartitionConditionBuilder"
> > },
> > "capacity" : "MEDIUM"
> > }
> >
> > 2017-10-23 16:19 GMT+08:00 op <52...@qq.com>:
> >
> > > hi shaofeng
> > > then how to use a new PartitionConditionBuilder？
> > > thanks！
> > >
> > >
> > > ------------------ 原始邮件 ------------------
> > > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > > 发送时间: 2017年10月17日(星期二) 晚上6:16
> > > 收件人: "dev"<de...@kylin.apache.org>;
> > >
> > > 主题: Re: data format when building increment cube
> > >
> > >
> > >
> > > It is not supported. You need to develop a new
> PartitionConditionBuilder.
> > > Please refer
> > > to org.apache.kylin.metadata.model.PartitionDesc.
> > > YearMonthDayPartitionConditionBuilder
> > >
> > > 2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
> > >
> > > >
> > > > hello everyone.
> > > > how can i define my own data format?
> > > > our hive tables are partitioned by month_id(format :yyyymm) and
> > > > subpartitioned by day_id(format ：dd)
> > > >
> > > > thanks!
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
回复： data format when building increment cube
Posted by op <52...@qq.com>.
on kylin web,there is no "Edit Json" ......... my version 2.0
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年10月23日(星期一) 下午5:53
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: data format when building increment cube
You can backup the metastore to local, then you will see the model json
file.
On Kylin web, move mouse to the model, click "Action" -> "Edit JSON", will
see the JSON content.
2017-10-23 17:23 GMT+08:00 op <52...@qq.com>:
> thank you shaofeng，but i cannot find the json file
> or you mean change it on kylin web？
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2017年10月23日(星期一) 下午5:09
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: data format when building increment cube
>
>
>
> If you create a new model from Kylin web, the model has a property
> "partition_condition_builder" which refers to the class that
> implements the PartitionConditionBuilder.
> You can manually change it to your customized implementation:
>
>
> {
> "uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
> "last_modified" : 1508748727953,
> "name" : "test_parition",
> "owner" : "admin",
> "is_draft" : false,
> "description" : "",
> "fact_table" : "DEFAULT.KYLIN_SALES",
> "lookups" : [ ],
> "dimensions" : [ {
> "table" : "KYLIN_SALES",
> "columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
> "LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID", "BUYER_ID",
> "OPS_USER_ID", "OPS_REGION" ]
> } ],
> "metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
> "filter_condition" : "",
> "partition_desc" : {
> "partition_date_column" : "KYLIN_SALES.PART_DT",
> "partition_time_column" : null,
> "partition_date_start" : 0,
> "partition_date_format" : "yyyy-MM-dd",
> "partition_time_format" : "HH:mm:ss",
> "partition_type" : "APPEND",
> "partition_condition_builder" :
> "org.apache.kylin.metadata.model.PartitionDesc$
> DefaultPartitionConditionBuilder"
> },
> "capacity" : "MEDIUM"
> }
>
> 2017-10-23 16:19 GMT+08:00 op <52...@qq.com>:
>
> > hi shaofeng
> > then how to use a new PartitionConditionBuilder？
> > thanks！
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > 发送时间: 2017年10月17日(星期二) 晚上6:16
> > 收件人: "dev"<de...@kylin.apache.org>;
> >
> > 主题: Re: data format when building increment cube
> >
> >
> >
> > It is not supported. You need to develop a new PartitionConditionBuilder.
> > Please refer
> > to org.apache.kylin.metadata.model.PartitionDesc.
> > YearMonthDayPartitionConditionBuilder
> >
> > 2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
> >
> > >
> > > hello everyone.
> > > how can i define my own data format?
> > > our hive tables are partitioned by month_id(format :yyyymm) and
> > > subpartitioned by day_id(format ：dd)
> > >
> > > thanks!
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
回复： data format when building increment cube
Posted by op <52...@qq.com>.
thx ！
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年10月23日(星期一) 下午5:53
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: data format when building increment cube
You can backup the metastore to local, then you will see the model json
file.
On Kylin web, move mouse to the model, click "Action" -> "Edit JSON", will
see the JSON content.
2017-10-23 17:23 GMT+08:00 op <52...@qq.com>:
> thank you shaofeng，but i cannot find the json file
> or you mean change it on kylin web？
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2017年10月23日(星期一) 下午5:09
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: data format when building increment cube
>
>
>
> If you create a new model from Kylin web, the model has a property
> "partition_condition_builder" which refers to the class that
> implements the PartitionConditionBuilder.
> You can manually change it to your customized implementation:
>
>
> {
> "uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
> "last_modified" : 1508748727953,
> "name" : "test_parition",
> "owner" : "admin",
> "is_draft" : false,
> "description" : "",
> "fact_table" : "DEFAULT.KYLIN_SALES",
> "lookups" : [ ],
> "dimensions" : [ {
> "table" : "KYLIN_SALES",
> "columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
> "LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID", "BUYER_ID",
> "OPS_USER_ID", "OPS_REGION" ]
> } ],
> "metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
> "filter_condition" : "",
> "partition_desc" : {
> "partition_date_column" : "KYLIN_SALES.PART_DT",
> "partition_time_column" : null,
> "partition_date_start" : 0,
> "partition_date_format" : "yyyy-MM-dd",
> "partition_time_format" : "HH:mm:ss",
> "partition_type" : "APPEND",
> "partition_condition_builder" :
> "org.apache.kylin.metadata.model.PartitionDesc$
> DefaultPartitionConditionBuilder"
> },
> "capacity" : "MEDIUM"
> }
>
> 2017-10-23 16:19 GMT+08:00 op <52...@qq.com>:
>
> > hi shaofeng
> > then how to use a new PartitionConditionBuilder？
> > thanks！
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > 发送时间: 2017年10月17日(星期二) 晚上6:16
> > 收件人: "dev"<de...@kylin.apache.org>;
> >
> > 主题: Re: data format when building increment cube
> >
> >
> >
> > It is not supported. You need to develop a new PartitionConditionBuilder.
> > Please refer
> > to org.apache.kylin.metadata.model.PartitionDesc.
> > YearMonthDayPartitionConditionBuilder
> >
> > 2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
> >
> > >
> > > hello everyone.
> > > how can i define my own data format?
> > > our hive tables are partitioned by month_id(format :yyyymm) and
> > > subpartitioned by day_id(format ：dd)
> > >
> > > thanks!
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
Re: data format when building increment cube
Posted by ShaoFeng Shi <sh...@apache.org>.
You can backup the metastore to local, then you will see the model json
file.
On Kylin web, move mouse to the model, click "Action" -> "Edit JSON", will
see the JSON content.
2017-10-23 17:23 GMT+08:00 op <52...@qq.com>:
> thank you shaofeng，but i cannot find the json file
> or you mean change it on kylin web？
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2017年10月23日(星期一) 下午5:09
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: data format when building increment cube
>
>
>
> If you create a new model from Kylin web, the model has a property
> "partition_condition_builder" which refers to the class that
> implements the PartitionConditionBuilder.
> You can manually change it to your customized implementation:
>
>
> {
> "uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
> "last_modified" : 1508748727953,
> "name" : "test_parition",
> "owner" : "admin",
> "is_draft" : false,
> "description" : "",
> "fact_table" : "DEFAULT.KYLIN_SALES",
> "lookups" : [ ],
> "dimensions" : [ {
> "table" : "KYLIN_SALES",
> "columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
> "LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID", "BUYER_ID",
> "OPS_USER_ID", "OPS_REGION" ]
> } ],
> "metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
> "filter_condition" : "",
> "partition_desc" : {
> "partition_date_column" : "KYLIN_SALES.PART_DT",
> "partition_time_column" : null,
> "partition_date_start" : 0,
> "partition_date_format" : "yyyy-MM-dd",
> "partition_time_format" : "HH:mm:ss",
> "partition_type" : "APPEND",
> "partition_condition_builder" :
> "org.apache.kylin.metadata.model.PartitionDesc$
> DefaultPartitionConditionBuilder"
> },
> "capacity" : "MEDIUM"
> }
>
> 2017-10-23 16:19 GMT+08:00 op <52...@qq.com>:
>
> > hi shaofeng
> > then how to use a new PartitionConditionBuilder？
> > thanks！
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> > 发送时间: 2017年10月17日(星期二) 晚上6:16
> > 收件人: "dev"<de...@kylin.apache.org>;
> >
> > 主题: Re: data format when building increment cube
> >
> >
> >
> > It is not supported. You need to develop a new PartitionConditionBuilder.
> > Please refer
> > to org.apache.kylin.metadata.model.PartitionDesc.
> > YearMonthDayPartitionConditionBuilder
> >
> > 2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
> >
> > >
> > > hello everyone.
> > > how can i define my own data format?
> > > our hive tables are partitioned by month_id(format :yyyymm) and
> > > subpartitioned by day_id(format ：dd)
> > >
> > > thanks!
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
回复： data format when building increment cube
Posted by op <52...@qq.com>.
thank you shaofeng，but i cannot find the json file
or you mean change it on kylin web？
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年10月23日(星期一) 下午5:09
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: data format when building increment cube
If you create a new model from Kylin web, the model has a property
"partition_condition_builder" which refers to the class that
implements the PartitionConditionBuilder.
You can manually change it to your customized implementation:
{
"uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
"last_modified" : 1508748727953,
"name" : "test_parition",
"owner" : "admin",
"is_draft" : false,
"description" : "",
"fact_table" : "DEFAULT.KYLIN_SALES",
"lookups" : [ ],
"dimensions" : [ {
"table" : "KYLIN_SALES",
"columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
"LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID", "BUYER_ID",
"OPS_USER_ID", "OPS_REGION" ]
} ],
"metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
"filter_condition" : "",
"partition_desc" : {
"partition_date_column" : "KYLIN_SALES.PART_DT",
"partition_time_column" : null,
"partition_date_start" : 0,
"partition_date_format" : "yyyy-MM-dd",
"partition_time_format" : "HH:mm:ss",
"partition_type" : "APPEND",
"partition_condition_builder" :
"org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
},
"capacity" : "MEDIUM"
}
2017-10-23 16:19 GMT+08:00 op <52...@qq.com>:
> hi shaofeng
> then how to use a new PartitionConditionBuilder？
> thanks！
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2017年10月17日(星期二) 晚上6:16
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: data format when building increment cube
>
>
>
> It is not supported. You need to develop a new PartitionConditionBuilder.
> Please refer
> to org.apache.kylin.metadata.model.PartitionDesc.
> YearMonthDayPartitionConditionBuilder
>
> 2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
>
> >
> > hello everyone.
> > how can i define my own data format?
> > our hive tables are partitioned by month_id(format :yyyymm) and
> > subpartitioned by day_id(format ：dd)
> >
> > thanks!
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
Re: data format when building increment cube
Posted by ShaoFeng Shi <sh...@apache.org>.
If you create a new model from Kylin web, the model has a property
"partition_condition_builder" which refers to the class that
implements the PartitionConditionBuilder.
You can manually change it to your customized implementation:
{
"uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
"last_modified" : 1508748727953,
"name" : "test_parition",
"owner" : "admin",
"is_draft" : false,
"description" : "",
"fact_table" : "DEFAULT.KYLIN_SALES",
"lookups" : [ ],
"dimensions" : [ {
"table" : "KYLIN_SALES",
"columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
"LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID", "BUYER_ID",
"OPS_USER_ID", "OPS_REGION" ]
} ],
"metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
"filter_condition" : "",
"partition_desc" : {
"partition_date_column" : "KYLIN_SALES.PART_DT",
"partition_time_column" : null,
"partition_date_start" : 0,
"partition_date_format" : "yyyy-MM-dd",
"partition_time_format" : "HH:mm:ss",
"partition_type" : "APPEND",
"partition_condition_builder" :
"org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
},
"capacity" : "MEDIUM"
}
2017-10-23 16:19 GMT+08:00 op <52...@qq.com>:
> hi shaofeng
> then how to use a new PartitionConditionBuilder？
> thanks！
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2017年10月17日(星期二) 晚上6:16
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: data format when building increment cube
>
>
>
> It is not supported. You need to develop a new PartitionConditionBuilder.
> Please refer
> to org.apache.kylin.metadata.model.PartitionDesc.
> YearMonthDayPartitionConditionBuilder
>
> 2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
>
> >
> > hello everyone.
> > how can i define my own data format?
> > our hive tables are partitioned by month_id(format :yyyymm) and
> > subpartitioned by day_id(format ：dd)
> >
> > thanks!
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
回复： data format when building increment cube
Posted by op <52...@qq.com>.
hi shaofeng
then how to use a new PartitionConditionBuilder？
thanks！
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年10月17日(星期二) 晚上6:16
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: data format when building increment cube
It is not supported. You need to develop a new PartitionConditionBuilder.
Please refer
to org.apache.kylin.metadata.model.PartitionDesc.YearMonthDayPartitionConditionBuilder
2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
>
> hello everyone.
> how can i define my own data format?
> our hive tables are partitioned by month_id(format :yyyymm) and
> subpartitioned by day_id(format ：dd)
>
> thanks!
>
--
Best regards,
Shaofeng Shi 史少锋
Re: data format when building increment cube
Posted by ShaoFeng Shi <sh...@apache.org>.
It is not supported. You need to develop a new PartitionConditionBuilder.
Please refer
to org.apache.kylin.metadata.model.PartitionDesc.YearMonthDayPartitionConditionBuilder
2017-10-17 18:06 GMT+08:00 op <52...@qq.com>:
>
> hello everyone.
> how can i define my own data format?
> our hive tables are partitioned by month_id(format :yyyymm) and
> subpartitioned by day_id(format ：dd)
>
> thanks!
>
--
Best regards,
Shaofeng Shi 史少锋