You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2023/03/01 11:57:32 UTC
Re: Support DDL and DML for kylin5
Hi Yang,
This is cool. Currently Kylin only sync the table metadata from Hive
metastore, which increases the complexity for the user. This proposal will
make it easier I think.
A quick question: if support this feature, will Kylin write back the schema
info to Hive? Thank you!
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
> Hi
> I submit a new feature about introduce DML and DDL in kylin5 in
> https://issues.apache.org/jira/browse/KYLIN-5466.
> Any comments are welcome!
>
> From Ted-Jiang
>
>
Re: Support DDL and DML for kylin5
Posted by Xiaoxiang Yu <xx...@apache.org>.
Thanks for reply. My workmates and I think it is a good feature. And I have another two questions.
1. I think it(KYLIN-5466) is not a small task, will you divide it into some small tasks(or PRs) in your ticket? Will you set ETA for this feature ?
2. Have you decided how to impelment your new feature? Will you provided some technical detail in your google doc?
Thx.
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2023-03-02 12:59:28, "Yang Jiang" <ya...@apache.org> wrote:
>Hi Xiaoxiang:
>
> Thanks for the comment.
> > 1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
> For now, not supported, But it doesn't affect in future development.
> > 2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
> Will support in `ALTER xxxx`
>
>In our environment ,The most common users case is:
>*show job status
> SHOW JOB job_id;
>*trigger build
> INSERT PARTITION project_name.model_test (2023-01-01, 2023-02-10);
>*create model
>
>*delete model
> DELETE MODEL project_name.model_name;
>*cancel job
> CANCEL JOB job_id;
>
>
>On 2023/03/02 02:16:15 Xiaoxiang Yu wrote:
>> Hi Yang Jiang,
>>
>>
>> I have check this google doc(https://docs.google.com/document/d/1Wa5Ih-rKi2Uqqg8cLfUqwuZBEfEryMwXcD55ezER_Fg ) .I think it is a good feature to me. Here is my questions:
>> 1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
>> 2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
>>
>>
>>
>> --
>>
>> Best wishes to you !
>> From ：Xiaoxiang Yu
>>
>>
>>
>>
>>
>> At 2023-03-02 09:25:30, "Yang Jiang" <ya...@apache.org> wrote:
>> >Hi Shaofeng:
>> > Currently not, kylin will only read meta data form hive. Only create meta data in kylin system like `model`, `job` ...
>> >
>> >On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
>> >> Hi Yang,
>> >>
>> >> This is cool. Currently Kylin only sync the table metadata from Hive
>> >> metastore, which increases the complexity for the user. This proposal will
>> >> make it easier I think.
>> >>
>> >> A quick question: if support this feature, will Kylin write back the schema
>> >> info to Hive? Thank you!
>> >>
>> >> Best regards,
>> >>
>> >> Shaofeng Shi 史少锋
>> >> Apache Kylin PMC,
>> >> Apache Incubator PMC,
>> >> Email: shaofengshi@apache.org
>> >>
>> >> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> >> Join Kylin user mail group: user-subscribe@kylin.apache.org
>> >> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>> >>
>> >>
>> >>
>> >>
>> >> Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
>> >>
>> >> > Hi
>> >> > I submit a new feature about introduce DML and DDL in kylin5 in
>> >> > https://issues.apache.org/jira/browse/KYLIN-5466.
>> >> > Any comments are welcome!
>> >> >
>> >> > From Ted-Jiang
>> >> >
>> >> >
>> >>
>>
Re: Support DDL and DML for kylin5
Posted by Yang Jiang <ya...@apache.org>.
Hi Xiaoxiang:
Thanks for the comment.
> 1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
For now, not supported, But it doesn't affect in future development.
> 2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
Will support in `ALTER xxxx`
In our environment ,The most common users case is:
*show job status
SHOW JOB job_id;
*trigger build
INSERT PARTITION project_name.model_test (2023-01-01, 2023-02-10);
*create model
*delete model
DELETE MODEL project_name.model_name;
*cancel job
CANCEL JOB job_id;
On 2023/03/02 02:16:15 Xiaoxiang Yu wrote:
> Hi Yang Jiang,
>
>
> I have check this google doc(https://docs.google.com/document/d/1Wa5Ih-rKi2Uqqg8cLfUqwuZBEfEryMwXcD55ezER_Fg ) .I think it is a good feature to me. Here is my questions:
> 1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
> 2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
>
>
>
> --
>
> Best wishes to you !
> From ：Xiaoxiang Yu
>
>
>
>
>
> At 2023-03-02 09:25:30, "Yang Jiang" <ya...@apache.org> wrote:
> >Hi Shaofeng:
> > Currently not, kylin will only read meta data form hive. Only create meta data in kylin system like `model`, `job` ...
> >
> >On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
> >> Hi Yang,
> >>
> >> This is cool. Currently Kylin only sync the table metadata from Hive
> >> metastore, which increases the complexity for the user. This proposal will
> >> make it easier I think.
> >>
> >> A quick question: if support this feature, will Kylin write back the schema
> >> info to Hive? Thank you!
> >>
> >> Best regards,
> >>
> >> Shaofeng Shi 史少锋
> >> Apache Kylin PMC,
> >> Apache Incubator PMC,
> >> Email: shaofengshi@apache.org
> >>
> >> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> >> Join Kylin user mail group: user-subscribe@kylin.apache.org
> >> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
> >>
> >>
> >>
> >>
> >> Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
> >>
> >> > Hi
> >> > I submit a new feature about introduce DML and DDL in kylin5 in
> >> > https://issues.apache.org/jira/browse/KYLIN-5466.
> >> > Any comments are welcome!
> >> >
> >> > From Ted-Jiang
> >> >
> >> >
> >>
>
Re: Support DDL and DML for kylin5
Posted by Xiaoxiang Yu <xx...@apache.org>.
Hi Yang Jiang,
I have check this google doc(https://docs.google.com/document/d/1Wa5Ih-rKi2Uqqg8cLfUqwuZBEfEryMwXcD55ezER_Fg ) .I think it is a good feature to me. Here is my questions:
1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2023-03-02 09:25:30, "Yang Jiang" <ya...@apache.org> wrote:
>Hi Shaofeng:
> Currently not, kylin will only read meta data form hive. Only create meta data in kylin system like `model`, `job` ...
>
>On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
>> Hi Yang,
>>
>> This is cool. Currently Kylin only sync the table metadata from Hive
>> metastore, which increases the complexity for the user. This proposal will
>> make it easier I think.
>>
>> A quick question: if support this feature, will Kylin write back the schema
>> info to Hive? Thank you!
>>
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC,
>> Apache Incubator PMC,
>> Email: shaofengshi@apache.org
>>
>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> Join Kylin user mail group: user-subscribe@kylin.apache.org
>> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>>
>>
>>
>>
>> Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
>>
>> > Hi
>> > I submit a new feature about introduce DML and DDL in kylin5 in
>> > https://issues.apache.org/jira/browse/KYLIN-5466.
>> > Any comments are welcome!
>> >
>> > From Ted-Jiang
>> >
>> >
>>
Re: Support DDL and DML for kylin5
Posted by ShaoFeng Shi <sh...@apache.org>.
I see; it doesn't operate the table schema metadata, just the Kylin
metadata like model, index, job, etc.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Yang Jiang <ya...@apache.org> 于2023年3月2日周四 09:25写道：
> Hi Shaofeng:
> Currently not, kylin will only read meta data form hive. Only create
> meta data in kylin system like `model`, `job` ...
>
> On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
> > Hi Yang,
> >
> > This is cool. Currently Kylin only sync the table metadata from Hive
> > metastore, which increases the complexity for the user. This proposal
> will
> > make it easier I think.
> >
> > A quick question: if support this feature, will Kylin write back the
> schema
> > info to Hive? Thank you!
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofengshi@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscribe@kylin.apache.org
> > Join Kylin dev mail group: dev-subscribe@kylin.apache.org
> >
> >
> >
> >
> > Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
> >
> > > Hi
> > > I submit a new feature about introduce DML and DDL in kylin5 in
> > > https://issues.apache.org/jira/browse/KYLIN-5466.
> > > Any comments are welcome!
> > >
> > > From Ted-Jiang
> > >
> > >
> >
>
Re: Support DDL and DML for kylin5
Posted by Yang Jiang <ya...@apache.org>.
Hi Shaofeng:
Currently not, kylin will only read meta data form hive. Only create meta data in kylin system like `model`, `job` ...
On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
> Hi Yang,
>
> This is cool. Currently Kylin only sync the table metadata from Hive
> metastore, which increases the complexity for the user. This proposal will
> make it easier I think.
>
> A quick question: if support this feature, will Kylin write back the schema
> info to Hive? Thank you!
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofengshi@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
>
> > Hi
> > I submit a new feature about introduce DML and DDL in kylin5 in
> > https://issues.apache.org/jira/browse/KYLIN-5466.
> > Any comments are welcome!
> >
> > From Ted-Jiang
> >
> >
>