You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by "Jiang, Yang" <ya...@ebay.com.INVALID> on 2023/02/28 06:05:15 UTC
Support DDL and DML for kylin5
Hi
I submit a new feature about introduce DML and DDL in kylin5 in https://issues.apache.org/jira/browse/KYLIN-5466.
Any comments are welcome!
From Ted-Jiang
Re: Support DDL and DML for kylin5
Posted by Xiaoxiang Yu <xx...@apache.org>.
Thanks for reply. My workmates and I think it is a good feature. And I have another two questions.
1. I think it(KYLIN-5466) is not a small task, will you divide it into some small tasks(or PRs) in your ticket? Will you set ETA for this feature ?
2. Have you decided how to impelment your new feature? Will you provided some technical detail in your google doc?
Thx.
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2023-03-02 12:59:28, "Yang Jiang" <ya...@apache.org> wrote:
>Hi Xiaoxiang:
>
> Thanks for the comment.
> > 1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
> For now, not supported, But it doesn't affect in future development.
> > 2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
> Will support in `ALTER xxxx`
>
>In our environment ,The most common users case is:
>*show job status
> SHOW JOB job_id;
>*trigger build
> INSERT PARTITION project_name.model_test (2023-01-01, 2023-02-10);
>*create model
>
>*delete model
> DELETE MODEL project_name.model_name;
>*cancel job
> CANCEL JOB job_id;
>
>
>On 2023/03/02 02:16:15 Xiaoxiang Yu wrote:
>> Hi Yang Jiang,
>>
>>
>> I have check this google doc(https://docs.google.com/document/d/1Wa5Ih-rKi2Uqqg8cLfUqwuZBEfEryMwXcD55ezER_Fg ) .I think it is a good feature to me. Here is my questions:
>> 1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
>> 2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
>>
>>
>>
>> --
>>
>> Best wishes to you !
>> From ：Xiaoxiang Yu
>>
>>
>>
>>
>>
>> At 2023-03-02 09:25:30, "Yang Jiang" <ya...@apache.org> wrote:
>> >Hi Shaofeng:
>> > Currently not, kylin will only read meta data form hive. Only create meta data in kylin system like `model`, `job` ...
>> >
>> >On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
>> >> Hi Yang,
>> >>
>> >> This is cool. Currently Kylin only sync the table metadata from Hive
>> >> metastore, which increases the complexity for the user. This proposal will
>> >> make it easier I think.
>> >>
>> >> A quick question: if support this feature, will Kylin write back the schema
>> >> info to Hive? Thank you!
>> >>
>> >> Best regards,
>> >>
>> >> Shaofeng Shi 史少锋
>> >> Apache Kylin PMC,
>> >> Apache Incubator PMC,
>> >> Email: shaofengshi@apache.org
>> >>
>> >> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> >> Join Kylin user mail group: user-subscribe@kylin.apache.org
>> >> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>> >>
>> >>
>> >>
>> >>
>> >> Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
>> >>
>> >> > Hi
>> >> > I submit a new feature about introduce DML and DDL in kylin5 in
>> >> > https://issues.apache.org/jira/browse/KYLIN-5466.
>> >> > Any comments are welcome!
>> >> >
>> >> > From Ted-Jiang
>> >> >
>> >> >
>> >>
>>
Re: Support DDL and DML for kylin5
Posted by Yang Jiang <ya...@apache.org>.
Hi Xiaoxiang:
Thanks for the comment.
> 1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
For now, not supported, But it doesn't affect in future development.
> 2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
Will support in `ALTER xxxx`
In our environment ,The most common users case is:
*show job status
SHOW JOB job_id;
*trigger build
INSERT PARTITION project_name.model_test (2023-01-01, 2023-02-10);
*create model
*delete model
DELETE MODEL project_name.model_name;
*cancel job
CANCEL JOB job_id;
On 2023/03/02 02:16:15 Xiaoxiang Yu wrote:
> Hi Yang Jiang,
>
>
> I have check this google doc(https://docs.google.com/document/d/1Wa5Ih-rKi2Uqqg8cLfUqwuZBEfEryMwXcD55ezER_Fg ) .I think it is a good feature to me. Here is my questions:
> 1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
> 2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
>
>
>
> --
>
> Best wishes to you !
> From ：Xiaoxiang Yu
>
>
>
>
>
> At 2023-03-02 09:25:30, "Yang Jiang" <ya...@apache.org> wrote:
> >Hi Shaofeng:
> > Currently not, kylin will only read meta data form hive. Only create meta data in kylin system like `model`, `job` ...
> >
> >On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
> >> Hi Yang,
> >>
> >> This is cool. Currently Kylin only sync the table metadata from Hive
> >> metastore, which increases the complexity for the user. This proposal will
> >> make it easier I think.
> >>
> >> A quick question: if support this feature, will Kylin write back the schema
> >> info to Hive? Thank you!
> >>
> >> Best regards,
> >>
> >> Shaofeng Shi 史少锋
> >> Apache Kylin PMC,
> >> Apache Incubator PMC,
> >> Email: shaofengshi@apache.org
> >>
> >> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> >> Join Kylin user mail group: user-subscribe@kylin.apache.org
> >> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
> >>
> >>
> >>
> >>
> >> Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
> >>
> >> > Hi
> >> > I submit a new feature about introduce DML and DDL in kylin5 in
> >> > https://issues.apache.org/jira/browse/KYLIN-5466.
> >> > Any comments are welcome!
> >> >
> >> > From Ted-Jiang
> >> >
> >> >
> >>
>
Re: Support DDL and DML for kylin5
Posted by Xiaoxiang Yu <xx...@apache.org>.
Hi Yang Jiang,
I have check this google doc(https://docs.google.com/document/d/1Wa5Ih-rKi2Uqqg8cLfUqwuZBEfEryMwXcD55ezER_Fg ) .I think it is a good feature to me. Here is my questions:
1. Will this DDL provide way to add/edit/delete RulebaseIndex(aggregate group) for Model?
2. Will this DDL provide way to add model level config (Hive provide such grammar : TBLPROPERTIES (property_name=property_value, ...) )
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2023-03-02 09:25:30, "Yang Jiang" <ya...@apache.org> wrote:
>Hi Shaofeng:
> Currently not, kylin will only read meta data form hive. Only create meta data in kylin system like `model`, `job` ...
>
>On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
>> Hi Yang,
>>
>> This is cool. Currently Kylin only sync the table metadata from Hive
>> metastore, which increases the complexity for the user. This proposal will
>> make it easier I think.
>>
>> A quick question: if support this feature, will Kylin write back the schema
>> info to Hive? Thank you!
>>
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC,
>> Apache Incubator PMC,
>> Email: shaofengshi@apache.org
>>
>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> Join Kylin user mail group: user-subscribe@kylin.apache.org
>> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>>
>>
>>
>>
>> Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
>>
>> > Hi
>> > I submit a new feature about introduce DML and DDL in kylin5 in
>> > https://issues.apache.org/jira/browse/KYLIN-5466.
>> > Any comments are welcome!
>> >
>> > From Ted-Jiang
>> >
>> >
>>
Re: Support DDL and DML for kylin5
Posted by ShaoFeng Shi <sh...@apache.org>.
I see; it doesn't operate the table schema metadata, just the Kylin
metadata like model, index, job, etc.
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Yang Jiang <ya...@apache.org> 于2023年3月2日周四 09:25写道：
> Hi Shaofeng:
> Currently not, kylin will only read meta data form hive. Only create
> meta data in kylin system like `model`, `job` ...
>
> On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
> > Hi Yang,
> >
> > This is cool. Currently Kylin only sync the table metadata from Hive
> > metastore, which increases the complexity for the user. This proposal
> will
> > make it easier I think.
> >
> > A quick question: if support this feature, will Kylin write back the
> schema
> > info to Hive? Thank you!
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC,
> > Apache Incubator PMC,
> > Email: shaofengshi@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscribe@kylin.apache.org
> > Join Kylin dev mail group: dev-subscribe@kylin.apache.org
> >
> >
> >
> >
> > Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
> >
> > > Hi
> > > I submit a new feature about introduce DML and DDL in kylin5 in
> > > https://issues.apache.org/jira/browse/KYLIN-5466.
> > > Any comments are welcome!
> > >
> > > From Ted-Jiang
> > >
> > >
> >
>
Re: Support DDL and DML for kylin5
Posted by Yang Jiang <ya...@apache.org>.
Hi Shaofeng:
Currently not, kylin will only read meta data form hive. Only create meta data in kylin system like `model`, `job` ...
On 2023/03/01 11:57:32 ShaoFeng Shi wrote:
> Hi Yang,
>
> This is cool. Currently Kylin only sync the table metadata from Hive
> metastore, which increases the complexity for the user. This proposal will
> make it easier I think.
>
> A quick question: if support this feature, will Kylin write back the schema
> info to Hive? Thank you!
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC,
> Apache Incubator PMC,
> Email: shaofengshi@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscribe@kylin.apache.org
> Join Kylin dev mail group: dev-subscribe@kylin.apache.org
>
>
>
>
> Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
>
> > Hi
> > I submit a new feature about introduce DML and DDL in kylin5 in
> > https://issues.apache.org/jira/browse/KYLIN-5466.
> > Any comments are welcome!
> >
> > From Ted-Jiang
> >
> >
>
Re: Support DDL and DML for kylin5
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi Yang,
This is cool. Currently Kylin only sync the table metadata from Hive
metastore, which increases the complexity for the user. This proposal will
make it easier I think.
A quick question: if support this feature, will Kylin write back the schema
info to Hive? Thank you!
Best regards,
Shaofeng Shi 史少锋
Apache Kylin PMC,
Apache Incubator PMC,
Email: shaofengshi@apache.org
Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscribe@kylin.apache.org
Join Kylin dev mail group: dev-subscribe@kylin.apache.org
Jiang, Yang <ya...@ebay.com.invalid> 于2023年2月28日周二 14:06写道：
> Hi
> I submit a new feature about introduce DML and DDL in kylin5 in
> https://issues.apache.org/jira/browse/KYLIN-5466.
> Any comments are welcome!
>
> From Ted-Jiang
>
>
Re: Support DDL and DML for kylin5
Posted by Li Yang <li...@apache.org>.
Great proposal!!
Looking forward to it. JIRA voted and watched. :-)
On Tue, Feb 28, 2023 at 2:06 PM Jiang, Yang <ya...@ebay.com.invalid>
wrote:
> Hi
> I submit a new feature about introduce DML and DDL in kylin5 in
> https://issues.apache.org/jira/browse/KYLIN-5466.
> Any comments are welcome!
>
> From Ted-Jiang
>
>