You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by yuxia <lu...@alumni.sjtu.edu.cn> on 2022/12/26 12:03:14 UTC
[DISCUSS] FLIP-282: Introduce Delete & Update API
Hi, devs.
I'd like to start a discussion about FLIP-282: Introduce Delete & Update API[1].
Row-Level SQL Delete & Update are becoming more and more important in modern big data workflow. The use cases include deleting a set of rows for regulatory compliance, updating a set of rows for data correction, etc.
So, in this FLIP, I want to introduce Delete & Update API to Flink in batch mode. With these interfaces, the external connectors will have ability to delete & update existing data in the corresponding storages.
Looking forwards to your feedback.
[1]: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
Best regards,
Yuxia
Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, all. I have started voting[1] for this FLIP. If you have question, you can reply in this discuss thread.
[1]: https://lists.apache.org/thread/3m91zys8dgnzyypoq88gkb2t55lqvgh1
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "yuxia" <lu...@alumni.sjtu.edu.cn>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期五, 2023年 1 月 06日 下午 8:16:21
主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Hi, all.
Jingsong raises a really good point. And after a offline discussion with Jingsong, we reach an agreement about it.
I post the modification for the FLIP in here:
1: remove all the modiction in ScanTableSource and DynamicTableSink which are really confusing.
2: Introduce an independent interface named SupportsRowLevelModificationScan for source, with which the table source scan can know whether the table source scan is for update or delete statement.
3: Introduce an interface named RowLevelModificationScanContext, the source which implements SupportsRowLevelModificationScan can then pass the RowLevelModificationScanContext providing
necessary informations to the sink.
I have updated all the modification to the FLIP-282[1].
Now, seems we have a consensus for this FLIP, I would like to start a vote on FLIP-282, which begins on the following Monday
(Jan 9th at 10:00 AM GMT) if there's no any other question. If you have any other question, please reply it in this discuss thread.[2]
[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
[2] https://lists.apache.org/thread/6h64v7v6gj916pkvmc3ql3vxxccr46r3
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Jingsong Li" <ji...@gmail.com>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期五, 2023年 1 月 06日 下午 2:08:26
主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Thanks yuxia for your explanation.
But what I mean is that this may lead to confusion for implementers
and users. You can use comments to explain it. However, a good
interface can make the mechanism clearer through code design.
So here, I still think an independent SupportsXX interface can make
the behavior more clear.
Best,
Jingsong
On Wed, Jan 4, 2023 at 10:56 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> Hi, Jingsong, thanks for your comments.
>
> ## About RowLevelDeleteMode
> That's really a good suggestion, now I have updated the FLIP to make RowLevelDeleteMode a higer level.
>
> ## About scope of addContextParameter
> Sorry for the confusing, now I have updated the FLIP to add more comments for it. The scope for the parameters is limited to the phase
> that Flink translates ranslates physical RelNode to ExecNode.
> It's possible to see all the other sources and sinks in a topo. For the order, if only one sink, the sink will be last one to see the parametes,
> the order for the sources is consistent to the order the table source nodes are translated to ExecNode.
> If there're multiple sinks for the case of StatementSet, the sink may also see the parameters added by the table sources that belong the statment
> added earlier.
>
> ## About scope of getScanPurpose
> Yes, all sources wil see this method. But it won't bring any compatibility issues for in here we just tell the source scan
> what's scan purpose without touching any other logic. If sources ignore this method, it just works as normally. So I think there's
> no necessary to add a new interface like SupportsXX.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Jingsong Li" <ji...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期二, 2023年 1 月 03日 下午 12:12:12
> 主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
>
> Thanks yuxia for the FLIP! It looks really good!
>
> I have three comments:
>
> ## RowLevelDeleteMode
>
> Can RowLevelDeleteMode be a higher level?
> `SupportsRowLevelDelete.RowLevelDeleteMode` is better than
> `SupportsRowLevelDelete.RowLevelDeleteInfo.RowLevelDeleteMode`.
> Same as `RowLevelUpdateMode`.
>
> ## Scope of addContextParameter
>
> I see that some of your comments are for sink, but can you make it
> clearer here? What exactly is its scope? For example, is it possible
> to see all the other sources and sinks in a topo? What is the order of
> seeing?
>
> ## Scope of getScanPurpose
>
> Will all sources see this method? Will there be compatibility issues?
> If sources ignore this method, will this cause strange phenomena?
>
> What I mean is: should another SupportsXX be created here to provide
> delete and update.
>
> Best,
> Jingsong
>
> On Thu, Dec 29, 2022 at 6:23 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> >
> > Hi, Lincoln Lee;
> > 1: Yes, it's a typo; Thanks for pointing out. I have fixed the typo.
> > 2: For stream users, assuming for delete, they will receive TableException("DELETE TABLE is not supported for streaming mode now"); Update is similar. I also update them to the FLIP.
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Lincoln Lee" <li...@gmail.com>
> > 收件人: "dev" <de...@flink.apache.org>
> > 发送时间: 星期三, 2022年 12 月 28日 上午 9:50:50
> > 主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
> >
> > Hi yuxia,
> >
> > Thanks for the proposal! I think it'll be very useful for users in batch
> > scenarios to cooperate with external systems.
> >
> > For the flip I have two questions:
> > 1. Is it a typo the default method 'default ScanPurpose getScanPurpose();'
> > without implementation in interface ScanContext?
> > 2. For stream users, what exceptions will be received for this unsupported
> > operations?
> >
> > Best,
> > Lincoln Lee
> >
> >
> > yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月26日周一 20:24写道：
> >
> > > Hi, devs.
> > >
> > > I'd like to start a discussion about FLIP-282: Introduce Delete & Update
> > > API[1].
> > >
> > > Row-Level SQL Delete & Update are becoming more and more important in
> > > modern big data workflow. The use cases include deleting a set of rows for
> > > regulatory compliance, updating a set of rows for data correction, etc.
> > > So, in this FLIP, I want to introduce Delete & Update API to Flink in
> > > batch mode. With these interfaces, the external connectors will have
> > > ability to delete & update existing data in the corresponding storages.
> > >
> > > Looking forwards to your feedback.
> > >
> > > [1]:
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
> > >
> > >
> > > Best regards,
> > > Yuxia
> > >
> > >
Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, all.
Jingsong raises a really good point. And after a offline discussion with Jingsong, we reach an agreement about it.
I post the modification for the FLIP in here:
1: remove all the modiction in ScanTableSource and DynamicTableSink which are really confusing.
2: Introduce an independent interface named SupportsRowLevelModificationScan for source, with which the table source scan can know whether the table source scan is for update or delete statement.
3: Introduce an interface named RowLevelModificationScanContext, the source which implements SupportsRowLevelModificationScan can then pass the RowLevelModificationScanContext providing
necessary informations to the sink.
I have updated all the modification to the FLIP-282[1].
Now, seems we have a consensus for this FLIP, I would like to start a vote on FLIP-282, which begins on the following Monday
(Jan 9th at 10:00 AM GMT) if there's no any other question. If you have any other question, please reply it in this discuss thread.[2]
[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
[2] https://lists.apache.org/thread/6h64v7v6gj916pkvmc3ql3vxxccr46r3
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Jingsong Li" <ji...@gmail.com>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期五, 2023年 1 月 06日 下午 2:08:26
主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Thanks yuxia for your explanation.
But what I mean is that this may lead to confusion for implementers
and users. You can use comments to explain it. However, a good
interface can make the mechanism clearer through code design.
So here, I still think an independent SupportsXX interface can make
the behavior more clear.
Best,
Jingsong
On Wed, Jan 4, 2023 at 10:56 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> Hi, Jingsong, thanks for your comments.
>
> ## About RowLevelDeleteMode
> That's really a good suggestion, now I have updated the FLIP to make RowLevelDeleteMode a higer level.
>
> ## About scope of addContextParameter
> Sorry for the confusing, now I have updated the FLIP to add more comments for it. The scope for the parameters is limited to the phase
> that Flink translates ranslates physical RelNode to ExecNode.
> It's possible to see all the other sources and sinks in a topo. For the order, if only one sink, the sink will be last one to see the parametes,
> the order for the sources is consistent to the order the table source nodes are translated to ExecNode.
> If there're multiple sinks for the case of StatementSet, the sink may also see the parameters added by the table sources that belong the statment
> added earlier.
>
> ## About scope of getScanPurpose
> Yes, all sources wil see this method. But it won't bring any compatibility issues for in here we just tell the source scan
> what's scan purpose without touching any other logic. If sources ignore this method, it just works as normally. So I think there's
> no necessary to add a new interface like SupportsXX.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Jingsong Li" <ji...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期二, 2023年 1 月 03日 下午 12:12:12
> 主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
>
> Thanks yuxia for the FLIP! It looks really good!
>
> I have three comments:
>
> ## RowLevelDeleteMode
>
> Can RowLevelDeleteMode be a higher level?
> `SupportsRowLevelDelete.RowLevelDeleteMode` is better than
> `SupportsRowLevelDelete.RowLevelDeleteInfo.RowLevelDeleteMode`.
> Same as `RowLevelUpdateMode`.
>
> ## Scope of addContextParameter
>
> I see that some of your comments are for sink, but can you make it
> clearer here? What exactly is its scope? For example, is it possible
> to see all the other sources and sinks in a topo? What is the order of
> seeing?
>
> ## Scope of getScanPurpose
>
> Will all sources see this method? Will there be compatibility issues?
> If sources ignore this method, will this cause strange phenomena?
>
> What I mean is: should another SupportsXX be created here to provide
> delete and update.
>
> Best,
> Jingsong
>
> On Thu, Dec 29, 2022 at 6:23 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> >
> > Hi, Lincoln Lee;
> > 1: Yes, it's a typo; Thanks for pointing out. I have fixed the typo.
> > 2: For stream users, assuming for delete, they will receive TableException("DELETE TABLE is not supported for streaming mode now"); Update is similar. I also update them to the FLIP.
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Lincoln Lee" <li...@gmail.com>
> > 收件人: "dev" <de...@flink.apache.org>
> > 发送时间: 星期三, 2022年 12 月 28日 上午 9:50:50
> > 主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
> >
> > Hi yuxia,
> >
> > Thanks for the proposal! I think it'll be very useful for users in batch
> > scenarios to cooperate with external systems.
> >
> > For the flip I have two questions:
> > 1. Is it a typo the default method 'default ScanPurpose getScanPurpose();'
> > without implementation in interface ScanContext?
> > 2. For stream users, what exceptions will be received for this unsupported
> > operations?
> >
> > Best,
> > Lincoln Lee
> >
> >
> > yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月26日周一 20:24写道：
> >
> > > Hi, devs.
> > >
> > > I'd like to start a discussion about FLIP-282: Introduce Delete & Update
> > > API[1].
> > >
> > > Row-Level SQL Delete & Update are becoming more and more important in
> > > modern big data workflow. The use cases include deleting a set of rows for
> > > regulatory compliance, updating a set of rows for data correction, etc.
> > > So, in this FLIP, I want to introduce Delete & Update API to Flink in
> > > batch mode. With these interfaces, the external connectors will have
> > > ability to delete & update existing data in the corresponding storages.
> > >
> > > Looking forwards to your feedback.
> > >
> > > [1]:
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
> > >
> > >
> > > Best regards,
> > > Yuxia
> > >
> > >
Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Posted by Jingsong Li <ji...@gmail.com>.
Thanks yuxia for your explanation.
But what I mean is that this may lead to confusion for implementers
and users. You can use comments to explain it. However, a good
interface can make the mechanism clearer through code design.
So here, I still think an independent SupportsXX interface can make
the behavior more clear.
Best,
Jingsong
On Wed, Jan 4, 2023 at 10:56 AM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> Hi, Jingsong, thanks for your comments.
>
> ## About RowLevelDeleteMode
> That's really a good suggestion, now I have updated the FLIP to make RowLevelDeleteMode a higer level.
>
> ## About scope of addContextParameter
> Sorry for the confusing, now I have updated the FLIP to add more comments for it. The scope for the parameters is limited to the phase
> that Flink translates ranslates physical RelNode to ExecNode.
> It's possible to see all the other sources and sinks in a topo. For the order, if only one sink, the sink will be last one to see the parametes,
> the order for the sources is consistent to the order the table source nodes are translated to ExecNode.
> If there're multiple sinks for the case of StatementSet, the sink may also see the parameters added by the table sources that belong the statment
> added earlier.
>
> ## About scope of getScanPurpose
> Yes, all sources wil see this method. But it won't bring any compatibility issues for in here we just tell the source scan
> what's scan purpose without touching any other logic. If sources ignore this method, it just works as normally. So I think there's
> no necessary to add a new interface like SupportsXX.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Jingsong Li" <ji...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期二, 2023年 1 月 03日 下午 12:12:12
> 主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
>
> Thanks yuxia for the FLIP! It looks really good!
>
> I have three comments:
>
> ## RowLevelDeleteMode
>
> Can RowLevelDeleteMode be a higher level?
> `SupportsRowLevelDelete.RowLevelDeleteMode` is better than
> `SupportsRowLevelDelete.RowLevelDeleteInfo.RowLevelDeleteMode`.
> Same as `RowLevelUpdateMode`.
>
> ## Scope of addContextParameter
>
> I see that some of your comments are for sink, but can you make it
> clearer here? What exactly is its scope? For example, is it possible
> to see all the other sources and sinks in a topo? What is the order of
> seeing?
>
> ## Scope of getScanPurpose
>
> Will all sources see this method? Will there be compatibility issues?
> If sources ignore this method, will this cause strange phenomena?
>
> What I mean is: should another SupportsXX be created here to provide
> delete and update.
>
> Best,
> Jingsong
>
> On Thu, Dec 29, 2022 at 6:23 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
> >
> > Hi, Lincoln Lee;
> > 1: Yes, it's a typo; Thanks for pointing out. I have fixed the typo.
> > 2: For stream users, assuming for delete, they will receive TableException("DELETE TABLE is not supported for streaming mode now"); Update is similar. I also update them to the FLIP.
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "Lincoln Lee" <li...@gmail.com>
> > 收件人: "dev" <de...@flink.apache.org>
> > 发送时间: 星期三, 2022年 12 月 28日 上午 9:50:50
> > 主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
> >
> > Hi yuxia,
> >
> > Thanks for the proposal! I think it'll be very useful for users in batch
> > scenarios to cooperate with external systems.
> >
> > For the flip I have two questions:
> > 1. Is it a typo the default method 'default ScanPurpose getScanPurpose();'
> > without implementation in interface ScanContext?
> > 2. For stream users, what exceptions will be received for this unsupported
> > operations?
> >
> > Best,
> > Lincoln Lee
> >
> >
> > yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月26日周一 20:24写道：
> >
> > > Hi, devs.
> > >
> > > I'd like to start a discussion about FLIP-282: Introduce Delete & Update
> > > API[1].
> > >
> > > Row-Level SQL Delete & Update are becoming more and more important in
> > > modern big data workflow. The use cases include deleting a set of rows for
> > > regulatory compliance, updating a set of rows for data correction, etc.
> > > So, in this FLIP, I want to introduce Delete & Update API to Flink in
> > > batch mode. With these interfaces, the external connectors will have
> > > ability to delete & update existing data in the corresponding storages.
> > >
> > > Looking forwards to your feedback.
> > >
> > > [1]:
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
> > >
> > >
> > > Best regards,
> > > Yuxia
> > >
> > >
Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, Jingsong, thanks for your comments.
## About RowLevelDeleteMode
That's really a good suggestion, now I have updated the FLIP to make RowLevelDeleteMode a higer level.
## About scope of addContextParameter
Sorry for the confusing, now I have updated the FLIP to add more comments for it. The scope for the parameters is limited to the phase
that Flink translates ranslates physical RelNode to ExecNode.
It's possible to see all the other sources and sinks in a topo. For the order, if only one sink, the sink will be last one to see the parametes,
the order for the sources is consistent to the order the table source nodes are translated to ExecNode.
If there're multiple sinks for the case of StatementSet, the sink may also see the parameters added by the table sources that belong the statment
added earlier.
## About scope of getScanPurpose
Yes, all sources wil see this method. But it won't bring any compatibility issues for in here we just tell the source scan
what's scan purpose without touching any other logic. If sources ignore this method, it just works as normally. So I think there's
no necessary to add a new interface like SupportsXX.
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Jingsong Li" <ji...@gmail.com>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期二, 2023年 1 月 03日 下午 12:12:12
主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Thanks yuxia for the FLIP! It looks really good!
I have three comments:
## RowLevelDeleteMode
Can RowLevelDeleteMode be a higher level?
`SupportsRowLevelDelete.RowLevelDeleteMode` is better than
`SupportsRowLevelDelete.RowLevelDeleteInfo.RowLevelDeleteMode`.
Same as `RowLevelUpdateMode`.
## Scope of addContextParameter
I see that some of your comments are for sink, but can you make it
clearer here? What exactly is its scope? For example, is it possible
to see all the other sources and sinks in a topo? What is the order of
seeing?
## Scope of getScanPurpose
Will all sources see this method? Will there be compatibility issues?
If sources ignore this method, will this cause strange phenomena?
What I mean is: should another SupportsXX be created here to provide
delete and update.
Best,
Jingsong
On Thu, Dec 29, 2022 at 6:23 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> Hi, Lincoln Lee;
> 1: Yes, it's a typo; Thanks for pointing out. I have fixed the typo.
> 2: For stream users, assuming for delete, they will receive TableException("DELETE TABLE is not supported for streaming mode now"); Update is similar. I also update them to the FLIP.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Lincoln Lee" <li...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期三, 2022年 12 月 28日 上午 9:50:50
> 主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
>
> Hi yuxia,
>
> Thanks for the proposal! I think it'll be very useful for users in batch
> scenarios to cooperate with external systems.
>
> For the flip I have two questions:
> 1. Is it a typo the default method 'default ScanPurpose getScanPurpose();'
> without implementation in interface ScanContext?
> 2. For stream users, what exceptions will be received for this unsupported
> operations?
>
> Best,
> Lincoln Lee
>
>
> yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月26日周一 20:24写道：
>
> > Hi, devs.
> >
> > I'd like to start a discussion about FLIP-282: Introduce Delete & Update
> > API[1].
> >
> > Row-Level SQL Delete & Update are becoming more and more important in
> > modern big data workflow. The use cases include deleting a set of rows for
> > regulatory compliance, updating a set of rows for data correction, etc.
> > So, in this FLIP, I want to introduce Delete & Update API to Flink in
> > batch mode. With these interfaces, the external connectors will have
> > ability to delete & update existing data in the corresponding storages.
> >
> > Looking forwards to your feedback.
> >
> > [1]:
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
> >
> >
> > Best regards,
> > Yuxia
> >
> >
Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Posted by Jingsong Li <ji...@gmail.com>.
Thanks yuxia for the FLIP! It looks really good!
I have three comments:
## RowLevelDeleteMode
Can RowLevelDeleteMode be a higher level?
`SupportsRowLevelDelete.RowLevelDeleteMode` is better than
`SupportsRowLevelDelete.RowLevelDeleteInfo.RowLevelDeleteMode`.
Same as `RowLevelUpdateMode`.
## Scope of addContextParameter
I see that some of your comments are for sink, but can you make it
clearer here? What exactly is its scope? For example, is it possible
to see all the other sources and sinks in a topo? What is the order of
seeing?
## Scope of getScanPurpose
Will all sources see this method? Will there be compatibility issues?
If sources ignore this method, will this cause strange phenomena?
What I mean is: should another SupportsXX be created here to provide
delete and update.
Best,
Jingsong
On Thu, Dec 29, 2022 at 6:23 PM yuxia <lu...@alumni.sjtu.edu.cn> wrote:
>
> Hi, Lincoln Lee;
> 1: Yes, it's a typo; Thanks for pointing out. I have fixed the typo.
> 2: For stream users, assuming for delete, they will receive TableException("DELETE TABLE is not supported for streaming mode now"); Update is similar. I also update them to the FLIP.
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "Lincoln Lee" <li...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期三, 2022年 12 月 28日 上午 9:50:50
> 主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
>
> Hi yuxia,
>
> Thanks for the proposal! I think it'll be very useful for users in batch
> scenarios to cooperate with external systems.
>
> For the flip I have two questions:
> 1. Is it a typo the default method 'default ScanPurpose getScanPurpose();'
> without implementation in interface ScanContext?
> 2. For stream users, what exceptions will be received for this unsupported
> operations?
>
> Best,
> Lincoln Lee
>
>
> yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月26日周一 20:24写道：
>
> > Hi, devs.
> >
> > I'd like to start a discussion about FLIP-282: Introduce Delete & Update
> > API[1].
> >
> > Row-Level SQL Delete & Update are becoming more and more important in
> > modern big data workflow. The use cases include deleting a set of rows for
> > regulatory compliance, updating a set of rows for data correction, etc.
> > So, in this FLIP, I want to introduce Delete & Update API to Flink in
> > batch mode. With these interfaces, the external connectors will have
> > ability to delete & update existing data in the corresponding storages.
> >
> > Looking forwards to your feedback.
> >
> > [1]:
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
> >
> >
> > Best regards,
> > Yuxia
> >
> >
Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, Lincoln Lee;
1: Yes, it's a typo; Thanks for pointing out. I have fixed the typo.
2: For stream users, assuming for delete, they will receive TableException("DELETE TABLE is not supported for streaming mode now"); Update is similar. I also update them to the FLIP.
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "Lincoln Lee" <li...@gmail.com>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期三, 2022年 12 月 28日 上午 9:50:50
主题: Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Hi yuxia,
Thanks for the proposal! I think it'll be very useful for users in batch
scenarios to cooperate with external systems.
For the flip I have two questions:
1. Is it a typo the default method 'default ScanPurpose getScanPurpose();'
without implementation in interface ScanContext?
2. For stream users, what exceptions will be received for this unsupported
operations?
Best,
Lincoln Lee
yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月26日周一 20:24写道：
> Hi, devs.
>
> I'd like to start a discussion about FLIP-282: Introduce Delete & Update
> API[1].
>
> Row-Level SQL Delete & Update are becoming more and more important in
> modern big data workflow. The use cases include deleting a set of rows for
> regulatory compliance, updating a set of rows for data correction, etc.
> So, in this FLIP, I want to introduce Delete & Update API to Flink in
> batch mode. With these interfaces, the external connectors will have
> ability to delete & update existing data in the corresponding storages.
>
> Looking forwards to your feedback.
>
> [1]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
>
>
> Best regards,
> Yuxia
>
>
Re: [DISCUSS] FLIP-282: Introduce Delete & Update API
Posted by Lincoln Lee <li...@gmail.com>.
Hi yuxia,
Thanks for the proposal! I think it'll be very useful for users in batch
scenarios to cooperate with external systems.
For the flip I have two questions:
1. Is it a typo the default method 'default ScanPurpose getScanPurpose();'
without implementation in interface ScanContext?
2. For stream users, what exceptions will be received for this unsupported
operations?
Best,
Lincoln Lee
yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月26日周一 20:24写道：
> Hi, devs.
>
> I'd like to start a discussion about FLIP-282: Introduce Delete & Update
> API[1].
>
> Row-Level SQL Delete & Update are becoming more and more important in
> modern big data workflow. The use cases include deleting a set of rows for
> regulatory compliance, updating a set of rows for data correction, etc.
> So, in this FLIP, I want to introduce Delete & Update API to Flink in
> batch mode. With these interfaces, the external connectors will have
> ability to delete & update existing data in the corresponding storages.
>
> Looking forwards to your feedback.
>
> [1]:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061
>
>
> Best regards,
> Yuxia
>
>