You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Junrui Lee <jr...@gmail.com> on 2024/07/26 03:02:36 UTC
[VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Hi everyone,
Thanks for all the feedback about FLIP-468: Introducing StreamGraph-Based
Job Submission [1]. The discussion thread can be found here [2].
The vote will be open for at least 72 hours unless there are any objections
or insufficient votes.
Best,
Junrui
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
[2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by David Morávek <da...@gmail.com>.
tldr, compiled plan needs to be stable and feature complete; covering sql
streaming, batch and datastream api
On Fri 9. 8. 2024 at 15:58, Jim Hughes <jh...@confluent.io.invalid> wrote:
> David, Junrui,
>
> Is there a brief description of the prerequisites for compiled plan-based
> submission?
>
> Cheers,
>
> Jim
>
> On Fri, Aug 9, 2024 at 3:55 AM David Morávek <dm...@apache.org> wrote:
>
> > Thanks, Junrui, for the summary, and Zhu, I appreciate the detailed
> offline
> > conversation. It was incredibly helpful in aligning on our north star for
> > job submission.
> >
> > We’ve agreed that FLIP-468 serves as a temporary solution with acceptable
> > risks regarding API stability, which are outweighed by the new features
> and
> > opportunities it brings. Our long-term goal remains to transition to a
> > compiled plan-based submission once the necessary prerequisites are in
> > place.
> >
> > Let’s proceed with the voting. I’m giving a +1 from my side.
> >
> > Thanks for bearing with me.
> >
> > Best,
> > D.
> >
> >
> > On Tue, Aug 6, 2024 at 4:04 PM Junrui Lee <jr...@gmail.com> wrote:
> >
> > > Hi everyone,
> > >
> > > David, Zhu and I had an offline discussion about FLIP-468 and more.
> Here
> > > are the key points:
> > > 1. Flink lacks a stable public REST API for job submission. Such a
> public
> > > REST API will allow users to create and store compiled plans of jobs,
> and
> > > to directly submit/resumit these plans. It can be helpful for making
> job
> > > plans more stable across versions, speeding up job submission, and
> > > supporting version-less job submission clients.
> > > 2. FLIP-468 mainly focuses on how to enable JobMaster/Scheduler to see
> > and
> > > understand StreamGraph. It is applicable for both SQL and DataStream
> > jobs.
> > > This FLIP will modify the existing jobgraph submission REST API. But
> that
> > > is an internal API for Flink clients and the change will not affect
> > users.
> > > Therefore, it does not conflict with the target of #1.
> > >
> > > The conclusion is:
> > > 1. We will continue with FLIP-468 as is.
> > > 2. Later we will consider how to introduce a pubic REST API for job
> > > submission. This will happen in another FLIP.
> > >
> > > Thanks David and Zhu for the inputs and thoughts.
> > > As the content of FLIP-468 is not changed, we will continue the voting
> > > using the previous voting ML[1].
> > >
> > > [1] https://lists.apache.org/thread/d7md9bprfhynjfq7vy8dqld2dn9lf0ob
> > >
> > > Best,
> > > Junrui
> > >
> > > David Morávek <dm...@apache.org> 于 2024年7月29日周一 16:53写道：
> > >
> > > > Hi everyone,
> > > >
> > > > I hope this message finds you well. I recently returned from a
> two-week
> > > > vacation and apologize for the delay in my response.
> > > >
> > > > There are still a few crucial questions, especially regarding how
> this
> > > will
> > > > work with streaming that need to be addressed. Therefore, I would
> like
> > to
> > > > temporarily cast a -1 vote until these outstanding issues are
> resolved.
> > > > I'll provide more detailed in the discussion thread.
> > > >
> > > > Thank you for your understanding.
> > > >
> > > > Best regards,
> > > > D.
> > > >
> > > > On Fri, Jul 26, 2024 at 9:30 AM Zhanghao Chen <
> > zhanghao.chen@outlook.com
> > > >
> > > > wrote:
> > > >
> > > > > Thanks for driving it. +1 (non-binding)
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > ________________________________
> > > > > From: Junrui Lee <jr...@gmail.com>
> > > > > Sent: Friday, July 26, 2024 11:02
> > > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > > Subject: [VOTE] FLIP-468: Introducing StreamGraph-Based Job
> > Submission
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > Thanks for all the feedback about FLIP-468: Introducing
> > > StreamGraph-Based
> > > > > Job Submission [1]. The discussion thread can be found here [2].
> > > > >
> > > > > The vote will be open for at least 72 hours unless there are any
> > > > objections
> > > > > or insufficient votes.
> > > > >
> > > > > Best,
> > > > >
> > > > > Junrui
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
> > > > >
> > > > > [2]
> https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by Jim Hughes <jh...@confluent.io.INVALID>.
David, Junrui,
Is there a brief description of the prerequisites for compiled plan-based
submission?
Cheers,
Jim
On Fri, Aug 9, 2024 at 3:55 AM David Morávek <dm...@apache.org> wrote:
> Thanks, Junrui, for the summary, and Zhu, I appreciate the detailed offline
> conversation. It was incredibly helpful in aligning on our north star for
> job submission.
>
> We’ve agreed that FLIP-468 serves as a temporary solution with acceptable
> risks regarding API stability, which are outweighed by the new features and
> opportunities it brings. Our long-term goal remains to transition to a
> compiled plan-based submission once the necessary prerequisites are in
> place.
>
> Let’s proceed with the voting. I’m giving a +1 from my side.
>
> Thanks for bearing with me.
>
> Best,
> D.
>
>
> On Tue, Aug 6, 2024 at 4:04 PM Junrui Lee <jr...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > David, Zhu and I had an offline discussion about FLIP-468 and more. Here
> > are the key points:
> > 1. Flink lacks a stable public REST API for job submission. Such a public
> > REST API will allow users to create and store compiled plans of jobs, and
> > to directly submit/resumit these plans. It can be helpful for making job
> > plans more stable across versions, speeding up job submission, and
> > supporting version-less job submission clients.
> > 2. FLIP-468 mainly focuses on how to enable JobMaster/Scheduler to see
> and
> > understand StreamGraph. It is applicable for both SQL and DataStream
> jobs.
> > This FLIP will modify the existing jobgraph submission REST API. But that
> > is an internal API for Flink clients and the change will not affect
> users.
> > Therefore, it does not conflict with the target of #1.
> >
> > The conclusion is:
> > 1. We will continue with FLIP-468 as is.
> > 2. Later we will consider how to introduce a pubic REST API for job
> > submission. This will happen in another FLIP.
> >
> > Thanks David and Zhu for the inputs and thoughts.
> > As the content of FLIP-468 is not changed, we will continue the voting
> > using the previous voting ML[1].
> >
> > [1] https://lists.apache.org/thread/d7md9bprfhynjfq7vy8dqld2dn9lf0ob
> >
> > Best,
> > Junrui
> >
> > David Morávek <dm...@apache.org> 于 2024年7月29日周一 16:53写道：
> >
> > > Hi everyone,
> > >
> > > I hope this message finds you well. I recently returned from a two-week
> > > vacation and apologize for the delay in my response.
> > >
> > > There are still a few crucial questions, especially regarding how this
> > will
> > > work with streaming that need to be addressed. Therefore, I would like
> to
> > > temporarily cast a -1 vote until these outstanding issues are resolved.
> > > I'll provide more detailed in the discussion thread.
> > >
> > > Thank you for your understanding.
> > >
> > > Best regards,
> > > D.
> > >
> > > On Fri, Jul 26, 2024 at 9:30 AM Zhanghao Chen <
> zhanghao.chen@outlook.com
> > >
> > > wrote:
> > >
> > > > Thanks for driving it. +1 (non-binding)
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > ________________________________
> > > > From: Junrui Lee <jr...@gmail.com>
> > > > Sent: Friday, July 26, 2024 11:02
> > > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > > Subject: [VOTE] FLIP-468: Introducing StreamGraph-Based Job
> Submission
> > > >
> > > > Hi everyone,
> > > >
> > > > Thanks for all the feedback about FLIP-468: Introducing
> > StreamGraph-Based
> > > > Job Submission [1]. The discussion thread can be found here [2].
> > > >
> > > > The vote will be open for at least 72 hours unless there are any
> > > objections
> > > > or insufficient votes.
> > > >
> > > > Best,
> > > >
> > > > Junrui
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
> > > >
> > > > [2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
> > > >
> > >
> >
>
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by David Morávek <dm...@apache.org>.
Thanks, Junrui, for the summary, and Zhu, I appreciate the detailed offline
conversation. It was incredibly helpful in aligning on our north star for
job submission.
We’ve agreed that FLIP-468 serves as a temporary solution with acceptable
risks regarding API stability, which are outweighed by the new features and
opportunities it brings. Our long-term goal remains to transition to a
compiled plan-based submission once the necessary prerequisites are in
place.
Let’s proceed with the voting. I’m giving a +1 from my side.
Thanks for bearing with me.
Best,
D.
On Tue, Aug 6, 2024 at 4:04 PM Junrui Lee <jr...@gmail.com> wrote:
> Hi everyone,
>
> David, Zhu and I had an offline discussion about FLIP-468 and more. Here
> are the key points:
> 1. Flink lacks a stable public REST API for job submission. Such a public
> REST API will allow users to create and store compiled plans of jobs, and
> to directly submit/resumit these plans. It can be helpful for making job
> plans more stable across versions, speeding up job submission, and
> supporting version-less job submission clients.
> 2. FLIP-468 mainly focuses on how to enable JobMaster/Scheduler to see and
> understand StreamGraph. It is applicable for both SQL and DataStream jobs.
> This FLIP will modify the existing jobgraph submission REST API. But that
> is an internal API for Flink clients and the change will not affect users.
> Therefore, it does not conflict with the target of #1.
>
> The conclusion is:
> 1. We will continue with FLIP-468 as is.
> 2. Later we will consider how to introduce a pubic REST API for job
> submission. This will happen in another FLIP.
>
> Thanks David and Zhu for the inputs and thoughts.
> As the content of FLIP-468 is not changed, we will continue the voting
> using the previous voting ML[1].
>
> [1] https://lists.apache.org/thread/d7md9bprfhynjfq7vy8dqld2dn9lf0ob
>
> Best,
> Junrui
>
> David Morávek <dm...@apache.org> 于 2024年7月29日周一 16:53写道：
>
> > Hi everyone,
> >
> > I hope this message finds you well. I recently returned from a two-week
> > vacation and apologize for the delay in my response.
> >
> > There are still a few crucial questions, especially regarding how this
> will
> > work with streaming that need to be addressed. Therefore, I would like to
> > temporarily cast a -1 vote until these outstanding issues are resolved.
> > I'll provide more detailed in the discussion thread.
> >
> > Thank you for your understanding.
> >
> > Best regards,
> > D.
> >
> > On Fri, Jul 26, 2024 at 9:30 AM Zhanghao Chen <zhanghao.chen@outlook.com
> >
> > wrote:
> >
> > > Thanks for driving it. +1 (non-binding)
> > >
> > > Best,
> > > Zhanghao Chen
> > > ________________________________
> > > From: Junrui Lee <jr...@gmail.com>
> > > Sent: Friday, July 26, 2024 11:02
> > > To: dev@flink.apache.org <de...@flink.apache.org>
> > > Subject: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
> > >
> > > Hi everyone,
> > >
> > > Thanks for all the feedback about FLIP-468: Introducing
> StreamGraph-Based
> > > Job Submission [1]. The discussion thread can be found here [2].
> > >
> > > The vote will be open for at least 72 hours unless there are any
> > objections
> > > or insufficient votes.
> > >
> > > Best,
> > >
> > > Junrui
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
> > >
> > > [2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
> > >
> >
>
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by Junrui Lee <jr...@gmail.com>.
Hi everyone,
David, Zhu and I had an offline discussion about FLIP-468 and more. Here
are the key points:
1. Flink lacks a stable public REST API for job submission. Such a public
REST API will allow users to create and store compiled plans of jobs, and
to directly submit/resumit these plans. It can be helpful for making job
plans more stable across versions, speeding up job submission, and
supporting version-less job submission clients.
2. FLIP-468 mainly focuses on how to enable JobMaster/Scheduler to see and
understand StreamGraph. It is applicable for both SQL and DataStream jobs.
This FLIP will modify the existing jobgraph submission REST API. But that
is an internal API for Flink clients and the change will not affect users.
Therefore, it does not conflict with the target of #1.
The conclusion is:
1. We will continue with FLIP-468 as is.
2. Later we will consider how to introduce a pubic REST API for job
submission. This will happen in another FLIP.
Thanks David and Zhu for the inputs and thoughts.
As the content of FLIP-468 is not changed, we will continue the voting
using the previous voting ML[1].
[1] https://lists.apache.org/thread/d7md9bprfhynjfq7vy8dqld2dn9lf0ob
Best,
Junrui
David Morávek <dm...@apache.org> 于 2024年7月29日周一 16:53写道：
> Hi everyone,
>
> I hope this message finds you well. I recently returned from a two-week
> vacation and apologize for the delay in my response.
>
> There are still a few crucial questions, especially regarding how this will
> work with streaming that need to be addressed. Therefore, I would like to
> temporarily cast a -1 vote until these outstanding issues are resolved.
> I'll provide more detailed in the discussion thread.
>
> Thank you for your understanding.
>
> Best regards,
> D.
>
> On Fri, Jul 26, 2024 at 9:30 AM Zhanghao Chen <zh...@outlook.com>
> wrote:
>
> > Thanks for driving it. +1 (non-binding)
> >
> > Best,
> > Zhanghao Chen
> > ________________________________
> > From: Junrui Lee <jr...@gmail.com>
> > Sent: Friday, July 26, 2024 11:02
> > To: dev@flink.apache.org <de...@flink.apache.org>
> > Subject: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
> >
> > Hi everyone,
> >
> > Thanks for all the feedback about FLIP-468: Introducing StreamGraph-Based
> > Job Submission [1]. The discussion thread can be found here [2].
> >
> > The vote will be open for at least 72 hours unless there are any
> objections
> > or insufficient votes.
> >
> > Best,
> >
> > Junrui
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
> >
> > [2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
> >
>
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by David Morávek <dm...@apache.org>.
Hi everyone,
I hope this message finds you well. I recently returned from a two-week
vacation and apologize for the delay in my response.
There are still a few crucial questions, especially regarding how this will
work with streaming that need to be addressed. Therefore, I would like to
temporarily cast a -1 vote until these outstanding issues are resolved.
I'll provide more detailed in the discussion thread.
Thank you for your understanding.
Best regards,
D.
On Fri, Jul 26, 2024 at 9:30 AM Zhanghao Chen <zh...@outlook.com>
wrote:
> Thanks for driving it. +1 (non-binding)
>
> Best,
> Zhanghao Chen
> ________________________________
> From: Junrui Lee <jr...@gmail.com>
> Sent: Friday, July 26, 2024 11:02
> To: dev@flink.apache.org <de...@flink.apache.org>
> Subject: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
>
> Hi everyone,
>
> Thanks for all the feedback about FLIP-468: Introducing StreamGraph-Based
> Job Submission [1]. The discussion thread can be found here [2].
>
> The vote will be open for at least 72 hours unless there are any objections
> or insufficient votes.
>
> Best,
>
> Junrui
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
>
> [2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
>
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by Zhanghao Chen <zh...@outlook.com>.
Thanks for driving it. +1 (non-binding)
Best,
Zhanghao Chen
________________________________
From: Junrui Lee <jr...@gmail.com>
Sent: Friday, July 26, 2024 11:02
To: dev@flink.apache.org <de...@flink.apache.org>
Subject: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Hi everyone,
Thanks for all the feedback about FLIP-468: Introducing StreamGraph-Based
Job Submission [1]. The discussion thread can be found here [2].
The vote will be open for at least 72 hours unless there are any objections
or insufficient votes.
Best,
Junrui
[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
[2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by Yuepeng Pan <pa...@apache.org>.
Thanks for driving it !+1(non-binding)
Best regards,
Yuepeng Pan
At 2024-07-26 11:49:41, "Rui Fan" <19...@gmail.com> wrote:
>Thanks Junrui for driving this proposal!
>
>+1(binding)
>
>Best,
>Rui
>
>On Fri, Jul 26, 2024 at 11:02 AM Junrui Lee <jr...@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Thanks for all the feedback about FLIP-468: Introducing StreamGraph-Based
>> Job Submission [1]. The discussion thread can be found here [2].
>>
>> The vote will be open for at least 72 hours unless there are any objections
>> or insufficient votes.
>>
>> Best,
>>
>> Junrui
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
>>
>> [2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
>>
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by Zhu Zhu <re...@gmail.com>.
+1 (binding)
Thanks,
Zhu
Ron Liu <ro...@gmail.com> 于2024年7月26日周五 11:57写道：
> +1(binding)
>
> Best,
> Ron
>
> Rui Fan <19...@gmail.com> 于2024年7月26日周五 11:50写道：
>
> > Thanks Junrui for driving this proposal!
> >
> > +1(binding)
> >
> > Best,
> > Rui
> >
> > On Fri, Jul 26, 2024 at 11:02 AM Junrui Lee <jr...@gmail.com> wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for all the feedback about FLIP-468: Introducing
> StreamGraph-Based
> > > Job Submission [1]. The discussion thread can be found here [2].
> > >
> > > The vote will be open for at least 72 hours unless there are any
> > objections
> > > or insufficient votes.
> > >
> > > Best,
> > >
> > > Junrui
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
> > >
> > > [2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
> > >
> >
>
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by Ron Liu <ro...@gmail.com>.
+1(binding)
Best,
Ron
Rui Fan <19...@gmail.com> 于2024年7月26日周五 11:50写道：
> Thanks Junrui for driving this proposal!
>
> +1(binding)
>
> Best,
> Rui
>
> On Fri, Jul 26, 2024 at 11:02 AM Junrui Lee <jr...@gmail.com> wrote:
>
> > Hi everyone,
> >
> > Thanks for all the feedback about FLIP-468: Introducing StreamGraph-Based
> > Job Submission [1]. The discussion thread can be found here [2].
> >
> > The vote will be open for at least 72 hours unless there are any
> objections
> > or insufficient votes.
> >
> > Best,
> >
> > Junrui
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
> >
> > [2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
> >
>
Re: [VOTE] FLIP-468: Introducing StreamGraph-Based Job Submission
Posted by Rui Fan <19...@gmail.com>.
Thanks Junrui for driving this proposal!
+1(binding)
Best,
Rui
On Fri, Jul 26, 2024 at 11:02 AM Junrui Lee <jr...@gmail.com> wrote:
> Hi everyone,
>
> Thanks for all the feedback about FLIP-468: Introducing StreamGraph-Based
> Job Submission [1]. The discussion thread can be found here [2].
>
> The vote will be open for at least 72 hours unless there are any objections
> or insufficient votes.
>
> Best,
>
> Junrui
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-468%3A+Introducing+StreamGraph-Based+Job+Submission
>
> [2] https://lists.apache.org/thread/l65mst4prx09rmloydn64z1w1zp8coyz
>