You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by JunRui Lee <jr...@gmail.com> on 2022/12/29 11:45:36 UTC
[DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Hi, devs,
I'd like to start a discussion about FLIP-283: Use adaptive batch
scheduler as default scheduler for batch jobs[1].
In FLIP-187, we introduced an adaptive batch scheduler. The adaptive
batch scheduler has stronger batch scheduling capabilities, including
automatically deciding parallelisms of job vertices for batch
jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3],
and speculative execution (FLIP-168)[4]. To further use the adaptive
batch scheduler to improve flink's batch capability, in this FLIP
we aim to make the adaptive batch scheduler as the default batch
scheduler.
Currently, users have to set some configuration of the adaptive
batch scheduler, which is not very convenient. To use the adaptive
batch scheduler as the default batch scheduler, we need to improve
the user's out-of-the-box experience. Therefore, we also need to
optimize the current adaptive batch scheduler configuration.
Looking forward to your feedback.
[1]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs
[2]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler
[3]:https://issues.apache.org/jira/browse/FLINK-29663
[4]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job
Best regards,
JunRui Lee
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Posted by Junrui Lee <jr...@gmail.com>.
Hi, all. I have started voting[1] for this FLIP. If you have any questions,
you can reply in this discussion thread.
[1]: https://lists.apache.org/thread/gdymm7pr2slzy9gqkfo97vn73496w0cj
Best regards,
Junrui
Xintong Song <to...@gmail.com> 于2023年1月4日周三 18:16写道：
> Thanks for the proposal.
>
> Another potential benefit I see in this FLIP is that it may reduce the
> complexity and maintenance overhead of the scheduler. During developing
> hybrid shuffle, we had to re-implement some similar logic to make both
> default and adaptive batch schedulers support the new shuffle type. For
> batch workloads, the functionality provided by adaptive batch scheduler
> should already be a superset of those provided by the default scheduler. If
> that can be confirmed from user feedback after making the adaptive batch
> scheduler default, we may consider making default scheduler streaming
> dedicated. That should significantly simplify the default scheduler,
> benefiting both the maintenance of the default scheduler and the
> development of new features in the batch area.
>
> So +1 from my side.
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jan 4, 2023 at 1:14 PM JunRui Lee <jr...@gmail.com> wrote:
>
> > Hi, Yuxia
> >
> > Thanks for the reply.
> >
> > > 1. About the configuration renaming. The old configurations ...
> >
> > Thanks for your suggestion. And this explanation has been updated
> > to the FLIP to describe this change more clearly. The old configurations
> > will be deprecated after the configuration renaming, and it still will be
> > considered to guarantee compatibility.
> >
> > > 2. I'm cursion in which case users will disable auto parallelism...
> >
> > Actually, the adaptive batch scheduler is not only used for automatic
> > parallelism derivation, but also for speculative execution. If users
> > want to use speculative execution and do not enable automatic
> > parallelism, they can configure the configuration
> > "execution.batch.adaptive.auto-parallelism.enabled" instead of
> > configuring the configuration "jobmanager.scheduler".
> >
> > In addition, “adaptive” represents a function, not just the automatic
> > derivation of parallelism, and more functions may be added in the
> > future. So we want users to focus on enabling or disabling the
> > parallelism derivation function itself rather than replacing the
> > scheduler type.
> >
> > Best,
> > JunRui
> >
> > Lijie Wang <wa...@gmail.com> 于2023年1月3日周二 11:03写道：
> >
> > > Hi Junrui,
> > >
> > > Thanks for driving this FLIP, + 1 for this proposal. I believe it will
> > > greatly improve the experiences of batch users.
> > >
> > > Best,
> > > Lijie
> > >
> > > Zhu Zhu <re...@gmail.com> 于2022年12月30日周五 12:40写道：
> > >
> > > > Hi Junrui,
> > > >
> > > > Thanks for creating this FLIP!
> > > >
> > > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in
> batch
> > > > scheduling, also with some must-have features like speculative
> > execution.
> > > > It will be great that users can easily use it, without required to
> > > knowing
> > > > the underlying scheduler and configuring some advanced items.
> > > >
> > > > So generally +1 for this proposal.
> > > >
> > > > Regarding the configuration key renaming, like yuxia mentioned, we
> > should
> > > > deprecate the old ones and add new ones with new names, to guarantee
> > > > compatibility.
> > > >
> > > > Thanks,
> > > > Zhu
> > > >
> > > > yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月30日周五 11:10写道：
> > > > >
> > > > > Hi, JunRui Lee.
> > > > >
> > > > > Thanks for driving this FLIP. It must a good improvement for batch
> > > > users' experiences.
> > > > > I have few questions about this FLIP:
> > > > > 1: About the configuration renaming. The old configurations will be
> > > > deprecated or removed directly? if user upgrade their Flink version,
> > > these
> > > > old configuration will still be considered or just ignored? If
> ignore,
> > > the
> > > > users may need to modify their configurations after they upgrade
> their
> > > > Flink.
> > > > >
> > > > > 2: I'm cursion in which case users will disable auto parallelism
> > > > derivation if they have enabled adaptive batch scheduler. IIUC, auto
> > > > parallelism derivation is what adaptive batch scheduler aim to do. If
> > use
> > > > want to diable auto parallelism derivation, can they just disable
> > > adaptive
> > > > batch scheduler.?
> > > > >
> > > > > Best regards,
> > > > > Yuxia
> > > > >
> > > > > ----- 原始邮件 -----
> > > > > 发件人: "JunRui Lee" <jr...@gmail.com>
> > > > > 收件人: "dev" <de...@flink.apache.org>
> > > > > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36
> > > > > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default
> > > > scheduler for batch jobs
> > > > >
> > > > > Hi, devs,
> > > > >
> > > > > I'd like to start a discussion about FLIP-283: Use adaptive batch
> > > > > scheduler as default scheduler for batch jobs[1].
> > > > >
> > > > > In FLIP-187, we introduced an adaptive batch scheduler. The
> adaptive
> > > > > batch scheduler has stronger batch scheduling capabilities,
> including
> > > > > automatically deciding parallelisms of job vertices for batch
> > > > > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3],
> > > > > and speculative execution (FLIP-168)[4]. To further use the
> adaptive
> > > > > batch scheduler to improve flink's batch capability, in this FLIP
> > > > > we aim to make the adaptive batch scheduler as the default batch
> > > > > scheduler.
> > > > >
> > > > > Currently, users have to set some configuration of the adaptive
> > > > > batch scheduler, which is not very convenient. To use the adaptive
> > > > > batch scheduler as the default batch scheduler, we need to improve
> > > > > the user's out-of-the-box experience. Therefore, we also need to
> > > > > optimize the current adaptive batch scheduler configuration.
> > > > >
> > > > > Looking forward to your feedback.
> > > > >
> > > > > [1]:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs
> > > > > [2]:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler
> > > > > [3]:https://issues.apache.org/jira/browse/FLINK-29663
> > > > > [4]:
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job
> > > > >
> > > > > Best regards,
> > > > > JunRui Lee
> > > >
> > >
> >
>
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Posted by Xintong Song <to...@gmail.com>.
Thanks for the proposal.
Another potential benefit I see in this FLIP is that it may reduce the
complexity and maintenance overhead of the scheduler. During developing
hybrid shuffle, we had to re-implement some similar logic to make both
default and adaptive batch schedulers support the new shuffle type. For
batch workloads, the functionality provided by adaptive batch scheduler
should already be a superset of those provided by the default scheduler. If
that can be confirmed from user feedback after making the adaptive batch
scheduler default, we may consider making default scheduler streaming
dedicated. That should significantly simplify the default scheduler,
benefiting both the maintenance of the default scheduler and the
development of new features in the batch area.
So +1 from my side.
Best,
Xintong
On Wed, Jan 4, 2023 at 1:14 PM JunRui Lee <jr...@gmail.com> wrote:
> Hi, Yuxia
>
> Thanks for the reply.
>
> > 1. About the configuration renaming. The old configurations ...
>
> Thanks for your suggestion. And this explanation has been updated
> to the FLIP to describe this change more clearly. The old configurations
> will be deprecated after the configuration renaming, and it still will be
> considered to guarantee compatibility.
>
> > 2. I'm cursion in which case users will disable auto parallelism...
>
> Actually, the adaptive batch scheduler is not only used for automatic
> parallelism derivation, but also for speculative execution. If users
> want to use speculative execution and do not enable automatic
> parallelism, they can configure the configuration
> "execution.batch.adaptive.auto-parallelism.enabled" instead of
> configuring the configuration "jobmanager.scheduler".
>
> In addition, “adaptive” represents a function, not just the automatic
> derivation of parallelism, and more functions may be added in the
> future. So we want users to focus on enabling or disabling the
> parallelism derivation function itself rather than replacing the
> scheduler type.
>
> Best,
> JunRui
>
> Lijie Wang <wa...@gmail.com> 于2023年1月3日周二 11:03写道：
>
> > Hi Junrui,
> >
> > Thanks for driving this FLIP, + 1 for this proposal. I believe it will
> > greatly improve the experiences of batch users.
> >
> > Best,
> > Lijie
> >
> > Zhu Zhu <re...@gmail.com> 于2022年12月30日周五 12:40写道：
> >
> > > Hi Junrui,
> > >
> > > Thanks for creating this FLIP!
> > >
> > > AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch
> > > scheduling, also with some must-have features like speculative
> execution.
> > > It will be great that users can easily use it, without required to
> > knowing
> > > the underlying scheduler and configuring some advanced items.
> > >
> > > So generally +1 for this proposal.
> > >
> > > Regarding the configuration key renaming, like yuxia mentioned, we
> should
> > > deprecate the old ones and add new ones with new names, to guarantee
> > > compatibility.
> > >
> > > Thanks,
> > > Zhu
> > >
> > > yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月30日周五 11:10写道：
> > > >
> > > > Hi, JunRui Lee.
> > > >
> > > > Thanks for driving this FLIP. It must a good improvement for batch
> > > users' experiences.
> > > > I have few questions about this FLIP:
> > > > 1: About the configuration renaming. The old configurations will be
> > > deprecated or removed directly? if user upgrade their Flink version,
> > these
> > > old configuration will still be considered or just ignored? If ignore,
> > the
> > > users may need to modify their configurations after they upgrade their
> > > Flink.
> > > >
> > > > 2: I'm cursion in which case users will disable auto parallelism
> > > derivation if they have enabled adaptive batch scheduler. IIUC, auto
> > > parallelism derivation is what adaptive batch scheduler aim to do. If
> use
> > > want to diable auto parallelism derivation, can they just disable
> > adaptive
> > > batch scheduler.?
> > > >
> > > > Best regards,
> > > > Yuxia
> > > >
> > > > ----- 原始邮件 -----
> > > > 发件人: "JunRui Lee" <jr...@gmail.com>
> > > > 收件人: "dev" <de...@flink.apache.org>
> > > > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36
> > > > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default
> > > scheduler for batch jobs
> > > >
> > > > Hi, devs,
> > > >
> > > > I'd like to start a discussion about FLIP-283: Use adaptive batch
> > > > scheduler as default scheduler for batch jobs[1].
> > > >
> > > > In FLIP-187, we introduced an adaptive batch scheduler. The adaptive
> > > > batch scheduler has stronger batch scheduling capabilities, including
> > > > automatically deciding parallelisms of job vertices for batch
> > > > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3],
> > > > and speculative execution (FLIP-168)[4]. To further use the adaptive
> > > > batch scheduler to improve flink's batch capability, in this FLIP
> > > > we aim to make the adaptive batch scheduler as the default batch
> > > > scheduler.
> > > >
> > > > Currently, users have to set some configuration of the adaptive
> > > > batch scheduler, which is not very convenient. To use the adaptive
> > > > batch scheduler as the default batch scheduler, we need to improve
> > > > the user's out-of-the-box experience. Therefore, we also need to
> > > > optimize the current adaptive batch scheduler configuration.
> > > >
> > > > Looking forward to your feedback.
> > > >
> > > > [1]:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs
> > > > [2]:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler
> > > > [3]:https://issues.apache.org/jira/browse/FLINK-29663
> > > > [4]:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job
> > > >
> > > > Best regards,
> > > > JunRui Lee
> > >
> >
>
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Posted by JunRui Lee <jr...@gmail.com>.
Hi, Yuxia
Thanks for the reply.
> 1. About the configuration renaming. The old configurations ...
Thanks for your suggestion. And this explanation has been updated
to the FLIP to describe this change more clearly. The old configurations
will be deprecated after the configuration renaming, and it still will be
considered to guarantee compatibility.
> 2. I'm cursion in which case users will disable auto parallelism...
Actually, the adaptive batch scheduler is not only used for automatic
parallelism derivation, but also for speculative execution. If users
want to use speculative execution and do not enable automatic
parallelism, they can configure the configuration
"execution.batch.adaptive.auto-parallelism.enabled" instead of
configuring the configuration "jobmanager.scheduler".
In addition, “adaptive” represents a function, not just the automatic
derivation of parallelism, and more functions may be added in the
future. So we want users to focus on enabling or disabling the
parallelism derivation function itself rather than replacing the
scheduler type.
Best,
JunRui
Lijie Wang <wa...@gmail.com> 于2023年1月3日周二 11:03写道：
> Hi Junrui,
>
> Thanks for driving this FLIP, + 1 for this proposal. I believe it will
> greatly improve the experiences of batch users.
>
> Best,
> Lijie
>
> Zhu Zhu <re...@gmail.com> 于2022年12月30日周五 12:40写道：
>
> > Hi Junrui,
> >
> > Thanks for creating this FLIP!
> >
> > AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch
> > scheduling, also with some must-have features like speculative execution.
> > It will be great that users can easily use it, without required to
> knowing
> > the underlying scheduler and configuring some advanced items.
> >
> > So generally +1 for this proposal.
> >
> > Regarding the configuration key renaming, like yuxia mentioned, we should
> > deprecate the old ones and add new ones with new names, to guarantee
> > compatibility.
> >
> > Thanks,
> > Zhu
> >
> > yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月30日周五 11:10写道：
> > >
> > > Hi, JunRui Lee.
> > >
> > > Thanks for driving this FLIP. It must a good improvement for batch
> > users' experiences.
> > > I have few questions about this FLIP:
> > > 1: About the configuration renaming. The old configurations will be
> > deprecated or removed directly? if user upgrade their Flink version,
> these
> > old configuration will still be considered or just ignored? If ignore,
> the
> > users may need to modify their configurations after they upgrade their
> > Flink.
> > >
> > > 2: I'm cursion in which case users will disable auto parallelism
> > derivation if they have enabled adaptive batch scheduler. IIUC, auto
> > parallelism derivation is what adaptive batch scheduler aim to do. If use
> > want to diable auto parallelism derivation, can they just disable
> adaptive
> > batch scheduler.?
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > ----- 原始邮件 -----
> > > 发件人: "JunRui Lee" <jr...@gmail.com>
> > > 收件人: "dev" <de...@flink.apache.org>
> > > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36
> > > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default
> > scheduler for batch jobs
> > >
> > > Hi, devs,
> > >
> > > I'd like to start a discussion about FLIP-283: Use adaptive batch
> > > scheduler as default scheduler for batch jobs[1].
> > >
> > > In FLIP-187, we introduced an adaptive batch scheduler. The adaptive
> > > batch scheduler has stronger batch scheduling capabilities, including
> > > automatically deciding parallelisms of job vertices for batch
> > > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3],
> > > and speculative execution (FLIP-168)[4]. To further use the adaptive
> > > batch scheduler to improve flink's batch capability, in this FLIP
> > > we aim to make the adaptive batch scheduler as the default batch
> > > scheduler.
> > >
> > > Currently, users have to set some configuration of the adaptive
> > > batch scheduler, which is not very convenient. To use the adaptive
> > > batch scheduler as the default batch scheduler, we need to improve
> > > the user's out-of-the-box experience. Therefore, we also need to
> > > optimize the current adaptive batch scheduler configuration.
> > >
> > > Looking forward to your feedback.
> > >
> > > [1]:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs
> > > [2]:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler
> > > [3]:https://issues.apache.org/jira/browse/FLINK-29663
> > > [4]:
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job
> > >
> > > Best regards,
> > > JunRui Lee
> >
>
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Posted by Lijie Wang <wa...@gmail.com>.
Hi Junrui,
Thanks for driving this FLIP, + 1 for this proposal. I believe it will
greatly improve the experiences of batch users.
Best,
Lijie
Zhu Zhu <re...@gmail.com> 于2022年12月30日周五 12:40写道：
> Hi Junrui,
>
> Thanks for creating this FLIP!
>
> AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch
> scheduling, also with some must-have features like speculative execution.
> It will be great that users can easily use it, without required to knowing
> the underlying scheduler and configuring some advanced items.
>
> So generally +1 for this proposal.
>
> Regarding the configuration key renaming, like yuxia mentioned, we should
> deprecate the old ones and add new ones with new names, to guarantee
> compatibility.
>
> Thanks,
> Zhu
>
> yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月30日周五 11:10写道：
> >
> > Hi, JunRui Lee.
> >
> > Thanks for driving this FLIP. It must a good improvement for batch
> users' experiences.
> > I have few questions about this FLIP:
> > 1: About the configuration renaming. The old configurations will be
> deprecated or removed directly? if user upgrade their Flink version, these
> old configuration will still be considered or just ignored? If ignore, the
> users may need to modify their configurations after they upgrade their
> Flink.
> >
> > 2: I'm cursion in which case users will disable auto parallelism
> derivation if they have enabled adaptive batch scheduler. IIUC, auto
> parallelism derivation is what adaptive batch scheduler aim to do. If use
> want to diable auto parallelism derivation, can they just disable adaptive
> batch scheduler.?
> >
> > Best regards,
> > Yuxia
> >
> > ----- 原始邮件 -----
> > 发件人: "JunRui Lee" <jr...@gmail.com>
> > 收件人: "dev" <de...@flink.apache.org>
> > 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36
> > 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default
> scheduler for batch jobs
> >
> > Hi, devs,
> >
> > I'd like to start a discussion about FLIP-283: Use adaptive batch
> > scheduler as default scheduler for batch jobs[1].
> >
> > In FLIP-187, we introduced an adaptive batch scheduler. The adaptive
> > batch scheduler has stronger batch scheduling capabilities, including
> > automatically deciding parallelisms of job vertices for batch
> > jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3],
> > and speculative execution (FLIP-168)[4]. To further use the adaptive
> > batch scheduler to improve flink's batch capability, in this FLIP
> > we aim to make the adaptive batch scheduler as the default batch
> > scheduler.
> >
> > Currently, users have to set some configuration of the adaptive
> > batch scheduler, which is not very convenient. To use the adaptive
> > batch scheduler as the default batch scheduler, we need to improve
> > the user's out-of-the-box experience. Therefore, we also need to
> > optimize the current adaptive batch scheduler configuration.
> >
> > Looking forward to your feedback.
> >
> > [1]:
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs
> > [2]:
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler
> > [3]:https://issues.apache.org/jira/browse/FLINK-29663
> > [4]:
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job
> >
> > Best regards,
> > JunRui Lee
>
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Posted by Zhu Zhu <re...@gmail.com>.
Hi Junrui,
Thanks for creating this FLIP!
AdaptiveBatchScheduler is more powerful than DefaultScheduler in batch
scheduling, also with some must-have features like speculative execution.
It will be great that users can easily use it, without required to knowing
the underlying scheduler and configuring some advanced items.
So generally +1 for this proposal.
Regarding the configuration key renaming, like yuxia mentioned, we should
deprecate the old ones and add new ones with new names, to guarantee
compatibility.
Thanks,
Zhu
yuxia <lu...@alumni.sjtu.edu.cn> 于2022年12月30日周五 11:10写道：
>
> Hi, JunRui Lee.
>
> Thanks for driving this FLIP. It must a good improvement for batch users' experiences.
> I have few questions about this FLIP:
> 1: About the configuration renaming. The old configurations will be deprecated or removed directly? if user upgrade their Flink version, these old configuration will still be considered or just ignored? If ignore, the users may need to modify their configurations after they upgrade their Flink.
>
> 2: I'm cursion in which case users will disable auto parallelism derivation if they have enabled adaptive batch scheduler. IIUC, auto parallelism derivation is what adaptive batch scheduler aim to do. If use want to diable auto parallelism derivation, can they just disable adaptive batch scheduler.?
>
> Best regards,
> Yuxia
>
> ----- 原始邮件 -----
> 发件人: "JunRui Lee" <jr...@gmail.com>
> 收件人: "dev" <de...@flink.apache.org>
> 发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36
> 主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
>
> Hi, devs,
>
> I'd like to start a discussion about FLIP-283: Use adaptive batch
> scheduler as default scheduler for batch jobs[1].
>
> In FLIP-187, we introduced an adaptive batch scheduler. The adaptive
> batch scheduler has stronger batch scheduling capabilities, including
> automatically deciding parallelisms of job vertices for batch
> jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3],
> and speculative execution (FLIP-168)[4]. To further use the adaptive
> batch scheduler to improve flink's batch capability, in this FLIP
> we aim to make the adaptive batch scheduler as the default batch
> scheduler.
>
> Currently, users have to set some configuration of the adaptive
> batch scheduler, which is not very convenient. To use the adaptive
> batch scheduler as the default batch scheduler, we need to improve
> the user's out-of-the-box experience. Therefore, we also need to
> optimize the current adaptive batch scheduler configuration.
>
> Looking forward to your feedback.
>
> [1]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs
> [2]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler
> [3]:https://issues.apache.org/jira/browse/FLINK-29663
> [4]:
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job
>
> Best regards,
> JunRui Lee
Re: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Posted by yuxia <lu...@alumni.sjtu.edu.cn>.
Hi, JunRui Lee.
Thanks for driving this FLIP. It must a good improvement for batch users' experiences.
I have few questions about this FLIP:
1: About the configuration renaming. The old configurations will be deprecated or removed directly? if user upgrade their Flink version, these old configuration will still be considered or just ignored? If ignore, the users may need to modify their configurations after they upgrade their Flink.
2: I'm cursion in which case users will disable auto parallelism derivation if they have enabled adaptive batch scheduler. IIUC, auto parallelism derivation is what adaptive batch scheduler aim to do. If use want to diable auto parallelism derivation, can they just disable adaptive batch scheduler.?
Best regards,
Yuxia
----- 原始邮件 -----
发件人: "JunRui Lee" <jr...@gmail.com>
收件人: "dev" <de...@flink.apache.org>
发送时间: 星期四, 2022年 12 月 29日 下午 7:45:36
主题: [DISCUSS] FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs
Hi, devs,
I'd like to start a discussion about FLIP-283: Use adaptive batch
scheduler as default scheduler for batch jobs[1].
In FLIP-187, we introduced an adaptive batch scheduler. The adaptive
batch scheduler has stronger batch scheduling capabilities, including
automatically deciding parallelisms of job vertices for batch
jobs (FLIP-187)[2], data balanced distribution (FLINK-29663)[3],
and speculative execution (FLIP-168)[4]. To further use the adaptive
batch scheduler to improve flink's batch capability, in this FLIP
we aim to make the adaptive batch scheduler as the default batch
scheduler.
Currently, users have to set some configuration of the adaptive
batch scheduler, which is not very convenient. To use the adaptive
batch scheduler as the default batch scheduler, we need to improve
the user's out-of-the-box experience. Therefore, we also need to
optimize the current adaptive batch scheduler configuration.
Looking forward to your feedback.
[1]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-283%3A+Use+adaptive+batch+scheduler+as+default+scheduler+for+batch+jobs
[2]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Scheduler
[3]:https://issues.apache.org/jira/browse/FLINK-29663
[4]:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-168%3A+Speculative+Execution+for+Batch+Job
Best regards,
JunRui Lee