You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Yadong Xie <vt...@gmail.com> on 2020/02/20 09:49:43 UTC
[VOTE] FLIP-101: Add Pending Slots Detail
Hi all
I want to start the vote for FLIP-101, which proposes to add pending slots
information to help users check which vertex/subtask is blocked.
To help everyone better understand the proposal, we spent some efforts on
making an online POC
previous web:
http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
POC web:
http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
The vote will last for at least 72 hours, following the consensus voting
process.
FLIP wiki:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
Discussion thread:
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
Thanks,
Yadong
Re: [VOTE] FLIP-101: Add Pending Slots Detail
Posted by Till Rohrmann <tr...@apache.org>.
Hi Yadong,
thanks for creating this FLIP. I like the idea to show the user if some
slots are missing in order to run an operator.
However, since we plan to rework the scheduling part and also how resources
are acquired extensively, I'm not entirely sure whether we should add this
functionality now. I think with the introduction of pipelined region
scheduling and declarative resource management, many implementation
details, e.g. pending slot requests, the decision with which parallelism
the operator should run, etc., will change. Hence, I would suggest to
postpone this change until we have implemented the scheduler changes.
A minor comment concerning API design: PendingSlotRequest is an internal
class which we should not expose via an RPC call. If this is really
necessary, then we should introduce a new type carrying only the required
information (excluding the assigned future, for example).
Cheers,
Till
On Tue, Feb 25, 2020 at 8:29 AM Yadong Xie <vt...@gmail.com> wrote:
> Hi all
>
> Thanks for all your suggestions!
>
> > Hi Jark and Kurt
>
> I have redesigned the UI and move the pending slots information from the
> tab to the vertex graph
>
> > Hi Zhijiang
>
> The slots id could not be used to associated with slot detail in the
> current version as lining said, we can add more detailed data(popup a modal
> or a drawer) in later versions.
> I have removed the slots id from the UI to avoid confusing users.
>
>
> here is the new POC URL:
>
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
>
> any comments are welcome
>
> lining jing <ji...@gmail.com> 于2020年2月24日周一 下午4:57写道：
>
> > Currently, users can't define slot resources. If there are pending slots,
> > the main reason is that there are not enough slots, so the slot size is
> > mainly displayed without more details. If slot resources can be set
> > dynamically later, more information could be shown.
> >
> > Zhijiang <wa...@aliyun.com.invalid> 于2020年2月21日周五 下午11:22写道：
> >
> > > Thanks Yadong for this FLIP!
> > >
> > > If I understood correctly, the motivation is based on the scenario of
> > > scheduled job state for long time in the well resourced cluster.
> > >
> > > It is meaningful for me if we can provide some useful infos to help
> users
> > > analyze the reason. I see from PoC that we can get the
> > > respective vertex which slot is pending and the detailed slot id info.
> > > I am curious that is it feasible to do some valid analysis from logs
> > based
> > > on the slot id? Or what's the expectation to use these infos?
> > >
> > > Best,
> > > Zhijiang
> > > ------------------------------------------------------------------
> > > From:Kurt Young <yk...@gmail.com>
> > > Send Time:2020 Feb. 21 (Fri.) 14:38
> > > To:dev <de...@flink.apache.org>
> > > Subject:Re: [VOTE] FLIP-101: Add Pending Slots Detail
> > >
> > > I agree with Jark, even if we have pending slots now, a dedicated tab
> > seems
> > > to be too much.
> > >
> > > Best,
> > > Kurt
> > >
> > >
> > > On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <im...@gmail.com> wrote:
> > >
> > > > Thanks Yadong,
> > > >
> > > > I think a pending slot view will be helpful. But will it be verbose
> > when
> > > > there is no pending slot, but a "pending slot" in the tab?
> > > > What do you think to show the pending slot page when click the "?" on
> > the
> > > > vertex status?
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Thu, 20 Feb 2020 at 17:50, Yadong Xie <vt...@gmail.com>
> wrote:
> > > >
> > > > > Hi all
> > > > >
> > > > > I want to start the vote for FLIP-101, which proposes to add
> pending
> > > > slots
> > > > > information to help users check which vertex/subtask is blocked.
> > > > >
> > > > > To help everyone better understand the proposal, we spent some
> > efforts
> > > on
> > > > > making an online POC
> > > > >
> > > > > previous web:
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > > > > POC web:
> > > > >
> > > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> > > > >
> > > > >
> > > > > The vote will last for at least 72 hours, following the consensus
> > > voting
> > > > > process.
> > > > >
> > > > > FLIP wiki:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> > > > >
> > > > > Discussion thread:
> > > > >
> > > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Yadong
> > > > >
> > > >
> > >
> > >
> >
>
Re: [VOTE] FLIP-101: Add Pending Slots Detail
Posted by Yadong Xie <vt...@gmail.com>.
Hi all
Thanks for all your suggestions!
> Hi Jark and Kurt
I have redesigned the UI and move the pending slots information from the
tab to the vertex graph
> Hi Zhijiang
The slots id could not be used to associated with slot detail in the
current version as lining said, we can add more detailed data(popup a modal
or a drawer) in later versions.
I have removed the slots id from the UI to avoid confusing users.
here is the new POC URL:
http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
any comments are welcome
lining jing <ji...@gmail.com> 于2020年2月24日周一 下午4:57写道：
> Currently, users can't define slot resources. If there are pending slots,
> the main reason is that there are not enough slots, so the slot size is
> mainly displayed without more details. If slot resources can be set
> dynamically later, more information could be shown.
>
> Zhijiang <wa...@aliyun.com.invalid> 于2020年2月21日周五 下午11:22写道：
>
> > Thanks Yadong for this FLIP!
> >
> > If I understood correctly, the motivation is based on the scenario of
> > scheduled job state for long time in the well resourced cluster.
> >
> > It is meaningful for me if we can provide some useful infos to help users
> > analyze the reason. I see from PoC that we can get the
> > respective vertex which slot is pending and the detailed slot id info.
> > I am curious that is it feasible to do some valid analysis from logs
> based
> > on the slot id? Or what's the expectation to use these infos?
> >
> > Best,
> > Zhijiang
> > ------------------------------------------------------------------
> > From:Kurt Young <yk...@gmail.com>
> > Send Time:2020 Feb. 21 (Fri.) 14:38
> > To:dev <de...@flink.apache.org>
> > Subject:Re: [VOTE] FLIP-101: Add Pending Slots Detail
> >
> > I agree with Jark, even if we have pending slots now, a dedicated tab
> seems
> > to be too much.
> >
> > Best,
> > Kurt
> >
> >
> > On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <im...@gmail.com> wrote:
> >
> > > Thanks Yadong,
> > >
> > > I think a pending slot view will be helpful. But will it be verbose
> when
> > > there is no pending slot, but a "pending slot" in the tab?
> > > What do you think to show the pending slot page when click the "?" on
> the
> > > vertex status?
> > >
> > > Best,
> > > Jark
> > >
> > > On Thu, 20 Feb 2020 at 17:50, Yadong Xie <vt...@gmail.com> wrote:
> > >
> > > > Hi all
> > > >
> > > > I want to start the vote for FLIP-101, which proposes to add pending
> > > slots
> > > > information to help users check which vertex/subtask is blocked.
> > > >
> > > > To help everyone better understand the proposal, we spent some
> efforts
> > on
> > > > making an online POC
> > > >
> > > > previous web:
> > > >
> > >
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > > > POC web:
> > > >
> > > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> > > >
> > > >
> > > > The vote will last for at least 72 hours, following the consensus
> > voting
> > > > process.
> > > >
> > > > FLIP wiki:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> > > >
> > > > Discussion thread:
> > > >
> > > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > > >
> > > > Thanks,
> > > >
> > > > Yadong
> > > >
> > >
> >
> >
>
Re: [VOTE] FLIP-101: Add Pending Slots Detail
Posted by lining jing <ji...@gmail.com>.
Currently, users can't define slot resources. If there are pending slots,
the main reason is that there are not enough slots, so the slot size is
mainly displayed without more details. If slot resources can be set
dynamically later, more information could be shown.
Zhijiang <wa...@aliyun.com.invalid> 于2020年2月21日周五 下午11:22写道：
> Thanks Yadong for this FLIP!
>
> If I understood correctly, the motivation is based on the scenario of
> scheduled job state for long time in the well resourced cluster.
>
> It is meaningful for me if we can provide some useful infos to help users
> analyze the reason. I see from PoC that we can get the
> respective vertex which slot is pending and the detailed slot id info.
> I am curious that is it feasible to do some valid analysis from logs based
> on the slot id? Or what's the expectation to use these infos?
>
> Best,
> Zhijiang
> ------------------------------------------------------------------
> From:Kurt Young <yk...@gmail.com>
> Send Time:2020 Feb. 21 (Fri.) 14:38
> To:dev <de...@flink.apache.org>
> Subject:Re: [VOTE] FLIP-101: Add Pending Slots Detail
>
> I agree with Jark, even if we have pending slots now, a dedicated tab seems
> to be too much.
>
> Best,
> Kurt
>
>
> On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <im...@gmail.com> wrote:
>
> > Thanks Yadong,
> >
> > I think a pending slot view will be helpful. But will it be verbose when
> > there is no pending slot, but a "pending slot" in the tab?
> > What do you think to show the pending slot page when click the "?" on the
> > vertex status?
> >
> > Best,
> > Jark
> >
> > On Thu, 20 Feb 2020 at 17:50, Yadong Xie <vt...@gmail.com> wrote:
> >
> > > Hi all
> > >
> > > I want to start the vote for FLIP-101, which proposes to add pending
> > slots
> > > information to help users check which vertex/subtask is blocked.
> > >
> > > To help everyone better understand the proposal, we spent some efforts
> on
> > > making an online POC
> > >
> > > previous web:
> > >
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > > POC web:
> > >
> > >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> > >
> > >
> > > The vote will last for at least 72 hours, following the consensus
> voting
> > > process.
> > >
> > > FLIP wiki:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> > >
> > > Discussion thread:
> > >
> > >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> > >
> > > Thanks,
> > >
> > > Yadong
> > >
> >
>
>
Re: [VOTE] FLIP-101: Add Pending Slots Detail
Posted by Zhijiang <wa...@aliyun.com.INVALID>.
Thanks Yadong for this FLIP!
If I understood correctly, the motivation is based on the scenario of scheduled job state for long time in the well resourced cluster.
It is meaningful for me if we can provide some useful infos to help users analyze the reason. I see from PoC that we can get the
respective vertex which slot is pending and the detailed slot id info.
I am curious that is it feasible to do some valid analysis from logs based on the slot id? Or what's the expectation to use these infos?
Best,
Zhijiang
------------------------------------------------------------------
From:Kurt Young <yk...@gmail.com>
Send Time:2020 Feb. 21 (Fri.) 14:38
To:dev <de...@flink.apache.org>
Subject:Re: [VOTE] FLIP-101: Add Pending Slots Detail
I agree with Jark, even if we have pending slots now, a dedicated tab seems
to be too much.
Best,
Kurt
On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <im...@gmail.com> wrote:
> Thanks Yadong,
>
> I think a pending slot view will be helpful. But will it be verbose when
> there is no pending slot, but a "pending slot" in the tab?
> What do you think to show the pending slot page when click the "?" on the
> vertex status?
>
> Best,
> Jark
>
> On Thu, 20 Feb 2020 at 17:50, Yadong Xie <vt...@gmail.com> wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-101, which proposes to add pending
> slots
> > information to help users check which vertex/subtask is blocked.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web:
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>
Re: [VOTE] FLIP-101: Add Pending Slots Detail
Posted by Kurt Young <yk...@gmail.com>.
I agree with Jark, even if we have pending slots now, a dedicated tab seems
to be too much.
Best,
Kurt
On Fri, Feb 21, 2020 at 2:12 PM Jark Wu <im...@gmail.com> wrote:
> Thanks Yadong,
>
> I think a pending slot view will be helpful. But will it be verbose when
> there is no pending slot, but a "pending slot" in the tab?
> What do you think to show the pending slot page when click the "?" on the
> vertex status?
>
> Best,
> Jark
>
> On Thu, 20 Feb 2020 at 17:50, Yadong Xie <vt...@gmail.com> wrote:
>
> > Hi all
> >
> > I want to start the vote for FLIP-101, which proposes to add pending
> slots
> > information to help users check which vertex/subtask is blocked.
> >
> > To help everyone better understand the proposal, we spent some efforts on
> > making an online POC
> >
> > previous web:
> >
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> > POC web:
> >
> >
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
> >
> >
> > The vote will last for at least 72 hours, following the consensus voting
> > process.
> >
> > FLIP wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
> >
> > Discussion thread:
> >
> >
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
> >
> > Thanks,
> >
> > Yadong
> >
>
Re: [VOTE] FLIP-101: Add Pending Slots Detail
Posted by Jark Wu <im...@gmail.com>.
Thanks Yadong,
I think a pending slot view will be helpful. But will it be verbose when
there is no pending slot, but a "pending slot" in the tab?
What do you think to show the pending slot page when click the "?" on the
vertex status?
Best,
Jark
On Thu, 20 Feb 2020 at 17:50, Yadong Xie <vt...@gmail.com> wrote:
> Hi all
>
> I want to start the vote for FLIP-101, which proposes to add pending slots
> information to help users check which vertex/subtask is blocked.
>
> To help everyone better understand the proposal, we spent some efforts on
> making an online POC
>
> previous web:
> http://101.132.122.69:8081/#/job/b88840a1e71a0535e1556b52c4c12fcc/overview
> POC web:
>
> http://101.132.122.69:8081/web/#/job/b88840a1e71a0535e1556b52c4c12fcc/pending-slots
>
>
> The vote will last for at least 72 hours, following the consensus voting
> process.
>
> FLIP wiki:
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-101%3A+Add+Pending+Slots+Detail
>
> Discussion thread:
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html
>
> Thanks,
>
> Yadong
>