You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by "Zhijiang(wangzhijiang999)" <wa...@aliyun.com.INVALID> on 2018/10/10 06:24:14 UTC

回复：[DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Thanks Fabian for proposing this topic.

It is very worth improving the web dashborad for showing more useful informations which can benefit flink users a lot.

Just two small personal concerns:
1. The start time and end time are already given, so it is easy to estimate the rough duration time. Is it necessary to show the duration information to occupy the space?
2. The job name given by users can be used for identification, and the job id is automatically generated in random. I am not sure whether this id is useful for further debugging. If not maybe we can ignore the job id from the dashboard?

Best,
Zhijiang
------------------------------------------------------------------
发件人：Jin Sun <is...@gmail.com>
发送时间：2018年10月10日(星期三) 01:10
收件人：dev <de...@flink.apache.org>
主　题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Great job! That would very helpful for debug.


I would suggest to use small icons for this Job Manager/Managers when there are too many instances (like a thousand)
May be we can also introduce locality,  that task managers belongs to same rack shows together?




Small icons can be like this:




On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org> wrote:
mation on the front
page. Your mock looks really promising to me since it shows some basic
metrics and cluster information at a glance. Apart from the the source
input and sink output metrics, all other required information should be
available to display it in the dashboard. Thus, your proposal should only
affect flink-runtime-web which should make it easier to realize.

I'm in favour of adding this feature to Flink's dashboard to make it
available to the whole community.

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by Till Rohrmann <tr...@apache.org>.
Hi Fabian,

yes the community is very much open and thankful for contributions to the
web UI including the technology used. What it could use is a person who
would really like to drive this since so far it was if it all someone's
side project.

Cheers,
Till

On Wed, Oct 10, 2018 at 11:28 AM Fabian Wollert <fa...@zalando.de> wrote:

> Hi everyone, thx for all the comments and feedback. Let me address
> everything individually:
>
> @Till: yes, for the start my plan would be to just touch the
> flink-runtime-web/web-dashboard repo/folder.
>
> @Jin Sun:
>
>    - smaller icons on increasing server counts: yes, thats also something i
>    already thought about. will keep it in mind when realizing the first
>    version!
>    - about locality: i searched quickly through the docs, but i could not
>    find anything regarding flink featuring rack awareness. Is this
> something
>    already implemented? If not, i think this will bloat the size of this
>    initial proposal. If its somewhere already included, we could implement
> it
>    for sure.
>
> @Zhijiang:the focus of this redesign was not yet including the job list in
> the lower half of the overview. as part of the redesign we can also think
> about optimising this list though, and removing unnecessary columns is
> usually the most easy thing to do. we can maybe create a separate ticket
> for this as well and discuss this issue there, to not bloat the initial
> discussion with too much topics.
>
> @Robert:
>
>    - Agreed that it might make sense to also show this on job level. Since
>    these metrics are probably gonna be introduced later only anyways, we
> can
>    discuss this maybe then separately after FLINK-9050
>    <https://issues.apache.org/jira/browse/FLINK-9050> (linked the wrong
>    ticket in my initial mail) is done.
>    - Rewriting the whole thing while doing this also came to my mind. What
>    i would like to do anyways (even if we stick for now to A1) is to remove
>    bower as a package manager (since its deprecated) and update bootstrap
> to
>    V4. I will check what the additional effort is to move to React/Redux.
>    We're working with this here at work as well, so implementing at least a
>    first MVP might be feasible as well, before getting to deep into A1
>    specifics. But that basically means that you guys are open to change the
>    underlying web/JS technology, yeah?
>
> Cheers
>
> --
>
>
> *Fabian WollertZalando SE*
>
> E-Mail: fabian@zalando.de
>
>
> Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
> rmetzger@apache.org>:
>
> > Hey Fabian,
> > thanks a lot for reaching out to the Flink community with this proposal!
> > (Posting to the ML instead of creating a JIRA is a good idea for such
> > questions -- you can create a ticket/tickets once the discussion here has
> > come to a conclusion)
> >
> > I have two comments:
> > - You are listing Records/Kb in and Records/Kb out as cluster-wide
> metrics.
> > I wonder whether we should rather show these metrics for each job,
> instead
> > of the entire cluster? (or maybe both). My concern is that the
> cluster-wide
> > metric is not really relevant as soon as you have jobs with different
> > characteristics running on one cluster
> > - You mention that the Flink UI is based on Angular 1. I've been thinking
> > for quite a while now whether we should actually rewrite / migrate the
> > Flink UI to React.
> > Do you think we can re-use most of the work you'd be doing for this
> change
> > when we migrate to React?
> >
> > Best,
> > Robert
> >
> >
> >
> > On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
> > <wa...@aliyun.com.invalid> wrote:
> >
> > > Thanks Fabian for proposing this topic.
> > >
> > > It is very worth improving the web dashborad for showing more useful
> > > informations which can benefit flink users a lot.
> > >
> > > Just two small personal concerns:
> > > 1. The start time and end time are already given, so it is easy to
> > > estimate the rough duration time. Is it necessary to show the duration
> > > information to occupy the space?
> > > 2. The job name given by users can be used for identification, and the
> > > job id is automatically generated in random. I am not sure whether this
> > id
> > > is useful for further debugging. If not maybe we can ignore the job id
> > from
> > > the dashboard?
> > >
> > > Best,
> > > Zhijiang
> > >
> > > ------------------------------------------------------------------
> > > 发件人：Jin Sun <is...@gmail.com>
> > > 发送时间：2018年10月10日(星期三) 01:10
> > > 收件人：dev <de...@flink.apache.org>
> > > 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal
> > >
> > > Great job! That would very helpful for debug.
> > >
> > >
> > >    - I would suggest to use small icons for this Job Manager/Managers
> > >    when there are too many instances (like a thousand)
> > >    - May be we can also introduce locality,  that task managers belongs
> > >    to same rack shows together?
> > >
> > >
> > >
> > >
> > >
> > > Small icons can be like this:
> > >
> > >
> > >
> > >
> > > On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org>
> wrote:
> > >
> > > mation on the front
> > > page. Your mock looks really promising to me since it shows some basic
> > > metrics and cluster information at a glance. Apart from the the source
> > > input and sink output metrics, all other required information should be
> > > available to display it in the dashboard. Thus, your proposal should
> only
> > > affect flink-runtime-web which should make it easier to realize.
> > >
> > > I'm in favour of adding this feature to Flink's dashboard to make it
> > > available to the whole community.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by lining jing <ji...@gmail.com>.
Ok

Shaoxuan Wang <ws...@gmail.com>于2018年11月6日 周二19:26写道：

> Lining,
> Thanks for the proposal.
> There is another ongoing ML (
>
> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Change-underlying-Frontend-Architecture-for-Flink-Web-Dashboard-td24902.html
> ),
> where YaDong has shared some sample code (
> https://github.com/vthinkxie/flink-runtime-web) which upgraded the Flink
> web UI to Angular 7.0. I would suggest we add all the new web UI features
> on top of that underlying framework. What do you think?
>
> Regards,
> Shaoxuan
>
>
>
> On Tue, Nov 6, 2018 at 6:37 PM lining jing <ji...@gmail.com> wrote:
>
> > We also need refactor single job show. Now, just can see vertex metrics.
> > And if you want see other informations, have to go to other page .First,
> we
> > need update vertex
> >  and operator show.
> >
> > - vertex show in https://issues.apache.org/jira/browse/FLINK-10802.
> > [image: image.png]
> > - operator show like
> > [image: image.png]
> > - add job dashbord  show faiover,  tm, operator overview.
> >
> >
> > Fabian Wollert <fa...@zalando.de> 于2018年10月29日周一 下午7:59写道：
> >
> >> sure, will do.
> >>
> >> --
> >>
> >>
> >> *Fabian WollertZalando SE*
> >>
> >> E-Mail: fabian@zalando.de
> >>
> >>
> >> Am Mo., 29. Okt. 2018 um 12:57 Uhr schrieb Chesnay Schepler <
> >> chesnay@apache.org>:
> >>
> >> > Please start the discussion in an entirely new thread; people may
> >> > discard this thread immediately since the first page is purely about
> the
> >> > layout of the WebUI.
> >> >
> >> > On 29.10.2018 12:39, Fabian Wollert wrote:
> >> > > Hi again,
> >> > >
> >> > > Chesnay correctly commented in the tickets that we first should
> >> discuss
> >> > > here, if changing the underlying technology for the Flink Web
> >> Dashboard
> >> > is
> >> > > a valid option at all. What are your thoughts about this?
> >> > >
> >> > > personally I agree with Till's comments in the ticket, Angular 1
> being
> >> > > basically outdated and is not having a large following anymore. From
> >> my
> >> > > experience the choice between Angular 2-7 or React is subjective,
> you
> >> can
> >> > > get things done with both. I personally only have experience with
> >> React,
> >> > so
> >> > > i personally would be faster to develop with this one. I currently
> >> have
> >> > not
> >> > > planned to learn Angular as well (being a more backend focused
> >> developer
> >> > in
> >> > > general) so if the decision would be to go with Angular, i would be
> >> > > unfortunately out of this rework of the Flink Dashboard most
> >> certainly.
> >> > >
> >> > > Cheers
> >> > > Fabian
> >> > >
> >> > > --
> >> > >
> >> > >
> >> > > *Fabian WollertZalando SE*
> >> > >
> >> > > E-Mail: fabian@zalando.de
> >> > >
> >> > >
> >> > > Am Mo., 29. Okt. 2018 um 09:21 Uhr schrieb Fabian Wollert <
> >> > fabian@zalando.de
> >> > >> :
> >> > >> Hi everyone,
> >> > >>
> >> > >> thx for all the feedback. I created now
> >> > >> https://issues.apache.org/jira/browse/FLINK-10705 with sub tickets
> >> to
> >> > >> tackle this. i also found some time this weekend and implemented
> the
> >> > first
> >> > >> draft, which i will post in the ticket (not sure if i get the
> >> pictures
> >> > to
> >> > >> work here in the mailing list :-D).
> >> > >>
> >> > >> Lets continue discussion in the tickets then.
> >> > >>
> >> > >> Since this is my first bigger contribution to Flink, please advise
> on
> >> > how
> >> > >> to handle tickets, and structure the work. But for now i will just
> >> > continue
> >> > >> to work on this, whenever i find free time.
> >> > >>
> >> > >> Cheers
> >> > >>
> >> > >> --
> >> > >>
> >> > >>
> >> > >> *Fabian WollertZalando SE*
> >> > >>
> >> > >> E-Mail: fabian@zalando.de
> >> > >>
> >> > >>
> >> > >> Am Sa., 27. Okt. 2018 um 17:15 Uhr schrieb Robert Metzger <
> >> > >> rmetzger@apache.org>:
> >> > >>
> >> > >>> Hey,
> >> > >>> Sorry for the delay.
> >> > >>>
> >> > >>> Yes -- I would be open to revisit the underlying technologies.
> >> > >>>
> >> > >>> Best,
> >> > >>> Robert
> >> > >>>
> >> > >>> On Wed, Oct 10, 2018 at 11:28 AM Fabian Wollert <
> fabian@zalando.de>
> >> > >>> wrote:
> >> > >>>
> >> > >>>> Hi everyone, thx for all the comments and feedback. Let me
> address
> >> > >>>> everything individually:
> >> > >>>>
> >> > >>>> @Till: yes, for the start my plan would be to just touch the
> >> > >>>> flink-runtime-web/web-dashboard repo/folder.
> >> > >>>>
> >> > >>>> @Jin Sun:
> >> > >>>>
> >> > >>>>     - smaller icons on increasing server counts: yes, thats also
> >> > >>> something i
> >> > >>>>     already thought about. will keep it in mind when realizing
> the
> >> > first
> >> > >>>>     version!
> >> > >>>>     - about locality: i searched quickly through the docs, but i
> >> could
> >> > >>> not
> >> > >>>>     find anything regarding flink featuring rack awareness. Is
> this
> >> > >>>> something
> >> > >>>>     already implemented? If not, i think this will bloat the size
> >> of
> >> > this
> >> > >>>>     initial proposal. If its somewhere already included, we could
> >> > >>> implement
> >> > >>>> it
> >> > >>>>     for sure.
> >> > >>>>
> >> > >>>> @Zhijiang:the focus of this redesign was not yet including the
> job
> >> > list
> >> > >>> in
> >> > >>>> the lower half of the overview. as part of the redesign we can
> also
> >> > >>> think
> >> > >>>> about optimising this list though, and removing unnecessary
> >> columns is
> >> > >>>> usually the most easy thing to do. we can maybe create a separate
> >> > ticket
> >> > >>>> for this as well and discuss this issue there, to not bloat the
> >> > initial
> >> > >>>> discussion with too much topics.
> >> > >>>>
> >> > >>>> @Robert:
> >> > >>>>
> >> > >>>>     - Agreed that it might make sense to also show this on job
> >> level.
> >> > >>> Since
> >> > >>>>     these metrics are probably gonna be introduced later only
> >> > anyways, we
> >> > >>>> can
> >> > >>>>     discuss this maybe then separately after FLINK-9050
> >> > >>>>     <https://issues.apache.org/jira/browse/FLINK-9050> (linked
> the
> >> > wrong
> >> > >>>>     ticket in my initial mail) is done.
> >> > >>>>     - Rewriting the whole thing while doing this also came to my
> >> mind.
> >> > >>> What
> >> > >>>>     i would like to do anyways (even if we stick for now to A1)
> is
> >> to
> >> > >>> remove
> >> > >>>>     bower as a package manager (since its deprecated) and update
> >> > >>> bootstrap
> >> > >>>> to
> >> > >>>>     V4. I will check what the additional effort is to move to
> >> > >>> React/Redux.
> >> > >>>>     We're working with this here at work as well, so implementing
> >> at
> >> > >>> least a
> >> > >>>>     first MVP might be feasible as well, before getting to deep
> >> into
> >> > A1
> >> > >>>>     specifics. But that basically means that you guys are open to
> >> > change
> >> > >>> the
> >> > >>>>     underlying web/JS technology, yeah?
> >> > >>>>
> >> > >>>> Cheers
> >> > >>>>
> >> > >>>> --
> >> > >>>>
> >> > >>>>
> >> > >>>> *Fabian WollertZalando SE*
> >> > >>>>
> >> > >>>> E-Mail: fabian@zalando.de
> >> > >>>>
> >> > >>>>
> >> > >>>> Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
> >> > >>>> rmetzger@apache.org>:
> >> > >>>>
> >> > >>>>> Hey Fabian,
> >> > >>>>> thanks a lot for reaching out to the Flink community with this
> >> > >>> proposal!
> >> > >>>>> (Posting to the ML instead of creating a JIRA is a good idea for
> >> such
> >> > >>>>> questions -- you can create a ticket/tickets once the discussion
> >> here
> >> > >>> has
> >> > >>>>> come to a conclusion)
> >> > >>>>>
> >> > >>>>> I have two comments:
> >> > >>>>> - You are listing Records/Kb in and Records/Kb out as
> cluster-wide
> >> > >>>> metrics.
> >> > >>>>> I wonder whether we should rather show these metrics for each
> job,
> >> > >>>> instead
> >> > >>>>> of the entire cluster? (or maybe both). My concern is that the
> >> > >>>> cluster-wide
> >> > >>>>> metric is not really relevant as soon as you have jobs with
> >> different
> >> > >>>>> characteristics running on one cluster
> >> > >>>>> - You mention that the Flink UI is based on Angular 1. I've been
> >> > >>> thinking
> >> > >>>>> for quite a while now whether we should actually rewrite /
> migrate
> >> > the
> >> > >>>>> Flink UI to React.
> >> > >>>>> Do you think we can re-use most of the work you'd be doing for
> >> this
> >> > >>>> change
> >> > >>>>> when we migrate to React?
> >> > >>>>>
> >> > >>>>> Best,
> >> > >>>>> Robert
> >> > >>>>>
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
> >> > >>>>> <wa...@aliyun.com.invalid> wrote:
> >> > >>>>>
> >> > >>>>>> Thanks Fabian for proposing this topic.
> >> > >>>>>>
> >> > >>>>>> It is very worth improving the web dashborad for showing more
> >> useful
> >> > >>>>>> informations which can benefit flink users a lot.
> >> > >>>>>>
> >> > >>>>>> Just two small personal concerns:
> >> > >>>>>> 1. The start time and end time are already given, so it is easy
> >> to
> >> > >>>>>> estimate the rough duration time. Is it necessary to show the
> >> > >>> duration
> >> > >>>>>> information to occupy the space?
> >> > >>>>>> 2. The job name given by users can be used for identification,
> >> and
> >> > >>> the
> >> > >>>>>> job id is automatically generated in random. I am not sure
> >> whether
> >> > >>> this
> >> > >>>>> id
> >> > >>>>>> is useful for further debugging. If not maybe we can ignore the
> >> job
> >> > >>> id
> >> > >>>>> from
> >> > >>>>>> the dashboard?
> >> > >>>>>>
> >> > >>>>>> Best,
> >> > >>>>>> Zhijiang
> >> > >>>>>>
> >> > >>>>>>
> >> ------------------------------------------------------------------
> >> > >>>>>> 发件人：Jin Sun <is...@gmail.com>
> >> > >>>>>> 发送时间：2018年10月10日(星期三) 01:10
> >> > >>>>>> 收件人：dev <de...@flink.apache.org>
> >> > >>>>>> 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement
> >> > >>> Proposal
> >> > >>>>>> Great job! That would very helpful for debug.
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>     - I would suggest to use small icons for this Job
> >> > >>> Manager/Managers
> >> > >>>>>>     when there are too many instances (like a thousand)
> >> > >>>>>>     - May be we can also introduce locality,  that task
> managers
> >> > >>> belongs
> >> > >>>>>>     to same rack shows together?
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> Small icons can be like this:
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>> On Oct 9, 2018, at 8:49 PM, Till Rohrmann <
> trohrmann@apache.org>
> >> > >>>> wrote:
> >> > >>>>>> mation on the front
> >> > >>>>>> page. Your mock looks really promising to me since it shows
> some
> >> > >>> basic
> >> > >>>>>> metrics and cluster information at a glance. Apart from the the
> >> > >>> source
> >> > >>>>>> input and sink output metrics, all other required information
> >> > >>> should be
> >> > >>>>>> available to display it in the dashboard. Thus, your proposal
> >> should
> >> > >>>> only
> >> > >>>>>> affect flink-runtime-web which should make it easier to
> realize.
> >> > >>>>>>
> >> > >>>>>> I'm in favour of adding this feature to Flink's dashboard to
> >> make it
> >> > >>>>>> available to the whole community.
> >> > >>>>>>
> >> > >>>>>>
> >> > >>>>>>
> >> >
> >> >
> >>
> >
>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by Shaoxuan Wang <ws...@gmail.com>.
Lining,
Thanks for the proposal.
There is another ongoing ML (
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Change-underlying-Frontend-Architecture-for-Flink-Web-Dashboard-td24902.html),
where YaDong has shared some sample code (
https://github.com/vthinkxie/flink-runtime-web) which upgraded the Flink
web UI to Angular 7.0. I would suggest we add all the new web UI features
on top of that underlying framework. What do you think?

Regards,
Shaoxuan



On Tue, Nov 6, 2018 at 6:37 PM lining jing <ji...@gmail.com> wrote:

> We also need refactor single job show. Now, just can see vertex metrics.
> And if you want see other informations, have to go to other page .First, we
> need update vertex
>  and operator show.
>
> - vertex show in https://issues.apache.org/jira/browse/FLINK-10802.
> [image: image.png]
> - operator show like
> [image: image.png]
> - add job dashbord  show faiover,  tm, operator overview.
>
>
> Fabian Wollert <fa...@zalando.de> 于2018年10月29日周一 下午7:59写道：
>
>> sure, will do.
>>
>> --
>>
>>
>> *Fabian WollertZalando SE*
>>
>> E-Mail: fabian@zalando.de
>>
>>
>> Am Mo., 29. Okt. 2018 um 12:57 Uhr schrieb Chesnay Schepler <
>> chesnay@apache.org>:
>>
>> > Please start the discussion in an entirely new thread; people may
>> > discard this thread immediately since the first page is purely about the
>> > layout of the WebUI.
>> >
>> > On 29.10.2018 12:39, Fabian Wollert wrote:
>> > > Hi again,
>> > >
>> > > Chesnay correctly commented in the tickets that we first should
>> discuss
>> > > here, if changing the underlying technology for the Flink Web
>> Dashboard
>> > is
>> > > a valid option at all. What are your thoughts about this?
>> > >
>> > > personally I agree with Till's comments in the ticket, Angular 1 being
>> > > basically outdated and is not having a large following anymore. From
>> my
>> > > experience the choice between Angular 2-7 or React is subjective, you
>> can
>> > > get things done with both. I personally only have experience with
>> React,
>> > so
>> > > i personally would be faster to develop with this one. I currently
>> have
>> > not
>> > > planned to learn Angular as well (being a more backend focused
>> developer
>> > in
>> > > general) so if the decision would be to go with Angular, i would be
>> > > unfortunately out of this rework of the Flink Dashboard most
>> certainly.
>> > >
>> > > Cheers
>> > > Fabian
>> > >
>> > > --
>> > >
>> > >
>> > > *Fabian WollertZalando SE*
>> > >
>> > > E-Mail: fabian@zalando.de
>> > >
>> > >
>> > > Am Mo., 29. Okt. 2018 um 09:21 Uhr schrieb Fabian Wollert <
>> > fabian@zalando.de
>> > >> :
>> > >> Hi everyone,
>> > >>
>> > >> thx for all the feedback. I created now
>> > >> https://issues.apache.org/jira/browse/FLINK-10705 with sub tickets
>> to
>> > >> tackle this. i also found some time this weekend and implemented the
>> > first
>> > >> draft, which i will post in the ticket (not sure if i get the
>> pictures
>> > to
>> > >> work here in the mailing list :-D).
>> > >>
>> > >> Lets continue discussion in the tickets then.
>> > >>
>> > >> Since this is my first bigger contribution to Flink, please advise on
>> > how
>> > >> to handle tickets, and structure the work. But for now i will just
>> > continue
>> > >> to work on this, whenever i find free time.
>> > >>
>> > >> Cheers
>> > >>
>> > >> --
>> > >>
>> > >>
>> > >> *Fabian WollertZalando SE*
>> > >>
>> > >> E-Mail: fabian@zalando.de
>> > >>
>> > >>
>> > >> Am Sa., 27. Okt. 2018 um 17:15 Uhr schrieb Robert Metzger <
>> > >> rmetzger@apache.org>:
>> > >>
>> > >>> Hey,
>> > >>> Sorry for the delay.
>> > >>>
>> > >>> Yes -- I would be open to revisit the underlying technologies.
>> > >>>
>> > >>> Best,
>> > >>> Robert
>> > >>>
>> > >>> On Wed, Oct 10, 2018 at 11:28 AM Fabian Wollert <fa...@zalando.de>
>> > >>> wrote:
>> > >>>
>> > >>>> Hi everyone, thx for all the comments and feedback. Let me address
>> > >>>> everything individually:
>> > >>>>
>> > >>>> @Till: yes, for the start my plan would be to just touch the
>> > >>>> flink-runtime-web/web-dashboard repo/folder.
>> > >>>>
>> > >>>> @Jin Sun:
>> > >>>>
>> > >>>>     - smaller icons on increasing server counts: yes, thats also
>> > >>> something i
>> > >>>>     already thought about. will keep it in mind when realizing the
>> > first
>> > >>>>     version!
>> > >>>>     - about locality: i searched quickly through the docs, but i
>> could
>> > >>> not
>> > >>>>     find anything regarding flink featuring rack awareness. Is this
>> > >>>> something
>> > >>>>     already implemented? If not, i think this will bloat the size
>> of
>> > this
>> > >>>>     initial proposal. If its somewhere already included, we could
>> > >>> implement
>> > >>>> it
>> > >>>>     for sure.
>> > >>>>
>> > >>>> @Zhijiang:the focus of this redesign was not yet including the job
>> > list
>> > >>> in
>> > >>>> the lower half of the overview. as part of the redesign we can also
>> > >>> think
>> > >>>> about optimising this list though, and removing unnecessary
>> columns is
>> > >>>> usually the most easy thing to do. we can maybe create a separate
>> > ticket
>> > >>>> for this as well and discuss this issue there, to not bloat the
>> > initial
>> > >>>> discussion with too much topics.
>> > >>>>
>> > >>>> @Robert:
>> > >>>>
>> > >>>>     - Agreed that it might make sense to also show this on job
>> level.
>> > >>> Since
>> > >>>>     these metrics are probably gonna be introduced later only
>> > anyways, we
>> > >>>> can
>> > >>>>     discuss this maybe then separately after FLINK-9050
>> > >>>>     <https://issues.apache.org/jira/browse/FLINK-9050> (linked the
>> > wrong
>> > >>>>     ticket in my initial mail) is done.
>> > >>>>     - Rewriting the whole thing while doing this also came to my
>> mind.
>> > >>> What
>> > >>>>     i would like to do anyways (even if we stick for now to A1) is
>> to
>> > >>> remove
>> > >>>>     bower as a package manager (since its deprecated) and update
>> > >>> bootstrap
>> > >>>> to
>> > >>>>     V4. I will check what the additional effort is to move to
>> > >>> React/Redux.
>> > >>>>     We're working with this here at work as well, so implementing
>> at
>> > >>> least a
>> > >>>>     first MVP might be feasible as well, before getting to deep
>> into
>> > A1
>> > >>>>     specifics. But that basically means that you guys are open to
>> > change
>> > >>> the
>> > >>>>     underlying web/JS technology, yeah?
>> > >>>>
>> > >>>> Cheers
>> > >>>>
>> > >>>> --
>> > >>>>
>> > >>>>
>> > >>>> *Fabian WollertZalando SE*
>> > >>>>
>> > >>>> E-Mail: fabian@zalando.de
>> > >>>>
>> > >>>>
>> > >>>> Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
>> > >>>> rmetzger@apache.org>:
>> > >>>>
>> > >>>>> Hey Fabian,
>> > >>>>> thanks a lot for reaching out to the Flink community with this
>> > >>> proposal!
>> > >>>>> (Posting to the ML instead of creating a JIRA is a good idea for
>> such
>> > >>>>> questions -- you can create a ticket/tickets once the discussion
>> here
>> > >>> has
>> > >>>>> come to a conclusion)
>> > >>>>>
>> > >>>>> I have two comments:
>> > >>>>> - You are listing Records/Kb in and Records/Kb out as cluster-wide
>> > >>>> metrics.
>> > >>>>> I wonder whether we should rather show these metrics for each job,
>> > >>>> instead
>> > >>>>> of the entire cluster? (or maybe both). My concern is that the
>> > >>>> cluster-wide
>> > >>>>> metric is not really relevant as soon as you have jobs with
>> different
>> > >>>>> characteristics running on one cluster
>> > >>>>> - You mention that the Flink UI is based on Angular 1. I've been
>> > >>> thinking
>> > >>>>> for quite a while now whether we should actually rewrite / migrate
>> > the
>> > >>>>> Flink UI to React.
>> > >>>>> Do you think we can re-use most of the work you'd be doing for
>> this
>> > >>>> change
>> > >>>>> when we migrate to React?
>> > >>>>>
>> > >>>>> Best,
>> > >>>>> Robert
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
>> > >>>>> <wa...@aliyun.com.invalid> wrote:
>> > >>>>>
>> > >>>>>> Thanks Fabian for proposing this topic.
>> > >>>>>>
>> > >>>>>> It is very worth improving the web dashborad for showing more
>> useful
>> > >>>>>> informations which can benefit flink users a lot.
>> > >>>>>>
>> > >>>>>> Just two small personal concerns:
>> > >>>>>> 1. The start time and end time are already given, so it is easy
>> to
>> > >>>>>> estimate the rough duration time. Is it necessary to show the
>> > >>> duration
>> > >>>>>> information to occupy the space?
>> > >>>>>> 2. The job name given by users can be used for identification,
>> and
>> > >>> the
>> > >>>>>> job id is automatically generated in random. I am not sure
>> whether
>> > >>> this
>> > >>>>> id
>> > >>>>>> is useful for further debugging. If not maybe we can ignore the
>> job
>> > >>> id
>> > >>>>> from
>> > >>>>>> the dashboard?
>> > >>>>>>
>> > >>>>>> Best,
>> > >>>>>> Zhijiang
>> > >>>>>>
>> > >>>>>>
>> ------------------------------------------------------------------
>> > >>>>>> 发件人：Jin Sun <is...@gmail.com>
>> > >>>>>> 发送时间：2018年10月10日(星期三) 01:10
>> > >>>>>> 收件人：dev <de...@flink.apache.org>
>> > >>>>>> 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement
>> > >>> Proposal
>> > >>>>>> Great job! That would very helpful for debug.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>     - I would suggest to use small icons for this Job
>> > >>> Manager/Managers
>> > >>>>>>     when there are too many instances (like a thousand)
>> > >>>>>>     - May be we can also introduce locality,  that task managers
>> > >>> belongs
>> > >>>>>>     to same rack shows together?
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Small icons can be like this:
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org>
>> > >>>> wrote:
>> > >>>>>> mation on the front
>> > >>>>>> page. Your mock looks really promising to me since it shows some
>> > >>> basic
>> > >>>>>> metrics and cluster information at a glance. Apart from the the
>> > >>> source
>> > >>>>>> input and sink output metrics, all other required information
>> > >>> should be
>> > >>>>>> available to display it in the dashboard. Thus, your proposal
>> should
>> > >>>> only
>> > >>>>>> affect flink-runtime-web which should make it easier to realize.
>> > >>>>>>
>> > >>>>>> I'm in favour of adding this feature to Flink's dashboard to
>> make it
>> > >>>>>> available to the whole community.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> >
>> >
>>
>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by lining jing <ji...@gmail.com>.
We also need refactor single job show. Now, just can see vertex metrics.
And if you want see other informations, have to go to other page .First, we
need update vertex
 and operator show.

- vertex show in https://issues.apache.org/jira/browse/FLINK-10802.
[image: image.png]
- operator show like
[image: image.png]
- add job dashbord  show faiover,  tm, operator overview.


Fabian Wollert <fa...@zalando.de> 于2018年10月29日周一 下午7:59写道：

> sure, will do.
>
> --
>
>
> *Fabian WollertZalando SE*
>
> E-Mail: fabian@zalando.de
>
>
> Am Mo., 29. Okt. 2018 um 12:57 Uhr schrieb Chesnay Schepler <
> chesnay@apache.org>:
>
> > Please start the discussion in an entirely new thread; people may
> > discard this thread immediately since the first page is purely about the
> > layout of the WebUI.
> >
> > On 29.10.2018 12:39, Fabian Wollert wrote:
> > > Hi again,
> > >
> > > Chesnay correctly commented in the tickets that we first should discuss
> > > here, if changing the underlying technology for the Flink Web Dashboard
> > is
> > > a valid option at all. What are your thoughts about this?
> > >
> > > personally I agree with Till's comments in the ticket, Angular 1 being
> > > basically outdated and is not having a large following anymore. From my
> > > experience the choice between Angular 2-7 or React is subjective, you
> can
> > > get things done with both. I personally only have experience with
> React,
> > so
> > > i personally would be faster to develop with this one. I currently have
> > not
> > > planned to learn Angular as well (being a more backend focused
> developer
> > in
> > > general) so if the decision would be to go with Angular, i would be
> > > unfortunately out of this rework of the Flink Dashboard most certainly.
> > >
> > > Cheers
> > > Fabian
> > >
> > > --
> > >
> > >
> > > *Fabian WollertZalando SE*
> > >
> > > E-Mail: fabian@zalando.de
> > >
> > >
> > > Am Mo., 29. Okt. 2018 um 09:21 Uhr schrieb Fabian Wollert <
> > fabian@zalando.de
> > >> :
> > >> Hi everyone,
> > >>
> > >> thx for all the feedback. I created now
> > >> https://issues.apache.org/jira/browse/FLINK-10705 with sub tickets to
> > >> tackle this. i also found some time this weekend and implemented the
> > first
> > >> draft, which i will post in the ticket (not sure if i get the pictures
> > to
> > >> work here in the mailing list :-D).
> > >>
> > >> Lets continue discussion in the tickets then.
> > >>
> > >> Since this is my first bigger contribution to Flink, please advise on
> > how
> > >> to handle tickets, and structure the work. But for now i will just
> > continue
> > >> to work on this, whenever i find free time.
> > >>
> > >> Cheers
> > >>
> > >> --
> > >>
> > >>
> > >> *Fabian WollertZalando SE*
> > >>
> > >> E-Mail: fabian@zalando.de
> > >>
> > >>
> > >> Am Sa., 27. Okt. 2018 um 17:15 Uhr schrieb Robert Metzger <
> > >> rmetzger@apache.org>:
> > >>
> > >>> Hey,
> > >>> Sorry for the delay.
> > >>>
> > >>> Yes -- I would be open to revisit the underlying technologies.
> > >>>
> > >>> Best,
> > >>> Robert
> > >>>
> > >>> On Wed, Oct 10, 2018 at 11:28 AM Fabian Wollert <fa...@zalando.de>
> > >>> wrote:
> > >>>
> > >>>> Hi everyone, thx for all the comments and feedback. Let me address
> > >>>> everything individually:
> > >>>>
> > >>>> @Till: yes, for the start my plan would be to just touch the
> > >>>> flink-runtime-web/web-dashboard repo/folder.
> > >>>>
> > >>>> @Jin Sun:
> > >>>>
> > >>>>     - smaller icons on increasing server counts: yes, thats also
> > >>> something i
> > >>>>     already thought about. will keep it in mind when realizing the
> > first
> > >>>>     version!
> > >>>>     - about locality: i searched quickly through the docs, but i
> could
> > >>> not
> > >>>>     find anything regarding flink featuring rack awareness. Is this
> > >>>> something
> > >>>>     already implemented? If not, i think this will bloat the size of
> > this
> > >>>>     initial proposal. If its somewhere already included, we could
> > >>> implement
> > >>>> it
> > >>>>     for sure.
> > >>>>
> > >>>> @Zhijiang:the focus of this redesign was not yet including the job
> > list
> > >>> in
> > >>>> the lower half of the overview. as part of the redesign we can also
> > >>> think
> > >>>> about optimising this list though, and removing unnecessary columns
> is
> > >>>> usually the most easy thing to do. we can maybe create a separate
> > ticket
> > >>>> for this as well and discuss this issue there, to not bloat the
> > initial
> > >>>> discussion with too much topics.
> > >>>>
> > >>>> @Robert:
> > >>>>
> > >>>>     - Agreed that it might make sense to also show this on job
> level.
> > >>> Since
> > >>>>     these metrics are probably gonna be introduced later only
> > anyways, we
> > >>>> can
> > >>>>     discuss this maybe then separately after FLINK-9050
> > >>>>     <https://issues.apache.org/jira/browse/FLINK-9050> (linked the
> > wrong
> > >>>>     ticket in my initial mail) is done.
> > >>>>     - Rewriting the whole thing while doing this also came to my
> mind.
> > >>> What
> > >>>>     i would like to do anyways (even if we stick for now to A1) is
> to
> > >>> remove
> > >>>>     bower as a package manager (since its deprecated) and update
> > >>> bootstrap
> > >>>> to
> > >>>>     V4. I will check what the additional effort is to move to
> > >>> React/Redux.
> > >>>>     We're working with this here at work as well, so implementing at
> > >>> least a
> > >>>>     first MVP might be feasible as well, before getting to deep into
> > A1
> > >>>>     specifics. But that basically means that you guys are open to
> > change
> > >>> the
> > >>>>     underlying web/JS technology, yeah?
> > >>>>
> > >>>> Cheers
> > >>>>
> > >>>> --
> > >>>>
> > >>>>
> > >>>> *Fabian WollertZalando SE*
> > >>>>
> > >>>> E-Mail: fabian@zalando.de
> > >>>>
> > >>>>
> > >>>> Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
> > >>>> rmetzger@apache.org>:
> > >>>>
> > >>>>> Hey Fabian,
> > >>>>> thanks a lot for reaching out to the Flink community with this
> > >>> proposal!
> > >>>>> (Posting to the ML instead of creating a JIRA is a good idea for
> such
> > >>>>> questions -- you can create a ticket/tickets once the discussion
> here
> > >>> has
> > >>>>> come to a conclusion)
> > >>>>>
> > >>>>> I have two comments:
> > >>>>> - You are listing Records/Kb in and Records/Kb out as cluster-wide
> > >>>> metrics.
> > >>>>> I wonder whether we should rather show these metrics for each job,
> > >>>> instead
> > >>>>> of the entire cluster? (or maybe both). My concern is that the
> > >>>> cluster-wide
> > >>>>> metric is not really relevant as soon as you have jobs with
> different
> > >>>>> characteristics running on one cluster
> > >>>>> - You mention that the Flink UI is based on Angular 1. I've been
> > >>> thinking
> > >>>>> for quite a while now whether we should actually rewrite / migrate
> > the
> > >>>>> Flink UI to React.
> > >>>>> Do you think we can re-use most of the work you'd be doing for this
> > >>>> change
> > >>>>> when we migrate to React?
> > >>>>>
> > >>>>> Best,
> > >>>>> Robert
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
> > >>>>> <wa...@aliyun.com.invalid> wrote:
> > >>>>>
> > >>>>>> Thanks Fabian for proposing this topic.
> > >>>>>>
> > >>>>>> It is very worth improving the web dashborad for showing more
> useful
> > >>>>>> informations which can benefit flink users a lot.
> > >>>>>>
> > >>>>>> Just two small personal concerns:
> > >>>>>> 1. The start time and end time are already given, so it is easy to
> > >>>>>> estimate the rough duration time. Is it necessary to show the
> > >>> duration
> > >>>>>> information to occupy the space?
> > >>>>>> 2. The job name given by users can be used for identification, and
> > >>> the
> > >>>>>> job id is automatically generated in random. I am not sure whether
> > >>> this
> > >>>>> id
> > >>>>>> is useful for further debugging. If not maybe we can ignore the
> job
> > >>> id
> > >>>>> from
> > >>>>>> the dashboard?
> > >>>>>>
> > >>>>>> Best,
> > >>>>>> Zhijiang
> > >>>>>>
> > >>>>>> ------------------------------------------------------------------
> > >>>>>> 发件人：Jin Sun <is...@gmail.com>
> > >>>>>> 发送时间：2018年10月10日(星期三) 01:10
> > >>>>>> 收件人：dev <de...@flink.apache.org>
> > >>>>>> 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement
> > >>> Proposal
> > >>>>>> Great job! That would very helpful for debug.
> > >>>>>>
> > >>>>>>
> > >>>>>>     - I would suggest to use small icons for this Job
> > >>> Manager/Managers
> > >>>>>>     when there are too many instances (like a thousand)
> > >>>>>>     - May be we can also introduce locality,  that task managers
> > >>> belongs
> > >>>>>>     to same rack shows together?
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Small icons can be like this:
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org>
> > >>>> wrote:
> > >>>>>> mation on the front
> > >>>>>> page. Your mock looks really promising to me since it shows some
> > >>> basic
> > >>>>>> metrics and cluster information at a glance. Apart from the the
> > >>> source
> > >>>>>> input and sink output metrics, all other required information
> > >>> should be
> > >>>>>> available to display it in the dashboard. Thus, your proposal
> should
> > >>>> only
> > >>>>>> affect flink-runtime-web which should make it easier to realize.
> > >>>>>>
> > >>>>>> I'm in favour of adding this feature to Flink's dashboard to make
> it
> > >>>>>> available to the whole community.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> >
> >
>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by Fabian Wollert <fa...@zalando.de>.
sure, will do.

--


*Fabian WollertZalando SE*

E-Mail: fabian@zalando.de


Am Mo., 29. Okt. 2018 um 12:57 Uhr schrieb Chesnay Schepler <
chesnay@apache.org>:

> Please start the discussion in an entirely new thread; people may
> discard this thread immediately since the first page is purely about the
> layout of the WebUI.
>
> On 29.10.2018 12:39, Fabian Wollert wrote:
> > Hi again,
> >
> > Chesnay correctly commented in the tickets that we first should discuss
> > here, if changing the underlying technology for the Flink Web Dashboard
> is
> > a valid option at all. What are your thoughts about this?
> >
> > personally I agree with Till's comments in the ticket, Angular 1 being
> > basically outdated and is not having a large following anymore. From my
> > experience the choice between Angular 2-7 or React is subjective, you can
> > get things done with both. I personally only have experience with React,
> so
> > i personally would be faster to develop with this one. I currently have
> not
> > planned to learn Angular as well (being a more backend focused developer
> in
> > general) so if the decision would be to go with Angular, i would be
> > unfortunately out of this rework of the Flink Dashboard most certainly.
> >
> > Cheers
> > Fabian
> >
> > --
> >
> >
> > *Fabian WollertZalando SE*
> >
> > E-Mail: fabian@zalando.de
> >
> >
> > Am Mo., 29. Okt. 2018 um 09:21 Uhr schrieb Fabian Wollert <
> fabian@zalando.de
> >> :
> >> Hi everyone,
> >>
> >> thx for all the feedback. I created now
> >> https://issues.apache.org/jira/browse/FLINK-10705 with sub tickets to
> >> tackle this. i also found some time this weekend and implemented the
> first
> >> draft, which i will post in the ticket (not sure if i get the pictures
> to
> >> work here in the mailing list :-D).
> >>
> >> Lets continue discussion in the tickets then.
> >>
> >> Since this is my first bigger contribution to Flink, please advise on
> how
> >> to handle tickets, and structure the work. But for now i will just
> continue
> >> to work on this, whenever i find free time.
> >>
> >> Cheers
> >>
> >> --
> >>
> >>
> >> *Fabian WollertZalando SE*
> >>
> >> E-Mail: fabian@zalando.de
> >>
> >>
> >> Am Sa., 27. Okt. 2018 um 17:15 Uhr schrieb Robert Metzger <
> >> rmetzger@apache.org>:
> >>
> >>> Hey,
> >>> Sorry for the delay.
> >>>
> >>> Yes -- I would be open to revisit the underlying technologies.
> >>>
> >>> Best,
> >>> Robert
> >>>
> >>> On Wed, Oct 10, 2018 at 11:28 AM Fabian Wollert <fa...@zalando.de>
> >>> wrote:
> >>>
> >>>> Hi everyone, thx for all the comments and feedback. Let me address
> >>>> everything individually:
> >>>>
> >>>> @Till: yes, for the start my plan would be to just touch the
> >>>> flink-runtime-web/web-dashboard repo/folder.
> >>>>
> >>>> @Jin Sun:
> >>>>
> >>>>     - smaller icons on increasing server counts: yes, thats also
> >>> something i
> >>>>     already thought about. will keep it in mind when realizing the
> first
> >>>>     version!
> >>>>     - about locality: i searched quickly through the docs, but i could
> >>> not
> >>>>     find anything regarding flink featuring rack awareness. Is this
> >>>> something
> >>>>     already implemented? If not, i think this will bloat the size of
> this
> >>>>     initial proposal. If its somewhere already included, we could
> >>> implement
> >>>> it
> >>>>     for sure.
> >>>>
> >>>> @Zhijiang:the focus of this redesign was not yet including the job
> list
> >>> in
> >>>> the lower half of the overview. as part of the redesign we can also
> >>> think
> >>>> about optimising this list though, and removing unnecessary columns is
> >>>> usually the most easy thing to do. we can maybe create a separate
> ticket
> >>>> for this as well and discuss this issue there, to not bloat the
> initial
> >>>> discussion with too much topics.
> >>>>
> >>>> @Robert:
> >>>>
> >>>>     - Agreed that it might make sense to also show this on job level.
> >>> Since
> >>>>     these metrics are probably gonna be introduced later only
> anyways, we
> >>>> can
> >>>>     discuss this maybe then separately after FLINK-9050
> >>>>     <https://issues.apache.org/jira/browse/FLINK-9050> (linked the
> wrong
> >>>>     ticket in my initial mail) is done.
> >>>>     - Rewriting the whole thing while doing this also came to my mind.
> >>> What
> >>>>     i would like to do anyways (even if we stick for now to A1) is to
> >>> remove
> >>>>     bower as a package manager (since its deprecated) and update
> >>> bootstrap
> >>>> to
> >>>>     V4. I will check what the additional effort is to move to
> >>> React/Redux.
> >>>>     We're working with this here at work as well, so implementing at
> >>> least a
> >>>>     first MVP might be feasible as well, before getting to deep into
> A1
> >>>>     specifics. But that basically means that you guys are open to
> change
> >>> the
> >>>>     underlying web/JS technology, yeah?
> >>>>
> >>>> Cheers
> >>>>
> >>>> --
> >>>>
> >>>>
> >>>> *Fabian WollertZalando SE*
> >>>>
> >>>> E-Mail: fabian@zalando.de
> >>>>
> >>>>
> >>>> Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
> >>>> rmetzger@apache.org>:
> >>>>
> >>>>> Hey Fabian,
> >>>>> thanks a lot for reaching out to the Flink community with this
> >>> proposal!
> >>>>> (Posting to the ML instead of creating a JIRA is a good idea for such
> >>>>> questions -- you can create a ticket/tickets once the discussion here
> >>> has
> >>>>> come to a conclusion)
> >>>>>
> >>>>> I have two comments:
> >>>>> - You are listing Records/Kb in and Records/Kb out as cluster-wide
> >>>> metrics.
> >>>>> I wonder whether we should rather show these metrics for each job,
> >>>> instead
> >>>>> of the entire cluster? (or maybe both). My concern is that the
> >>>> cluster-wide
> >>>>> metric is not really relevant as soon as you have jobs with different
> >>>>> characteristics running on one cluster
> >>>>> - You mention that the Flink UI is based on Angular 1. I've been
> >>> thinking
> >>>>> for quite a while now whether we should actually rewrite / migrate
> the
> >>>>> Flink UI to React.
> >>>>> Do you think we can re-use most of the work you'd be doing for this
> >>>> change
> >>>>> when we migrate to React?
> >>>>>
> >>>>> Best,
> >>>>> Robert
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
> >>>>> <wa...@aliyun.com.invalid> wrote:
> >>>>>
> >>>>>> Thanks Fabian for proposing this topic.
> >>>>>>
> >>>>>> It is very worth improving the web dashborad for showing more useful
> >>>>>> informations which can benefit flink users a lot.
> >>>>>>
> >>>>>> Just two small personal concerns:
> >>>>>> 1. The start time and end time are already given, so it is easy to
> >>>>>> estimate the rough duration time. Is it necessary to show the
> >>> duration
> >>>>>> information to occupy the space?
> >>>>>> 2. The job name given by users can be used for identification, and
> >>> the
> >>>>>> job id is automatically generated in random. I am not sure whether
> >>> this
> >>>>> id
> >>>>>> is useful for further debugging. If not maybe we can ignore the job
> >>> id
> >>>>> from
> >>>>>> the dashboard?
> >>>>>>
> >>>>>> Best,
> >>>>>> Zhijiang
> >>>>>>
> >>>>>> ------------------------------------------------------------------
> >>>>>> 发件人：Jin Sun <is...@gmail.com>
> >>>>>> 发送时间：2018年10月10日(星期三) 01:10
> >>>>>> 收件人：dev <de...@flink.apache.org>
> >>>>>> 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement
> >>> Proposal
> >>>>>> Great job! That would very helpful for debug.
> >>>>>>
> >>>>>>
> >>>>>>     - I would suggest to use small icons for this Job
> >>> Manager/Managers
> >>>>>>     when there are too many instances (like a thousand)
> >>>>>>     - May be we can also introduce locality,  that task managers
> >>> belongs
> >>>>>>     to same rack shows together?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Small icons can be like this:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org>
> >>>> wrote:
> >>>>>> mation on the front
> >>>>>> page. Your mock looks really promising to me since it shows some
> >>> basic
> >>>>>> metrics and cluster information at a glance. Apart from the the
> >>> source
> >>>>>> input and sink output metrics, all other required information
> >>> should be
> >>>>>> available to display it in the dashboard. Thus, your proposal should
> >>>> only
> >>>>>> affect flink-runtime-web which should make it easier to realize.
> >>>>>>
> >>>>>> I'm in favour of adding this feature to Flink's dashboard to make it
> >>>>>> available to the whole community.
> >>>>>>
> >>>>>>
> >>>>>>
>
>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by Chesnay Schepler <ch...@apache.org>.
Please start the discussion in an entirely new thread; people may 
discard this thread immediately since the first page is purely about the 
layout of the WebUI.

On 29.10.2018 12:39, Fabian Wollert wrote:
> Hi again,
>
> Chesnay correctly commented in the tickets that we first should discuss
> here, if changing the underlying technology for the Flink Web Dashboard is
> a valid option at all. What are your thoughts about this?
>
> personally I agree with Till's comments in the ticket, Angular 1 being
> basically outdated and is not having a large following anymore. From my
> experience the choice between Angular 2-7 or React is subjective, you can
> get things done with both. I personally only have experience with React, so
> i personally would be faster to develop with this one. I currently have not
> planned to learn Angular as well (being a more backend focused developer in
> general) so if the decision would be to go with Angular, i would be
> unfortunately out of this rework of the Flink Dashboard most certainly.
>
> Cheers
> Fabian
>
> --
>
>
> *Fabian WollertZalando SE*
>
> E-Mail: fabian@zalando.de
>
>
> Am Mo., 29. Okt. 2018 um 09:21 Uhr schrieb Fabian Wollert <fabian@zalando.de
>> :
>> Hi everyone,
>>
>> thx for all the feedback. I created now
>> https://issues.apache.org/jira/browse/FLINK-10705 with sub tickets to
>> tackle this. i also found some time this weekend and implemented the first
>> draft, which i will post in the ticket (not sure if i get the pictures to
>> work here in the mailing list :-D).
>>
>> Lets continue discussion in the tickets then.
>>
>> Since this is my first bigger contribution to Flink, please advise on how
>> to handle tickets, and structure the work. But for now i will just continue
>> to work on this, whenever i find free time.
>>
>> Cheers
>>
>> --
>>
>>
>> *Fabian WollertZalando SE*
>>
>> E-Mail: fabian@zalando.de
>>
>>
>> Am Sa., 27. Okt. 2018 um 17:15 Uhr schrieb Robert Metzger <
>> rmetzger@apache.org>:
>>
>>> Hey,
>>> Sorry for the delay.
>>>
>>> Yes -- I would be open to revisit the underlying technologies.
>>>
>>> Best,
>>> Robert
>>>
>>> On Wed, Oct 10, 2018 at 11:28 AM Fabian Wollert <fa...@zalando.de>
>>> wrote:
>>>
>>>> Hi everyone, thx for all the comments and feedback. Let me address
>>>> everything individually:
>>>>
>>>> @Till: yes, for the start my plan would be to just touch the
>>>> flink-runtime-web/web-dashboard repo/folder.
>>>>
>>>> @Jin Sun:
>>>>
>>>>     - smaller icons on increasing server counts: yes, thats also
>>> something i
>>>>     already thought about. will keep it in mind when realizing the first
>>>>     version!
>>>>     - about locality: i searched quickly through the docs, but i could
>>> not
>>>>     find anything regarding flink featuring rack awareness. Is this
>>>> something
>>>>     already implemented? If not, i think this will bloat the size of this
>>>>     initial proposal. If its somewhere already included, we could
>>> implement
>>>> it
>>>>     for sure.
>>>>
>>>> @Zhijiang:the focus of this redesign was not yet including the job list
>>> in
>>>> the lower half of the overview. as part of the redesign we can also
>>> think
>>>> about optimising this list though, and removing unnecessary columns is
>>>> usually the most easy thing to do. we can maybe create a separate ticket
>>>> for this as well and discuss this issue there, to not bloat the initial
>>>> discussion with too much topics.
>>>>
>>>> @Robert:
>>>>
>>>>     - Agreed that it might make sense to also show this on job level.
>>> Since
>>>>     these metrics are probably gonna be introduced later only anyways, we
>>>> can
>>>>     discuss this maybe then separately after FLINK-9050
>>>>     <https://issues.apache.org/jira/browse/FLINK-9050> (linked the wrong
>>>>     ticket in my initial mail) is done.
>>>>     - Rewriting the whole thing while doing this also came to my mind.
>>> What
>>>>     i would like to do anyways (even if we stick for now to A1) is to
>>> remove
>>>>     bower as a package manager (since its deprecated) and update
>>> bootstrap
>>>> to
>>>>     V4. I will check what the additional effort is to move to
>>> React/Redux.
>>>>     We're working with this here at work as well, so implementing at
>>> least a
>>>>     first MVP might be feasible as well, before getting to deep into A1
>>>>     specifics. But that basically means that you guys are open to change
>>> the
>>>>     underlying web/JS technology, yeah?
>>>>
>>>> Cheers
>>>>
>>>> --
>>>>
>>>>
>>>> *Fabian WollertZalando SE*
>>>>
>>>> E-Mail: fabian@zalando.de
>>>>
>>>>
>>>> Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
>>>> rmetzger@apache.org>:
>>>>
>>>>> Hey Fabian,
>>>>> thanks a lot for reaching out to the Flink community with this
>>> proposal!
>>>>> (Posting to the ML instead of creating a JIRA is a good idea for such
>>>>> questions -- you can create a ticket/tickets once the discussion here
>>> has
>>>>> come to a conclusion)
>>>>>
>>>>> I have two comments:
>>>>> - You are listing Records/Kb in and Records/Kb out as cluster-wide
>>>> metrics.
>>>>> I wonder whether we should rather show these metrics for each job,
>>>> instead
>>>>> of the entire cluster? (or maybe both). My concern is that the
>>>> cluster-wide
>>>>> metric is not really relevant as soon as you have jobs with different
>>>>> characteristics running on one cluster
>>>>> - You mention that the Flink UI is based on Angular 1. I've been
>>> thinking
>>>>> for quite a while now whether we should actually rewrite / migrate the
>>>>> Flink UI to React.
>>>>> Do you think we can re-use most of the work you'd be doing for this
>>>> change
>>>>> when we migrate to React?
>>>>>
>>>>> Best,
>>>>> Robert
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
>>>>> <wa...@aliyun.com.invalid> wrote:
>>>>>
>>>>>> Thanks Fabian for proposing this topic.
>>>>>>
>>>>>> It is very worth improving the web dashborad for showing more useful
>>>>>> informations which can benefit flink users a lot.
>>>>>>
>>>>>> Just two small personal concerns:
>>>>>> 1. The start time and end time are already given, so it is easy to
>>>>>> estimate the rough duration time. Is it necessary to show the
>>> duration
>>>>>> information to occupy the space?
>>>>>> 2. The job name given by users can be used for identification, and
>>> the
>>>>>> job id is automatically generated in random. I am not sure whether
>>> this
>>>>> id
>>>>>> is useful for further debugging. If not maybe we can ignore the job
>>> id
>>>>> from
>>>>>> the dashboard?
>>>>>>
>>>>>> Best,
>>>>>> Zhijiang
>>>>>>
>>>>>> ------------------------------------------------------------------
>>>>>> 发件人：Jin Sun <is...@gmail.com>
>>>>>> 发送时间：2018年10月10日(星期三) 01:10
>>>>>> 收件人：dev <de...@flink.apache.org>
>>>>>> 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement
>>> Proposal
>>>>>> Great job! That would very helpful for debug.
>>>>>>
>>>>>>
>>>>>>     - I would suggest to use small icons for this Job
>>> Manager/Managers
>>>>>>     when there are too many instances (like a thousand)
>>>>>>     - May be we can also introduce locality,  that task managers
>>> belongs
>>>>>>     to same rack shows together?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Small icons can be like this:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org>
>>>> wrote:
>>>>>> mation on the front
>>>>>> page. Your mock looks really promising to me since it shows some
>>> basic
>>>>>> metrics and cluster information at a glance. Apart from the the
>>> source
>>>>>> input and sink output metrics, all other required information
>>> should be
>>>>>> available to display it in the dashboard. Thus, your proposal should
>>>> only
>>>>>> affect flink-runtime-web which should make it easier to realize.
>>>>>>
>>>>>> I'm in favour of adding this feature to Flink's dashboard to make it
>>>>>> available to the whole community.
>>>>>>
>>>>>>
>>>>>>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by Fabian Wollert <fa...@zalando.de>.
Hi again,

Chesnay correctly commented in the tickets that we first should discuss
here, if changing the underlying technology for the Flink Web Dashboard is
a valid option at all. What are your thoughts about this?

personally I agree with Till's comments in the ticket, Angular 1 being
basically outdated and is not having a large following anymore. From my
experience the choice between Angular 2-7 or React is subjective, you can
get things done with both. I personally only have experience with React, so
i personally would be faster to develop with this one. I currently have not
planned to learn Angular as well (being a more backend focused developer in
general) so if the decision would be to go with Angular, i would be
unfortunately out of this rework of the Flink Dashboard most certainly.

Cheers
Fabian

--


*Fabian WollertZalando SE*

E-Mail: fabian@zalando.de


Am Mo., 29. Okt. 2018 um 09:21 Uhr schrieb Fabian Wollert <fabian@zalando.de
>:

> Hi everyone,
>
> thx for all the feedback. I created now
> https://issues.apache.org/jira/browse/FLINK-10705 with sub tickets to
> tackle this. i also found some time this weekend and implemented the first
> draft, which i will post in the ticket (not sure if i get the pictures to
> work here in the mailing list :-D).
>
> Lets continue discussion in the tickets then.
>
> Since this is my first bigger contribution to Flink, please advise on how
> to handle tickets, and structure the work. But for now i will just continue
> to work on this, whenever i find free time.
>
> Cheers
>
> --
>
>
> *Fabian WollertZalando SE*
>
> E-Mail: fabian@zalando.de
>
>
> Am Sa., 27. Okt. 2018 um 17:15 Uhr schrieb Robert Metzger <
> rmetzger@apache.org>:
>
>> Hey,
>> Sorry for the delay.
>>
>> Yes -- I would be open to revisit the underlying technologies.
>>
>> Best,
>> Robert
>>
>> On Wed, Oct 10, 2018 at 11:28 AM Fabian Wollert <fa...@zalando.de>
>> wrote:
>>
>> > Hi everyone, thx for all the comments and feedback. Let me address
>> > everything individually:
>> >
>> > @Till: yes, for the start my plan would be to just touch the
>> > flink-runtime-web/web-dashboard repo/folder.
>> >
>> > @Jin Sun:
>> >
>> >    - smaller icons on increasing server counts: yes, thats also
>> something i
>> >    already thought about. will keep it in mind when realizing the first
>> >    version!
>> >    - about locality: i searched quickly through the docs, but i could
>> not
>> >    find anything regarding flink featuring rack awareness. Is this
>> > something
>> >    already implemented? If not, i think this will bloat the size of this
>> >    initial proposal. If its somewhere already included, we could
>> implement
>> > it
>> >    for sure.
>> >
>> > @Zhijiang:the focus of this redesign was not yet including the job list
>> in
>> > the lower half of the overview. as part of the redesign we can also
>> think
>> > about optimising this list though, and removing unnecessary columns is
>> > usually the most easy thing to do. we can maybe create a separate ticket
>> > for this as well and discuss this issue there, to not bloat the initial
>> > discussion with too much topics.
>> >
>> > @Robert:
>> >
>> >    - Agreed that it might make sense to also show this on job level.
>> Since
>> >    these metrics are probably gonna be introduced later only anyways, we
>> > can
>> >    discuss this maybe then separately after FLINK-9050
>> >    <https://issues.apache.org/jira/browse/FLINK-9050> (linked the wrong
>> >    ticket in my initial mail) is done.
>> >    - Rewriting the whole thing while doing this also came to my mind.
>> What
>> >    i would like to do anyways (even if we stick for now to A1) is to
>> remove
>> >    bower as a package manager (since its deprecated) and update
>> bootstrap
>> > to
>> >    V4. I will check what the additional effort is to move to
>> React/Redux.
>> >    We're working with this here at work as well, so implementing at
>> least a
>> >    first MVP might be feasible as well, before getting to deep into A1
>> >    specifics. But that basically means that you guys are open to change
>> the
>> >    underlying web/JS technology, yeah?
>> >
>> > Cheers
>> >
>> > --
>> >
>> >
>> > *Fabian WollertZalando SE*
>> >
>> > E-Mail: fabian@zalando.de
>> >
>> >
>> > Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
>> > rmetzger@apache.org>:
>> >
>> > > Hey Fabian,
>> > > thanks a lot for reaching out to the Flink community with this
>> proposal!
>> > > (Posting to the ML instead of creating a JIRA is a good idea for such
>> > > questions -- you can create a ticket/tickets once the discussion here
>> has
>> > > come to a conclusion)
>> > >
>> > > I have two comments:
>> > > - You are listing Records/Kb in and Records/Kb out as cluster-wide
>> > metrics.
>> > > I wonder whether we should rather show these metrics for each job,
>> > instead
>> > > of the entire cluster? (or maybe both). My concern is that the
>> > cluster-wide
>> > > metric is not really relevant as soon as you have jobs with different
>> > > characteristics running on one cluster
>> > > - You mention that the Flink UI is based on Angular 1. I've been
>> thinking
>> > > for quite a while now whether we should actually rewrite / migrate the
>> > > Flink UI to React.
>> > > Do you think we can re-use most of the work you'd be doing for this
>> > change
>> > > when we migrate to React?
>> > >
>> > > Best,
>> > > Robert
>> > >
>> > >
>> > >
>> > > On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
>> > > <wa...@aliyun.com.invalid> wrote:
>> > >
>> > > > Thanks Fabian for proposing this topic.
>> > > >
>> > > > It is very worth improving the web dashborad for showing more useful
>> > > > informations which can benefit flink users a lot.
>> > > >
>> > > > Just two small personal concerns:
>> > > > 1. The start time and end time are already given, so it is easy to
>> > > > estimate the rough duration time. Is it necessary to show the
>> duration
>> > > > information to occupy the space?
>> > > > 2. The job name given by users can be used for identification, and
>> the
>> > > > job id is automatically generated in random. I am not sure whether
>> this
>> > > id
>> > > > is useful for further debugging. If not maybe we can ignore the job
>> id
>> > > from
>> > > > the dashboard?
>> > > >
>> > > > Best,
>> > > > Zhijiang
>> > > >
>> > > > ------------------------------------------------------------------
>> > > > 发件人：Jin Sun <is...@gmail.com>
>> > > > 发送时间：2018年10月10日(星期三) 01:10
>> > > > 收件人：dev <de...@flink.apache.org>
>> > > > 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement
>> Proposal
>> > > >
>> > > > Great job! That would very helpful for debug.
>> > > >
>> > > >
>> > > >    - I would suggest to use small icons for this Job
>> Manager/Managers
>> > > >    when there are too many instances (like a thousand)
>> > > >    - May be we can also introduce locality,  that task managers
>> belongs
>> > > >    to same rack shows together?
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > Small icons can be like this:
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org>
>> > wrote:
>> > > >
>> > > > mation on the front
>> > > > page. Your mock looks really promising to me since it shows some
>> basic
>> > > > metrics and cluster information at a glance. Apart from the the
>> source
>> > > > input and sink output metrics, all other required information
>> should be
>> > > > available to display it in the dashboard. Thus, your proposal should
>> > only
>> > > > affect flink-runtime-web which should make it easier to realize.
>> > > >
>> > > > I'm in favour of adding this feature to Flink's dashboard to make it
>> > > > available to the whole community.
>> > > >
>> > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by Fabian Wollert <fa...@zalando.de>.
Hi everyone,

thx for all the feedback. I created now
https://issues.apache.org/jira/browse/FLINK-10705 with sub tickets to
tackle this. i also found some time this weekend and implemented the first
draft, which i will post in the ticket (not sure if i get the pictures to
work here in the mailing list :-D).

Lets continue discussion in the tickets then.

Since this is my first bigger contribution to Flink, please advise on how
to handle tickets, and structure the work. But for now i will just continue
to work on this, whenever i find free time.

Cheers

--


*Fabian WollertZalando SE*

E-Mail: fabian@zalando.de


Am Sa., 27. Okt. 2018 um 17:15 Uhr schrieb Robert Metzger <
rmetzger@apache.org>:

> Hey,
> Sorry for the delay.
>
> Yes -- I would be open to revisit the underlying technologies.
>
> Best,
> Robert
>
> On Wed, Oct 10, 2018 at 11:28 AM Fabian Wollert <fa...@zalando.de> wrote:
>
> > Hi everyone, thx for all the comments and feedback. Let me address
> > everything individually:
> >
> > @Till: yes, for the start my plan would be to just touch the
> > flink-runtime-web/web-dashboard repo/folder.
> >
> > @Jin Sun:
> >
> >    - smaller icons on increasing server counts: yes, thats also
> something i
> >    already thought about. will keep it in mind when realizing the first
> >    version!
> >    - about locality: i searched quickly through the docs, but i could not
> >    find anything regarding flink featuring rack awareness. Is this
> > something
> >    already implemented? If not, i think this will bloat the size of this
> >    initial proposal. If its somewhere already included, we could
> implement
> > it
> >    for sure.
> >
> > @Zhijiang:the focus of this redesign was not yet including the job list
> in
> > the lower half of the overview. as part of the redesign we can also think
> > about optimising this list though, and removing unnecessary columns is
> > usually the most easy thing to do. we can maybe create a separate ticket
> > for this as well and discuss this issue there, to not bloat the initial
> > discussion with too much topics.
> >
> > @Robert:
> >
> >    - Agreed that it might make sense to also show this on job level.
> Since
> >    these metrics are probably gonna be introduced later only anyways, we
> > can
> >    discuss this maybe then separately after FLINK-9050
> >    <https://issues.apache.org/jira/browse/FLINK-9050> (linked the wrong
> >    ticket in my initial mail) is done.
> >    - Rewriting the whole thing while doing this also came to my mind.
> What
> >    i would like to do anyways (even if we stick for now to A1) is to
> remove
> >    bower as a package manager (since its deprecated) and update bootstrap
> > to
> >    V4. I will check what the additional effort is to move to React/Redux.
> >    We're working with this here at work as well, so implementing at
> least a
> >    first MVP might be feasible as well, before getting to deep into A1
> >    specifics. But that basically means that you guys are open to change
> the
> >    underlying web/JS technology, yeah?
> >
> > Cheers
> >
> > --
> >
> >
> > *Fabian WollertZalando SE*
> >
> > E-Mail: fabian@zalando.de
> >
> >
> > Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
> > rmetzger@apache.org>:
> >
> > > Hey Fabian,
> > > thanks a lot for reaching out to the Flink community with this
> proposal!
> > > (Posting to the ML instead of creating a JIRA is a good idea for such
> > > questions -- you can create a ticket/tickets once the discussion here
> has
> > > come to a conclusion)
> > >
> > > I have two comments:
> > > - You are listing Records/Kb in and Records/Kb out as cluster-wide
> > metrics.
> > > I wonder whether we should rather show these metrics for each job,
> > instead
> > > of the entire cluster? (or maybe both). My concern is that the
> > cluster-wide
> > > metric is not really relevant as soon as you have jobs with different
> > > characteristics running on one cluster
> > > - You mention that the Flink UI is based on Angular 1. I've been
> thinking
> > > for quite a while now whether we should actually rewrite / migrate the
> > > Flink UI to React.
> > > Do you think we can re-use most of the work you'd be doing for this
> > change
> > > when we migrate to React?
> > >
> > > Best,
> > > Robert
> > >
> > >
> > >
> > > On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
> > > <wa...@aliyun.com.invalid> wrote:
> > >
> > > > Thanks Fabian for proposing this topic.
> > > >
> > > > It is very worth improving the web dashborad for showing more useful
> > > > informations which can benefit flink users a lot.
> > > >
> > > > Just two small personal concerns:
> > > > 1. The start time and end time are already given, so it is easy to
> > > > estimate the rough duration time. Is it necessary to show the
> duration
> > > > information to occupy the space?
> > > > 2. The job name given by users can be used for identification, and
> the
> > > > job id is automatically generated in random. I am not sure whether
> this
> > > id
> > > > is useful for further debugging. If not maybe we can ignore the job
> id
> > > from
> > > > the dashboard?
> > > >
> > > > Best,
> > > > Zhijiang
> > > >
> > > > ------------------------------------------------------------------
> > > > 发件人：Jin Sun <is...@gmail.com>
> > > > 发送时间：2018年10月10日(星期三) 01:10
> > > > 收件人：dev <de...@flink.apache.org>
> > > > 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement
> Proposal
> > > >
> > > > Great job! That would very helpful for debug.
> > > >
> > > >
> > > >    - I would suggest to use small icons for this Job Manager/Managers
> > > >    when there are too many instances (like a thousand)
> > > >    - May be we can also introduce locality,  that task managers
> belongs
> > > >    to same rack shows together?
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Small icons can be like this:
> > > >
> > > >
> > > >
> > > >
> > > > On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org>
> > wrote:
> > > >
> > > > mation on the front
> > > > page. Your mock looks really promising to me since it shows some
> basic
> > > > metrics and cluster information at a glance. Apart from the the
> source
> > > > input and sink output metrics, all other required information should
> be
> > > > available to display it in the dashboard. Thus, your proposal should
> > only
> > > > affect flink-runtime-web which should make it easier to realize.
> > > >
> > > > I'm in favour of adding this feature to Flink's dashboard to make it
> > > > available to the whole community.
> > > >
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by Robert Metzger <rm...@apache.org>.
Hey,
Sorry for the delay.

Yes -- I would be open to revisit the underlying technologies.

Best,
Robert

On Wed, Oct 10, 2018 at 11:28 AM Fabian Wollert <fa...@zalando.de> wrote:

> Hi everyone, thx for all the comments and feedback. Let me address
> everything individually:
>
> @Till: yes, for the start my plan would be to just touch the
> flink-runtime-web/web-dashboard repo/folder.
>
> @Jin Sun:
>
>    - smaller icons on increasing server counts: yes, thats also something i
>    already thought about. will keep it in mind when realizing the first
>    version!
>    - about locality: i searched quickly through the docs, but i could not
>    find anything regarding flink featuring rack awareness. Is this
> something
>    already implemented? If not, i think this will bloat the size of this
>    initial proposal. If its somewhere already included, we could implement
> it
>    for sure.
>
> @Zhijiang:the focus of this redesign was not yet including the job list in
> the lower half of the overview. as part of the redesign we can also think
> about optimising this list though, and removing unnecessary columns is
> usually the most easy thing to do. we can maybe create a separate ticket
> for this as well and discuss this issue there, to not bloat the initial
> discussion with too much topics.
>
> @Robert:
>
>    - Agreed that it might make sense to also show this on job level. Since
>    these metrics are probably gonna be introduced later only anyways, we
> can
>    discuss this maybe then separately after FLINK-9050
>    <https://issues.apache.org/jira/browse/FLINK-9050> (linked the wrong
>    ticket in my initial mail) is done.
>    - Rewriting the whole thing while doing this also came to my mind. What
>    i would like to do anyways (even if we stick for now to A1) is to remove
>    bower as a package manager (since its deprecated) and update bootstrap
> to
>    V4. I will check what the additional effort is to move to React/Redux.
>    We're working with this here at work as well, so implementing at least a
>    first MVP might be feasible as well, before getting to deep into A1
>    specifics. But that basically means that you guys are open to change the
>    underlying web/JS technology, yeah?
>
> Cheers
>
> --
>
>
> *Fabian WollertZalando SE*
>
> E-Mail: fabian@zalando.de
>
>
> Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
> rmetzger@apache.org>:
>
> > Hey Fabian,
> > thanks a lot for reaching out to the Flink community with this proposal!
> > (Posting to the ML instead of creating a JIRA is a good idea for such
> > questions -- you can create a ticket/tickets once the discussion here has
> > come to a conclusion)
> >
> > I have two comments:
> > - You are listing Records/Kb in and Records/Kb out as cluster-wide
> metrics.
> > I wonder whether we should rather show these metrics for each job,
> instead
> > of the entire cluster? (or maybe both). My concern is that the
> cluster-wide
> > metric is not really relevant as soon as you have jobs with different
> > characteristics running on one cluster
> > - You mention that the Flink UI is based on Angular 1. I've been thinking
> > for quite a while now whether we should actually rewrite / migrate the
> > Flink UI to React.
> > Do you think we can re-use most of the work you'd be doing for this
> change
> > when we migrate to React?
> >
> > Best,
> > Robert
> >
> >
> >
> > On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
> > <wa...@aliyun.com.invalid> wrote:
> >
> > > Thanks Fabian for proposing this topic.
> > >
> > > It is very worth improving the web dashborad for showing more useful
> > > informations which can benefit flink users a lot.
> > >
> > > Just two small personal concerns:
> > > 1. The start time and end time are already given, so it is easy to
> > > estimate the rough duration time. Is it necessary to show the duration
> > > information to occupy the space?
> > > 2. The job name given by users can be used for identification, and the
> > > job id is automatically generated in random. I am not sure whether this
> > id
> > > is useful for further debugging. If not maybe we can ignore the job id
> > from
> > > the dashboard?
> > >
> > > Best,
> > > Zhijiang
> > >
> > > ------------------------------------------------------------------
> > > 发件人：Jin Sun <is...@gmail.com>
> > > 发送时间：2018年10月10日(星期三) 01:10
> > > 收件人：dev <de...@flink.apache.org>
> > > 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal
> > >
> > > Great job! That would very helpful for debug.
> > >
> > >
> > >    - I would suggest to use small icons for this Job Manager/Managers
> > >    when there are too many instances (like a thousand)
> > >    - May be we can also introduce locality,  that task managers belongs
> > >    to same rack shows together?
> > >
> > >
> > >
> > >
> > >
> > > Small icons can be like this:
> > >
> > >
> > >
> > >
> > > On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org>
> wrote:
> > >
> > > mation on the front
> > > page. Your mock looks really promising to me since it shows some basic
> > > metrics and cluster information at a glance. Apart from the the source
> > > input and sink output metrics, all other required information should be
> > > available to display it in the dashboard. Thus, your proposal should
> only
> > > affect flink-runtime-web which should make it easier to realize.
> > >
> > > I'm in favour of adding this feature to Flink's dashboard to make it
> > > available to the whole community.
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by Fabian Wollert <fa...@zalando.de>.
Hi everyone, thx for all the comments and feedback. Let me address
everything individually:

@Till: yes, for the start my plan would be to just touch the
flink-runtime-web/web-dashboard repo/folder.

@Jin Sun:

   - smaller icons on increasing server counts: yes, thats also something i
   already thought about. will keep it in mind when realizing the first
   version!
   - about locality: i searched quickly through the docs, but i could not
   find anything regarding flink featuring rack awareness. Is this something
   already implemented? If not, i think this will bloat the size of this
   initial proposal. If its somewhere already included, we could implement it
   for sure.

@Zhijiang:the focus of this redesign was not yet including the job list in
the lower half of the overview. as part of the redesign we can also think
about optimising this list though, and removing unnecessary columns is
usually the most easy thing to do. we can maybe create a separate ticket
for this as well and discuss this issue there, to not bloat the initial
discussion with too much topics.

@Robert:

   - Agreed that it might make sense to also show this on job level. Since
   these metrics are probably gonna be introduced later only anyways, we can
   discuss this maybe then separately after FLINK-9050
   <https://issues.apache.org/jira/browse/FLINK-9050> (linked the wrong
   ticket in my initial mail) is done.
   - Rewriting the whole thing while doing this also came to my mind. What
   i would like to do anyways (even if we stick for now to A1) is to remove
   bower as a package manager (since its deprecated) and update bootstrap to
   V4. I will check what the additional effort is to move to React/Redux.
   We're working with this here at work as well, so implementing at least a
   first MVP might be feasible as well, before getting to deep into A1
   specifics. But that basically means that you guys are open to change the
   underlying web/JS technology, yeah?

Cheers

--


*Fabian WollertZalando SE*

E-Mail: fabian@zalando.de


Am Mi., 10. Okt. 2018 um 08:41 Uhr schrieb Robert Metzger <
rmetzger@apache.org>:

> Hey Fabian,
> thanks a lot for reaching out to the Flink community with this proposal!
> (Posting to the ML instead of creating a JIRA is a good idea for such
> questions -- you can create a ticket/tickets once the discussion here has
> come to a conclusion)
>
> I have two comments:
> - You are listing Records/Kb in and Records/Kb out as cluster-wide metrics.
> I wonder whether we should rather show these metrics for each job, instead
> of the entire cluster? (or maybe both). My concern is that the cluster-wide
> metric is not really relevant as soon as you have jobs with different
> characteristics running on one cluster
> - You mention that the Flink UI is based on Angular 1. I've been thinking
> for quite a while now whether we should actually rewrite / migrate the
> Flink UI to React.
> Do you think we can re-use most of the work you'd be doing for this change
> when we migrate to React?
>
> Best,
> Robert
>
>
>
> On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
> <wa...@aliyun.com.invalid> wrote:
>
> > Thanks Fabian for proposing this topic.
> >
> > It is very worth improving the web dashborad for showing more useful
> > informations which can benefit flink users a lot.
> >
> > Just two small personal concerns:
> > 1. The start time and end time are already given, so it is easy to
> > estimate the rough duration time. Is it necessary to show the duration
> > information to occupy the space?
> > 2. The job name given by users can be used for identification, and the
> > job id is automatically generated in random. I am not sure whether this
> id
> > is useful for further debugging. If not maybe we can ignore the job id
> from
> > the dashboard?
> >
> > Best,
> > Zhijiang
> >
> > ------------------------------------------------------------------
> > 发件人：Jin Sun <is...@gmail.com>
> > 发送时间：2018年10月10日(星期三) 01:10
> > 收件人：dev <de...@flink.apache.org>
> > 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal
> >
> > Great job! That would very helpful for debug.
> >
> >
> >    - I would suggest to use small icons for this Job Manager/Managers
> >    when there are too many instances (like a thousand)
> >    - May be we can also introduce locality,  that task managers belongs
> >    to same rack shows together?
> >
> >
> >
> >
> >
> > Small icons can be like this:
> >
> >
> >
> >
> > On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org> wrote:
> >
> > mation on the front
> > page. Your mock looks really promising to me since it shows some basic
> > metrics and cluster information at a glance. Apart from the the source
> > input and sink output metrics, all other required information should be
> > available to display it in the dashboard. Thus, your proposal should only
> > affect flink-runtime-web which should make it easier to realize.
> >
> > I'm in favour of adding this feature to Flink's dashboard to make it
> > available to the whole community.
> >
> >
> >
>

Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal

Posted by Robert Metzger <rm...@apache.org>.
Hey Fabian,
thanks a lot for reaching out to the Flink community with this proposal!
(Posting to the ML instead of creating a JIRA is a good idea for such
questions -- you can create a ticket/tickets once the discussion here has
come to a conclusion)

I have two comments:
- You are listing Records/Kb in and Records/Kb out as cluster-wide metrics.
I wonder whether we should rather show these metrics for each job, instead
of the entire cluster? (or maybe both). My concern is that the cluster-wide
metric is not really relevant as soon as you have jobs with different
characteristics running on one cluster
- You mention that the Flink UI is based on Angular 1. I've been thinking
for quite a while now whether we should actually rewrite / migrate the
Flink UI to React.
Do you think we can re-use most of the work you'd be doing for this change
when we migrate to React?

Best,
Robert



On Wed, Oct 10, 2018 at 8:24 AM Zhijiang(wangzhijiang999)
<wa...@aliyun.com.invalid> wrote:

> Thanks Fabian for proposing this topic.
>
> It is very worth improving the web dashborad for showing more useful
> informations which can benefit flink users a lot.
>
> Just two small personal concerns:
> 1. The start time and end time are already given, so it is easy to
> estimate the rough duration time. Is it necessary to show the duration
> information to occupy the space?
> 2. The job name given by users can be used for identification, and the
> job id is automatically generated in random. I am not sure whether this id
> is useful for further debugging. If not maybe we can ignore the job id from
> the dashboard?
>
> Best,
> Zhijiang
>
> ------------------------------------------------------------------
> 发件人：Jin Sun <is...@gmail.com>
> 发送时间：2018年10月10日(星期三) 01:10
> 收件人：dev <de...@flink.apache.org>
> 主 题：Re: [DISCUSS] Flink Cluster Overview Dashboard Improvement Proposal
>
> Great job! That would very helpful for debug.
>
>
>    - I would suggest to use small icons for this Job Manager/Managers
>    when there are too many instances (like a thousand)
>    - May be we can also introduce locality,  that task managers belongs
>    to same rack shows together?
>
>
>
>
>
> Small icons can be like this:
>
>
>
>
> On Oct 9, 2018, at 8:49 PM, Till Rohrmann <tr...@apache.org> wrote:
>
> mation on the front
> page. Your mock looks really promising to me since it shows some basic
> metrics and cluster information at a glance. Apart from the the source
> input and sink output metrics, all other required information should be
> available to display it in the dashboard. Thus, your proposal should only
> affect flink-runtime-web which should make it easier to realize.
>
> I'm in favour of adding this feature to Flink's dashboard to make it
> available to the whole community.
>
>
>