You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by CalvinKirs <ac...@163.com> on 2021/06/18 04:23:42 UTC
[DISCUSS]Remove zk monitoring
Hi guys:
Currently, since the registry is transformed into a plug-in architecture, each plug-in has to implement the corresponding extension interface, but for monitoring, we can't guarantee that each plug-in can provide the same monitoring metrics.
In addition, the monitoring of the registry is not part of Dolphin's management in my opinion.
So I suggest to remove the registry monitoring, what do you think about it.
Best Wishes！
CalvinKirs, Apache DolphinScheduler PMC
Re: [DISCUSS]Remove zk monitoring
Posted by Lidong Dai <da...@gmail.com>.
yes, more detailed metrics of DolphinScheduler, including task running
status, will also be integrated with Skywalking.
there will be a large number of metrics that need to be monitored, I think
Best Regards
---------------
DolphinScheduler PMC
Lidong Dai
lidongdai@apache.org
---------------
On Fri, Jun 18, 2021 at 10:49 PM Sheng Wu <wu...@gmail.com> wrote:
> Try to integrate with 3rd party system. like SkyWalking contributed
> dolphinscheduler server monitoring, you could set up zookeeper
> monitoring(not tracing) by using SkyWalking directly.
> If you need any help from the SkyWalking community, ping us on GitHub issue
> or mail list.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Sheng Wu <wu...@gmail.com> 于2021年6月18日周五 下午10:47写道：
>
> > This isn't a thing you could take care of by yourself.
> > A project like SkyWalking has provided consistent metrics analysis and
> > visualization solutions for a long time.
> >
> > Don't recommend to re-implement the scheduling unrelated thing.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > Lidong Dai <da...@gmail.com> 于2021年6月18日周五 下午10:39写道：
> >
> >> hi,
> >> Monitoring is very important and necessary , how about providing a
> choice,
> >> in the front end can show different monitor metrics depending on the
> >> back-end configuration?
> >>
> >>
> >>
> >> Best Regards
> >> ---------------
> >> DolphinScheduler PMC
> >> Lidong Dai
> >> lidongdai@apache.org
> >> ---------------
> >>
> >>
> >> On Fri, Jun 18, 2021 at 12:23 PM CalvinKirs <ac...@163.com> wrote:
> >>
> >> > Hi guys:
> >> >
> >> >
> >> > Currently, since the registry is transformed into a plug-in
> >> architecture,
> >> > each plug-in has to implement the corresponding extension interface,
> but
> >> > for monitoring, we can't guarantee that each plug-in can provide the
> >> same
> >> > monitoring metrics.
> >> > In addition, the monitoring of the registry is not part of Dolphin's
> >> > management in my opinion.
> >> >
> >> >
> >> > So I suggest to remove the registry monitoring, what do you think
> about
> >> it.
> >> >
> >> >
> >> > Best Wishes！
> >> > CalvinKirs, Apache DolphinScheduler PMC
> >> >
> >> >
> >>
> >
>
Re: [DISCUSS]Remove zk monitoring
Posted by Sheng Wu <wu...@gmail.com>.
Try to integrate with 3rd party system. like SkyWalking contributed
dolphinscheduler server monitoring, you could set up zookeeper
monitoring(not tracing) by using SkyWalking directly.
If you need any help from the SkyWalking community, ping us on GitHub issue
or mail list.
Sheng Wu 吴晟
Twitter, wusheng1108
Sheng Wu <wu...@gmail.com> 于2021年6月18日周五 下午10:47写道：
> This isn't a thing you could take care of by yourself.
> A project like SkyWalking has provided consistent metrics analysis and
> visualization solutions for a long time.
>
> Don't recommend to re-implement the scheduling unrelated thing.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Lidong Dai <da...@gmail.com> 于2021年6月18日周五 下午10:39写道：
>
>> hi,
>> Monitoring is very important and necessary , how about providing a choice,
>> in the front end can show different monitor metrics depending on the
>> back-end configuration?
>>
>>
>>
>> Best Regards
>> ---------------
>> DolphinScheduler PMC
>> Lidong Dai
>> lidongdai@apache.org
>> ---------------
>>
>>
>> On Fri, Jun 18, 2021 at 12:23 PM CalvinKirs <ac...@163.com> wrote:
>>
>> > Hi guys:
>> >
>> >
>> > Currently, since the registry is transformed into a plug-in
>> architecture,
>> > each plug-in has to implement the corresponding extension interface, but
>> > for monitoring, we can't guarantee that each plug-in can provide the
>> same
>> > monitoring metrics.
>> > In addition, the monitoring of the registry is not part of Dolphin's
>> > management in my opinion.
>> >
>> >
>> > So I suggest to remove the registry monitoring, what do you think about
>> it.
>> >
>> >
>> > Best Wishes！
>> > CalvinKirs, Apache DolphinScheduler PMC
>> >
>> >
>>
>
Re: [DISCUSS]Remove zk monitoring
Posted by CalvinKirs <ac...@163.com>.
Cc
Best Wishes！
CalvinKirs, Apache DolphinScheduler PMC
On 06/18/2021 23:52，CalvinKirs<ac...@163.com> wrote：
Dear Sheng:
Sorry, thank you very much for your reminder.
What I mean is that system monitoring functions like DB or registry should be given to APM system like SkyWalking to monitor and manage, not because DS uses these external components, so DS should also implement monitoring of them.
In addition, for the monitoring of DS itself, we have already implemented the basic metrics monitoring by interfacing with SkyWalking in 1.3.6. If we need more perfect metrics monitoring, we can invite two community contributors to complete it together.
Back to the original point, I strongly agree with your statement: Don't recommend to re-implement the scheduling unrelated thing.
Best Wishes！
CalvinKirs, Apache DolphinScheduler PMC
On 06/18/2021 23:19，Sheng Wu<wu...@gmail.com> wrote：
Hi Kirs
You accidentally reply to my personal mail, rather than the mail list.
Whether DS will provide full-stack observability/monitoring capabilities
relies on community decision, but `professional APM` seems a very strange
term.
Metrics and performance will impact DS's end users, so, the community
should make a call.
Last distributed tracing and monitoring are pure contributed by SkyWalking
committer accidentally, we should rely on the next accidental contribution.
From a tech perspective speaking, any service with Prometheus endpoint or
OpenTelemetry integration, SkyWalking could adapt, and you don't have to
suffer Grafana's GPL concerns.
Sheng Wu 吴晟
Twitter, wusheng1108
CalvinKirs <ac...@163.com> 于2021年6月18日周五 下午11:09写道：
Dear Sheng:
Deeply thanks for your reply.
Yes, I totally agree with you,Monitoring is important. But the monitoring
of those external components should be left to a professional APM system to
be managed centrally. We only need to focus on our functions.
Kirs
邮箱：acm_master@163.com
<https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=Kirs&uid=acm_master%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22%E9%82%AE%E7%AE%B1%EF%BC%9Aacm_master%40163.com%22%5D>
---- Replied Message ----
From Sheng Wu<wu...@gmail.com> <wu...@gmail.com>
Date 06/18/2021 22:47
To dev <de...@dolphinscheduler.apache.org> <de...@dolphinscheduler.apache.org>
Subject Re: [DISCUSS]Remove zk monitoring
This isn't a thing you could take care of by yourself.
A project like SkyWalking has provided consistent metrics analysis and
visualization solutions for a long time.
Don't recommend to re-implement the scheduling unrelated thing.
Sheng Wu 吴晟
Twitter, wusheng1108
Lidong Dai <da...@gmail.com> 于2021年6月18日周五 下午10:39写道：
hi,
Monitoring is very important and necessary , how about providing a
choice,
in the front end can show different monitor metrics depending on the
back-end configuration?
Best Regards
---------------
DolphinScheduler PMC
Lidong Dai
lidongdai@apache.org
---------------
On Fri, Jun 18, 2021 at 12:23 PM CalvinKirs <ac...@163.com> wrote:
Hi guys:
Currently, since the registry is transformed into a plug-in
architecture,
each plug-in has to implement the corresponding extension interface,
but
for monitoring, we can't guarantee that each plug-in can provide the
same
monitoring metrics.
In addition, the monitoring of the registry is not part of Dolphin's
management in my opinion.
So I suggest to remove the registry monitoring, what do you think
about
it.
Best Wishes！
CalvinKirs, Apache DolphinScheduler PMC
Re: [DISCUSS]Remove zk monitoring
Posted by CalvinKirs <ac...@163.com>.
Dear Sheng:
Sorry, thank you very much for your reminder.
What I mean is that system monitoring functions like DB or registry should be given to APM system like SkyWalking to monitor and manage, not because DS uses these external components, so DS should also implement monitoring of them.
In addition, for the monitoring of DS itself, we have already implemented the basic metrics monitoring by interfacing with SkyWalking in 1.3.6. If we need more perfect metrics monitoring, we can invite two community contributors to complete it together.
Back to the original point, I strongly agree with your statement: Don't recommend to re-implement the scheduling unrelated thing.
Best Wishes！
CalvinKirs, Apache DolphinScheduler PMC
On 06/18/2021 23:19，Sheng Wu<wu...@gmail.com> wrote：
Hi Kirs
You accidentally reply to my personal mail, rather than the mail list.
Whether DS will provide full-stack observability/monitoring capabilities
relies on community decision, but `professional APM` seems a very strange
term.
Metrics and performance will impact DS's end users, so, the community
should make a call.
Last distributed tracing and monitoring are pure contributed by SkyWalking
committer accidentally, we should rely on the next accidental contribution.
From a tech perspective speaking, any service with Prometheus endpoint or
OpenTelemetry integration, SkyWalking could adapt, and you don't have to
suffer Grafana's GPL concerns.
Sheng Wu 吴晟
Twitter, wusheng1108
CalvinKirs <ac...@163.com> 于2021年6月18日周五 下午11:09写道：
Dear Sheng:
Deeply thanks for your reply.
Yes, I totally agree with you,Monitoring is important. But the monitoring
of those external components should be left to a professional APM system to
be managed centrally. We only need to focus on our functions.
Kirs
邮箱：acm_master@163.com
<https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=Kirs&uid=acm_master%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22%E9%82%AE%E7%AE%B1%EF%BC%9Aacm_master%40163.com%22%5D>
---- Replied Message ----
From Sheng Wu<wu...@gmail.com> <wu...@gmail.com>
Date 06/18/2021 22:47
To dev <de...@dolphinscheduler.apache.org> <de...@dolphinscheduler.apache.org>
Subject Re: [DISCUSS]Remove zk monitoring
This isn't a thing you could take care of by yourself.
A project like SkyWalking has provided consistent metrics analysis and
visualization solutions for a long time.
Don't recommend to re-implement the scheduling unrelated thing.
Sheng Wu 吴晟
Twitter, wusheng1108
Lidong Dai <da...@gmail.com> 于2021年6月18日周五 下午10:39写道：
hi,
Monitoring is very important and necessary , how about providing a
choice,
in the front end can show different monitor metrics depending on the
back-end configuration?
Best Regards
---------------
DolphinScheduler PMC
Lidong Dai
lidongdai@apache.org
---------------
On Fri, Jun 18, 2021 at 12:23 PM CalvinKirs <ac...@163.com> wrote:
Hi guys:
Currently, since the registry is transformed into a plug-in
architecture,
each plug-in has to implement the corresponding extension interface,
but
for monitoring, we can't guarantee that each plug-in can provide the
same
monitoring metrics.
In addition, the monitoring of the registry is not part of Dolphin's
management in my opinion.
So I suggest to remove the registry monitoring, what do you think
about
it.
Best Wishes！
CalvinKirs, Apache DolphinScheduler PMC
Re: [DISCUSS]Remove zk monitoring
Posted by Sheng Wu <wu...@gmail.com>.
Hi Kirs
You accidentally reply to my personal mail, rather than the mail list.
Whether DS will provide full-stack observability/monitoring capabilities
relies on community decision, but `professional APM` seems a very strange
term.
Metrics and performance will impact DS's end users, so, the community
should make a call.
Last distributed tracing and monitoring are pure contributed by SkyWalking
committer accidentally, we should rely on the next accidental contribution.
From a tech perspective speaking, any service with Prometheus endpoint or
OpenTelemetry integration, SkyWalking could adapt, and you don't have to
suffer Grafana's GPL concerns.
Sheng Wu 吴晟
Twitter, wusheng1108
CalvinKirs <ac...@163.com> 于2021年6月18日周五 下午11:09写道：
> Dear Sheng:
> Deeply thanks for your reply.
> Yes, I totally agree with you,Monitoring is important. But the monitoring
> of those external components should be left to a professional APM system to
> be managed centrally. We only need to focus on our functions.
> Kirs
> 邮箱：acm_master@163.com
>
> <https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=Kirs&uid=acm_master%40163.com&iconUrl=https%3A%2F%2Fmail-online.nosdn.127.net%2Fqiyelogo%2FdefaultAvatar.png&items=%5B%22%E9%82%AE%E7%AE%B1%EF%BC%9Aacm_master%40163.com%22%5D>
> ---- Replied Message ----
> From Sheng Wu<wu...@gmail.com> <wu...@gmail.com>
> Date 06/18/2021 22:47
> To dev <de...@dolphinscheduler.apache.org> <de...@dolphinscheduler.apache.org>
> Subject Re: [DISCUSS]Remove zk monitoring
> This isn't a thing you could take care of by yourself.
> A project like SkyWalking has provided consistent metrics analysis and
> visualization solutions for a long time.
>
> Don't recommend to re-implement the scheduling unrelated thing.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> Lidong Dai <da...@gmail.com> 于2021年6月18日周五 下午10:39写道：
>
> > hi,
> > Monitoring is very important and necessary , how about providing a
> choice,
> > in the front end can show different monitor metrics depending on the
> > back-end configuration?
> >
> >
> >
> > Best Regards
> > ---------------
> > DolphinScheduler PMC
> > Lidong Dai
> > lidongdai@apache.org
> > ---------------
> >
> >
> > On Fri, Jun 18, 2021 at 12:23 PM CalvinKirs <ac...@163.com> wrote:
> >
> > > Hi guys:
> > >
> > >
> > > Currently, since the registry is transformed into a plug-in
> architecture,
> > > each plug-in has to implement the corresponding extension interface,
> but
> > > for monitoring, we can't guarantee that each plug-in can provide the
> same
> > > monitoring metrics.
> > > In addition, the monitoring of the registry is not part of Dolphin's
> > > management in my opinion.
> > >
> > >
> > > So I suggest to remove the registry monitoring, what do you think
> about
> > it.
> > >
> > >
> > > Best Wishes！
> > > CalvinKirs, Apache DolphinScheduler PMC
> > >
> > >
> >
>
Re: [DISCUSS]Remove zk monitoring
Posted by Sheng Wu <wu...@gmail.com>.
This isn't a thing you could take care of by yourself.
A project like SkyWalking has provided consistent metrics analysis and
visualization solutions for a long time.
Don't recommend to re-implement the scheduling unrelated thing.
Sheng Wu 吴晟
Twitter, wusheng1108
Lidong Dai <da...@gmail.com> 于2021年6月18日周五 下午10:39写道：
> hi,
> Monitoring is very important and necessary , how about providing a choice,
> in the front end can show different monitor metrics depending on the
> back-end configuration?
>
>
>
> Best Regards
> ---------------
> DolphinScheduler PMC
> Lidong Dai
> lidongdai@apache.org
> ---------------
>
>
> On Fri, Jun 18, 2021 at 12:23 PM CalvinKirs <ac...@163.com> wrote:
>
> > Hi guys:
> >
> >
> > Currently, since the registry is transformed into a plug-in architecture,
> > each plug-in has to implement the corresponding extension interface, but
> > for monitoring, we can't guarantee that each plug-in can provide the same
> > monitoring metrics.
> > In addition, the monitoring of the registry is not part of Dolphin's
> > management in my opinion.
> >
> >
> > So I suggest to remove the registry monitoring, what do you think about
> it.
> >
> >
> > Best Wishes！
> > CalvinKirs, Apache DolphinScheduler PMC
> >
> >
>
Re: [DISCUSS]Remove zk monitoring
Posted by Lidong Dai <da...@gmail.com>.
hi,
Monitoring is very important and necessary , how about providing a choice,
in the front end can show different monitor metrics depending on the
back-end configuration?
Best Regards
---------------
DolphinScheduler PMC
Lidong Dai
lidongdai@apache.org
---------------
On Fri, Jun 18, 2021 at 12:23 PM CalvinKirs <ac...@163.com> wrote:
> Hi guys:
>
>
> Currently, since the registry is transformed into a plug-in architecture,
> each plug-in has to implement the corresponding extension interface, but
> for monitoring, we can't guarantee that each plug-in can provide the same
> monitoring metrics.
> In addition, the monitoring of the registry is not part of Dolphin's
> management in my opinion.
>
>
> So I suggest to remove the registry monitoring, what do you think about it.
>
>
> Best Wishes！
> CalvinKirs, Apache DolphinScheduler PMC
>
>