You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@shardingsphere.apache.org by "zhangliang@apache.org" <zh...@apache.org> on 2020/06/12 08:23:38 UTC
[ElasticJob] Discuss new features of ElasticJob 3.0
Hi all,
We are beginning to develop ElasticJob, and want to add some new features
in version 3.0.
I want discuss the feature of it.
Some ideas for new features:
- Support one time trigger for ElasticJob lite
- Support job dependency based one sharding item
- Use native ZooKeeper API instead of curator
Some ideas for new architecture:
- Refactor Job API, just keep SimpleJob, make other job types introduce via
SPI
- Redesign Job domain, make job as top level interface, which composite by
task (sharding item)
- Refactor event trace module
- Split event trace and elasticjob-core module
- Open SPI for event trace data storage
- Support prometheus storage
------------------
Liang Zhang (John)
Apache ShardingSphere & Dubbo
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by "zhangliang@apache.org" <zh...@apache.org>.
This is a new feature about alarm. We need more discuss about this.
------------------
Liang Zhang (John)
Apache ShardingSphere & Dubbo
李好 <da...@gmail.com> 于2020年6月15日周一 下午1:39写道：
> Hi liang :
>
> I am looking foward to seeing the new Feature.
>
> I wrote a simple job-system for visual configuration before. At
> reality bussiness,most of
> people seems care more about when the task finish and what the task
> result.So I suggest add a function interface for sending detail
> message,example email,phone-note,message-push after the task finish.
> people can decide to if turn off or turn on on method and error level.
>
> Regards
> Jian
>
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by ������ <da...@gmail.com>.
Hi liang :
I am looking foward to seeing the new Feature.
I wrote a simple job-system for visual configuration before. At reality bussiness,most of
people seems care more about when the task finish and what the task result.So I suggest add a function interface for sending detail message,example email,phone-note,message-push after the task finish.
people can decide to if turn off or turn on on method and error level.
Regards
Jian
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by guo jiwei <te...@apache.org>.
Hi,
Looking forward to the new features.
BTW, If we can remove quartz and support our own schedule engine
someday, it will be a great feature.
Regards
Jiwei Guo (Tboy)
On Fri, Jun 12, 2020 at 4:24 PM zhangliang@apache.org <zh...@apache.org>
wrote:
> Hi all,
>
> We are beginning to develop ElasticJob, and want to add some new features
> in version 3.0.
>
> I want discuss the feature of it.
>
> Some ideas for new features:
>
> - Support one time trigger for ElasticJob lite
> - Support job dependency based one sharding item
> - Use native ZooKeeper API instead of curator
>
> Some ideas for new architecture:
>
> - Refactor Job API, just keep SimpleJob, make other job types introduce via
> SPI
> - Redesign Job domain, make job as top level interface, which composite by
> task (sharding item)
> - Refactor event trace module
> - Split event trace and elasticjob-core module
> - Open SPI for event trace data storage
> - Support prometheus storage
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by Sheng Wu <wu...@gmail.com>.
zhangliang@apache.org <zh...@apache.org>于2020年6月15日 周一17:58写道：
> How about spilt them?
> For event trace data (about job execute timestamp and result), persist into
> MySQL or other databases, for metrics, persist into Prometheus.
I think separating them will make you can't use Grafana. I think alarm,
monitor and observability should be an important and separated topic.
Sheng
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>
>
> Sheng Wu <wu...@gmail.com> 于2020年6月15日周一 下午2:02写道：
>
> > What event trace data? Prometheus is focusing on metrics only. Only
> > statistic data is suitable.
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > zhangliang@apache.org <zh...@apache.org> 于2020年6月15日周一 下午1:34写道：
> >
> > > Hi Sheng,
> > >
> > > The mean of `Support prometheus storage` is putting the event trace
> data
> > > info prometheus ecosystem, we can use push gateway to collect the data
> > and
> > > use grafana to display the result.
> > >
> > > ------------------
> > >
> > > Liang Zhang (John)
> > > Apache ShardingSphere & Dubbo
> > >
> > >
> > > Sheng Wu <wu...@gmail.com> 于2020年6月15日周一 上午9:26写道：
> > >
> > > > Hi Liang
> > > >
> > > > About SkyWalking, that is another topic. It is fine to do that later.
> > > > My question is, what do you mean `Support prometheus storage`.
> > Prometheus
> > > > is a monitoring system, basically. What does it storage you mean?
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > zhangliang@apache.org <zh...@apache.org> 于2020年6月13日周六
> 上午1:28写道：
> > > >
> > > > > For persist job event trace. We can consider about use SkyWalking
> to
> > > > trace
> > > > > the job event, is it possible?
> > > > >
> > > > > ------------------
> > > > >
> > > > > Liang Zhang (John)
> > > > > Apache ShardingSphere & Dubbo
> > > > >
> > > > >
> > > > > Sheng Wu <wu...@gmail.com> 于2020年6月12日周五 下午4:54写道：
> > > > >
> > > > > > zhangliang@apache.org <zh...@apache.org> 于2020年6月12日周五
> > > 下午4:24写道：
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We are beginning to develop ElasticJob, and want to add some
> new
> > > > > features
> > > > > > > in version 3.0.
> > > > > > >
> > > > > > > I want discuss the feature of it.
> > > > > > >
> > > > > > > Some ideas for new features:
> > > > > > >
> > > > > > > - Support one time trigger for ElasticJob lite
> > > > > > > - Support job dependency based one sharding item
> > > > > > > - Use native ZooKeeper API instead of curator
> > > > > > >
> > > > > > > Some ideas for new architecture:
> > > > > > >
> > > > > > > - Refactor Job API, just keep SimpleJob, make other job types
> > > > introduce
> > > > > > via
> > > > > > > SPI
> > > > > > > - Redesign Job domain, make job as top level interface, which
> > > > composite
> > > > > > by
> > > > > > > task (sharding item)
> > > > > > > - Refactor event trace module
> > > > > > > - Split event trace and elasticjob-core module
> > > > > > > - Open SPI for event trace data storage
> > > > > > > - Support prometheus storage
> > > > > > >
> > > > > >
> > > > > > What is Prometheus storage?
> > > > > >
> > > > > >
> > > > > > Sheng Wu 吴晟
> > > > > > Twitter, wusheng1108
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > ------------------
> > > > > > >
> > > > > > > Liang Zhang (John)
> > > > > > > Apache ShardingSphere & Dubbo
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Sheng Wu 吴晟
Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by "zhangliang@apache.org" <zh...@apache.org>.
How about spilt them?
For event trace data (about job execute timestamp and result), persist into
MySQL or other databases, for metrics, persist into Prometheus.
------------------
Liang Zhang (John)
Apache ShardingSphere & Dubbo
Sheng Wu <wu...@gmail.com> 于2020年6月15日周一 下午2:02写道：
> What event trace data? Prometheus is focusing on metrics only. Only
> statistic data is suitable.
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> zhangliang@apache.org <zh...@apache.org> 于2020年6月15日周一 下午1:34写道：
>
> > Hi Sheng,
> >
> > The mean of `Support prometheus storage` is putting the event trace data
> > info prometheus ecosystem, we can use push gateway to collect the data
> and
> > use grafana to display the result.
> >
> > ------------------
> >
> > Liang Zhang (John)
> > Apache ShardingSphere & Dubbo
> >
> >
> > Sheng Wu <wu...@gmail.com> 于2020年6月15日周一 上午9:26写道：
> >
> > > Hi Liang
> > >
> > > About SkyWalking, that is another topic. It is fine to do that later.
> > > My question is, what do you mean `Support prometheus storage`.
> Prometheus
> > > is a monitoring system, basically. What does it storage you mean?
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > zhangliang@apache.org <zh...@apache.org> 于2020年6月13日周六 上午1:28写道：
> > >
> > > > For persist job event trace. We can consider about use SkyWalking to
> > > trace
> > > > the job event, is it possible?
> > > >
> > > > ------------------
> > > >
> > > > Liang Zhang (John)
> > > > Apache ShardingSphere & Dubbo
> > > >
> > > >
> > > > Sheng Wu <wu...@gmail.com> 于2020年6月12日周五 下午4:54写道：
> > > >
> > > > > zhangliang@apache.org <zh...@apache.org> 于2020年6月12日周五
> > 下午4:24写道：
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We are beginning to develop ElasticJob, and want to add some new
> > > > features
> > > > > > in version 3.0.
> > > > > >
> > > > > > I want discuss the feature of it.
> > > > > >
> > > > > > Some ideas for new features:
> > > > > >
> > > > > > - Support one time trigger for ElasticJob lite
> > > > > > - Support job dependency based one sharding item
> > > > > > - Use native ZooKeeper API instead of curator
> > > > > >
> > > > > > Some ideas for new architecture:
> > > > > >
> > > > > > - Refactor Job API, just keep SimpleJob, make other job types
> > > introduce
> > > > > via
> > > > > > SPI
> > > > > > - Redesign Job domain, make job as top level interface, which
> > > composite
> > > > > by
> > > > > > task (sharding item)
> > > > > > - Refactor event trace module
> > > > > > - Split event trace and elasticjob-core module
> > > > > > - Open SPI for event trace data storage
> > > > > > - Support prometheus storage
> > > > > >
> > > > >
> > > > > What is Prometheus storage?
> > > > >
> > > > >
> > > > > Sheng Wu 吴晟
> > > > > Twitter, wusheng1108
> > > > >
> > > > >
> > > > > >
> > > > > > ------------------
> > > > > >
> > > > > > Liang Zhang (John)
> > > > > > Apache ShardingSphere & Dubbo
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by Sheng Wu <wu...@gmail.com>.
What event trace data? Prometheus is focusing on metrics only. Only
statistic data is suitable.
Sheng Wu 吴晟
Twitter, wusheng1108
zhangliang@apache.org <zh...@apache.org> 于2020年6月15日周一 下午1:34写道：
> Hi Sheng,
>
> The mean of `Support prometheus storage` is putting the event trace data
> info prometheus ecosystem, we can use push gateway to collect the data and
> use grafana to display the result.
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>
>
> Sheng Wu <wu...@gmail.com> 于2020年6月15日周一 上午9:26写道：
>
> > Hi Liang
> >
> > About SkyWalking, that is another topic. It is fine to do that later.
> > My question is, what do you mean `Support prometheus storage`. Prometheus
> > is a monitoring system, basically. What does it storage you mean?
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > zhangliang@apache.org <zh...@apache.org> 于2020年6月13日周六 上午1:28写道：
> >
> > > For persist job event trace. We can consider about use SkyWalking to
> > trace
> > > the job event, is it possible?
> > >
> > > ------------------
> > >
> > > Liang Zhang (John)
> > > Apache ShardingSphere & Dubbo
> > >
> > >
> > > Sheng Wu <wu...@gmail.com> 于2020年6月12日周五 下午4:54写道：
> > >
> > > > zhangliang@apache.org <zh...@apache.org> 于2020年6月12日周五
> 下午4:24写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > We are beginning to develop ElasticJob, and want to add some new
> > > features
> > > > > in version 3.0.
> > > > >
> > > > > I want discuss the feature of it.
> > > > >
> > > > > Some ideas for new features:
> > > > >
> > > > > - Support one time trigger for ElasticJob lite
> > > > > - Support job dependency based one sharding item
> > > > > - Use native ZooKeeper API instead of curator
> > > > >
> > > > > Some ideas for new architecture:
> > > > >
> > > > > - Refactor Job API, just keep SimpleJob, make other job types
> > introduce
> > > > via
> > > > > SPI
> > > > > - Redesign Job domain, make job as top level interface, which
> > composite
> > > > by
> > > > > task (sharding item)
> > > > > - Refactor event trace module
> > > > > - Split event trace and elasticjob-core module
> > > > > - Open SPI for event trace data storage
> > > > > - Support prometheus storage
> > > > >
> > > >
> > > > What is Prometheus storage?
> > > >
> > > >
> > > > Sheng Wu 吴晟
> > > > Twitter, wusheng1108
> > > >
> > > >
> > > > >
> > > > > ------------------
> > > > >
> > > > > Liang Zhang (John)
> > > > > Apache ShardingSphere & Dubbo
> > > > >
> > > >
> > >
> >
>
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by "zhangliang@apache.org" <zh...@apache.org>.
Hi Sheng,
The mean of `Support prometheus storage` is putting the event trace data
info prometheus ecosystem, we can use push gateway to collect the data and
use grafana to display the result.
------------------
Liang Zhang (John)
Apache ShardingSphere & Dubbo
Sheng Wu <wu...@gmail.com> 于2020年6月15日周一 上午9:26写道：
> Hi Liang
>
> About SkyWalking, that is another topic. It is fine to do that later.
> My question is, what do you mean `Support prometheus storage`. Prometheus
> is a monitoring system, basically. What does it storage you mean?
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> zhangliang@apache.org <zh...@apache.org> 于2020年6月13日周六 上午1:28写道：
>
> > For persist job event trace. We can consider about use SkyWalking to
> trace
> > the job event, is it possible?
> >
> > ------------------
> >
> > Liang Zhang (John)
> > Apache ShardingSphere & Dubbo
> >
> >
> > Sheng Wu <wu...@gmail.com> 于2020年6月12日周五 下午4:54写道：
> >
> > > zhangliang@apache.org <zh...@apache.org> 于2020年6月12日周五 下午4:24写道：
> > >
> > > > Hi all,
> > > >
> > > > We are beginning to develop ElasticJob, and want to add some new
> > features
> > > > in version 3.0.
> > > >
> > > > I want discuss the feature of it.
> > > >
> > > > Some ideas for new features:
> > > >
> > > > - Support one time trigger for ElasticJob lite
> > > > - Support job dependency based one sharding item
> > > > - Use native ZooKeeper API instead of curator
> > > >
> > > > Some ideas for new architecture:
> > > >
> > > > - Refactor Job API, just keep SimpleJob, make other job types
> introduce
> > > via
> > > > SPI
> > > > - Redesign Job domain, make job as top level interface, which
> composite
> > > by
> > > > task (sharding item)
> > > > - Refactor event trace module
> > > > - Split event trace and elasticjob-core module
> > > > - Open SPI for event trace data storage
> > > > - Support prometheus storage
> > > >
> > >
> > > What is Prometheus storage?
> > >
> > >
> > > Sheng Wu 吴晟
> > > Twitter, wusheng1108
> > >
> > >
> > > >
> > > > ------------------
> > > >
> > > > Liang Zhang (John)
> > > > Apache ShardingSphere & Dubbo
> > > >
> > >
> >
>
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by Sheng Wu <wu...@gmail.com>.
Hi Liang
About SkyWalking, that is another topic. It is fine to do that later.
My question is, what do you mean `Support prometheus storage`. Prometheus
is a monitoring system, basically. What does it storage you mean?
Sheng Wu 吴晟
Twitter, wusheng1108
zhangliang@apache.org <zh...@apache.org> 于2020年6月13日周六 上午1:28写道：
> For persist job event trace. We can consider about use SkyWalking to trace
> the job event, is it possible?
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>
>
> Sheng Wu <wu...@gmail.com> 于2020年6月12日周五 下午4:54写道：
>
> > zhangliang@apache.org <zh...@apache.org> 于2020年6月12日周五 下午4:24写道：
> >
> > > Hi all,
> > >
> > > We are beginning to develop ElasticJob, and want to add some new
> features
> > > in version 3.0.
> > >
> > > I want discuss the feature of it.
> > >
> > > Some ideas for new features:
> > >
> > > - Support one time trigger for ElasticJob lite
> > > - Support job dependency based one sharding item
> > > - Use native ZooKeeper API instead of curator
> > >
> > > Some ideas for new architecture:
> > >
> > > - Refactor Job API, just keep SimpleJob, make other job types introduce
> > via
> > > SPI
> > > - Redesign Job domain, make job as top level interface, which composite
> > by
> > > task (sharding item)
> > > - Refactor event trace module
> > > - Split event trace and elasticjob-core module
> > > - Open SPI for event trace data storage
> > > - Support prometheus storage
> > >
> >
> > What is Prometheus storage?
> >
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > >
> > > ------------------
> > >
> > > Liang Zhang (John)
> > > Apache ShardingSphere & Dubbo
> > >
> >
>
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by "zhangliang@apache.org" <zh...@apache.org>.
For persist job event trace. We can consider about use SkyWalking to trace
the job event, is it possible?
------------------
Liang Zhang (John)
Apache ShardingSphere & Dubbo
Sheng Wu <wu...@gmail.com> 于2020年6月12日周五 下午4:54写道：
> zhangliang@apache.org <zh...@apache.org> 于2020年6月12日周五 下午4:24写道：
>
> > Hi all,
> >
> > We are beginning to develop ElasticJob, and want to add some new features
> > in version 3.0.
> >
> > I want discuss the feature of it.
> >
> > Some ideas for new features:
> >
> > - Support one time trigger for ElasticJob lite
> > - Support job dependency based one sharding item
> > - Use native ZooKeeper API instead of curator
> >
> > Some ideas for new architecture:
> >
> > - Refactor Job API, just keep SimpleJob, make other job types introduce
> via
> > SPI
> > - Redesign Job domain, make job as top level interface, which composite
> by
> > task (sharding item)
> > - Refactor event trace module
> > - Split event trace and elasticjob-core module
> > - Open SPI for event trace data storage
> > - Support prometheus storage
> >
>
> What is Prometheus storage?
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> >
> > ------------------
> >
> > Liang Zhang (John)
> > Apache ShardingSphere & Dubbo
> >
>
Re: [ElasticJob] Discuss new features of ElasticJob 3.0
Posted by Sheng Wu <wu...@gmail.com>.
zhangliang@apache.org <zh...@apache.org> 于2020年6月12日周五 下午4:24写道：
> Hi all,
>
> We are beginning to develop ElasticJob, and want to add some new features
> in version 3.0.
>
> I want discuss the feature of it.
>
> Some ideas for new features:
>
> - Support one time trigger for ElasticJob lite
> - Support job dependency based one sharding item
> - Use native ZooKeeper API instead of curator
>
> Some ideas for new architecture:
>
> - Refactor Job API, just keep SimpleJob, make other job types introduce via
> SPI
> - Redesign Job domain, make job as top level interface, which composite by
> task (sharding item)
> - Refactor event trace module
> - Split event trace and elasticjob-core module
> - Open SPI for event trace data storage
> - Support prometheus storage
>
What is Prometheus storage?
Sheng Wu 吴晟
Twitter, wusheng1108
>
> ------------------
>
> Liang Zhang (John)
> Apache ShardingSphere & Dubbo
>