You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by 阿根廷拉 <48...@qq.com> on 2019/12/09 13:46:38 UTC
回复： [Proposal] Improve CI / CD process
Which CI tool to choose, Is the CI locally built or the public network built ?
------------------ 原始邮件 ------------------
发件人: "William Guo Wei"<william-guowei@qq.com>;
发送时间: 2019年12月9日(星期一) 晚上9:29
收件人: "dev"<dev@dolphinscheduler.apache.org>;
主题: Re: [Proposal] Improve CI / CD process
Very good proposal! Very glad to see new proposal after meetup.
Here are some questions:
1. About the bar of UT. I thought now UT is below 40%, totally agree to improve that. Other people opinion?
2. I noticed that you mentioned some AWS Servers to support this CICD process. How many extra resource besides Apache offered do you need if we want to finish this CICD.
3. I am curious about license check. Do we have some way to do it automatically or semi-auto way?
William Guo Wei
------------------ Original ------------------
From: Xiaochun Liu <liuxiaochun@apache.org>
Date: Mon,Dec 9,2019 8:47 PM
To: dev <dev@dolphinscheduler.apache.org>
Subject: Re: [Proposal] Improve CI / CD process
Hi,
all, the CI / CD process in DolphinScheduler only has build check and license check. The other parts are missing now.
Here is my proposal:
The overall idea of CI / CD is shown in the following graph. There are two main triggering methods, submitting PR / MR trigger and night time trigger. The night time trigger triggers to run once a day, and the task is relatively heavy.
The content of CI / CD to be tested or checked is shown in the graph below.
The above is my overall thinking. The details of the CI / CD plan for the next version are as follows:
1) The ut needs to be improved. The goal of the proportion of UT is to reach 60%. At least the newly added PR cannot be lower than this. If it feels unreasonable, we can vote to discuss it. The modules involved in this supplementary UT are api server, common, alert, and some master and worker UTs.
2) checkstyle check, there are two kinds of google and Alibaba, personal preference is alibaba (more comprehensive), this can be discussed separately
3) spotbugs check
4) Front-end eslint check
5) Coveralls code coverage report. When submitting a PR, it will be automatically displayed in the comments, showing the impact of the new part on the current code coverage, and the current test coverage will be displayed on the github home page.
If have other ideas, we can discuss together ~~ Thank you
Xiaochun Liu
liuxiaochun@apache.org
Re: Re: [Proposal] Improve CI / CD process
Posted by Sheng Wu <wu...@gmail.com>.
I think you should not do this kind of mock, it is pointless.
This is why e2e or integration tests existing. You need to verify data in
the database or through your query API.
Sheng Wu 吴晟
Twitter, wusheng1108
qiao zhanwei <qi...@outlook.com> 于2019年12月11日周三 上午10:45写道：
> I know, for example, service, conroller is the test logic, you can mock
> the corresponding dao and service,
> but dao is a direct database operation, you need to test the sql
> statement,
> how does this mock? Do you need to connect to the database or zk?
>
>
> —————————————
> DolphinScheduler(Incubator) PPMC
> Zhanwei Qiao 乔占卫
>
> qiaozhanwei@outlook.com
>
> From: Xiaochun Liu<ma...@apache.org>
> Date: 2019-12-11 10:37
> To: qiao zhanwei<ma...@outlook.com>
> CC: dev<ma...@dolphinscheduler.apache.org>; Xiaochun Liu<mailto:
> khadgar.mage@gmail.com>
> Subject: Re: [Proposal] Improve CI / CD process
> question :
> DAO UT needs to connect to db or zk，Github action can do it ？
>
> Mockito can mock the dependence
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) Committer
> Xiaochun Liu 刘小春
> liuxiaochun@apache.org<ma...@apache.org>
> ---------------
>
>
>
> 在 2019年12月11日，上午10:33，qiao zhanwei <qiaozhanwei@outlook.com<mailto:
> qiaozhanwei@outlook.com>> 写道：
>
> Hi Xiaochun :
> question :
> DAO UT needs to connect to db or zk，Github action can do it ？
>
> CI/CD is a persistent process
> agree the first step is to do a license, spotbugs ，checkstyle ,
> Front-end eslint check and part UT
>
>
> —————————————
> DolphinScheduler(Incubator) PPMC
> Zhanwei Qiao 乔占卫
>
> qiaozhanwei@outlook.com<ma...@outlook.com>
>
> From: lidong dai<ma...@gmail.com>
> Date: 2019-12-10 18:52
> To: dev<ma...@dolphinscheduler.apache.org>; Xiaochun Liu<mailto:
> khadgar.mage@gmail.com>
> Subject: Re: [Proposal] Improve CI / CD process
> hi xiaochun，
> What do you think the hardware configuration of CI server should be？
>
>
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) PPMC
> Lidong Dai 代立冬
> dailidong66@gmail.com<ma...@gmail.com>
> ---------------
>
>
> Xiaochun Liu <li...@apache.org>>
> 于2019年12月9日周一 下午11:30写道：
>
> > @ William Guo Wei
> > 1、60% is actually not a particularly good ratio. I personally feel that
> > 80% should be okay, but considering the current UT situation, I don't
> dare
> > set too high a goal, we can go step by step.
> > 2、The AWS in the picture is an example. In fact, it can be any cloud
> host,
> > as long as the external network can access it. The main purpose is to
> > trigger the deployment of the demo environment by github action.
> > And this is not the goal of the next version for the time being, you can
> > ignore it for the time being and evaluate it later.
> > 3、License test has been automated for a long time in the github action.
> >
> > @阿根廷拉
> > We use github action tas ci tool, public network built
> >
> > Xiaochun Liu
> > liuxiaochun@apache.org<ma...@apache.org>
> >
> >
> >
> > > 在 2019年12月9日，下午9:46，阿根廷拉 <48...@qq.com>>
> 写道：
> > >
> > > Which CI tool to choose, Is the CI locally built or the public
> > network built ?
> > >
> > >
> > >
> > >
> > > ------------------ 原始邮件 ------------------
> > > 发件人: "William Guo Wei"<william-guowei@qq.com<mailto:
> william-guowei@qq.com>>;
> > > 发送时间: 2019年12月9日(星期一) 晚上9:29
> > > 收件人: "dev"<dev@dolphinscheduler.apache.org<mailto:
> dev@dolphinscheduler.apache.org>>;
> > >
> > > 主题: Re: [Proposal] Improve CI / CD process
> > >
> > >
> > >
> > > Very good proposal! Very glad to see new proposal after meetup.
> > > Here are some questions:
> > > 1. About the bar of UT. I thought now UT is below 40%, totally agree to
> > improve that. Other people opinion?
> > > 2. I noticed that you mentioned some AWS Servers to support this CICD
> > process. How many extra resource besides Apache offered do you need if we
> > want to finish this CICD.
> > > 3. I am curious about license check. Do we have some way to do it
> > automatically or semi-auto way?
> > >
> > >
> > >
> > > William Guo Wei
> > >
> > >
> > > ------------------ Original ------------------
> > > From: Xiaochun Liu <liuxiaochun@apache.org<mailto:
> liuxiaochun@apache.org>>
> > > Date: Mon,Dec 9,2019 8:47 PM
> > > To: dev <dev@dolphinscheduler.apache.org<mailto:
> dev@dolphinscheduler.apache.org>>
> > > Subject: Re: [Proposal] Improve CI / CD process
> > >
> > >
> > >
> > >
> > >
> > > Hi,
> > > all, the CI / CD process in DolphinScheduler only has build
> check
> > and license check. The other parts are missing now.
> > >
> > >
> > > Here is my proposal:
> > > The overall idea of CI / CD is shown in the following graph. There
> are
> > two main triggering methods, submitting PR / MR trigger and night time
> > trigger. The night time trigger triggers to run once a day, and the task
> is
> > relatively heavy.
> > >
> > > The content of CI / CD to be tested or checked is shown in the graph
> > below.
> > >
> > >
> > >
> > >
> > >
> > > The above is my overall thinking. The details of the CI / CD plan for
> > the next version are as follows:
> > > 1) The ut needs to be improved. The goal of the proportion of UT
> > is to reach 60%. At least the newly added PR cannot be lower than this.
> If
> > it feels unreasonable, we can vote to discuss it. The modules involved in
> > this supplementary UT are api server, common, alert, and some master and
> > worker UTs.
> > > 2) checkstyle check, there are two kinds of google and Alibaba,
> > personal preference is alibaba (more comprehensive), this can be
> discussed
> > separately
> > > 3) spotbugs check
> > > 4) Front-end eslint check
> > > 5) Coveralls code coverage report. When submitting a PR, it will
> > be automatically displayed in the comments, showing the impact of the new
> > part on the current code coverage, and the current test coverage will be
> > displayed on the github home page.
> > >
> > >
> > > If have other ideas, we can discuss together ~~ Thank you
> > >
> > > Xiaochun Liu
> > > liuxiaochun@apache.org<ma...@apache.org>
> >
> >
>
>
Re: Re: [Proposal] Improve CI / CD process
Posted by qiao zhanwei <qi...@outlook.com>.
I know, for example, service, conroller is the test logic, you can mock the corresponding dao and service,
but dao is a direct database operation, you need to test the sql statement,
how does this mock? Do you need to connect to the database or zk?
—————————————
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
From: Xiaochun Liu<ma...@apache.org>
Date: 2019-12-11 10:37
To: qiao zhanwei<ma...@outlook.com>
CC: dev<ma...@dolphinscheduler.apache.org>; Xiaochun Liu<ma...@gmail.com>
Subject: Re: [Proposal] Improve CI / CD process
question :
DAO UT needs to connect to db or zk，Github action can do it ？
Mockito can mock the dependence
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org<ma...@apache.org>
---------------
在 2019年12月11日，上午10:33，qiao zhanwei <qi...@outlook.com>> 写道：
Hi Xiaochun :
question :
DAO UT needs to connect to db or zk，Github action can do it ？
CI/CD is a persistent process
agree the first step is to do a license, spotbugs ，checkstyle , Front-end eslint check and part UT
—————————————
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com<ma...@outlook.com>
From: lidong dai<ma...@gmail.com>
Date: 2019-12-10 18:52
To: dev<ma...@dolphinscheduler.apache.org>; Xiaochun Liu<ma...@gmail.com>
Subject: Re: [Proposal] Improve CI / CD process
hi xiaochun，
What do you think the hardware configuration of CI server should be？
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com<ma...@gmail.com>
---------------
Xiaochun Liu <li...@apache.org>> 于2019年12月9日周一 下午11:30写道：
> @ William Guo Wei
> 1、60% is actually not a particularly good ratio. I personally feel that
> 80% should be okay, but considering the current UT situation, I don't dare
> set too high a goal, we can go step by step.
> 2、The AWS in the picture is an example. In fact, it can be any cloud host,
> as long as the external network can access it. The main purpose is to
> trigger the deployment of the demo environment by github action.
> And this is not the goal of the next version for the time being, you can
> ignore it for the time being and evaluate it later.
> 3、License test has been automated for a long time in the github action.
>
> @阿根廷拉
> We use github action tas ci tool, public network built
>
> Xiaochun Liu
> liuxiaochun@apache.org<ma...@apache.org>
>
>
>
> > 在 2019年12月9日，下午9:46，阿根廷拉 <48...@qq.com>> 写道：
> >
> > Which CI tool to choose, Is the CI locally built or the public
> network built ?
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "William Guo Wei"<wi...@qq.com>>;
> > 发送时间: 2019年12月9日(星期一) 晚上9:29
> > 收件人: "dev"<de...@dolphinscheduler.apache.org>>;
> >
> > 主题: Re: [Proposal] Improve CI / CD process
> >
> >
> >
> > Very good proposal! Very glad to see new proposal after meetup.
> > Here are some questions:
> > 1. About the bar of UT. I thought now UT is below 40%, totally agree to
> improve that. Other people opinion?
> > 2. I noticed that you mentioned some AWS Servers to support this CICD
> process. How many extra resource besides Apache offered do you need if we
> want to finish this CICD.
> > 3. I am curious about license check. Do we have some way to do it
> automatically or semi-auto way?
> >
> >
> >
> > William Guo Wei
> >
> >
> > ------------------ Original ------------------
> > From: Xiaochun Liu <li...@apache.org>>
> > Date: Mon,Dec 9,2019 8:47 PM
> > To: dev <de...@dolphinscheduler.apache.org>>
> > Subject: Re: [Proposal] Improve CI / CD process
> >
> >
> >
> >
> >
> > Hi,
> > all, the CI / CD process in DolphinScheduler only has build check
> and license check. The other parts are missing now.
> >
> >
> > Here is my proposal:
> > The overall idea of CI / CD is shown in the following graph. There are
> two main triggering methods, submitting PR / MR trigger and night time
> trigger. The night time trigger triggers to run once a day, and the task is
> relatively heavy.
> >
> > The content of CI / CD to be tested or checked is shown in the graph
> below.
> >
> >
> >
> >
> >
> > The above is my overall thinking. The details of the CI / CD plan for
> the next version are as follows:
> > 1) The ut needs to be improved. The goal of the proportion of UT
> is to reach 60%. At least the newly added PR cannot be lower than this. If
> it feels unreasonable, we can vote to discuss it. The modules involved in
> this supplementary UT are api server, common, alert, and some master and
> worker UTs.
> > 2) checkstyle check, there are two kinds of google and Alibaba,
> personal preference is alibaba (more comprehensive), this can be discussed
> separately
> > 3) spotbugs check
> > 4) Front-end eslint check
> > 5) Coveralls code coverage report. When submitting a PR, it will
> be automatically displayed in the comments, showing the impact of the new
> part on the current code coverage, and the current test coverage will be
> displayed on the github home page.
> >
> >
> > If have other ideas, we can discuss together ~~ Thank you
> >
> > Xiaochun Liu
> > liuxiaochun@apache.org<ma...@apache.org>
>
>
Re: [Proposal] Improve CI / CD process
Posted by Xiaochun Liu <li...@apache.org>.
> question :
> DAO UT needs to connect to db or zk，Github action can do it ？
Mockito can mock the dependence
Best Regards
---------------
DolphinScheduler(Incubator) Committer
Xiaochun Liu 刘小春
liuxiaochun@apache.org
---------------
> 在 2019年12月11日，上午10:33，qiao zhanwei <qi...@outlook.com> 写道：
>
> Hi Xiaochun :
> question :
> DAO UT needs to connect to db or zk，Github action can do it ？
>
> CI/CD is a persistent process
> agree the first step is to do a license, spotbugs ，checkstyle , Front-end eslint check and part UT
>
>
> —————————————
> DolphinScheduler(Incubator) PPMC
> Zhanwei Qiao 乔占卫
>
> qiaozhanwei@outlook.com <ma...@outlook.com>
>
> From: lidong dai <ma...@gmail.com>
> Date: 2019-12-10 18:52
> To: dev <ma...@dolphinscheduler.apache.org>; Xiaochun Liu <ma...@gmail.com>
> Subject: Re: [Proposal] Improve CI / CD process
> hi xiaochun，
> What do you think the hardware configuration of CI server should be？
>
>
>
> Best Regards
> ---------------
> DolphinScheduler(Incubator) PPMC
> Lidong Dai 代立冬
> dailidong66@gmail.com
> ---------------
>
>
> Xiaochun Liu <li...@apache.org> 于2019年12月9日周一 下午11:30写道：
>
> > @ William Guo Wei
> > 1、60% is actually not a particularly good ratio. I personally feel that
> > 80% should be okay, but considering the current UT situation, I don't dare
> > set too high a goal, we can go step by step.
> > 2、The AWS in the picture is an example. In fact, it can be any cloud host,
> > as long as the external network can access it. The main purpose is to
> > trigger the deployment of the demo environment by github action.
> > And this is not the goal of the next version for the time being, you can
> > ignore it for the time being and evaluate it later.
> > 3、License test has been automated for a long time in the github action.
> >
> > @阿根廷拉
> > We use github action tas ci tool, public network built
> >
> > Xiaochun Liu
> > liuxiaochun@apache.org
> >
> >
> >
> > > 在 2019年12月9日，下午9:46，阿根廷拉 <48...@qq.com> 写道：
> > >
> > > Which CI tool to choose, Is the CI locally built or the public
> > network built ?
> > >
> > >
> > >
> > >
> > > ------------------ 原始邮件 ------------------
> > > 发件人: "William Guo Wei"<william-guowei@qq.com>;
> > > 发送时间: 2019年12月9日(星期一) 晚上9:29
> > > 收件人: "dev"<dev@dolphinscheduler.apache.org>;
> > >
> > > 主题: Re: [Proposal] Improve CI / CD process
> > >
> > >
> > >
> > > Very good proposal! Very glad to see new proposal after meetup.
> > > Here are some questions:
> > > 1. About the bar of UT. I thought now UT is below 40%, totally agree to
> > improve that. Other people opinion?
> > > 2. I noticed that you mentioned some AWS Servers to support this CICD
> > process. How many extra resource besides Apache offered do you need if we
> > want to finish this CICD.
> > > 3. I am curious about license check. Do we have some way to do it
> > automatically or semi-auto way?
> > >
> > >
> > >
> > > William Guo Wei
> > >
> > >
> > > ------------------ Original ------------------
> > > From: Xiaochun Liu <liuxiaochun@apache.org>
> > > Date: Mon,Dec 9,2019 8:47 PM
> > > To: dev <dev@dolphinscheduler.apache.org>
> > > Subject: Re: [Proposal] Improve CI / CD process
> > >
> > >
> > >
> > >
> > >
> > > Hi,
> > > all, the CI / CD process in DolphinScheduler only has build check
> > and license check. The other parts are missing now.
> > >
> > >
> > > Here is my proposal:
> > > The overall idea of CI / CD is shown in the following graph. There are
> > two main triggering methods, submitting PR / MR trigger and night time
> > trigger. The night time trigger triggers to run once a day, and the task is
> > relatively heavy.
> > >
> > > The content of CI / CD to be tested or checked is shown in the graph
> > below.
> > >
> > >
> > >
> > >
> > >
> > > The above is my overall thinking. The details of the CI / CD plan for
> > the next version are as follows:
> > > 1) The ut needs to be improved. The goal of the proportion of UT
> > is to reach 60%. At least the newly added PR cannot be lower than this. If
> > it feels unreasonable, we can vote to discuss it. The modules involved in
> > this supplementary UT are api server, common, alert, and some master and
> > worker UTs.
> > > 2) checkstyle check, there are two kinds of google and Alibaba,
> > personal preference is alibaba (more comprehensive), this can be discussed
> > separately
> > > 3) spotbugs check
> > > 4) Front-end eslint check
> > > 5) Coveralls code coverage report. When submitting a PR, it will
> > be automatically displayed in the comments, showing the impact of the new
> > part on the current code coverage, and the current test coverage will be
> > displayed on the github home page.
> > >
> > >
> > > If have other ideas, we can discuss together ~~ Thank you
> > >
> > > Xiaochun Liu
> > > liuxiaochun@apache.org
> >
> >
Re: Re: [Proposal] Improve CI / CD process
Posted by qiao zhanwei <qi...@outlook.com>.
Hi Xiaochun :
question :
DAO UT needs to connect to db or zk，Github action can do it ？
CI/CD is a persistent process
agree the first step is to do a license, spotbugs ，checkstyle , Front-end eslint check and part UT
—————————————
DolphinScheduler(Incubator) PPMC
Zhanwei Qiao 乔占卫
qiaozhanwei@outlook.com
From: lidong dai<ma...@gmail.com>
Date: 2019-12-10 18:52
To: dev<ma...@dolphinscheduler.apache.org>; Xiaochun Liu<ma...@gmail.com>
Subject: Re: [Proposal] Improve CI / CD process
hi xiaochun，
What do you think the hardware configuration of CI server should be？
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
Xiaochun Liu <li...@apache.org> 于2019年12月9日周一 下午11:30写道：
> @ William Guo Wei
> 1、60% is actually not a particularly good ratio. I personally feel that
> 80% should be okay, but considering the current UT situation, I don't dare
> set too high a goal, we can go step by step.
> 2、The AWS in the picture is an example. In fact, it can be any cloud host,
> as long as the external network can access it. The main purpose is to
> trigger the deployment of the demo environment by github action.
> And this is not the goal of the next version for the time being, you can
> ignore it for the time being and evaluate it later.
> 3、License test has been automated for a long time in the github action.
>
> @阿根廷拉
> We use github action tas ci tool, public network built
>
> Xiaochun Liu
> liuxiaochun@apache.org
>
>
>
> > 在 2019年12月9日，下午9:46，阿根廷拉 <48...@qq.com> 写道：
> >
> > Which CI tool to choose, Is the CI locally built or the public
> network built ?
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "William Guo Wei"<william-guowei@qq.com>;
> > 发送时间: 2019年12月9日(星期一) 晚上9:29
> > 收件人: "dev"<dev@dolphinscheduler.apache.org>;
> >
> > 主题: Re: [Proposal] Improve CI / CD process
> >
> >
> >
> > Very good proposal! Very glad to see new proposal after meetup.
> > Here are some questions:
> > 1. About the bar of UT. I thought now UT is below 40%, totally agree to
> improve that. Other people opinion?
> > 2. I noticed that you mentioned some AWS Servers to support this CICD
> process. How many extra resource besides Apache offered do you need if we
> want to finish this CICD.
> > 3. I am curious about license check. Do we have some way to do it
> automatically or semi-auto way?
> >
> >
> >
> > William Guo Wei
> >
> >
> > ------------------ Original ------------------
> > From: Xiaochun Liu <liuxiaochun@apache.org>
> > Date: Mon,Dec 9,2019 8:47 PM
> > To: dev <dev@dolphinscheduler.apache.org>
> > Subject: Re: [Proposal] Improve CI / CD process
> >
> >
> >
> >
> >
> > Hi,
> > all, the CI / CD process in DolphinScheduler only has build check
> and license check. The other parts are missing now.
> >
> >
> > Here is my proposal:
> > The overall idea of CI / CD is shown in the following graph. There are
> two main triggering methods, submitting PR / MR trigger and night time
> trigger. The night time trigger triggers to run once a day, and the task is
> relatively heavy.
> >
> > The content of CI / CD to be tested or checked is shown in the graph
> below.
> >
> >
> >
> >
> >
> > The above is my overall thinking. The details of the CI / CD plan for
> the next version are as follows:
> > 1) The ut needs to be improved. The goal of the proportion of UT
> is to reach 60%. At least the newly added PR cannot be lower than this. If
> it feels unreasonable, we can vote to discuss it. The modules involved in
> this supplementary UT are api server, common, alert, and some master and
> worker UTs.
> > 2) checkstyle check, there are two kinds of google and Alibaba,
> personal preference is alibaba (more comprehensive), this can be discussed
> separately
> > 3) spotbugs check
> > 4) Front-end eslint check
> > 5) Coveralls code coverage report. When submitting a PR, it will
> be automatically displayed in the comments, showing the impact of the new
> part on the current code coverage, and the current test coverage will be
> displayed on the github home page.
> >
> >
> > If have other ideas, we can discuss together ~~ Thank you
> >
> > Xiaochun Liu
> > liuxiaochun@apache.org
>
>
Re: [Proposal] Improve CI / CD process
Posted by lidong dai <da...@gmail.com>.
hi xiaochun，
What do you think the hardware configuration of CI server should be？
Best Regards
---------------
DolphinScheduler(Incubator) PPMC
Lidong Dai 代立冬
dailidong66@gmail.com
---------------
Xiaochun Liu <li...@apache.org> 于2019年12月9日周一 下午11:30写道：
> @ William Guo Wei
> 1、60% is actually not a particularly good ratio. I personally feel that
> 80% should be okay, but considering the current UT situation, I don't dare
> set too high a goal, we can go step by step.
> 2、The AWS in the picture is an example. In fact, it can be any cloud host,
> as long as the external network can access it. The main purpose is to
> trigger the deployment of the demo environment by github action.
> And this is not the goal of the next version for the time being, you can
> ignore it for the time being and evaluate it later.
> 3、License test has been automated for a long time in the github action.
>
> @阿根廷拉
> We use github action tas ci tool, public network built
>
> Xiaochun Liu
> liuxiaochun@apache.org
>
>
>
> > 在 2019年12月9日，下午9:46，阿根廷拉 <48...@qq.com> 写道：
> >
> > Which CI tool to choose, Is the CI locally built or the public
> network built ?
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "William Guo Wei"<william-guowei@qq.com>;
> > 发送时间: 2019年12月9日(星期一) 晚上9:29
> > 收件人: "dev"<dev@dolphinscheduler.apache.org>;
> >
> > 主题: Re: [Proposal] Improve CI / CD process
> >
> >
> >
> > Very good proposal! Very glad to see new proposal after meetup.
> > Here are some questions:
> > 1. About the bar of UT. I thought now UT is below 40%, totally agree to
> improve that. Other people opinion?
> > 2. I noticed that you mentioned some AWS Servers to support this CICD
> process. How many extra resource besides Apache offered do you need if we
> want to finish this CICD.
> > 3. I am curious about license check. Do we have some way to do it
> automatically or semi-auto way?
> >
> >
> >
> > William Guo Wei
> >
> >
> > ------------------ Original ------------------
> > From: Xiaochun Liu <liuxiaochun@apache.org>
> > Date: Mon,Dec 9,2019 8:47 PM
> > To: dev <dev@dolphinscheduler.apache.org>
> > Subject: Re: [Proposal] Improve CI / CD process
> >
> >
> >
> >
> >
> > Hi,
> > all, the CI / CD process in DolphinScheduler only has build check
> and license check. The other parts are missing now.
> >
> >
> > Here is my proposal:
> > The overall idea of CI / CD is shown in the following graph. There are
> two main triggering methods, submitting PR / MR trigger and night time
> trigger. The night time trigger triggers to run once a day, and the task is
> relatively heavy.
> >
> > The content of CI / CD to be tested or checked is shown in the graph
> below.
> >
> >
> >
> >
> >
> > The above is my overall thinking. The details of the CI / CD plan for
> the next version are as follows:
> > 1) The ut needs to be improved. The goal of the proportion of UT
> is to reach 60%. At least the newly added PR cannot be lower than this. If
> it feels unreasonable, we can vote to discuss it. The modules involved in
> this supplementary UT are api server, common, alert, and some master and
> worker UTs.
> > 2) checkstyle check, there are two kinds of google and Alibaba,
> personal preference is alibaba (more comprehensive), this can be discussed
> separately
> > 3) spotbugs check
> > 4) Front-end eslint check
> > 5) Coveralls code coverage report. When submitting a PR, it will
> be automatically displayed in the comments, showing the impact of the new
> part on the current code coverage, and the current test coverage will be
> displayed on the github home page.
> >
> >
> > If have other ideas, we can discuss together ~~ Thank you
> >
> > Xiaochun Liu
> > liuxiaochun@apache.org
>
>
Re: [Proposal] Improve CI / CD process
Posted by Xiaochun Liu <li...@apache.org>.
@ William Guo Wei
1、60% is actually not a particularly good ratio. I personally feel that 80% should be okay, but considering the current UT situation, I don't dare set too high a goal, we can go step by step.
2、The AWS in the picture is an example. In fact, it can be any cloud host, as long as the external network can access it. The main purpose is to trigger the deployment of the demo environment by github action.
And this is not the goal of the next version for the time being, you can ignore it for the time being and evaluate it later.
3、License test has been automated for a long time in the github action.
@阿根廷拉
We use github action tas ci tool, public network built
Xiaochun Liu
liuxiaochun@apache.org
> 在 2019年12月9日，下午9:46，阿根廷拉 <48...@qq.com> 写道：
>
> Which CI tool to choose, Is the CI locally built or the public network built ?
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "William Guo Wei"<william-guowei@qq.com>;
> 发送时间: 2019年12月9日(星期一) 晚上9:29
> 收件人: "dev"<dev@dolphinscheduler.apache.org>;
>
> 主题: Re: [Proposal] Improve CI / CD process
>
>
>
> Very good proposal! Very glad to see new proposal after meetup.
> Here are some questions:
> 1. About the bar of UT. I thought now UT is below 40%, totally agree to improve that. Other people opinion?
> 2. I noticed that you mentioned some AWS Servers to support this CICD process. How many extra resource besides Apache offered do you need if we want to finish this CICD.
> 3. I am curious about license check. Do we have some way to do it automatically or semi-auto way?
>
>
>
> William Guo Wei
>
>
> ------------------ Original ------------------
> From: Xiaochun Liu <liuxiaochun@apache.org>
> Date: Mon,Dec 9,2019 8:47 PM
> To: dev <dev@dolphinscheduler.apache.org>
> Subject: Re: [Proposal] Improve CI / CD process
>
>
>
>
>
> Hi,
> all, the CI / CD process in DolphinScheduler only has build check and license check. The other parts are missing now.
>
>
> Here is my proposal:
> The overall idea of CI / CD is shown in the following graph. There are two main triggering methods, submitting PR / MR trigger and night time trigger. The night time trigger triggers to run once a day, and the task is relatively heavy.
>
> The content of CI / CD to be tested or checked is shown in the graph below.
>
>
>
>
>
> The above is my overall thinking. The details of the CI / CD plan for the next version are as follows:
> 1) The ut needs to be improved. The goal of the proportion of UT is to reach 60%. At least the newly added PR cannot be lower than this. If it feels unreasonable, we can vote to discuss it. The modules involved in this supplementary UT are api server, common, alert, and some master and worker UTs.
> 2) checkstyle check, there are two kinds of google and Alibaba, personal preference is alibaba (more comprehensive), this can be discussed separately
> 3) spotbugs check
> 4) Front-end eslint check
> 5) Coveralls code coverage report. When submitting a PR, it will be automatically displayed in the comments, showing the impact of the new part on the current code coverage, and the current test coverage will be displayed on the github home page.
>
>
> If have other ideas, we can discuss together ~~ Thank you
>
> Xiaochun Liu
> liuxiaochun@apache.org