You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@dolphinscheduler.apache.org by Yichao Yang <10...@qq.com> on 2020/08/05 10:45:21 UTC

Re： [DISCUSS] UT specification of DS

Hi,


As my first email said, we should first clarify the concept of unit test for DS and the purpose of writing unit test.


The concept of unit test is essentially unit testing, and its purpose is to verify the functional correctness of a method or module in DS as quickly and simply as possible.


To ensure the correctness of all functions and processes of DS, we can not completely rely on unit test (for example: in any company or organization, if we want to test the correctness of all functions of a software, we can't only rely on unit test, there will be more testing methods). So that's why there are integration testing, system testing, and E2E testing.


I think it can be combined with the views put forward at present.


1. Unit test:


Test scope: The current ut test scope remains unchanged, which can include some simple tests of database and ZK.


Objective: To verify the correctness of the method simply and quickly. (The simplicity here means that the logic is simplified and the code is easy to read, rather than simple to develop)


Using tools: Use mockito, powermockito, springrunner and other framework tests to write unit tests.


2. Integration test:


Test scope: Integration test of each module between single modules.


Objective: To make up for the small coverage of unit testing and meet the requirements of cross module testing between single modules.


Method: After the completion of simple UT, some integration tests can be added, which can be completed in UT module or E2E module. It is recommended to complete in UT module. Ut with the same function can be completed in a test file (or separate the folder of unit test and integration test).


Using tools: Use mockito, powermockito, springrunner and other framework tests to write integration tests.


3. E2E test:


Test scope: All end-to-end components for&nbsp;DS can be completed in this part, including front-end, ZK, worker and master, API server and log server, alert server and so on.


Purpose: The essence of the above components is end-to-end communication, and test responsibilities can be more clearly divided. It can satisfy the test between modules (servers).


Method: Subdivide modules under E2E module, such as front end, ZK, alertserver, etc


Using tools: Use the starter of each server.


I understand that the above three test methods can basically cover the DS test scenario.



----------------------


正如我的第一封邮件所说，我们应该先明确对于 ds 来说，unit test 的概念和编写 unit test 想要达到的目的。
unit test 概念本质上就是单元测试，目的也是为了最快最简单的验证 ds 中一个方法或者模块的功能正确性。
保障 ds 全部功能正确性和整体流程正确性并不能完全依赖 unit test（举个例子：在任何公司或者组织，如果想要测试一个软件的全部功能正确性，是不可能只依赖于 unit test 的，还会有更多的测试方法）。所以这也是为什么会有集成测试，系统测试，e2e测试。


我觉得可以结合目前提出的观点：
1.单元测试：
测试范围：目前的 ut 测试范围不变，其可以包含数据库以及 zk 的一些`简单`测试
目的：简单并且快速验证方法正确性（这里的简单是指逻辑简化，代码易读，而不是指开发简单）；
使用工具：使用 mockito，powermockito，springrunner 等框架测试编写单元测试


2.集成测试：
测试范围：单个 module 之间的各个模块的集成测试
目的：弥补单元测试覆盖范围小的不足，满足单个 module 之间跨模块测试的需求
方式：可以在简单 ut 完成后，加入一些集成测试，这部分可以在 ut 模块完成，也可以在 e2e 模块完成，推荐在 ut 模块中完成，相同功能ut可以在一个test文件中完成（或者单独划分 unit test 和集成测试的文件夹也可以）
使用工具：使用 mockito，powermockito，springrunner 等框架测试编写集成测试


3.e2e测试：
测试范围：所有和 ds 涉及到端对端的都可以在这部分完成，包含前端，zk，worker 和 master，api server 和 log server，alertserver 的端对端等。
目的：上述组件之间本质就是端对端的通信方式，并且可以更清楚的划分测试职责，更清晰并且细致的测试到每一个功能。满足跨 module（server）之间的测试。
方式：e2e module 下细分 module，比如前端，zk，alertserver 等
使用工具：使用各个 server 的 starter测试。



If you have any better suggestions or opinions, welcome to put forward~


Best,
Yichao Yang




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "dev"                                                                                    <acm_master@163.com&gt;;
发送时间:&nbsp;2020年8月5日(星期三) 中午11:00
收件人:&nbsp;"dev@dolphinscheduler.apache.org"<dev@dolphinscheduler.apache.org&gt;;

主题:&nbsp;Re: [DISCUSS] UT specification of DS



I think you have misunderstood what I mean. I personally think that zk plays a very important role in our project, so I think that zk must be reflected in the unit test. Currently we use the stand-alone version of zk, which will increase us The complexity and instability of the test, so a better approach is to use mock zk. As for whether a real environment needs to be used in e2e, I personally think it is a bit redundant. After all, the simulated client has already done this part of the work.
————————
我想您误解了我的意思，我认为，zk在我们的项目中承担了相当重要的角色，所以保证这块的正确性非常重要，ut本身应该保证它的一个简单性，可测性。这点我很同意@yangyichao同学的观点，我们应该在ut环节确保它的正确性，而不是在e2e环节，Apache Curator本身提供了zk的模拟客户端，这会简化我们的测试流程与复杂度，同时保证了ut的可测性、稳定性。我们完全可以在ut环节完成逻辑正确性的校验。您觉得呢？
Best&nbsp; wishes！
CalvinKirs


On 08/5/2020 10:41，wu shaoj<gabrywu@apache.org&gt; wrote：
I don't think so. We should use UT to guarantee the whole logic, if all the UT test passed, then most logic should be correct.
The purpose of UT is not only to cover a function for simple test, which is not enough.

---------------
我觉得UT就是用来做测试的、用来保证编写的逻辑是否正确的，不能单纯为了开发简单点，就简化UT。
其实测试分为单元测试、集成测试等几个步骤。单元测试可以尽量简化，保证基本逻辑正确，在此基础上增加集成测试，尽量测试各种场景。



On 2020/7/31, 22:36, "CalvinKirs" <acm_master@163.com&gt; wrote:

I agree with you, but some aspects need to be discussed. We should not put the test logic of zk and database in the e2e test. This is a bit cumbersome and does not meet the unit test specification. Regarding the unit test of the server module, I will come up with a plan and complete it later.


Best&nbsp; wishes！
CalvinKirs


On 07/31/2020 21:07，Yichao Yang<1048262223@qq.com&gt; wrote：
Hi all,


The purpose of this discussion is to discuss the UT specification of DS.


My thoughts are as follows:


1. About DS ut test (maybe include database)


Using framework: mockito, powermockito
Reason: The purpose of UT is to cover a function for simple test. We can not or do not need to just place all the test verification process on ut. Therefore, it is necessary to restore the essence of UT and simplify the difficulty of writing UT.
For example: If UT is used to test the whole process of an API, there will be a lot of conditions to be considered. It will be difficult to consider each condition. However, if we only direct each ut to the corresponding method and cover this method as completely as possible, the error rate of functional testing will be greatly reduced.


2. About ZK, front end test, (may be include database)


Module: put this part of the test in the E2E module


Reason: because these components are end-to-end originally, which is conducive to module division.



-------------------


各位社区同学们，大家好：


这次讨论的目的：讨论关于ds的ut编写规范。


我的想法如下：
1.关于ds ut测试（可以包含数据库）：
使用框架：Mockito，PowerMockito
目的：还原ut的本质，简化ut，但依然发挥ut的最大作用，并且降低ut编写的难度。
原因：ut的目的就是为了简单的测试覆盖一个函数，我们不可能也不需要把所有的测试验证过程都寄托于ut。所以还原ut的本质，简化ut编写难度是一个相对较好的选择。
举个例子：如果ut用来测试一个api的整个流程，其中需要考虑的条件判断会非常多，去吧每一种条件都考虑到会比较难。但是如果我们将每个ut只正对对应的方法上，并将这个方法覆盖尽可能完全，那么功能性测试的错误率会大大降低。


2.关于zk，前端测试，（可以包含数据库）
模块：将这部分的测试放在e2e模块
原因：因为这些都是数据端对端的测试，利于模块划分。


If you have any opinions or suggestions, welcome to put forward.


Best,
Yichao Yang

Re: [DISCUSS] UT specification of DS

Posted by CalvinKirs <ac...@163.com>.
+1
The summary is very good.


Best  wishes！
CalvinKirs


On 08/5/2020 18:45，Yichao Yang<10...@qq.com> wrote：
Hi,


As my first email said, we should first clarify the concept of unit test for DS and the purpose of writing unit test.


The concept of unit test is essentially unit testing, and its purpose is to verify the functional correctness of a method or module in DS as quickly and simply as possible.


To ensure the correctness of all functions and processes of DS, we can not completely rely on unit test (for example: in any company or organization, if we want to test the correctness of all functions of a software, we can't only rely on unit test, there will be more testing methods). So that's why there are integration testing, system testing, and E2E testing.


I think it can be combined with the views put forward at present.


1. Unit test:


Test scope: The current ut test scope remains unchanged, which can include some simple tests of database and ZK.


Objective: To verify the correctness of the method simply and quickly. (The simplicity here means that the logic is simplified and the code is easy to read, rather than simple to develop)


Using tools: Use mockito, powermockito, springrunner and other framework tests to write unit tests.


2. Integration test:


Test scope: Integration test of each module between single modules.


Objective: To make up for the small coverage of unit testing and meet the requirements of cross module testing between single modules.


Method: After the completion of simple UT, some integration tests can be added, which can be completed in UT module or E2E module. It is recommended to complete in UT module. Ut with the same function can be completed in a test file (or separate the folder of unit test and integration test).


Using tools: Use mockito, powermockito, springrunner and other framework tests to write integration tests.


3. E2E test:


Test scope: All end-to-end components for&nbsp;DS can be completed in this part, including front-end, ZK, worker and master, API server and log server, alert server and so on.


Purpose: The essence of the above components is end-to-end communication, and test responsibilities can be more clearly divided. It can satisfy the test between modules (servers).


Method: Subdivide modules under E2E module, such as front end, ZK, alertserver, etc


Using tools: Use the starter of each server.


I understand that the above three test methods can basically cover the DS test scenario.



----------------------


正如我的第一封邮件所说，我们应该先明确对于 ds 来说，unit test 的概念和编写 unit test 想要达到的目的。
unit test 概念本质上就是单元测试，目的也是为了最快最简单的验证 ds 中一个方法或者模块的功能正确性。
保障 ds 全部功能正确性和整体流程正确性并不能完全依赖 unit test（举个例子：在任何公司或者组织，如果想要测试一个软件的全部功能正确性，是不可能只依赖于 unit test 的，还会有更多的测试方法）。所以这也是为什么会有集成测试，系统测试，e2e测试。


我觉得可以结合目前提出的观点：
1.单元测试：
测试范围：目前的 ut 测试范围不变，其可以包含数据库以及 zk 的一些`简单`测试
目的：简单并且快速验证方法正确性（这里的简单是指逻辑简化，代码易读，而不是指开发简单）；
使用工具：使用 mockito，powermockito，springrunner 等框架测试编写单元测试


2.集成测试：
测试范围：单个 module 之间的各个模块的集成测试
目的：弥补单元测试覆盖范围小的不足，满足单个 module 之间跨模块测试的需求
方式：可以在简单 ut 完成后，加入一些集成测试，这部分可以在 ut 模块完成，也可以在 e2e 模块完成，推荐在 ut 模块中完成，相同功能ut可以在一个test文件中完成（或者单独划分 unit test 和集成测试的文件夹也可以）
使用工具：使用 mockito，powermockito，springrunner 等框架测试编写集成测试


3.e2e测试：
测试范围：所有和 ds 涉及到端对端的都可以在这部分完成，包含前端，zk，worker 和 master，api server 和 log server，alertserver 的端对端等。
目的：上述组件之间本质就是端对端的通信方式，并且可以更清楚的划分测试职责，更清晰并且细致的测试到每一个功能。满足跨 module（server）之间的测试。
方式：e2e module 下细分 module，比如前端，zk，alertserver 等
使用工具：使用各个 server 的 starter测试。



If you have any better suggestions or opinions, welcome to put forward~


Best,
Yichao Yang




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "dev"                                                                                    <acm_master@163.com&gt;;
发送时间:&nbsp;2020年8月5日(星期三) 中午11:00
收件人:&nbsp;"dev@dolphinscheduler.apache.org"<dev@dolphinscheduler.apache.org&gt;;

主题:&nbsp;Re: [DISCUSS] UT specification of DS



I think you have misunderstood what I mean. I personally think that zk plays a very important role in our project, so I think that zk must be reflected in the unit test. Currently we use the stand-alone version of zk, which will increase us The complexity and instability of the test, so a better approach is to use mock zk. As for whether a real environment needs to be used in e2e, I personally think it is a bit redundant. After all, the simulated client has already done this part of the work.
————————
我想您误解了我的意思，我认为，zk在我们的项目中承担了相当重要的角色，所以保证这块的正确性非常重要，ut本身应该保证它的一个简单性，可测性。这点我很同意@yangyichao同学的观点，我们应该在ut环节确保它的正确性，而不是在e2e环节，Apache Curator本身提供了zk的模拟客户端，这会简化我们的测试流程与复杂度，同时保证了ut的可测性、稳定性。我们完全可以在ut环节完成逻辑正确性的校验。您觉得呢？
Best&nbsp; wishes！
CalvinKirs


On 08/5/2020 10:41，wu shaoj<gabrywu@apache.org&gt; wrote：
I don't think so. We should use UT to guarantee the whole logic, if all the UT test passed, then most logic should be correct.
The purpose of UT is not only to cover a function for simple test, which is not enough.

---------------
我觉得UT就是用来做测试的、用来保证编写的逻辑是否正确的，不能单纯为了开发简单点，就简化UT。
其实测试分为单元测试、集成测试等几个步骤。单元测试可以尽量简化，保证基本逻辑正确，在此基础上增加集成测试，尽量测试各种场景。



On 2020/7/31, 22:36, "CalvinKirs" <acm_master@163.com&gt; wrote:

I agree with you, but some aspects need to be discussed. We should not put the test logic of zk and database in the e2e test. This is a bit cumbersome and does not meet the unit test specification. Regarding the unit test of the server module, I will come up with a plan and complete it later.


Best&nbsp; wishes！
CalvinKirs


On 07/31/2020 21:07，Yichao Yang<1048262223@qq.com&gt; wrote：
Hi all,


The purpose of this discussion is to discuss the UT specification of DS.


My thoughts are as follows:


1. About DS ut test (maybe include database)


Using framework: mockito, powermockito
Reason: The purpose of UT is to cover a function for simple test. We can not or do not need to just place all the test verification process on ut. Therefore, it is necessary to restore the essence of UT and simplify the difficulty of writing UT.
For example: If UT is used to test the whole process of an API, there will be a lot of conditions to be considered. It will be difficult to consider each condition. However, if we only direct each ut to the corresponding method and cover this method as completely as possible, the error rate of functional testing will be greatly reduced.


2. About ZK, front end test, (may be include database)


Module: put this part of the test in the E2E module


Reason: because these components are end-to-end originally, which is conducive to module division.



-------------------


各位社区同学们，大家好：


这次讨论的目的：讨论关于ds的ut编写规范。


我的想法如下：
1.关于ds ut测试（可以包含数据库）：
使用框架：Mockito，PowerMockito
目的：还原ut的本质，简化ut，但依然发挥ut的最大作用，并且降低ut编写的难度。
原因：ut的目的就是为了简单的测试覆盖一个函数，我们不可能也不需要把所有的测试验证过程都寄托于ut。所以还原ut的本质，简化ut编写难度是一个相对较好的选择。
举个例子：如果ut用来测试一个api的整个流程，其中需要考虑的条件判断会非常多，去吧每一种条件都考虑到会比较难。但是如果我们将每个ut只正对对应的方法上，并将这个方法覆盖尽可能完全，那么功能性测试的错误率会大大降低。


2.关于zk，前端测试，（可以包含数据库）
模块：将这部分的测试放在e2e模块
原因：因为这些都是数据端对端的测试，利于模块划分。


If you have any opinions or suggestions, welcome to put forward.


Best,
Yichao Yang

Re: Re： [DISCUSS] UT specification of DS

Posted by wu shaoj <ga...@apache.org>.
Yes, Integration test is absolutly needed. UT is the basic test method, which should be designed to cover the most case. If some bugs found, the first thing to do is to design the UT or Integration test, not to fix only


On 2020/8/5, 18:46, "Yichao Yang" <10...@qq.com> wrote:

    Hi,


    As my first email said, we should first clarify the concept of unit test for DS and the purpose of writing unit test.


    The concept of unit test is essentially unit testing, and its purpose is to verify the functional correctness of a method or module in DS as quickly and simply as possible.


    To ensure the correctness of all functions and processes of DS, we can not completely rely on unit test (for example: in any company or organization, if we want to test the correctness of all functions of a software, we can't only rely on unit test, there will be more testing methods). So that's why there are integration testing, system testing, and E2E testing.


    I think it can be combined with the views put forward at present.


    1. Unit test:


    Test scope: The current ut test scope remains unchanged, which can include some simple tests of database and ZK.


    Objective: To verify the correctness of the method simply and quickly. (The simplicity here means that the logic is simplified and the code is easy to read, rather than simple to develop)


    Using tools: Use mockito, powermockito, springrunner and other framework tests to write unit tests.


    2. Integration test:


    Test scope: Integration test of each module between single modules.


    Objective: To make up for the small coverage of unit testing and meet the requirements of cross module testing between single modules.


    Method: After the completion of simple UT, some integration tests can be added, which can be completed in UT module or E2E module. It is recommended to complete in UT module. Ut with the same function can be completed in a test file (or separate the folder of unit test and integration test).


    Using tools: Use mockito, powermockito, springrunner and other framework tests to write integration tests.


    3. E2E test:


    Test scope: All end-to-end components for&nbsp;DS can be completed in this part, including front-end, ZK, worker and master, API server and log server, alert server and so on.


    Purpose: The essence of the above components is end-to-end communication, and test responsibilities can be more clearly divided. It can satisfy the test between modules (servers).


    Method: Subdivide modules under E2E module, such as front end, ZK, alertserver, etc


    Using tools: Use the starter of each server.


    I understand that the above three test methods can basically cover the DS test scenario.



    ----------------------


    正如我的第一封邮件所说，我们应该先明确对于 ds 来说，unit test 的概念和编写 unit test 想要达到的目的。
    unit test 概念本质上就是单元测试，目的也是为了最快最简单的验证 ds 中一个方法或者模块的功能正确性。
    保障 ds 全部功能正确性和整体流程正确性并不能完全依赖 unit test（举个例子：在任何公司或者组织，如果想要测试一个软件的全部功能正确性，是不可能只依赖于 unit test 的，还会有更多的测试方法）。所以这也是为什么会有集成测试，系统测试，e2e测试。


    我觉得可以结合目前提出的观点：
    1.单元测试：
    测试范围：目前的 ut 测试范围不变，其可以包含数据库以及 zk 的一些`简单`测试
    目的：简单并且快速验证方法正确性（这里的简单是指逻辑简化，代码易读，而不是指开发简单）；
    使用工具：使用 mockito，powermockito，springrunner 等框架测试编写单元测试


    2.集成测试：
    测试范围：单个 module 之间的各个模块的集成测试
    目的：弥补单元测试覆盖范围小的不足，满足单个 module 之间跨模块测试的需求
    方式：可以在简单 ut 完成后，加入一些集成测试，这部分可以在 ut 模块完成，也可以在 e2e 模块完成，推荐在 ut 模块中完成，相同功能ut可以在一个test文件中完成（或者单独划分 unit test 和集成测试的文件夹也可以）
    使用工具：使用 mockito，powermockito，springrunner 等框架测试编写集成测试


    3.e2e测试：
    测试范围：所有和 ds 涉及到端对端的都可以在这部分完成，包含前端，zk，worker 和 master，api server 和 log server，alertserver 的端对端等。
    目的：上述组件之间本质就是端对端的通信方式，并且可以更清楚的划分测试职责，更清晰并且细致的测试到每一个功能。满足跨 module（server）之间的测试。
    方式：e2e module 下细分 module，比如前端，zk，alertserver 等
    使用工具：使用各个 server 的 starter测试。



    If you have any better suggestions or opinions, welcome to put forward~


    Best,
    Yichao Yang




    ------------------&nbsp;原始邮件&nbsp;------------------
    发件人:                                                                                                                        "dev"                                                                                    <acm_master@163.com&gt;;
    发送时间:&nbsp;2020年8月5日(星期三) 中午11:00
    收件人:&nbsp;"dev@dolphinscheduler.apache.org"<dev@dolphinscheduler.apache.org&gt;;

    主题:&nbsp;Re: [DISCUSS] UT specification of DS



    I think you have misunderstood what I mean. I personally think that zk plays a very important role in our project, so I think that zk must be reflected in the unit test. Currently we use the stand-alone version of zk, which will increase us The complexity and instability of the test, so a better approach is to use mock zk. As for whether a real environment needs to be used in e2e, I personally think it is a bit redundant. After all, the simulated client has already done this part of the work.
    ————————
    我想您误解了我的意思，我认为，zk在我们的项目中承担了相当重要的角色，所以保证这块的正确性非常重要，ut本身应该保证它的一个简单性，可测性。这点我很同意@yangyichao同学的观点，我们应该在ut环节确保它的正确性，而不是在e2e环节，Apache Curator本身提供了zk的模拟客户端，这会简化我们的测试流程与复杂度，同时保证了ut的可测性、稳定性。我们完全可以在ut环节完成逻辑正确性的校验。您觉得呢？
    Best&nbsp; wishes！
    CalvinKirs


    On 08/5/2020 10:41，wu shaoj<gabrywu@apache.org&gt; wrote：
    I don't think so. We should use UT to guarantee the whole logic, if all the UT test passed, then most logic should be correct.
    The purpose of UT is not only to cover a function for simple test, which is not enough.

    ---------------
    我觉得UT就是用来做测试的、用来保证编写的逻辑是否正确的，不能单纯为了开发简单点，就简化UT。
    其实测试分为单元测试、集成测试等几个步骤。单元测试可以尽量简化，保证基本逻辑正确，在此基础上增加集成测试，尽量测试各种场景。



    On 2020/7/31, 22:36, "CalvinKirs" <acm_master@163.com&gt; wrote:

    I agree with you, but some aspects need to be discussed. We should not put the test logic of zk and database in the e2e test. This is a bit cumbersome and does not meet the unit test specification. Regarding the unit test of the server module, I will come up with a plan and complete it later.


    Best&nbsp; wishes！
    CalvinKirs


    On 07/31/2020 21:07，Yichao Yang<1048262223@qq.com&gt; wrote：
    Hi all,


    The purpose of this discussion is to discuss the UT specification of DS.


    My thoughts are as follows:


    1. About DS ut test (maybe include database)


    Using framework: mockito, powermockito
    Reason: The purpose of UT is to cover a function for simple test. We can not or do not need to just place all the test verification process on ut. Therefore, it is necessary to restore the essence of UT and simplify the difficulty of writing UT.
    For example: If UT is used to test the whole process of an API, there will be a lot of conditions to be considered. It will be difficult to consider each condition. However, if we only direct each ut to the corresponding method and cover this method as completely as possible, the error rate of functional testing will be greatly reduced.


    2. About ZK, front end test, (may be include database)


    Module: put this part of the test in the E2E module


    Reason: because these components are end-to-end originally, which is conducive to module division.



    -------------------


    各位社区同学们，大家好：


    这次讨论的目的：讨论关于ds的ut编写规范。


    我的想法如下：
    1.关于ds ut测试（可以包含数据库）：
    使用框架：Mockito，PowerMockito
    目的：还原ut的本质，简化ut，但依然发挥ut的最大作用，并且降低ut编写的难度。
    原因：ut的目的就是为了简单的测试覆盖一个函数，我们不可能也不需要把所有的测试验证过程都寄托于ut。所以还原ut的本质，简化ut编写难度是一个相对较好的选择。
    举个例子：如果ut用来测试一个api的整个流程，其中需要考虑的条件判断会非常多，去吧每一种条件都考虑到会比较难。但是如果我们将每个ut只正对对应的方法上，并将这个方法覆盖尽可能完全，那么功能性测试的错误率会大大降低。


    2.关于zk，前端测试，（可以包含数据库）
    模块：将这部分的测试放在e2e模块
    原因：因为这些都是数据端对端的测试，利于模块划分。


    If you have any opinions or suggestions, welcome to put forward.


    Best,
    Yichao Yang