You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@skywalking.apache.org by Sheng Wu <wu...@apache.org> on 2020/03/30 09:28:26 UTC
[DISCUSSION] Meaning of test coverage, how/whether should we improve it?
Hi Dev Team.
I am looking at the Coverage dashboard[1] this afternoon.
26%+- seems a static status of SkyWalking main repo. Seems not "good"(for
normal people). Especially the coverage is caused by storage/core/receiver,
less than 15% usually.
But at the same time, like my twitter description[2], we have 101(excluded
3 CI tasks) e2e and agent plugin tests. These are exact covering
the storage/core/receiver.
From my understanding, besides these 3 test requirements[3][4][5], the e2e
tests exactly are covering storage/core/receiver.
*Logically, we have a very high coverage rate, which matches my
feeling (maybe others too)*.
The thing bothering me is, how we should describe this to a new
contributor? Such as
1. What parts of contributions should require UT coverage?
2. What e2e should be added?
3. Where is the balance and tradeoff between (1) and (2)?
I was trying to add more tests for core module(class
MetricsStreamProcessor#add), and nearly success yesterday night, but when I
was reviewing my own test case, I was asking myself, what are the points of
those UTs? I was mocking nearly all dependencies, so the logic is in the
tested and untested at the same time. Then I decided to revert and delete
those tests.
WDYT? Open my mind to hear more ideas.
[1] https://codecov.io/gh/apache/skywalking
[2] https://twitter.com/wusheng1108/status/1243798832690782211
[3] https://github.com/apache/skywalking/issues/4455
[4] https://github.com/apache/skywalking/issues/4530
[5] https://github.com/apache/skywalking/issues/4576
Sheng Wu 吴晟
Twitter, wusheng1108
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by Sheng Wu <wu...@gmail.com>.
Juan Pan <pa...@apache.org>于2020年3月30日 周一下午6:59写道：
> Hi,
> I understand your concern. From my experience as PPMC of Apache
> ShardingSphere, both UT and e2e are necessary.
>
>
> UT focuses on a specific module, i.e., the part of the whole project,
> especially like SW core module,
> which is the most important and less changed. Those UTs guarantee each
> small part of the project would run well.
> Someday, if one contributor modifies them, the corresponding UTs will help
> check what he changed.
> In this condition, the contributor does not need to care about other
> modules or e2e, which simplifies his concern.
Basically, current core structure is, core is a driver and linked,
actually. The purpose of those codes are building a straming process graph.
If we set the UT for core, we need to mock at least, 10-15 services, but do
nothing. I can't see the meaning of building a 500-600 lines mock codes.
This topic is more about meaningless UT, better for coverage, but not for
feature.
You could take a look of the class I mentioned, and try to write a case,
you will see what I mean. It will be fun.😃
>
>
> On the other hand, e2e will connect almost all of the modules together.
> We want to make sure the whole project can work well, which is not the
> responsibility of UTs.
>
>
> In a word, UTs focus the local, and e2e keeps the watch on the whole
> project, IMO.
>
>
> I’d like to listen to other voices from the community. :)
>
>
> Juan Pan (Trista)
>
> Senior DBA & PPMC of Apache ShardingSphere(Incubating)
> E-mail: panjuan@apache.org
>
>
>
>
> On 03/30/2020 17:28，Sheng Wu<wu...@apache.org> wrote：
> Hi Dev Team.
>
> I am looking at the Coverage dashboard[1] this afternoon.
> 26%+- seems a static status of SkyWalking main repo. Seems not "good"(for
> normal people). Especially the coverage is caused by storage/core/receiver,
> less than 15% usually.
> But at the same time, like my twitter description[2], we have 101(excluded
> 3 CI tasks) e2e and agent plugin tests. These are exact covering
> the storage/core/receiver.
>
> From my understanding, besides these 3 test requirements[3][4][5], the e2e
> tests exactly are covering storage/core/receiver.
> *Logically, we have a very high coverage rate, which matches my
> feeling (maybe others too)*.
>
> The thing bothering me is, how we should describe this to a new
> contributor? Such as
> 1. What parts of contributions should require UT coverage?
> 2. What e2e should be added?
> 3. Where is the balance and tradeoff between (1) and (2)?
>
> I was trying to add more tests for core module(class
> MetricsStreamProcessor#add), and nearly success yesterday night, but when I
> was reviewing my own test case, I was asking myself, what are the points of
> those UTs? I was mocking nearly all dependencies, so the logic is in the
> tested and untested at the same time. Then I decided to revert and delete
> those tests.
>
> WDYT? Open my mind to hear more ideas.
>
> [1] https://codecov.io/gh/apache/skywalking
> [2] https://twitter.com/wusheng1108/status/1243798832690782211
> [3] https://github.com/apache/skywalking/issues/4455
> [4] https://github.com/apache/skywalking/issues/4530
> [5] https://github.com/apache/skywalking/issues/4576
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
--
Sheng Wu 吴晟
Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108
Re:[DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by Juan Pan <pa...@apache.org>.
Hi,
I understand your concern. From my experience as PPMC of Apache ShardingSphere, both UT and e2e are necessary.
UT focuses on a specific module, i.e., the part of the whole project, especially like SW core module,
which is the most important and less changed. Those UTs guarantee each small part of the project would run well.
Someday, if one contributor modifies them, the corresponding UTs will help check what he changed.
In this condition, the contributor does not need to care about other modules or e2e, which simplifies his concern.
On the other hand, e2e will connect almost all of the modules together.
We want to make sure the whole project can work well, which is not the responsibility of UTs.
In a word, UTs focus the local, and e2e keeps the watch on the whole project, IMO.
I’d like to listen to other voices from the community. :)
Juan Pan (Trista)
Senior DBA & PPMC of Apache ShardingSphere(Incubating)
E-mail: panjuan@apache.org
On 03/30/2020 17:28，Sheng Wu<wu...@apache.org> wrote：
Hi Dev Team.
I am looking at the Coverage dashboard[1] this afternoon.
26%+- seems a static status of SkyWalking main repo. Seems not "good"(for
normal people). Especially the coverage is caused by storage/core/receiver,
less than 15% usually.
But at the same time, like my twitter description[2], we have 101(excluded
3 CI tasks) e2e and agent plugin tests. These are exact covering
the storage/core/receiver.
From my understanding, besides these 3 test requirements[3][4][5], the e2e
tests exactly are covering storage/core/receiver.
*Logically, we have a very high coverage rate, which matches my
feeling (maybe others too)*.
The thing bothering me is, how we should describe this to a new
contributor? Such as
1. What parts of contributions should require UT coverage?
2. What e2e should be added?
3. Where is the balance and tradeoff between (1) and (2)?
I was trying to add more tests for core module(class
MetricsStreamProcessor#add), and nearly success yesterday night, but when I
was reviewing my own test case, I was asking myself, what are the points of
those UTs? I was mocking nearly all dependencies, so the logic is in the
tested and untested at the same time. Then I decided to revert and delete
those tests.
WDYT? Open my mind to hear more ideas.
[1] https://codecov.io/gh/apache/skywalking
[2] https://twitter.com/wusheng1108/status/1243798832690782211
[3] https://github.com/apache/skywalking/issues/4455
[4] https://github.com/apache/skywalking/issues/4530
[5] https://github.com/apache/skywalking/issues/4576
Sheng Wu 吴晟
Twitter, wusheng1108
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by Sheng Wu <wu...@apache.org>.
I think the key is not how gracefully shutdown, it is when should shutdown,
and how to trigger it.
kezhenxu94@163 <ke...@163.com>于2020年3月31日 周二下午10:18写道：
> Jacoco agent writes the execution data to the files, if the targeted JVM
> does not shut down gracefully, the file is corrupted and cannot be
> analyzed, see
> https://github.com/jacoco/jacoco/issues/901#issuecomment-508442946
>
> > On Mar 31, 2020, at 21:43, Sheng Wu <wu...@gmail.com> wrote:
> >
> > Hi Zhenxu
> >
> > Could you share about why is gracefully shutdown important?
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > kezhenxu94@163 <ke...@163.com> 于2020年3月31日周二 下午9:36写道：
> >
> >> I did a quick POC locally, it turns out the solution is possible, the
> code
> >> lines executed in the E2E can be tracked and analyzed [0] (I run a
> >> simple/JDK E2E test, and the coverage increase ~3.42%), I’ve also tagged
> >> the coverage with “E2E” [1].
> >>
> >> The next step would be chained them up in the workflow of GitHub
> Actions,
> >> but one problems as follow should be fixed before it is ready:
> >>
> >> Gracefully shutdown mechanism of every component, including backend, web
> >> UI, service provider of E2E, among which the last two are easy to fix,
> as
> >> for the backend, I’ve tried but did not successful in a normal way, will
> >> take a detailed look when I get some time
> >>
> >>
> >> [0]
> >>
> https://codecov.io/github/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9
> >> [1]
> >>
> https://codecov.io/gh/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9/build
> >>
> >>> On Mar 31, 2020, at 07:56, Sheng Wu <wu...@gmail.com> wrote:
> >>>
> >>> kezhenxu94@163 <ke...@163.com> 于2020年3月30日周一 下午11:59写道：
> >>>
> >>>>
> >>>>
> >>>>> On Mar 30, 2020, at 21:58, Sheng Wu <wu...@gmail.com>
> wrote:
> >>>>>
> >>>>> Sheng Wu <wu...@gmail.com>于2020年3月30日 周一下午9:55写道：
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> kezhenxu94@apache <ke...@apache.org>于2020年3月30日 周一下午9:11写道：
> >>>>>>
> >>>>>>> In essence, code coverage is a measurement of the code lines that
> >>>> **have
> >>>>>>> been executed** during the tests, and the code lines that are
> >> executed
> >>>> in
> >>>>>>> the tests are **covered** actually, no matter whether they are
> >> counted
> >>>> in
> >>>>>>> the coverage rate (~26%), my idea is to let the coverage tool
> >>>> (Cobertura in
> >>>>>>> our case) **notice** this and count them in the final coverage, and
> >>>> IIRC,
> >>>>>>> Jacoco provides an Java agent that does this exactly, I’ll do some
> >>>> research
> >>>>>>> on this and forward the conclusion here, or open pull requests
> >>>> directly if
> >>>>>>> it works overall.
> >>>>>>
> >>>>>>
> >>>>>> It must be an agent, this is the only case it could measure the
> lines.
> >>>>>> If this really could work in this way, should be very interesting.
> >>>>>>
> >>>>>
> >>>>> My concern will be, how to aggregate the coverage across 104 GitHub
> >>>>> Actions.
> >>>>> It will be hard to have all result files in one place.
> >>>>>
> >>>>> Zhenxu
> >>>>> WDYT?
> >>>>>
> >>>>
> >>>> That’s not a concern, Codecov is able to merge them automatically
> >>>>
> >>>
> >>> Good to know :)
> >>>
> >>>
> >>> Sheng Wu 吴晟
> >>> Twitter, wusheng1108
> >>>
> >>>
> >>>>
> >>>>> Sheng
> >>>>>
> >>>>>
> >>>>>> Sheng
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> GitHub @kezhenxu94
> >>>>>>> Apache SkyWalking, Apache Dubbo
> >>>>>>>
> >>>>>>>> On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
> >>>>>>>>
> >>>>>>>> Hi Dev Team.
> >>>>>>>>
> >>>>>>>> I am looking at the Coverage dashboard[1] this afternoon.
> >>>>>>>> 26%+- seems a static status of SkyWalking main repo. Seems not
> >>>>>>> "good"(for
> >>>>>>>> normal people). Especially the coverage is caused by
> >>>>>>> storage/core/receiver,
> >>>>>>>> less than 15% usually.
> >>>>>>>> But at the same time, like my twitter description[2], we have
> >>>>>>> 101(excluded
> >>>>>>>> 3 CI tasks) e2e and agent plugin tests. These are exact covering
> >>>>>>>> the storage/core/receiver.
> >>>>>>>>
> >>>>>>>> From my understanding, besides these 3 test requirements[3][4][5],
> >> the
> >>>>>>> e2e
> >>>>>>>> tests exactly are covering storage/core/receiver.
> >>>>>>>> *Logically, we have a very high coverage rate, which matches my
> >>>>>>>> feeling (maybe others too)*.
> >>>>>>>>
> >>>>>>>> The thing bothering me is, how we should describe this to a new
> >>>>>>>> contributor? Such as
> >>>>>>>> 1. What parts of contributions should require UT coverage?
> >>>>>>>> 2. What e2e should be added?
> >>>>>>>> 3. Where is the balance and tradeoff between (1) and (2)?
> >>>>>>>>
> >>>>>>>> I was trying to add more tests for core module(class
> >>>>>>>> MetricsStreamProcessor#add), and nearly success yesterday night,
> but
> >>>>>>> when I
> >>>>>>>> was reviewing my own test case, I was asking myself, what are the
> >>>>>>> points of
> >>>>>>>> those UTs? I was mocking nearly all dependencies, so the logic is
> in
> >>>> the
> >>>>>>>> tested and untested at the same time. Then I decided to revert and
> >>>>>>> delete
> >>>>>>>> those tests.
> >>>>>>>>
> >>>>>>>> WDYT? Open my mind to hear more ideas.
> >>>>>>>>
> >>>>>>>> [1] https://codecov.io/gh/apache/skywalking
> >>>>>>>> [2] https://twitter.com/wusheng1108/status/1243798832690782211
> >>>>>>>> [3] https://github.com/apache/skywalking/issues/4455
> >>>>>>>> [4] https://github.com/apache/skywalking/issues/4530
> >>>>>>>> [5] https://github.com/apache/skywalking/issues/4576
> >>>>>>>>
> >>>>>>>> Sheng Wu 吴晟
> >>>>>>>> Twitter, wusheng1108
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>> Sheng Wu 吴晟
> >>>>>>
> >>>>>> Apache SkyWalking
> >>>>>> Apache Incubator
> >>>>>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >>>>>> Zipkin
> >>>>>> Twitter, wusheng1108
> >>>>>>
> >>>>> --
> >>>>> Sheng Wu 吴晟
> >>>>>
> >>>>> Apache SkyWalking
> >>>>> Apache Incubator
> >>>>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >>>>> Zipkin
> >>>>> Twitter, wusheng1108
> >>
> >>
> >>
>
> --
Sheng Wu 吴晟
Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by "kezhenxu94@163" <ke...@163.com>.
Jacoco agent writes the execution data to the files, if the targeted JVM does not shut down gracefully, the file is corrupted and cannot be analyzed, see https://github.com/jacoco/jacoco/issues/901#issuecomment-508442946
> On Mar 31, 2020, at 21:43, Sheng Wu <wu...@gmail.com> wrote:
>
> Hi Zhenxu
>
> Could you share about why is gracefully shutdown important?
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
> kezhenxu94@163 <ke...@163.com> 于2020年3月31日周二 下午9:36写道：
>
>> I did a quick POC locally, it turns out the solution is possible, the code
>> lines executed in the E2E can be tracked and analyzed [0] (I run a
>> simple/JDK E2E test, and the coverage increase ~3.42%), I’ve also tagged
>> the coverage with “E2E” [1].
>>
>> The next step would be chained them up in the workflow of GitHub Actions,
>> but one problems as follow should be fixed before it is ready:
>>
>> Gracefully shutdown mechanism of every component, including backend, web
>> UI, service provider of E2E, among which the last two are easy to fix, as
>> for the backend, I’ve tried but did not successful in a normal way, will
>> take a detailed look when I get some time
>>
>>
>> [0]
>> https://codecov.io/github/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9
>> [1]
>> https://codecov.io/gh/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9/build
>>
>>> On Mar 31, 2020, at 07:56, Sheng Wu <wu...@gmail.com> wrote:
>>>
>>> kezhenxu94@163 <ke...@163.com> 于2020年3月30日周一 下午11:59写道：
>>>
>>>>
>>>>
>>>>> On Mar 30, 2020, at 21:58, Sheng Wu <wu...@gmail.com> wrote:
>>>>>
>>>>> Sheng Wu <wu...@gmail.com>于2020年3月30日 周一下午9:55写道：
>>>>>
>>>>>>
>>>>>>
>>>>>> kezhenxu94@apache <ke...@apache.org>于2020年3月30日 周一下午9:11写道：
>>>>>>
>>>>>>> In essence, code coverage is a measurement of the code lines that
>>>> **have
>>>>>>> been executed** during the tests, and the code lines that are
>> executed
>>>> in
>>>>>>> the tests are **covered** actually, no matter whether they are
>> counted
>>>> in
>>>>>>> the coverage rate (~26%), my idea is to let the coverage tool
>>>> (Cobertura in
>>>>>>> our case) **notice** this and count them in the final coverage, and
>>>> IIRC,
>>>>>>> Jacoco provides an Java agent that does this exactly, I’ll do some
>>>> research
>>>>>>> on this and forward the conclusion here, or open pull requests
>>>> directly if
>>>>>>> it works overall.
>>>>>>
>>>>>>
>>>>>> It must be an agent, this is the only case it could measure the lines.
>>>>>> If this really could work in this way, should be very interesting.
>>>>>>
>>>>>
>>>>> My concern will be, how to aggregate the coverage across 104 GitHub
>>>>> Actions.
>>>>> It will be hard to have all result files in one place.
>>>>>
>>>>> Zhenxu
>>>>> WDYT?
>>>>>
>>>>
>>>> That’s not a concern, Codecov is able to merge them automatically
>>>>
>>>
>>> Good to know :)
>>>
>>>
>>> Sheng Wu 吴晟
>>> Twitter, wusheng1108
>>>
>>>
>>>>
>>>>> Sheng
>>>>>
>>>>>
>>>>>> Sheng
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> GitHub @kezhenxu94
>>>>>>> Apache SkyWalking, Apache Dubbo
>>>>>>>
>>>>>>>> On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
>>>>>>>>
>>>>>>>> Hi Dev Team.
>>>>>>>>
>>>>>>>> I am looking at the Coverage dashboard[1] this afternoon.
>>>>>>>> 26%+- seems a static status of SkyWalking main repo. Seems not
>>>>>>> "good"(for
>>>>>>>> normal people). Especially the coverage is caused by
>>>>>>> storage/core/receiver,
>>>>>>>> less than 15% usually.
>>>>>>>> But at the same time, like my twitter description[2], we have
>>>>>>> 101(excluded
>>>>>>>> 3 CI tasks) e2e and agent plugin tests. These are exact covering
>>>>>>>> the storage/core/receiver.
>>>>>>>>
>>>>>>>> From my understanding, besides these 3 test requirements[3][4][5],
>> the
>>>>>>> e2e
>>>>>>>> tests exactly are covering storage/core/receiver.
>>>>>>>> *Logically, we have a very high coverage rate, which matches my
>>>>>>>> feeling (maybe others too)*.
>>>>>>>>
>>>>>>>> The thing bothering me is, how we should describe this to a new
>>>>>>>> contributor? Such as
>>>>>>>> 1. What parts of contributions should require UT coverage?
>>>>>>>> 2. What e2e should be added?
>>>>>>>> 3. Where is the balance and tradeoff between (1) and (2)?
>>>>>>>>
>>>>>>>> I was trying to add more tests for core module(class
>>>>>>>> MetricsStreamProcessor#add), and nearly success yesterday night, but
>>>>>>> when I
>>>>>>>> was reviewing my own test case, I was asking myself, what are the
>>>>>>> points of
>>>>>>>> those UTs? I was mocking nearly all dependencies, so the logic is in
>>>> the
>>>>>>>> tested and untested at the same time. Then I decided to revert and
>>>>>>> delete
>>>>>>>> those tests.
>>>>>>>>
>>>>>>>> WDYT? Open my mind to hear more ideas.
>>>>>>>>
>>>>>>>> [1] https://codecov.io/gh/apache/skywalking
>>>>>>>> [2] https://twitter.com/wusheng1108/status/1243798832690782211
>>>>>>>> [3] https://github.com/apache/skywalking/issues/4455
>>>>>>>> [4] https://github.com/apache/skywalking/issues/4530
>>>>>>>> [5] https://github.com/apache/skywalking/issues/4576
>>>>>>>>
>>>>>>>> Sheng Wu 吴晟
>>>>>>>> Twitter, wusheng1108
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>> Sheng Wu 吴晟
>>>>>>
>>>>>> Apache SkyWalking
>>>>>> Apache Incubator
>>>>>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
>>>>>> Zipkin
>>>>>> Twitter, wusheng1108
>>>>>>
>>>>> --
>>>>> Sheng Wu 吴晟
>>>>>
>>>>> Apache SkyWalking
>>>>> Apache Incubator
>>>>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
>>>>> Zipkin
>>>>> Twitter, wusheng1108
>>
>>
>>
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by Sheng Wu <wu...@gmail.com>.
Hi Zhenxu
Could you share about why is gracefully shutdown important?
Sheng Wu 吴晟
Twitter, wusheng1108
kezhenxu94@163 <ke...@163.com> 于2020年3月31日周二 下午9:36写道：
> I did a quick POC locally, it turns out the solution is possible, the code
> lines executed in the E2E can be tracked and analyzed [0] (I run a
> simple/JDK E2E test, and the coverage increase ~3.42%), I’ve also tagged
> the coverage with “E2E” [1].
>
> The next step would be chained them up in the workflow of GitHub Actions,
> but one problems as follow should be fixed before it is ready:
>
> Gracefully shutdown mechanism of every component, including backend, web
> UI, service provider of E2E, among which the last two are easy to fix, as
> for the backend, I’ve tried but did not successful in a normal way, will
> take a detailed look when I get some time
>
>
> [0]
> https://codecov.io/github/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9
> [1]
> https://codecov.io/gh/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9/build
>
> > On Mar 31, 2020, at 07:56, Sheng Wu <wu...@gmail.com> wrote:
> >
> > kezhenxu94@163 <ke...@163.com> 于2020年3月30日周一 下午11:59写道：
> >
> >>
> >>
> >>> On Mar 30, 2020, at 21:58, Sheng Wu <wu...@gmail.com> wrote:
> >>>
> >>> Sheng Wu <wu...@gmail.com>于2020年3月30日 周一下午9:55写道：
> >>>
> >>>>
> >>>>
> >>>> kezhenxu94@apache <ke...@apache.org>于2020年3月30日 周一下午9:11写道：
> >>>>
> >>>>> In essence, code coverage is a measurement of the code lines that
> >> **have
> >>>>> been executed** during the tests, and the code lines that are
> executed
> >> in
> >>>>> the tests are **covered** actually, no matter whether they are
> counted
> >> in
> >>>>> the coverage rate (~26%), my idea is to let the coverage tool
> >> (Cobertura in
> >>>>> our case) **notice** this and count them in the final coverage, and
> >> IIRC,
> >>>>> Jacoco provides an Java agent that does this exactly, I’ll do some
> >> research
> >>>>> on this and forward the conclusion here, or open pull requests
> >> directly if
> >>>>> it works overall.
> >>>>
> >>>>
> >>>> It must be an agent, this is the only case it could measure the lines.
> >>>> If this really could work in this way, should be very interesting.
> >>>>
> >>>
> >>> My concern will be, how to aggregate the coverage across 104 GitHub
> >>> Actions.
> >>> It will be hard to have all result files in one place.
> >>>
> >>> Zhenxu
> >>> WDYT?
> >>>
> >>
> >> That’s not a concern, Codecov is able to merge them automatically
> >>
> >
> > Good to know :)
> >
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> >>
> >>> Sheng
> >>>
> >>>
> >>>> Sheng
> >>>>
> >>>>
> >>>>>
> >>>>>
> >>>>> GitHub @kezhenxu94
> >>>>> Apache SkyWalking, Apache Dubbo
> >>>>>
> >>>>>> On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
> >>>>>>
> >>>>>> Hi Dev Team.
> >>>>>>
> >>>>>> I am looking at the Coverage dashboard[1] this afternoon.
> >>>>>> 26%+- seems a static status of SkyWalking main repo. Seems not
> >>>>> "good"(for
> >>>>>> normal people). Especially the coverage is caused by
> >>>>> storage/core/receiver,
> >>>>>> less than 15% usually.
> >>>>>> But at the same time, like my twitter description[2], we have
> >>>>> 101(excluded
> >>>>>> 3 CI tasks) e2e and agent plugin tests. These are exact covering
> >>>>>> the storage/core/receiver.
> >>>>>>
> >>>>>> From my understanding, besides these 3 test requirements[3][4][5],
> the
> >>>>> e2e
> >>>>>> tests exactly are covering storage/core/receiver.
> >>>>>> *Logically, we have a very high coverage rate, which matches my
> >>>>>> feeling (maybe others too)*.
> >>>>>>
> >>>>>> The thing bothering me is, how we should describe this to a new
> >>>>>> contributor? Such as
> >>>>>> 1. What parts of contributions should require UT coverage?
> >>>>>> 2. What e2e should be added?
> >>>>>> 3. Where is the balance and tradeoff between (1) and (2)?
> >>>>>>
> >>>>>> I was trying to add more tests for core module(class
> >>>>>> MetricsStreamProcessor#add), and nearly success yesterday night, but
> >>>>> when I
> >>>>>> was reviewing my own test case, I was asking myself, what are the
> >>>>> points of
> >>>>>> those UTs? I was mocking nearly all dependencies, so the logic is in
> >> the
> >>>>>> tested and untested at the same time. Then I decided to revert and
> >>>>> delete
> >>>>>> those tests.
> >>>>>>
> >>>>>> WDYT? Open my mind to hear more ideas.
> >>>>>>
> >>>>>> [1] https://codecov.io/gh/apache/skywalking
> >>>>>> [2] https://twitter.com/wusheng1108/status/1243798832690782211
> >>>>>> [3] https://github.com/apache/skywalking/issues/4455
> >>>>>> [4] https://github.com/apache/skywalking/issues/4530
> >>>>>> [5] https://github.com/apache/skywalking/issues/4576
> >>>>>>
> >>>>>> Sheng Wu 吴晟
> >>>>>> Twitter, wusheng1108
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>> Sheng Wu 吴晟
> >>>>
> >>>> Apache SkyWalking
> >>>> Apache Incubator
> >>>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >>>> Zipkin
> >>>> Twitter, wusheng1108
> >>>>
> >>> --
> >>> Sheng Wu 吴晟
> >>>
> >>> Apache SkyWalking
> >>> Apache Incubator
> >>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >>> Zipkin
> >>> Twitter, wusheng1108
>
>
>
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by "kezhenxu94@163" <ke...@163.com>.
I did a quick POC locally, it turns out the solution is possible, the code lines executed in the E2E can be tracked and analyzed [0] (I run a simple/JDK E2E test, and the coverage increase ~3.42%), I’ve also tagged the coverage with “E2E” [1].
The next step would be chained them up in the workflow of GitHub Actions, but one problems as follow should be fixed before it is ready:
Gracefully shutdown mechanism of every component, including backend, web UI, service provider of E2E, among which the last two are easy to fix, as for the backend, I’ve tried but did not successful in a normal way, will take a detailed look when I get some time
[0] https://codecov.io/github/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9
[1] https://codecov.io/gh/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9/build
> On Mar 31, 2020, at 07:56, Sheng Wu <wu...@gmail.com> wrote:
>
> kezhenxu94@163 <ke...@163.com> 于2020年3月30日周一 下午11:59写道：
>
>>
>>
>>> On Mar 30, 2020, at 21:58, Sheng Wu <wu...@gmail.com> wrote:
>>>
>>> Sheng Wu <wu...@gmail.com>于2020年3月30日 周一下午9:55写道：
>>>
>>>>
>>>>
>>>> kezhenxu94@apache <ke...@apache.org>于2020年3月30日 周一下午9:11写道：
>>>>
>>>>> In essence, code coverage is a measurement of the code lines that
>> **have
>>>>> been executed** during the tests, and the code lines that are executed
>> in
>>>>> the tests are **covered** actually, no matter whether they are counted
>> in
>>>>> the coverage rate (~26%), my idea is to let the coverage tool
>> (Cobertura in
>>>>> our case) **notice** this and count them in the final coverage, and
>> IIRC,
>>>>> Jacoco provides an Java agent that does this exactly, I’ll do some
>> research
>>>>> on this and forward the conclusion here, or open pull requests
>> directly if
>>>>> it works overall.
>>>>
>>>>
>>>> It must be an agent, this is the only case it could measure the lines.
>>>> If this really could work in this way, should be very interesting.
>>>>
>>>
>>> My concern will be, how to aggregate the coverage across 104 GitHub
>>> Actions.
>>> It will be hard to have all result files in one place.
>>>
>>> Zhenxu
>>> WDYT?
>>>
>>
>> That’s not a concern, Codecov is able to merge them automatically
>>
>
> Good to know :)
>
>
> Sheng Wu 吴晟
> Twitter, wusheng1108
>
>
>>
>>> Sheng
>>>
>>>
>>>> Sheng
>>>>
>>>>
>>>>>
>>>>>
>>>>> GitHub @kezhenxu94
>>>>> Apache SkyWalking, Apache Dubbo
>>>>>
>>>>>> On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
>>>>>>
>>>>>> Hi Dev Team.
>>>>>>
>>>>>> I am looking at the Coverage dashboard[1] this afternoon.
>>>>>> 26%+- seems a static status of SkyWalking main repo. Seems not
>>>>> "good"(for
>>>>>> normal people). Especially the coverage is caused by
>>>>> storage/core/receiver,
>>>>>> less than 15% usually.
>>>>>> But at the same time, like my twitter description[2], we have
>>>>> 101(excluded
>>>>>> 3 CI tasks) e2e and agent plugin tests. These are exact covering
>>>>>> the storage/core/receiver.
>>>>>>
>>>>>> From my understanding, besides these 3 test requirements[3][4][5], the
>>>>> e2e
>>>>>> tests exactly are covering storage/core/receiver.
>>>>>> *Logically, we have a very high coverage rate, which matches my
>>>>>> feeling (maybe others too)*.
>>>>>>
>>>>>> The thing bothering me is, how we should describe this to a new
>>>>>> contributor? Such as
>>>>>> 1. What parts of contributions should require UT coverage?
>>>>>> 2. What e2e should be added?
>>>>>> 3. Where is the balance and tradeoff between (1) and (2)?
>>>>>>
>>>>>> I was trying to add more tests for core module(class
>>>>>> MetricsStreamProcessor#add), and nearly success yesterday night, but
>>>>> when I
>>>>>> was reviewing my own test case, I was asking myself, what are the
>>>>> points of
>>>>>> those UTs? I was mocking nearly all dependencies, so the logic is in
>> the
>>>>>> tested and untested at the same time. Then I decided to revert and
>>>>> delete
>>>>>> those tests.
>>>>>>
>>>>>> WDYT? Open my mind to hear more ideas.
>>>>>>
>>>>>> [1] https://codecov.io/gh/apache/skywalking
>>>>>> [2] https://twitter.com/wusheng1108/status/1243798832690782211
>>>>>> [3] https://github.com/apache/skywalking/issues/4455
>>>>>> [4] https://github.com/apache/skywalking/issues/4530
>>>>>> [5] https://github.com/apache/skywalking/issues/4576
>>>>>>
>>>>>> Sheng Wu 吴晟
>>>>>> Twitter, wusheng1108
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>> Sheng Wu 吴晟
>>>>
>>>> Apache SkyWalking
>>>> Apache Incubator
>>>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
>>>> Zipkin
>>>> Twitter, wusheng1108
>>>>
>>> --
>>> Sheng Wu 吴晟
>>>
>>> Apache SkyWalking
>>> Apache Incubator
>>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
>>> Zipkin
>>> Twitter, wusheng1108
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by Sheng Wu <wu...@gmail.com>.
kezhenxu94@163 <ke...@163.com> 于2020年3月30日周一 下午11:59写道：
>
>
> > On Mar 30, 2020, at 21:58, Sheng Wu <wu...@gmail.com> wrote:
> >
> > Sheng Wu <wu...@gmail.com>于2020年3月30日 周一下午9:55写道：
> >
> >>
> >>
> >> kezhenxu94@apache <ke...@apache.org>于2020年3月30日 周一下午9:11写道：
> >>
> >>> In essence, code coverage is a measurement of the code lines that
> **have
> >>> been executed** during the tests, and the code lines that are executed
> in
> >>> the tests are **covered** actually, no matter whether they are counted
> in
> >>> the coverage rate (~26%), my idea is to let the coverage tool
> (Cobertura in
> >>> our case) **notice** this and count them in the final coverage, and
> IIRC,
> >>> Jacoco provides an Java agent that does this exactly, I’ll do some
> research
> >>> on this and forward the conclusion here, or open pull requests
> directly if
> >>> it works overall.
> >>
> >>
> >> It must be an agent, this is the only case it could measure the lines.
> >> If this really could work in this way, should be very interesting.
> >>
> >
> > My concern will be, how to aggregate the coverage across 104 GitHub
> > Actions.
> > It will be hard to have all result files in one place.
> >
> > Zhenxu
> > WDYT?
> >
>
> That’s not a concern, Codecov is able to merge them automatically
>
Good to know :)
Sheng Wu 吴晟
Twitter, wusheng1108
>
> > Sheng
> >
> >
> >> Sheng
> >>
> >>
> >>>
> >>>
> >>> GitHub @kezhenxu94
> >>> Apache SkyWalking, Apache Dubbo
> >>>
> >>>> On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
> >>>>
> >>>> Hi Dev Team.
> >>>>
> >>>> I am looking at the Coverage dashboard[1] this afternoon.
> >>>> 26%+- seems a static status of SkyWalking main repo. Seems not
> >>> "good"(for
> >>>> normal people). Especially the coverage is caused by
> >>> storage/core/receiver,
> >>>> less than 15% usually.
> >>>> But at the same time, like my twitter description[2], we have
> >>> 101(excluded
> >>>> 3 CI tasks) e2e and agent plugin tests. These are exact covering
> >>>> the storage/core/receiver.
> >>>>
> >>>> From my understanding, besides these 3 test requirements[3][4][5], the
> >>> e2e
> >>>> tests exactly are covering storage/core/receiver.
> >>>> *Logically, we have a very high coverage rate, which matches my
> >>>> feeling (maybe others too)*.
> >>>>
> >>>> The thing bothering me is, how we should describe this to a new
> >>>> contributor? Such as
> >>>> 1. What parts of contributions should require UT coverage?
> >>>> 2. What e2e should be added?
> >>>> 3. Where is the balance and tradeoff between (1) and (2)?
> >>>>
> >>>> I was trying to add more tests for core module(class
> >>>> MetricsStreamProcessor#add), and nearly success yesterday night, but
> >>> when I
> >>>> was reviewing my own test case, I was asking myself, what are the
> >>> points of
> >>>> those UTs? I was mocking nearly all dependencies, so the logic is in
> the
> >>>> tested and untested at the same time. Then I decided to revert and
> >>> delete
> >>>> those tests.
> >>>>
> >>>> WDYT? Open my mind to hear more ideas.
> >>>>
> >>>> [1] https://codecov.io/gh/apache/skywalking
> >>>> [2] https://twitter.com/wusheng1108/status/1243798832690782211
> >>>> [3] https://github.com/apache/skywalking/issues/4455
> >>>> [4] https://github.com/apache/skywalking/issues/4530
> >>>> [5] https://github.com/apache/skywalking/issues/4576
> >>>>
> >>>> Sheng Wu 吴晟
> >>>> Twitter, wusheng1108
> >>>
> >>>
> >>>
> >>>
> >>> --
> >> Sheng Wu 吴晟
> >>
> >> Apache SkyWalking
> >> Apache Incubator
> >> Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >> Zipkin
> >> Twitter, wusheng1108
> >>
> > --
> > Sheng Wu 吴晟
> >
> > Apache SkyWalking
> > Apache Incubator
> > Apache ShardingSphere, ECharts, DolphinScheduler podlings
> > Zipkin
> > Twitter, wusheng1108
>
>
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by "kezhenxu94@163" <ke...@163.com>.
> On Mar 30, 2020, at 21:58, Sheng Wu <wu...@gmail.com> wrote:
>
> Sheng Wu <wu...@gmail.com>于2020年3月30日 周一下午9:55写道：
>
>>
>>
>> kezhenxu94@apache <ke...@apache.org>于2020年3月30日 周一下午9:11写道：
>>
>>> In essence, code coverage is a measurement of the code lines that **have
>>> been executed** during the tests, and the code lines that are executed in
>>> the tests are **covered** actually, no matter whether they are counted in
>>> the coverage rate (~26%), my idea is to let the coverage tool (Cobertura in
>>> our case) **notice** this and count them in the final coverage, and IIRC,
>>> Jacoco provides an Java agent that does this exactly, I’ll do some research
>>> on this and forward the conclusion here, or open pull requests directly if
>>> it works overall.
>>
>>
>> It must be an agent, this is the only case it could measure the lines.
>> If this really could work in this way, should be very interesting.
>>
>
> My concern will be, how to aggregate the coverage across 104 GitHub
> Actions.
> It will be hard to have all result files in one place.
>
> Zhenxu
> WDYT?
>
That’s not a concern, Codecov is able to merge them automatically
> Sheng
>
>
>> Sheng
>>
>>
>>>
>>>
>>> GitHub @kezhenxu94
>>> Apache SkyWalking, Apache Dubbo
>>>
>>>> On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
>>>>
>>>> Hi Dev Team.
>>>>
>>>> I am looking at the Coverage dashboard[1] this afternoon.
>>>> 26%+- seems a static status of SkyWalking main repo. Seems not
>>> "good"(for
>>>> normal people). Especially the coverage is caused by
>>> storage/core/receiver,
>>>> less than 15% usually.
>>>> But at the same time, like my twitter description[2], we have
>>> 101(excluded
>>>> 3 CI tasks) e2e and agent plugin tests. These are exact covering
>>>> the storage/core/receiver.
>>>>
>>>> From my understanding, besides these 3 test requirements[3][4][5], the
>>> e2e
>>>> tests exactly are covering storage/core/receiver.
>>>> *Logically, we have a very high coverage rate, which matches my
>>>> feeling (maybe others too)*.
>>>>
>>>> The thing bothering me is, how we should describe this to a new
>>>> contributor? Such as
>>>> 1. What parts of contributions should require UT coverage?
>>>> 2. What e2e should be added?
>>>> 3. Where is the balance and tradeoff between (1) and (2)?
>>>>
>>>> I was trying to add more tests for core module(class
>>>> MetricsStreamProcessor#add), and nearly success yesterday night, but
>>> when I
>>>> was reviewing my own test case, I was asking myself, what are the
>>> points of
>>>> those UTs? I was mocking nearly all dependencies, so the logic is in the
>>>> tested and untested at the same time. Then I decided to revert and
>>> delete
>>>> those tests.
>>>>
>>>> WDYT? Open my mind to hear more ideas.
>>>>
>>>> [1] https://codecov.io/gh/apache/skywalking
>>>> [2] https://twitter.com/wusheng1108/status/1243798832690782211
>>>> [3] https://github.com/apache/skywalking/issues/4455
>>>> [4] https://github.com/apache/skywalking/issues/4530
>>>> [5] https://github.com/apache/skywalking/issues/4576
>>>>
>>>> Sheng Wu 吴晟
>>>> Twitter, wusheng1108
>>>
>>>
>>>
>>>
>>> --
>> Sheng Wu 吴晟
>>
>> Apache SkyWalking
>> Apache Incubator
>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
>> Zipkin
>> Twitter, wusheng1108
>>
> --
> Sheng Wu 吴晟
>
> Apache SkyWalking
> Apache Incubator
> Apache ShardingSphere, ECharts, DolphinScheduler podlings
> Zipkin
> Twitter, wusheng1108
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by Sheng Wu <wu...@gmail.com>.
Sheng Wu <wu...@gmail.com>于2020年3月30日 周一下午9:55写道：
>
>
> kezhenxu94@apache <ke...@apache.org>于2020年3月30日 周一下午9:11写道：
>
>> In essence, code coverage is a measurement of the code lines that **have
>> been executed** during the tests, and the code lines that are executed in
>> the tests are **covered** actually, no matter whether they are counted in
>> the coverage rate (~26%), my idea is to let the coverage tool (Cobertura in
>> our case) **notice** this and count them in the final coverage, and IIRC,
>> Jacoco provides an Java agent that does this exactly, I’ll do some research
>> on this and forward the conclusion here, or open pull requests directly if
>> it works overall.
>
>
> It must be an agent, this is the only case it could measure the lines.
> If this really could work in this way, should be very interesting.
>
My concern will be, how to aggregate the coverage across 104 GitHub
Actions.
It will be hard to have all result files in one place.
Zhenxu
WDYT?
Sheng
> Sheng
>
>
>>
>>
>> GitHub @kezhenxu94
>> Apache SkyWalking, Apache Dubbo
>>
>> > On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
>> >
>> > Hi Dev Team.
>> >
>> > I am looking at the Coverage dashboard[1] this afternoon.
>> > 26%+- seems a static status of SkyWalking main repo. Seems not
>> "good"(for
>> > normal people). Especially the coverage is caused by
>> storage/core/receiver,
>> > less than 15% usually.
>> > But at the same time, like my twitter description[2], we have
>> 101(excluded
>> > 3 CI tasks) e2e and agent plugin tests. These are exact covering
>> > the storage/core/receiver.
>> >
>> > From my understanding, besides these 3 test requirements[3][4][5], the
>> e2e
>> > tests exactly are covering storage/core/receiver.
>> > *Logically, we have a very high coverage rate, which matches my
>> > feeling (maybe others too)*.
>> >
>> > The thing bothering me is, how we should describe this to a new
>> > contributor? Such as
>> > 1. What parts of contributions should require UT coverage?
>> > 2. What e2e should be added?
>> > 3. Where is the balance and tradeoff between (1) and (2)?
>> >
>> > I was trying to add more tests for core module(class
>> > MetricsStreamProcessor#add), and nearly success yesterday night, but
>> when I
>> > was reviewing my own test case, I was asking myself, what are the
>> points of
>> > those UTs? I was mocking nearly all dependencies, so the logic is in the
>> > tested and untested at the same time. Then I decided to revert and
>> delete
>> > those tests.
>> >
>> > WDYT? Open my mind to hear more ideas.
>> >
>> > [1] https://codecov.io/gh/apache/skywalking
>> > [2] https://twitter.com/wusheng1108/status/1243798832690782211
>> > [3] https://github.com/apache/skywalking/issues/4455
>> > [4] https://github.com/apache/skywalking/issues/4530
>> > [5] https://github.com/apache/skywalking/issues/4576
>> >
>> > Sheng Wu 吴晟
>> > Twitter, wusheng1108
>>
>>
>>
>>
>> --
> Sheng Wu 吴晟
>
> Apache SkyWalking
> Apache Incubator
> Apache ShardingSphere, ECharts, DolphinScheduler podlings
> Zipkin
> Twitter, wusheng1108
>
--
Sheng Wu 吴晟
Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by Sheng Wu <wu...@gmail.com>.
kezhenxu94@apache <ke...@apache.org>于2020年3月30日 周一下午9:11写道：
> In essence, code coverage is a measurement of the code lines that **have
> been executed** during the tests, and the code lines that are executed in
> the tests are **covered** actually, no matter whether they are counted in
> the coverage rate (~26%), my idea is to let the coverage tool (Cobertura in
> our case) **notice** this and count them in the final coverage, and IIRC,
> Jacoco provides an Java agent that does this exactly, I’ll do some research
> on this and forward the conclusion here, or open pull requests directly if
> it works overall.
It must be an agent, this is the only case it could measure the lines.
If this really could work in this way, should be very interesting.
Sheng
>
>
> GitHub @kezhenxu94
> Apache SkyWalking, Apache Dubbo
>
> > On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
> >
> > Hi Dev Team.
> >
> > I am looking at the Coverage dashboard[1] this afternoon.
> > 26%+- seems a static status of SkyWalking main repo. Seems not "good"(for
> > normal people). Especially the coverage is caused by
> storage/core/receiver,
> > less than 15% usually.
> > But at the same time, like my twitter description[2], we have
> 101(excluded
> > 3 CI tasks) e2e and agent plugin tests. These are exact covering
> > the storage/core/receiver.
> >
> > From my understanding, besides these 3 test requirements[3][4][5], the
> e2e
> > tests exactly are covering storage/core/receiver.
> > *Logically, we have a very high coverage rate, which matches my
> > feeling (maybe others too)*.
> >
> > The thing bothering me is, how we should describe this to a new
> > contributor? Such as
> > 1. What parts of contributions should require UT coverage?
> > 2. What e2e should be added?
> > 3. Where is the balance and tradeoff between (1) and (2)?
> >
> > I was trying to add more tests for core module(class
> > MetricsStreamProcessor#add), and nearly success yesterday night, but
> when I
> > was reviewing my own test case, I was asking myself, what are the points
> of
> > those UTs? I was mocking nearly all dependencies, so the logic is in the
> > tested and untested at the same time. Then I decided to revert and delete
> > those tests.
> >
> > WDYT? Open my mind to hear more ideas.
> >
> > [1] https://codecov.io/gh/apache/skywalking
> > [2] https://twitter.com/wusheng1108/status/1243798832690782211
> > [3] https://github.com/apache/skywalking/issues/4455
> > [4] https://github.com/apache/skywalking/issues/4530
> > [5] https://github.com/apache/skywalking/issues/4576
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
>
>
>
>
> --
Sheng Wu 吴晟
Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108
Re: [DISCUSSION] Meaning of test coverage, how/whether should we
improve it?
Posted by "kezhenxu94@apache" <ke...@apache.org>.
In essence, code coverage is a measurement of the code lines that **have been executed** during the tests, and the code lines that are executed in the tests are **covered** actually, no matter whether they are counted in the coverage rate (~26%), my idea is to let the coverage tool (Cobertura in our case) **notice** this and count them in the final coverage, and IIRC, Jacoco provides an Java agent that does this exactly, I’ll do some research on this and forward the conclusion here, or open pull requests directly if it works overall.
GitHub @kezhenxu94
Apache SkyWalking, Apache Dubbo
> On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
>
> Hi Dev Team.
>
> I am looking at the Coverage dashboard[1] this afternoon.
> 26%+- seems a static status of SkyWalking main repo. Seems not "good"(for
> normal people). Especially the coverage is caused by storage/core/receiver,
> less than 15% usually.
> But at the same time, like my twitter description[2], we have 101(excluded
> 3 CI tasks) e2e and agent plugin tests. These are exact covering
> the storage/core/receiver.
>
> From my understanding, besides these 3 test requirements[3][4][5], the e2e
> tests exactly are covering storage/core/receiver.
> *Logically, we have a very high coverage rate, which matches my
> feeling (maybe others too)*.
>
> The thing bothering me is, how we should describe this to a new
> contributor? Such as
> 1. What parts of contributions should require UT coverage?
> 2. What e2e should be added?
> 3. Where is the balance and tradeoff between (1) and (2)?
>
> I was trying to add more tests for core module(class
> MetricsStreamProcessor#add), and nearly success yesterday night, but when I
> was reviewing my own test case, I was asking myself, what are the points of
> those UTs? I was mocking nearly all dependencies, so the logic is in the
> tested and untested at the same time. Then I decided to revert and delete
> those tests.
>
> WDYT? Open my mind to hear more ideas.
>
> [1] https://codecov.io/gh/apache/skywalking
> [2] https://twitter.com/wusheng1108/status/1243798832690782211
> [3] https://github.com/apache/skywalking/issues/4455
> [4] https://github.com/apache/skywalking/issues/4530
> [5] https://github.com/apache/skywalking/issues/4576
>
> Sheng Wu 吴晟
> Twitter, wusheng1108