Re: [DISCUSSION] Meaning of test coverage, how/whether should we improve it?

[Sheng Wu <wu...@apache.org>]

I think the key is not how gracefully shutdown, it is when should shutdown,
and how to trigger it.

kezhenxu94@163 <ke...@163.com>于2020年3月31日 周二下午10:18写道：

> Jacoco agent writes the execution data to the files, if the targeted JVM
> does not shut down gracefully, the file is corrupted and cannot be
> analyzed, see
> https://github.com/jacoco/jacoco/issues/901#issuecomment-508442946
>
> > On Mar 31, 2020, at 21:43, Sheng Wu <wu...@gmail.com> wrote:
> >
> > Hi Zhenxu
> >
> > Could you share about why is gracefully shutdown important?
> >
> > Sheng Wu 吴晟
> > Twitter, wusheng1108
> >
> >
> > kezhenxu94@163 <ke...@163.com> 于2020年3月31日周二 下午9:36写道：
> >
> >> I did a quick POC locally, it turns out the solution is possible, the
> code
> >> lines executed in the E2E can be tracked and analyzed [0] (I run a
> >> simple/JDK E2E test, and the coverage increase ~3.42%), I’ve also tagged
> >> the coverage with “E2E” [1].
> >>
> >> The next step would be chained them up in the workflow of GitHub
> Actions,
> >> but one problems as follow should be fixed before it is ready:
> >>
> >> Gracefully shutdown mechanism of every component, including backend, web
> >> UI, service provider of E2E, among which the last two are easy to fix,
> as
> >> for the backend, I’ve tried but did not successful in a normal way, will
> >> take a detailed look when I get some time
> >>
> >>
> >> [0]
> >>
> https://codecov.io/github/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9
> >> [1]
> >>
> https://codecov.io/gh/apache/skywalking/commit/e55073e51970ecb50d014094e4e9b8dcc7d8c0d9/build
> >>
> >>> On Mar 31, 2020, at 07:56, Sheng Wu <wu...@gmail.com> wrote:
> >>>
> >>> kezhenxu94@163 <ke...@163.com> 于2020年3月30日周一 下午11:59写道：
> >>>
> >>>>
> >>>>
> >>>>> On Mar 30, 2020, at 21:58, Sheng Wu <wu...@gmail.com>
> wrote:
> >>>>>
> >>>>> Sheng Wu <wu...@gmail.com>于2020年3月30日 周一下午9:55写道：
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> kezhenxu94@apache <ke...@apache.org>于2020年3月30日 周一下午9:11写道：
> >>>>>>
> >>>>>>> In essence, code coverage is a measurement of the code lines that
> >>>> **have
> >>>>>>> been executed** during the tests, and the code lines that are
> >> executed
> >>>> in
> >>>>>>> the tests are **covered** actually, no matter whether they are
> >> counted
> >>>> in
> >>>>>>> the coverage rate (~26%), my idea is to let the coverage tool
> >>>> (Cobertura in
> >>>>>>> our case) **notice** this and count them in the final coverage, and
> >>>> IIRC,
> >>>>>>> Jacoco provides an Java agent that does this exactly, I’ll do some
> >>>> research
> >>>>>>> on this and forward the conclusion here, or open pull requests
> >>>> directly if
> >>>>>>> it works overall.
> >>>>>>
> >>>>>>
> >>>>>> It must be an agent, this is the only case it could measure the
> lines.
> >>>>>> If this really could work in this way, should be very interesting.
> >>>>>>
> >>>>>
> >>>>> My concern will be, how to aggregate the coverage across 104 GitHub
> >>>>> Actions.
> >>>>> It will be hard to have all result files in one place.
> >>>>>
> >>>>> Zhenxu
> >>>>> WDYT?
> >>>>>
> >>>>
> >>>> That’s not a concern, Codecov is able to merge them automatically
> >>>>
> >>>
> >>> Good to know :)
> >>>
> >>>
> >>> Sheng Wu 吴晟
> >>> Twitter, wusheng1108
> >>>
> >>>
> >>>>
> >>>>> Sheng
> >>>>>
> >>>>>
> >>>>>> Sheng
> >>>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> GitHub @kezhenxu94
> >>>>>>> Apache SkyWalking, Apache Dubbo
> >>>>>>>
> >>>>>>>> On Mar 30, 2020, at 17:28, Sheng Wu <wu...@apache.org> wrote:
> >>>>>>>>
> >>>>>>>> Hi Dev Team.
> >>>>>>>>
> >>>>>>>> I am looking at the Coverage dashboard[1] this afternoon.
> >>>>>>>> 26%+- seems a static status of SkyWalking main repo. Seems not
> >>>>>>> "good"(for
> >>>>>>>> normal people). Especially the coverage is caused by
> >>>>>>> storage/core/receiver,
> >>>>>>>> less than 15% usually.
> >>>>>>>> But at the same time, like my twitter description[2], we have
> >>>>>>> 101(excluded
> >>>>>>>> 3 CI tasks) e2e and agent plugin tests. These are exact covering
> >>>>>>>> the storage/core/receiver.
> >>>>>>>>
> >>>>>>>> From my understanding, besides these 3 test requirements[3][4][5],
> >> the
> >>>>>>> e2e
> >>>>>>>> tests exactly are covering storage/core/receiver.
> >>>>>>>> *Logically, we have a very high coverage rate, which matches my
> >>>>>>>> feeling (maybe others too)*.
> >>>>>>>>
> >>>>>>>> The thing bothering me is, how we should describe this to a new
> >>>>>>>> contributor? Such as
> >>>>>>>> 1. What parts of contributions should require UT coverage?
> >>>>>>>> 2. What e2e should be added?
> >>>>>>>> 3. Where is the balance and tradeoff between (1) and (2)?
> >>>>>>>>
> >>>>>>>> I was trying to add more tests for core module(class
> >>>>>>>> MetricsStreamProcessor#add), and nearly success yesterday night,
> but
> >>>>>>> when I
> >>>>>>>> was reviewing my own test case, I was asking myself, what are the
> >>>>>>> points of
> >>>>>>>> those UTs? I was mocking nearly all dependencies, so the logic is
> in
> >>>> the
> >>>>>>>> tested and untested at the same time. Then I decided to revert and
> >>>>>>> delete
> >>>>>>>> those tests.
> >>>>>>>>
> >>>>>>>> WDYT? Open my mind to hear more ideas.
> >>>>>>>>
> >>>>>>>> [1] https://codecov.io/gh/apache/skywalking
> >>>>>>>> [2] https://twitter.com/wusheng1108/status/1243798832690782211
> >>>>>>>> [3] https://github.com/apache/skywalking/issues/4455
> >>>>>>>> [4] https://github.com/apache/skywalking/issues/4530
> >>>>>>>> [5] https://github.com/apache/skywalking/issues/4576
> >>>>>>>>
> >>>>>>>> Sheng Wu 吴晟
> >>>>>>>> Twitter, wusheng1108
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>> Sheng Wu 吴晟
> >>>>>>
> >>>>>> Apache SkyWalking
> >>>>>> Apache Incubator
> >>>>>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >>>>>> Zipkin
> >>>>>> Twitter, wusheng1108
> >>>>>>
> >>>>> --
> >>>>> Sheng Wu 吴晟
> >>>>>
> >>>>> Apache SkyWalking
> >>>>> Apache Incubator
> >>>>> Apache ShardingSphere, ECharts, DolphinScheduler podlings
> >>>>> Zipkin
> >>>>> Twitter, wusheng1108
> >>
> >>
> >>
>
> --
Sheng Wu 吴晟

Apache SkyWalking
Apache Incubator
Apache ShardingSphere, ECharts, DolphinScheduler podlings
Zipkin
Twitter, wusheng1108