Jacoco and Codecov.io integration for Code Coverage reports

[Nicolò Boschi <bo...@gmail.com>]

Dear community,

Currently we don't have an aggregate code coverage report. There's a Jacoco
configuration to generate the report after unit tests but it is not
easily readable and understandable.

Having a better system would bring a couple of interesting aspects:
1. We can block a pull request if the code coverage decreases - that means
there are no tests for the new code.
2. We can have an overview for each modules and discover which features are
less covered and add new tests

I would like to introduce a new tool that reads, aggregates, stores and
reports the code coverage for the whole codebase. This tool is Codecov (
https://about.codecov.io/) that is one of the most popular tools for open
source projects.
It brings multiple advantages:
1. It aggregates all the Jacoco reports in a single report
2. It keeps the coverage history
3. It is well integrated with Github

The idea is the following:
- Add this profiling in the CI - for each test suite and upload the results
to codecov.io
- Enable the codecov report that add a comment in the pull


Implementation pull: https://github.com/apache/pulsar/pull/14600
Codecov comment in the PR:
https://github.com/apache/pulsar/pull/14600#issuecomment-1061636495
Codecov full report:
https://codecov.io/gh/apache/pulsar/tree/0fa212d40b2bf166b20f3155c40fdb9624f577ed

Initially there were some doubts around whether the jacoco agent would have
added a significant overhead (execution time + memory usage). I've seen
(and you can see the checks times in my pull) that it is not relevant in
the Pulsar codebase.

The pull only adds Jacoco coverage for unit tests. Integrating jacoco with
the integration tests (inside docker) will be more complex but it is
possible. We can do it in a second moment.
You basically need to add the jacoco agent inside the docker container,
enabling it in the java command and downloading the report files before the
container stops.


Note: Codecov is free for open source projects (
https://about.codecov.io/pricing/). It is required to install the app on
Github Marketplace (https://github.com/marketplace/codecov).

BR,
Nicolò Boschi

Re: Jacoco and Codecov.io integration for Code Coverage reports

[Nicolò Boschi <bo...@gmail.com>]

Thanks Enrico for the feedback.

I would like to merge the PR to complete the setup and reach more feedback
for other contributors.
Once we complete the setup, the Codecov bot will add a comment on every new
pull request


Nicolò Boschi


Il giorno mar 22 mar 2022 alle ore 18:32 Enrico Olivelli <
eolivelli@gmail.com> ha scritto:

> Nicolò,
> Great stuff indeed.
> Some comments inline below
>
> Thanks
>
>
> Il Mer 16 Mar 2022, 16:45 Nicolò Boschi <bo...@gmail.com> ha scritto:
>
> > Dear community,
> >
> > Currently we don't have an aggregate code coverage report. There's a
> Jacoco
> > configuration to generate the report after unit tests but it is not
> > easily readable and understandable.
> >
> > Having a better system would bring a couple of interesting aspects:
> > 1. We can block a pull request if the code coverage decreases - that
> means
> > there are no tests for the new code.
> >
>
> I am afraid that this would lead to some automatic excessive nitpicking.
> Let's keep this in the 'future works' basket
>
> 2. We can have an overview for each modules and discover which features are
> > less covered and add new tests
> >
> > I would like to introduce a new tool that reads, aggregates, stores and
> > reports the code coverage for the whole codebase. This tool is Codecov (
> > https://about.codecov.io/) that is one of the most popular tools for
> open
> > source projects.
> > It brings multiple advantages:
> > 1. It aggregates all the Jacoco reports in a single report
> > 2. It keeps the coverage history
> > 3. It is well integrated with Github
> >
> > The idea is the following:
> > - Add this profiling in the CI - for each test suite and upload the
> results
> > to codecov.io
> > - Enable the codecov report that add a comment in the pull
> >
> >
> > Implementation pull: https://github.com/apache/pulsar/pull/14600
> > Codecov comment in the PR:
> > https://github.com/apache/pulsar/pull/14600#issuecomment-1061636495
> > Codecov full report:
> >
> >
> https://codecov.io/gh/apache/pulsar/tree/0fa212d40b2bf166b20f3155c40fdb9624f577ed
> >
> > Initially there were some doubts around whether the jacoco agent would
> have
> > added a significant overhead (execution time + memory usage). I've seen
> > (and you can see the checks times in my pull) that it is not relevant in
> > the Pulsar codebase.
> >
> > The pull only adds Jacoco coverage for unit tests. Integrating jacoco
> with
> > the integration tests (inside docker) will be more complex but it is
> > possible. We can do it in a second moment.
> > You basically need to add the jacoco agent inside the docker container,
> > enabling it in the java command and downloading the report files before
> the
> > container stops.
> >
> >
> > Note: Codecov is free for open source projects (
> > https://about.codecov.io/pricing/). It is required to install the app on
> > Github Marketplace (https://github.com/marketplace/codecov).
> >
>
> Is it something we should ask to ASF infra probably.
>
> The PMC can deal with this
>
> Enrico
>
>
> > BR,
> > Nicolò Boschi
> >
>

Re: Jacoco and Codecov.io integration for Code Coverage reports

[Enrico Olivelli <eo...@gmail.com>]

Nicolò,
Great stuff indeed.
Some comments inline below

Thanks


Il Mer 16 Mar 2022, 16:45 Nicolò Boschi <bo...@gmail.com> ha scritto:

> Dear community,
>
> Currently we don't have an aggregate code coverage report. There's a Jacoco
> configuration to generate the report after unit tests but it is not
> easily readable and understandable.
>
> Having a better system would bring a couple of interesting aspects:
> 1. We can block a pull request if the code coverage decreases - that means
> there are no tests for the new code.
>

I am afraid that this would lead to some automatic excessive nitpicking.
Let's keep this in the 'future works' basket

2. We can have an overview for each modules and discover which features are
> less covered and add new tests
>
> I would like to introduce a new tool that reads, aggregates, stores and
> reports the code coverage for the whole codebase. This tool is Codecov (
> https://about.codecov.io/) that is one of the most popular tools for open
> source projects.
> It brings multiple advantages:
> 1. It aggregates all the Jacoco reports in a single report
> 2. It keeps the coverage history
> 3. It is well integrated with Github
>
> The idea is the following:
> - Add this profiling in the CI - for each test suite and upload the results
> to codecov.io
> - Enable the codecov report that add a comment in the pull
>
>
> Implementation pull: https://github.com/apache/pulsar/pull/14600
> Codecov comment in the PR:
> https://github.com/apache/pulsar/pull/14600#issuecomment-1061636495
> Codecov full report:
>
> https://codecov.io/gh/apache/pulsar/tree/0fa212d40b2bf166b20f3155c40fdb9624f577ed
>
> Initially there were some doubts around whether the jacoco agent would have
> added a significant overhead (execution time + memory usage). I've seen
> (and you can see the checks times in my pull) that it is not relevant in
> the Pulsar codebase.
>
> The pull only adds Jacoco coverage for unit tests. Integrating jacoco with
> the integration tests (inside docker) will be more complex but it is
> possible. We can do it in a second moment.
> You basically need to add the jacoco agent inside the docker container,
> enabling it in the java command and downloading the report files before the
> container stops.
>
>
> Note: Codecov is free for open source projects (
> https://about.codecov.io/pricing/). It is required to install the app on
> Github Marketplace (https://github.com/marketplace/codecov).
>

Is it something we should ask to ASF infra probably.

The PMC can deal with this

Enrico


> BR,
> Nicolò Boschi
>