You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by tison <wa...@gmail.com> on 2023/03/21 10:50:17 UTC
Unstable codecov action
For example
https://github.com/apache/pulsar/actions/runs/4454158774/jobs/7867745340?pr=19842
I'm wondering if anyone cares about the report and if it helps you during
the coding or reviewing process? Now it generates a few of noise but I just
omit the report it gives ;-)
For the issue itself, it seems some artifacts don't retain properly.
Best,
tison.
Re: Unstable codecov action
Posted by Lari Hotari <lh...@apache.org>.
Update regarding Codecov improvements for apache/pulsar CI:
- fixed issue with Jacoco coverage data not getting stored in files:
https://github.com/apache/pulsar/pull/19947
This seemed to be a broader issues since the reported total code coverage increased to about 72.8% with this fix, example https://app.codecov.io/gh/apache/pulsar/pull/19947/tree .
There's a workaround for Codecov upload issue in progress. More details in the comment
https://github.com/apache/pulsar/issues/19952#issuecomment-1487997039 .
This is waiting for ASF Infra to resolve https://issues.apache.org/jira/browse/INFRA-24399 .
After this, I believe that Codecov will be reasonably stable in our CI. Actions will be needed for individuals for adding a Codecov upload token for builds in personal forks. I'll add instructions for that while resolving #19952.
-Lari
On 2023/03/24 09:51:51 Lari Hotari wrote:
> Thanks for sharing the pain. That's the first step in improving something that is painful.
>
> For the flaky tests GitHub Actions workflow pulsar-ci-flaky.yaml, the Codecov upload should be a separate job in the workflow so that the upload could be retried separately without running all tests. This type of approach is already used in the main GitHub Actions workflow, "Pulsar CI".
> Contributions are welcome!
>
> We could also consider disabling codecov for pull request builds until someone who cares about test code coverage metrics picks up the work.
>
> Code coverage is the first metric that most will ask about tests. It's not the only metric that matter, but it is something that helps understand what parts of the code isn't even run in our tests. It will also help plan improvements to tests.
>
> Codecov upload fails very frequently with errors such as https://github.com/codecov/codecov-action/issues/837 and https://github.com/codecov/codecov-action/issues/598
> One possible resolution is https://community.codecov.com/t/upload-issues-unable-to-locate-build-via-github-actions-api/3954 .
> It's possible to make the codecov upload more stable by providing a token. This should be done for the master branch build so that the baseline code coverage metrics would succeed. For pull requests, the solution is to make the codecov upload retryable also in pulsar-ci-flaky.yaml. In addition, it could be made optional for builds in own forks.
>
> We should find a way as a development community to get code coverage metrics solution working. It is valuable even if an individual developer doesn't care about it at the moment.
> We need more Pulsar contributors to stand up that care about the quality aspects of our code base. Any volunteers?
>
> -Lari
>
> On 2023/03/21 10:50:17 tison wrote:
> > For example
> > https://github.com/apache/pulsar/actions/runs/4454158774/jobs/7867745340?pr=19842
> >
> > I'm wondering if anyone cares about the report and if it helps you during
> > the coding or reviewing process? Now it generates a few of noise but I just
> > omit the report it gives ;-)
> >
> > For the issue itself, it seems some artifacts don't retain properly.
> >
> > Best,
> > tison.
> >
>
Re: Unstable codecov action
Posted by Lari Hotari <lh...@apache.org>.
Thanks for sharing the pain. That's the first step in improving something that is painful.
For the flaky tests GitHub Actions workflow pulsar-ci-flaky.yaml, the Codecov upload should be a separate job in the workflow so that the upload could be retried separately without running all tests. This type of approach is already used in the main GitHub Actions workflow, "Pulsar CI".
Contributions are welcome!
We could also consider disabling codecov for pull request builds until someone who cares about test code coverage metrics picks up the work.
Code coverage is the first metric that most will ask about tests. It's not the only metric that matter, but it is something that helps understand what parts of the code isn't even run in our tests. It will also help plan improvements to tests.
Codecov upload fails very frequently with errors such as https://github.com/codecov/codecov-action/issues/837 and https://github.com/codecov/codecov-action/issues/598
One possible resolution is https://community.codecov.com/t/upload-issues-unable-to-locate-build-via-github-actions-api/3954 .
It's possible to make the codecov upload more stable by providing a token. This should be done for the master branch build so that the baseline code coverage metrics would succeed. For pull requests, the solution is to make the codecov upload retryable also in pulsar-ci-flaky.yaml. In addition, it could be made optional for builds in own forks.
We should find a way as a development community to get code coverage metrics solution working. It is valuable even if an individual developer doesn't care about it at the moment.
We need more Pulsar contributors to stand up that care about the quality aspects of our code base. Any volunteers?
-Lari
On 2023/03/21 10:50:17 tison wrote:
> For example
> https://github.com/apache/pulsar/actions/runs/4454158774/jobs/7867745340?pr=19842
>
> I'm wondering if anyone cares about the report and if it helps you during
> the coding or reviewing process? Now it generates a few of noise but I just
> omit the report it gives ;-)
>
> For the issue itself, it seems some artifacts don't retain properly.
>
> Best,
> tison.
>
Re: Unstable codecov action
Posted by Yunze Xu <yz...@streamnative.io.INVALID>.
The codecov is really annoying to me and never helps me during the
coding or reviewing process.
Just a complaint from me. I'm also curious how many developers benefit from it.
Thanks,
Yunze
On Tue, Mar 21, 2023 at 6:50 PM tison <wa...@gmail.com> wrote:
>
> For example
> https://github.com/apache/pulsar/actions/runs/4454158774/jobs/7867745340?pr=19842
>
> I'm wondering if anyone cares about the report and if it helps you during
> the coding or reviewing process? Now it generates a few of noise but I just
> omit the report it gives ;-)
>
> For the issue itself, it seems some artifacts don't retain properly.
>
> Best,
> tison.