You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2023/01/17 15:39:15 UTC
[GitHub] [pulsar] lhotari opened a new pull request, #19263: [feat][ci] Collect code coverage for integration tests
lhotari opened a new pull request, #19263:
URL: https://github.com/apache/pulsar/pull/19263
### Motivation
Code coverage metrics don't currently consider integration tests and systems tests that are run as part of the Pulsar CI pipeline. There hasn't been a way to measure the total code coverage of Pulsar CI pipeline tests although it would be useful to see what parts of Pulsar code base are tested and what aren't.
### Modifications
- Add solution to Pulsar Testcontainer based integration tests so that the Jacoco agent gets activated for processes that are run inside Docker containers
- A directory from the host is mounted in the docker container. This directory is used for storing the Jacoco exec files.
- The code coverage should be uploaded to codecov for all public code repositories. This is now changed so that it's possible to get coverage metrics for builds running in a fork.
- Graceful shutdown of Java processes is needed for collecting Jacoco exec data since by default, the Jacoco agent dumps the exec data to a file in a shutdown hook.
- For processes managed with supervisor, it's necessary to run "supervisorctl stop all" to trigger a graceful shutdown
- It was necessary to stop Zookeeper last for preventing shutdown getting stuck for Brokers or Bookies.
- Creating the Jacoco XML report needed for Codecov is created using Jacoco command line tools.
- The script has been added as part of `build/run_integration_group.sh` script.
- The jar files of the Docker image is needed to create the XML report from the exec files.
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. Please attach the local preview screenshots (run `sh start.sh` at `pulsar/site2/website`) to your PR description, or else your PR might not get merged. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #19263:
URL: https://github.com/apache/pulsar/pull/19263#issuecomment-1385624273
@yaalsn Please review this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] aymkhalil commented on pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
aymkhalil commented on PR #19263:
URL: https://github.com/apache/pulsar/pull/19263#issuecomment-1386059133
I wonder if including integration tests in the code coverage report would hide gaps in unit test coverage (unless the latter coverage metric was already broken :/)
Is there a possibility to separate the coverage report for unit vs integ tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #19263:
URL: https://github.com/apache/pulsar/pull/19263#issuecomment-1386018657
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #19263:
URL: https://github.com/apache/pulsar/pull/19263#issuecomment-1386095634
To experiment, I enabled it for my fork at https://app.codecov.io/github/lhotari/pulsar/flags . For apache/pulsar , I don't have access so I'll have to create a ASF ticket for handling that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #19263:
URL: https://github.com/apache/pulsar/pull/19263#issuecomment-1385795480
Codecov results of the currently running PR build in the fork will be available at https://app.codecov.io/github/lhotari/pulsar/commit/23c1865976414a166e4ff2d6c5874937390cfc9e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #19263:
URL: https://github.com/apache/pulsar/pull/19263#issuecomment-1386089839
@aymkhalil The docs for the flags feature in Codecov: https://docs.codecov.com/docs/flags . It seems that the full feature is disabled by default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #19263:
URL: https://github.com/apache/pulsar/pull/19263#issuecomment-1386074189
> I wonder if including integration tests in the code coverage report would hide gaps in unit test coverage (unless the latter coverage metric was already broken :/)
>
> Is there a possibility to separate the coverage report for unit vs integ tests?
Yes, it should be possible to view unit test coverage separately. They are reported to Codecob with different flags.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari merged pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
lhotari merged PR #19263:
URL: https://github.com/apache/pulsar/pull/19263
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #19263:
URL: https://github.com/apache/pulsar/pull/19263#issuecomment-1385780263
Example of code coverage results for an earlier experiment: https://app.codecov.io/github/lhotari/pulsar/commit/369dec198b5cf6b98cd5c52a0c32126974a8e9ac/tree
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] lhotari commented on pull request #19263: [feat][ci] Collect code coverage for integration tests from docker containers
Posted by GitBox <gi...@apache.org>.
lhotari commented on PR #19263:
URL: https://github.com/apache/pulsar/pull/19263#issuecomment-1385819776
#19264 will also be needed to fix coverage results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org