You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2021/04/07 20:55:10 UTC
[GitHub] [incubator-mxnet] barry-jin opened a new pull request #20138: [CI] Fix codecoverage
barry-jin opened a new pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138
## Description ##
Looks like in some PRs, code coverage report will be submitted to some unknown places, like:
https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fcentos-cpu/detail/PR-20105/8/pipeline/173/
Commit head is 622f0b3, but the report will be submitted to https://codecov.io/github/apache/incubator-mxnet/commit/395b8c9c168e7941c9d96e3a4a1328783cc72956
https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20107/4/pipeline/295
Commit head is cab1562, but the report will be submitted to https://codecov.io/github/apache/incubator-mxnet/commit/b5b7dbc2374ceb7b595943b8d13abd60ba32841b
So, this PR will add codecovArgs back.
## Checklist ##
### Essentials ###
- [x] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
- [x] Changes are complete (i.e. I finished coding on this PR)
- [x] All changes have test coverage
- [x] Code is well-documented
### Changes ###
- [ ] Feature1, tests, (and when applicable, API doc)
- [ ] Feature2, tests, (and when applicable, API doc)
## Comments ##
- If this change is a backward incompatible change, why must this change be made.
- Interesting edge cases to note here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on a change in pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
barry-jin commented on a change in pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#discussion_r610034846
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -783,11 +783,11 @@ unittest_ubuntu_python3_cpu_onednn() {
export MXNET_SUBGRAPH_VERBOSE=0
export MXNET_ENABLE_CYTHON=0
export DMLC_LOG_STACK_TRACE_DEPTH=100
- OMP_NUM_THREADS=$(expr $(nproc) / 4) pytest -m 'not serial' -k 'not test_operator' -n 4 --durations=50 --cov-report xml:tests_unittest.xml --verbose tests/python/unittest
+ OMP_NUM_THREADS=$(expr $(nproc) / 4) pytest -m 'not serial' -k 'not test_operator' -n 4 --durations=50 --cov=./contrib --cov=./python/mxnet --cov-report xml:tests_unittest.xml --verbose tests/python/unittest
MXNET_ENGINE_TYPE=NaiveEngine \
- OMP_NUM_THREADS=$(expr $(nproc) / 4) pytest -m 'not serial' -k 'test_operator' -n 4 --durations=50 --cov-report xml:tests_unittest.xml --cov-append --verbose tests/python/unittest
- pytest -m 'serial' --durations=50 --cov-report xml:tests_unittest.xml --cov-append --verbose tests/python/unittest
- pytest --durations=50 --cov-report xml:tests_mkl.xml --verbose tests/python/mkl
+ OMP_NUM_THREADS=$(expr $(nproc) / 4) pytest -m 'not serial' -k 'test_operator' -n 4 --durations=50 --cov=./contrib --cov=./python/mxnet --cov-report xml:tests_unittest.xml --cov-append --verbose tests/python/unittest
+ pytest -m 'serial' --durations=50 --cov=./contrib --cov=./python/mxnet --cov-report xml:tests_unittest.xml --cov-append --verbose tests/python/unittest
+ pytest --durations=50 --cov=./contrib --cov=./python/mxnet --cov-report xml:tests_mkl.xml --verbose tests/python/mkl
Review comment:
Thanks for the suggestion. I will update it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-816894937
@mxnet-bot run ci [centos-cpu, windows-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
barry-jin commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-816343910
@mxnet-bot run ci [unix-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#discussion_r610026556
##########
File path: ci/Jenkinsfile_utils.groovy
##########
@@ -112,7 +112,18 @@ def get_git_commit_hash() {
}
def publish_test_coverage() {
- sh "curl -s https://codecov.io/bash | bash"
+ // CodeCovs auto detection has trouble with our CIs PR validation due the merging strategy
+ git_commit_hash = get_git_commit_hash()
+
+ if (env.CHANGE_ID) {
+ // PR execution
+ codecovArgs = "-f '*.xml' -B ${env.CHANGE_TARGET} -P ${env.CHANGE_ID} -C ${git_commit_hash}"
Review comment:
Why is this filter necessary? I think codecov is able to find the reports on its own and also there might be other coverage reports that are not xml Files - like gcov..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-817051835
Is there a way to automatically invoke coverage as part of the pytest stuff? @leezu maybe you have some ideas?
To me it seems a bit odd that the only way is to do all this via invoke arguments. Is there not some way we can do this in the pytest modules or configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-815272659
FYI @josephevans https://github.com/apache/incubator-mxnet/commit/2c4f5aad26b4d7aefafaf6263ccf4aa11096d2fd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] leezu commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
leezu commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-816946290
@mxnet-bot run ci [unix-cpu, windows-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
marcoabreu commented on a change in pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#discussion_r610025602
##########
File path: ci/docker/runtime_functions.sh
##########
@@ -783,11 +783,11 @@ unittest_ubuntu_python3_cpu_onednn() {
export MXNET_SUBGRAPH_VERBOSE=0
export MXNET_ENABLE_CYTHON=0
export DMLC_LOG_STACK_TRACE_DEPTH=100
- OMP_NUM_THREADS=$(expr $(nproc) / 4) pytest -m 'not serial' -k 'not test_operator' -n 4 --durations=50 --cov-report xml:tests_unittest.xml --verbose tests/python/unittest
+ OMP_NUM_THREADS=$(expr $(nproc) / 4) pytest -m 'not serial' -k 'not test_operator' -n 4 --durations=50 --cov=./contrib --cov=./python/mxnet --cov-report xml:tests_unittest.xml --verbose tests/python/unittest
MXNET_ENGINE_TYPE=NaiveEngine \
- OMP_NUM_THREADS=$(expr $(nproc) / 4) pytest -m 'not serial' -k 'test_operator' -n 4 --durations=50 --cov-report xml:tests_unittest.xml --cov-append --verbose tests/python/unittest
- pytest -m 'serial' --durations=50 --cov-report xml:tests_unittest.xml --cov-append --verbose tests/python/unittest
- pytest --durations=50 --cov-report xml:tests_mkl.xml --verbose tests/python/mkl
+ OMP_NUM_THREADS=$(expr $(nproc) / 4) pytest -m 'not serial' -k 'test_operator' -n 4 --durations=50 --cov=./contrib --cov=./python/mxnet --cov-report xml:tests_unittest.xml --cov-append --verbose tests/python/unittest
+ pytest -m 'serial' --durations=50 --cov=./contrib --cov=./python/mxnet --cov-report xml:tests_unittest.xml --cov-append --verbose tests/python/unittest
+ pytest --durations=50 --cov=./contrib --cov=./python/mxnet --cov-report xml:tests_mkl.xml --verbose tests/python/mkl
Review comment:
Can't we have this in some configuration file instead of inlining it everywhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-815256702
Hey @barry-jin , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:
- To trigger all jobs: @mxnet-bot run ci [all]
- To trigger specific jobs: @mxnet-bot run ci [job1, job2]
***
**CI supported jobs**: [windows-gpu, miscellaneous, unix-cpu, website, unix-gpu, centos-cpu, clang, centos-gpu, sanity, windows-cpu, edge]
***
_Note_:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-816895020
Jenkins CI successfully triggered : [centos-cpu, windows-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin edited a comment on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
barry-jin edited a comment on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-816343082
> I hope this is just a test :)
Yes, the log shows "Python coveragepy not found". Looking into [codecov script](https://github.com/codecov/codecov-bash/blob/54be24366124fc91af6609e18c62f511b51f1889/codecov#L1234-L1252), it relies on coverage package to combine these xml reports. So, I add this test to check if its the root cause. Later I will remove "pip install coverage==5.5" and update the Jenkins environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
barry-jin commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-816890359
The coverage report should be fixed now https://app.codecov.io/gh/apache/incubator-mxnet/compare/20138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
barry-jin commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-816343082
> I hope this is just a test :)
Yes, it shows "Python coveragepy not found". Looking into [codecov script](https://github.com/codecov/codecov-bash/blob/54be24366124fc91af6609e18c62f511b51f1889/codecov#L1234-L1252), it relies on coverage package to combine these xml reports. So, I add this test to check if its the root cause. Later I will remove "pip install coverage==5.5" and update the Jenkins environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-816946487
Jenkins CI successfully triggered : [unix-cpu, windows-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] mxnet-bot commented on pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
mxnet-bot commented on pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#issuecomment-816343927
Jenkins CI successfully triggered : [unix-cpu]
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on a change in pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
barry-jin commented on a change in pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#discussion_r610044704
##########
File path: ci/Jenkinsfile_utils.groovy
##########
@@ -112,7 +112,18 @@ def get_git_commit_hash() {
}
def publish_test_coverage() {
- sh "curl -s https://codecov.io/bash | bash"
+ // CodeCovs auto detection has trouble with our CIs PR validation due the merging strategy
+ git_commit_hash = get_git_commit_hash()
+
+ if (env.CHANGE_ID) {
+ // PR execution
+ codecovArgs = "-f '*.xml' -B ${env.CHANGE_TARGET} -P ${env.CHANGE_ID} -C ${git_commit_hash}"
Review comment:
It's not necessary. I will remove it.
The strange thing is that the codecoverage xml report was generated during the test, but codecov script didn't find the xml file.
https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20138/2/pipeline/283
When we use nose to run these tests, there will have seperate xml report files generated, like nosetests_unittest.xml and nosetests_mkl.xml, but the final submitted report is coverage.xml
You can find in
https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-19876/1/pipeline/297
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin commented on a change in pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
barry-jin commented on a change in pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138#discussion_r610044704
##########
File path: ci/Jenkinsfile_utils.groovy
##########
@@ -112,7 +112,18 @@ def get_git_commit_hash() {
}
def publish_test_coverage() {
- sh "curl -s https://codecov.io/bash | bash"
+ // CodeCovs auto detection has trouble with our CIs PR validation due the merging strategy
+ git_commit_hash = get_git_commit_hash()
+
+ if (env.CHANGE_ID) {
+ // PR execution
+ codecovArgs = "-f '*.xml' -B ${env.CHANGE_TARGET} -P ${env.CHANGE_ID} -C ${git_commit_hash}"
Review comment:
It's not necessary. I will remove it.
The strange thing is that the codecoverage xml report was generated during the test, but codecov script didn't find the xml file.
https://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-20138/2/pipeline/283
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-mxnet] barry-jin closed pull request #20138: [CI] Fix codecoverage
Posted by GitBox <gi...@apache.org>.
barry-jin closed pull request #20138:
URL: https://github.com/apache/incubator-mxnet/pull/20138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org