You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@geode.apache.org by GitBox <gi...@apache.org> on 2020/06/17 21:13:32 UTC
[GitHub] [geode] rhoughton-pivot opened a new pull request #5271: Add Mass Test Run pipeline.
rhoughton-pivot opened a new pull request #5271:
URL: https://github.com/apache/geode/pull/5271
This commit includes:
* Creating a new test run pipeline that runs the distributed test
multiple number of times.
* Update the meta pipeline to add mass test run pipeline.
* Logs into concourse using Vault creds to get fly token.
* Report-generation job runs often, but only generates a report every
N iterations.
* Use 'try' around test job, so we can trigger on it's completion state
Thank you for submitting a contribution to Apache Geode.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
- [X] Has your PR been rebased against the latest commit within the target branch (typically `develop`)?
- [X] Is your initial contribution a single, squashed commit?
- [X] Does `gradlew build` run cleanly?
- [n/a] Have you written or updated unit tests to verify your changes?
- [n/a] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
### Note:
Please ensure that once the PR is submitted, check Concourse for build issues and
submit an update to your PR as soon as possible. If you need help, please send an
email to dev@geode.apache.org.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] rhoughton-pivot commented on a change in pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
rhoughton-pivot commented on a change in pull request #5271:
URL: https://github.com/apache/geode/pull/5271#discussion_r442429688
##########
File path: ci/pipelines/shared/jinja.variables.yml
##########
@@ -67,6 +67,7 @@ java_test_versions:
metadata:
initial_version: 1.14.0-((semver-prerelease-token)).0
+ mass_test_run_iterations: 200
Review comment:
Why would I run 100 jobs against SHA `abcdef` and then run a report containing 200 jobs mixed of `abcdef` and `123456` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] mhansonp commented on pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
mhansonp commented on pull request #5271:
URL: https://github.com/apache/geode/pull/5271#issuecomment-645653390
Shall I merge it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] rhoughton-pivot commented on a change in pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
rhoughton-pivot commented on a change in pull request #5271:
URL: https://github.com/apache/geode/pull/5271#discussion_r442429688
##########
File path: ci/pipelines/shared/jinja.variables.yml
##########
@@ -67,6 +67,7 @@ java_test_versions:
metadata:
initial_version: 1.14.0-((semver-prerelease-token)).0
+ mass_test_run_iterations: 200
Review comment:
Why would I run 100 jobs of `abcdef` and then run a report containing 200 jobs mixed of `abcdef` and `123456` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] onichols-pivotal commented on a change in pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
onichols-pivotal commented on a change in pull request #5271:
URL: https://github.com/apache/geode/pull/5271#discussion_r441858177
##########
File path: ci/pipelines/meta/deploy_meta.sh
##########
@@ -256,6 +256,7 @@ set +x
if [[ "${GEODE_FORK}" != "${UPSTREAM_FORK}" ]]; then
echo "Disabling unnecessary jobs for forks."
pauseJobs ${META_PIPELINE} set-reaper-pipeline
+ pauseJobs ${META_PIPELINE} set-mass-test-run-pipeline
Review comment:
Please also add `enableFeature mass-test-run` to `ci/pipelines/meta/deploy_meta.sh` line ~285 just below where `pr` is enabled
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] rhoughton-pivot merged pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
rhoughton-pivot merged pull request #5271:
URL: https://github.com/apache/geode/pull/5271
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] mhansonp commented on a change in pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
mhansonp commented on a change in pull request #5271:
URL: https://github.com/apache/geode/pull/5271#discussion_r441853889
##########
File path: ci/bin/concourse_job_performance.py
##########
@@ -58,7 +60,7 @@ def main(url, team, pipeline, job, number_of_builds, authorization_cookie, threa
builds = get_builds_summary_sheet(url, team, pipeline, job, number_of_builds+10, authorization_cookie)
build_to_examine = get_builds_to_examine(builds, number_of_builds)
- expected_failed_builds = [int(b['name']) for b in build_to_examine if b['status'] == 'failed']
+ expected_failed_builds = [int(b['name']) for b in build_to_examine if b['status'] in ['failed', 'succeeded']]
Review comment:
You might want to rename the variable if now you are looking at failed and succeeded builds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] mhansonp edited a comment on pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
mhansonp edited a comment on pull request #5271:
URL: https://github.com/apache/geode/pull/5271#issuecomment-645653390
Shall I merge it? It seems like it didn't change much..
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] mhansonp commented on a change in pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
mhansonp commented on a change in pull request #5271:
URL: https://github.com/apache/geode/pull/5271#discussion_r441907733
##########
File path: ci/pipelines/shared/jinja.variables.yml
##########
@@ -67,6 +67,7 @@ java_test_versions:
metadata:
initial_version: 1.14.0-((semver-prerelease-token)).0
+ mass_test_run_iterations: 200
Review comment:
Basically, 200 gives us a big enough sample size to tell the 1 off from the 2 or more. I think running 100 might be fine. Especially if the report generator script is doing a last 200 check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] mhansonp commented on a change in pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
mhansonp commented on a change in pull request #5271:
URL: https://github.com/apache/geode/pull/5271#discussion_r441907733
##########
File path: ci/pipelines/shared/jinja.variables.yml
##########
@@ -67,6 +67,7 @@ java_test_versions:
metadata:
initial_version: 1.14.0-((semver-prerelease-token)).0
+ mass_test_run_iterations: 200
Review comment:
Basically, 200 gives us a big enough sample size to tell the 1 off from the 2 or more. I think running 100 might be fine. Especially if the code is doing a last 200 check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] rhoughton-pivot commented on a change in pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
rhoughton-pivot commented on a change in pull request #5271:
URL: https://github.com/apache/geode/pull/5271#discussion_r441861412
##########
File path: ci/pipelines/meta/deploy_meta.sh
##########
@@ -256,6 +256,7 @@ set +x
if [[ "${GEODE_FORK}" != "${UPSTREAM_FORK}" ]]; then
echo "Disabling unnecessary jobs for forks."
pauseJobs ${META_PIPELINE} set-reaper-pipeline
+ pauseJobs ${META_PIPELINE} set-mass-test-run-pipeline
Review comment:
done
##########
File path: ci/bin/concourse_job_performance.py
##########
@@ -58,7 +60,7 @@ def main(url, team, pipeline, job, number_of_builds, authorization_cookie, threa
builds = get_builds_summary_sheet(url, team, pipeline, job, number_of_builds+10, authorization_cookie)
build_to_examine = get_builds_to_examine(builds, number_of_builds)
- expected_failed_builds = [int(b['name']) for b in build_to_examine if b['status'] == 'failed']
+ expected_failed_builds = [int(b['name']) for b in build_to_examine if b['status'] in ['failed', 'succeeded']]
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] onichols-pivotal commented on a change in pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
onichols-pivotal commented on a change in pull request #5271:
URL: https://github.com/apache/geode/pull/5271#discussion_r441869161
##########
File path: ci/pipelines/shared/jinja.variables.yml
##########
@@ -67,6 +67,7 @@ java_test_versions:
metadata:
initial_version: 1.14.0-((semver-prerelease-token)).0
+ mass_test_run_iterations: 200
Review comment:
@mhansonp I know there's been discussion in the past about how much value we get out of 100 vs 200. Can you mention the benefits of 200 vs 100? My back-of-the-envelope calculations suggest 200 takes about 4 days, while 100 could be finished in a weekend (and about $900 cheaper, if anyone cares)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [geode] mhansonp commented on a change in pull request #5271: Add Mass Test Run pipeline.
Posted by GitBox <gi...@apache.org>.
mhansonp commented on a change in pull request #5271:
URL: https://github.com/apache/geode/pull/5271#discussion_r442430968
##########
File path: ci/pipelines/shared/jinja.variables.yml
##########
@@ -67,6 +67,7 @@ java_test_versions:
metadata:
initial_version: 1.14.0-((semver-prerelease-token)).0
+ mass_test_run_iterations: 200
Review comment:
So you would use the SHA as normal, but you would run 100 tests and report against the last 200. That will give us a historical result set for the last two weeks. Or to just keep it simple, we can just do 100 and report on 100.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org