You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "andreydevyatkin (via GitHub)" <gi...@apache.org> on 2024/02/26 20:39:34 UTC
[PR] Fix issue creation for grafana alerts [beam]
andreydevyatkin opened a new pull request, #30424:
URL: https://github.com/apache/beam/pull/30424
This is a continuation of the following PRs:
- #30345
- #30344
- #30327
This PR fixes the bug with issue creation for grafana alerts and the following changes have been added:
- Enhanced the logic to determine if the threshold has been exceeded and moved from Grafana to the `github_runs_prefetcher` script
- If the issue was closed, only consider new workflow runs after the issue's `closed_at` date
- Changed Grafana alert rule to "alert per row"
- Re-open the issue if the workflow became flaky again
### How has it been tested?
Tested on the fork https://github.com/andreydevyatkin/beam
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "andreydevyatkin (via GitHub)" <gi...@apache.org>.
andreydevyatkin commented on code in PR #30424:
URL: https://github.com/apache/beam/pull/30424#discussion_r1504544020
##########
.test-infra/metrics/grafana/dashboards/GA-Post-Commits_status_dashboard.json:
##########
@@ -6605,7 +6605,7 @@
"hide": false,
"metricColumn": "none",
"rawQuery": true,
- "rawSql": "with workflows as (\n\twith temp as (SELECT t1.workflow_id, t1.job_name, t1.job_yml_filename, t1.dashboard_category, t1.run1, t2.run1id ,\nt1.run2, t2.run2id ,\nt1.run3, t2.run3id ,\nt1.run4, t2.run4id ,\nt1.run5, t2.run5id ,\nt1.run6, t2.run6id ,\nt1.run7, t2.run7id ,\nt1.run8, t2.run8id ,\nt1.run9, t2.run9id ,\nt1.run10, t2.run10id \nFROM (SELECT * FROM crosstab('SELECT github_workflows.workflow_id, name, filename, dashboard_category, run_number, status \n FROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs.workflow_id = github_workflows.workflow_id\nORDER BY 1,5 DESC\n','SELECT m from generate_series(1,10) m')\nAS c1(workflow_id text, job_name text, job_yml_filename text, dashboard_category text, run1 text, run2 text,run3 text, \n run4 text ,run5 text,run6 text, run7 text, \n run8 text ,run9 text,run10 text) \n\t ) AS t1\n JOIN (SELECT * FROM crosstab('SELECT name, status, github_
workflow_runs.url AS run_url\nFROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs.workflow_id = github_workflows.workflow_id\nORDER BY 1,3 DESC\n\n')\nAS c2(job_name text, run1id text, run2id text,run3id text, \n run4id text ,run5id text,run6id text, run7id text, \n run8id text ,run9id text,run10id text) \n\t\t\t ) AS t2\n ON t1.job_name = t2.job_name\n\t\t )\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'failure' then 0 else 4 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'failure' then 0 else 4 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in_progress' then 2 when run3 like 'queued' then 2 w
hen run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'failure' then 0 else 4 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'failure' then 0 else 4 end as run_4,\n case when run5 like 'success' then 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'failure' then 0 else 4 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'failure' then 0 else 4 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when run7 like 'cancelled' then 3 when run7 like 'fail
ure' then 0 else 4 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'failure' then 0 else 4 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' then 2 when run9 like 'cancelled' then 3 when run9 like 'failure' then 0 else 4 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'failure' then 0 else 4 end as run_10\n from\n temp\n where\n dashboard_category = 'runners_python'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n
run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
+ "rawSql": "with workflows as (\n\twith temp as (SELECT t1.workflow_id, t1.workflow_name, t1.workflow_filename, t1.dashboard_category, t1.run1, t2.run1id ,\nt1.run2, t2.run2id ,\nt1.run3, t2.run3id ,\nt1.run4, t2.run4id ,\nt1.run5, t2.run5id ,\nt1.run6, t2.run6id ,\nt1.run7, t2.run7id ,\nt1.run8, t2.run8id ,\nt1.run9, t2.run9id ,\nt1.run10, t2.run10id \nFROM (SELECT * FROM crosstab('SELECT github_workflows.workflow_id, name as workflow_name, filename AS workflow_name, dashboard_category, run_number, status \n FROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs.workflow_id = github_workflows.workflow_id\nORDER BY 1,5 DESC\n','SELECT m from generate_series(1,10) m')\nAS c1(workflow_id text, workflow_name text, workflow_filename text, dashboard_category text, run1 text, run2 text,run3 text, \n run4 text ,run5 text,run6 text, run7 text, \n run8 text ,run9 text,run10 text) \n\t ) AS t1\n JOIN (SELECT
* FROM crosstab('SELECT name, status, github_workflow_runs.url AS run_url\nFROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs.workflow_id = github_workflows.workflow_id\nORDER BY 1,3 DESC\n\n')\nAS c2(workflow_name text, run1id text, run2id text,run3id text, \n run4id text ,run5id text,run6id text, run7id text, \n run8id text ,run9id text,run10id text) \n\t\t\t ) AS t2\n ON t1.workflow_name = t2.workflow_name\n\t\t )\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'failure' then 0 else 4 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'failure' then 0 else 4 end as run_2,\n case when run3 like 'success' then 1 when ru
n3 like 'in_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'failure' then 0 else 4 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'failure' then 0 else 4 end as run_4,\n case when run5 like 'success' then 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'failure' then 0 else 4 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'failure' then 0 else 4 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting'
then 2 when run7 like 'cancelled' then 3 when run7 like 'failure' then 0 else 4 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'failure' then 0 else 4 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' then 2 when run9 like 'cancelled' then 3 when run9 like 'failure' then 0 else 4 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'failure' then 0 else 4 end as run_10\n from\n temp\n where\n dashboard_category = 'runners_python'\n)\nselect\n workflow_name,\n workflow_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5
,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
Review Comment:
They were missed in the previous PR, so I fixed them as part of this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "andreydevyatkin (via GitHub)" <gi...@apache.org>.
andreydevyatkin commented on code in PR #30424:
URL: https://github.com/apache/beam/pull/30424#discussion_r1504561031
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py:
##########
@@ -404,16 +454,11 @@ def save_workflows(workflows):
workflow.url,
workflow.category,
workflow.threshold,
+ workflow.is_flaky,
)
)
- run_number = 1
- for run in workflow.runs:
- if run.id != 0:
Review Comment:
The idea behind this check is to prevent adding runs with the same run_id, because the implementation used to assign the "0" as the default id. Now this check is no longer applicable because we generate uuid for the id field (https://github.com/apache/beam/pull/30424/files#diff-a99a5e5ac1dc4bf7f4d42360354deec95414d660122bd96214d190b92fee2340R390)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #30424:
URL: https://github.com/apache/beam/pull/30424#discussion_r1504506481
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py:
##########
@@ -404,16 +454,11 @@ def save_workflows(workflows):
workflow.url,
workflow.category,
workflow.threshold,
+ workflow.is_flaky,
)
)
- run_number = 1
- for run in workflow.runs:
- if run.id != 0:
Review Comment:
Is it safe to ignore this run.id check? I assume that's filtering out some kind of cancelled or in progress run
##########
.test-infra/metrics/grafana/dashboards/GA-Post-Commits_status_dashboard.json:
##########
@@ -6605,7 +6605,7 @@
"hide": false,
"metricColumn": "none",
"rawQuery": true,
- "rawSql": "with workflows as (\n\twith temp as (SELECT t1.workflow_id, t1.job_name, t1.job_yml_filename, t1.dashboard_category, t1.run1, t2.run1id ,\nt1.run2, t2.run2id ,\nt1.run3, t2.run3id ,\nt1.run4, t2.run4id ,\nt1.run5, t2.run5id ,\nt1.run6, t2.run6id ,\nt1.run7, t2.run7id ,\nt1.run8, t2.run8id ,\nt1.run9, t2.run9id ,\nt1.run10, t2.run10id \nFROM (SELECT * FROM crosstab('SELECT github_workflows.workflow_id, name, filename, dashboard_category, run_number, status \n FROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs.workflow_id = github_workflows.workflow_id\nORDER BY 1,5 DESC\n','SELECT m from generate_series(1,10) m')\nAS c1(workflow_id text, job_name text, job_yml_filename text, dashboard_category text, run1 text, run2 text,run3 text, \n run4 text ,run5 text,run6 text, run7 text, \n run8 text ,run9 text,run10 text) \n\t ) AS t1\n JOIN (SELECT * FROM crosstab('SELECT name, status, github_
workflow_runs.url AS run_url\nFROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs.workflow_id = github_workflows.workflow_id\nORDER BY 1,3 DESC\n\n')\nAS c2(job_name text, run1id text, run2id text,run3id text, \n run4id text ,run5id text,run6id text, run7id text, \n run8id text ,run9id text,run10id text) \n\t\t\t ) AS t2\n ON t1.job_name = t2.job_name\n\t\t )\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'failure' then 0 else 4 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'failure' then 0 else 4 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in_progress' then 2 when run3 like 'queued' then 2 w
hen run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'failure' then 0 else 4 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'failure' then 0 else 4 end as run_4,\n case when run5 like 'success' then 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'failure' then 0 else 4 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'failure' then 0 else 4 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when run7 like 'cancelled' then 3 when run7 like 'fail
ure' then 0 else 4 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'failure' then 0 else 4 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' then 2 when run9 like 'cancelled' then 3 when run9 like 'failure' then 0 else 4 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'failure' then 0 else 4 end as run_10\n from\n temp\n where\n dashboard_category = 'runners_python'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n
run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
+ "rawSql": "with workflows as (\n\twith temp as (SELECT t1.workflow_id, t1.workflow_name, t1.workflow_filename, t1.dashboard_category, t1.run1, t2.run1id ,\nt1.run2, t2.run2id ,\nt1.run3, t2.run3id ,\nt1.run4, t2.run4id ,\nt1.run5, t2.run5id ,\nt1.run6, t2.run6id ,\nt1.run7, t2.run7id ,\nt1.run8, t2.run8id ,\nt1.run9, t2.run9id ,\nt1.run10, t2.run10id \nFROM (SELECT * FROM crosstab('SELECT github_workflows.workflow_id, name as workflow_name, filename AS workflow_name, dashboard_category, run_number, status \n FROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs.workflow_id = github_workflows.workflow_id\nORDER BY 1,5 DESC\n','SELECT m from generate_series(1,10) m')\nAS c1(workflow_id text, workflow_name text, workflow_filename text, dashboard_category text, run1 text, run2 text,run3 text, \n run4 text ,run5 text,run6 text, run7 text, \n run8 text ,run9 text,run10 text) \n\t ) AS t1\n JOIN (SELECT
* FROM crosstab('SELECT name, status, github_workflow_runs.url AS run_url\nFROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs.workflow_id = github_workflows.workflow_id\nORDER BY 1,3 DESC\n\n')\nAS c2(workflow_name text, run1id text, run2id text,run3id text, \n run4id text ,run5id text,run6id text, run7id text, \n run8id text ,run9id text,run10id text) \n\t\t\t ) AS t2\n ON t1.workflow_name = t2.workflow_name\n\t\t )\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'failure' then 0 else 4 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'failure' then 0 else 4 end as run_2,\n case when run3 like 'success' then 1 when ru
n3 like 'in_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'failure' then 0 else 4 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'failure' then 0 else 4 end as run_4,\n case when run5 like 'success' then 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'failure' then 0 else 4 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'failure' then 0 else 4 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting'
then 2 when run7 like 'cancelled' then 3 when run7 like 'failure' then 0 else 4 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'failure' then 0 else 4 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' then 2 when run9 like 'cancelled' then 3 when run9 like 'failure' then 0 else 4 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'failure' then 0 else 4 end as run_10\n from\n temp\n where\n dashboard_category = 'runners_python'\n)\nselect\n workflow_name,\n workflow_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5
,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
Review Comment:
Is there a reason we only made this change to some of the queries, not all of them?
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py:
##########
@@ -143,6 +144,41 @@ def enhance_workflow(workflow):
print(f"No yaml file found for workflow: {workflow.name}")
+async def check_workflow_flakiness(workflow):
+ def filter_workflow_runs(run, issue):
+ started_at = datetime.strptime(run.started_at, "%Y-%m-%dT%H:%M:%SZ")
+ closed_at = datetime.strptime(issue["closed_at"], "%Y-%m-%dT%H:%M:%SZ")
+ if started_at > closed_at:
+ return True
+ return False
+
+ if not len(workflow.runs):
+ return False
+
+ url = f"https://api.github.com/repos/{GIT_ORG}/beam/issues"
+ headers = {"Authorization": get_token()}
+ semaphore = asyncio.Semaphore(5)
+ workflow_runs = workflow.runs
+ params = {
+ "state": "closed",
+ "labels": f"flaky_test,workflow_id: {workflow.id}",
+ }
+ response = await fetch(url, semaphore, params, headers)
+ if len(response):
+ print(f"Found a recently closed issue for the {workflow.name} workflow")
+ workflow_runs = [run for run in workflow_runs if filter_workflow_runs(run, response[0])]
+
+ print(f"Number of workflow runs to consider: {len(workflow_runs)}")
+ success_rate = 1.0
+ if len(workflow_runs):
+ failed_runs = list(filter(lambda r: r.status == "failure", workflow_runs))
+ print(f"Number of failed workflow runs: {len(failed_runs)}")
+ success_rate -= (len(failed_runs) / len(workflow_runs))
+
+ print(f"Success rate: {success_rate}")
+ return True if success_rate < workflow.threshold else False
Review Comment:
Can we also wait until there are at least 3 failures to trigger? Could be just bad luck with an extremely flaky test otherwise, or a timing issue with when the fix went in, and its more helpful to have a consistent signal IMO
##########
.test-infra/tools/flaky_test_detection.py:
##########
@@ -103,17 +97,23 @@ def main():
token = os.environ["GITHUB_TOKEN"]
auth = Auth.Token(token)
g = Github(auth=auth)
- repo = g.get_repo(f"{GIT_ORG}/{GIT_REPO}")
+ repo = g.get_repo(f"{GIT_ORG}/beam")
alerts = get_grafana_alerts()
open_issues = repo.get_issues(state="open", labels=["flaky_test"])
- workflow_ids = extract_workflow_id_from_issue_label(open_issues)
+ closed_issues = repo.get_issues(state="closed", labels=["flaky_test"])
+ workflow_open_issues = get_workflow_issues(open_issues)
+ workflow_closed_issues = get_workflow_issues(closed_issues)
for alert in alerts:
- if alert.workflow_id not in workflow_ids:
+ if alert.workflow_id in workflow_closed_issues.keys():
+ issue = workflow_closed_issues[alert.workflow_id]
+ if issue:
+ issue.edit(state="open")
Review Comment:
Can we also add a comment saying something like "reopening since the workflow is still flaky"
##########
.test-infra/tools/flaky_test_detection.py:
##########
@@ -65,11 +63,7 @@ def create_github_issue(repo, alert):
print(f"Body: {body}")
print(f"Labels: {labels}")
print("___")
-
- if READ_ONLY == "true":
- print("READ_ONLY is true, not creating issue")
- else:
- repo.create_issue(title=title, body=body, labels=labels)
+ repo.create_issue(title=title, body=body, labels=labels)
Review Comment:
Can we leave the READ_ONLY setting? Seems useful for local debugging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "andreydevyatkin (via GitHub)" <gi...@apache.org>.
andreydevyatkin commented on code in PR #30424:
URL: https://github.com/apache/beam/pull/30424#discussion_r1504660326
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py:
##########
@@ -143,6 +144,41 @@ def enhance_workflow(workflow):
print(f"No yaml file found for workflow: {workflow.name}")
+async def check_workflow_flakiness(workflow):
+ def filter_workflow_runs(run, issue):
+ started_at = datetime.strptime(run.started_at, "%Y-%m-%dT%H:%M:%SZ")
+ closed_at = datetime.strptime(issue["closed_at"], "%Y-%m-%dT%H:%M:%SZ")
+ if started_at > closed_at:
+ return True
+ return False
+
+ if not len(workflow.runs):
+ return False
+
+ url = f"https://api.github.com/repos/{GIT_ORG}/beam/issues"
+ headers = {"Authorization": get_token()}
+ semaphore = asyncio.Semaphore(5)
+ workflow_runs = workflow.runs
+ params = {
+ "state": "closed",
+ "labels": f"flaky_test,workflow_id: {workflow.id}",
+ }
+ response = await fetch(url, semaphore, params, headers)
+ if len(response):
+ print(f"Found a recently closed issue for the {workflow.name} workflow")
+ workflow_runs = [run for run in workflow_runs if filter_workflow_runs(run, response[0])]
+
+ print(f"Number of workflow runs to consider: {len(workflow_runs)}")
+ success_rate = 1.0
+ if len(workflow_runs):
+ failed_runs = list(filter(lambda r: r.status == "failure", workflow_runs))
+ print(f"Number of failed workflow runs: {len(failed_runs)}")
+ success_rate -= (len(failed_runs) / len(workflow_runs))
+
+ print(f"Success rate: {success_rate}")
+ return True if success_rate < workflow.threshold else False
Review Comment:
Should we limit number of runs instead of number of failures?
Let's say we have a threshold=0.8 and the following sequence of results (s - success , f - failure): s, f, s, f, s. So, in fact we have 5 runs that seem suspicious and the actual success rate is 0.6 - shouldn't the result be a reason to create an issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "andreydevyatkin (via GitHub)" <gi...@apache.org>.
andreydevyatkin commented on code in PR #30424:
URL: https://github.com/apache/beam/pull/30424#discussion_r1508075573
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py:
##########
@@ -143,6 +144,41 @@ def enhance_workflow(workflow):
print(f"No yaml file found for workflow: {workflow.name}")
+async def check_workflow_flakiness(workflow):
+ def filter_workflow_runs(run, issue):
+ started_at = datetime.strptime(run.started_at, "%Y-%m-%dT%H:%M:%SZ")
+ closed_at = datetime.strptime(issue["closed_at"], "%Y-%m-%dT%H:%M:%SZ")
+ if started_at > closed_at:
+ return True
+ return False
+
+ if not len(workflow.runs):
+ return False
+
+ url = f"https://api.github.com/repos/{GIT_ORG}/beam/issues"
+ headers = {"Authorization": get_token()}
+ semaphore = asyncio.Semaphore(5)
+ workflow_runs = workflow.runs
+ params = {
+ "state": "closed",
+ "labels": f"flaky_test,workflow_id: {workflow.id}",
+ }
+ response = await fetch(url, semaphore, params, headers)
+ if len(response):
+ print(f"Found a recently closed issue for the {workflow.name} workflow")
+ workflow_runs = [run for run in workflow_runs if filter_workflow_runs(run, response[0])]
+
+ print(f"Number of workflow runs to consider: {len(workflow_runs)}")
+ success_rate = 1.0
+ if len(workflow_runs):
+ failed_runs = list(filter(lambda r: r.status == "failure", workflow_runs))
+ print(f"Number of failed workflow runs: {len(failed_runs)}")
+ success_rate -= (len(failed_runs) / len(workflow_runs))
+
+ print(f"Success rate: {success_rate}")
+ return True if success_rate < workflow.threshold else False
Review Comment:
Added a limit on the min number of runs to consider https://github.com/apache/beam/pull/30424/files#diff-a99a5e5ac1dc4bf7f4d42360354deec95414d660122bd96214d190b92fee2340R172
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #30424:
URL: https://github.com/apache/beam/pull/30424#discussion_r1507673562
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py:
##########
@@ -143,6 +144,41 @@ def enhance_workflow(workflow):
print(f"No yaml file found for workflow: {workflow.name}")
+async def check_workflow_flakiness(workflow):
+ def filter_workflow_runs(run, issue):
+ started_at = datetime.strptime(run.started_at, "%Y-%m-%dT%H:%M:%SZ")
+ closed_at = datetime.strptime(issue["closed_at"], "%Y-%m-%dT%H:%M:%SZ")
+ if started_at > closed_at:
+ return True
+ return False
+
+ if not len(workflow.runs):
+ return False
+
+ url = f"https://api.github.com/repos/{GIT_ORG}/beam/issues"
+ headers = {"Authorization": get_token()}
+ semaphore = asyncio.Semaphore(5)
+ workflow_runs = workflow.runs
+ params = {
+ "state": "closed",
+ "labels": f"flaky_test,workflow_id: {workflow.id}",
+ }
+ response = await fetch(url, semaphore, params, headers)
+ if len(response):
+ print(f"Found a recently closed issue for the {workflow.name} workflow")
+ workflow_runs = [run for run in workflow_runs if filter_workflow_runs(run, response[0])]
+
+ print(f"Number of workflow runs to consider: {len(workflow_runs)}")
+ success_rate = 1.0
+ if len(workflow_runs):
+ failed_runs = list(filter(lambda r: r.status == "failure", workflow_runs))
+ print(f"Number of failed workflow runs: {len(failed_runs)}")
+ success_rate -= (len(failed_runs) / len(workflow_runs))
+
+ print(f"Success rate: {success_rate}")
+ return True if success_rate < workflow.threshold else False
Review Comment:
I think I'm more worried about the case where:
1) Change to fix the workflow goes in, but maybe all pieces aren't fully deployed or something like that where the change isn't instantaneous. This would depend on the workflow.
2) We go f, s, s, s, s, ....
Right now, this would fire immediately after the first failure. On the other hand, if we go:
f, f, f, f, f ...., I'd rather fire after the second or third failure instead of waiting 5 or 10 runs. Maybe just waiting for 2 failures is enough (vs 3), but either way gating on # failures instead of # runs gets us to fire a little faster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "andreydevyatkin (via GitHub)" <gi...@apache.org>.
andreydevyatkin commented on code in PR #30424:
URL: https://github.com/apache/beam/pull/30424#discussion_r1504561031
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py:
##########
@@ -404,16 +454,11 @@ def save_workflows(workflows):
workflow.url,
workflow.category,
workflow.threshold,
+ workflow.is_flaky,
)
)
- run_number = 1
- for run in workflow.runs:
- if run.id != 0:
Review Comment:
The idea behind this check is to prevent adding runs with the same run_id, because the implementation used to assign the "0" as the default id and run_id is a PK in the database. Now this check is no longer applicable because we generate uuid for the id field (https://github.com/apache/beam/pull/30424/files#diff-a99a5e5ac1dc4bf7f4d42360354deec95414d660122bd96214d190b92fee2340R390)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #30424:
URL: https://github.com/apache/beam/pull/30424#discussion_r1507757932
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/main.py:
##########
@@ -143,6 +144,41 @@ def enhance_workflow(workflow):
print(f"No yaml file found for workflow: {workflow.name}")
+async def check_workflow_flakiness(workflow):
+ def filter_workflow_runs(run, issue):
+ started_at = datetime.strptime(run.started_at, "%Y-%m-%dT%H:%M:%SZ")
+ closed_at = datetime.strptime(issue["closed_at"], "%Y-%m-%dT%H:%M:%SZ")
+ if started_at > closed_at:
+ return True
+ return False
+
+ if not len(workflow.runs):
+ return False
+
+ url = f"https://api.github.com/repos/{GIT_ORG}/beam/issues"
+ headers = {"Authorization": get_token()}
+ semaphore = asyncio.Semaphore(5)
+ workflow_runs = workflow.runs
+ params = {
+ "state": "closed",
+ "labels": f"flaky_test,workflow_id: {workflow.id}",
+ }
+ response = await fetch(url, semaphore, params, headers)
+ if len(response):
+ print(f"Found a recently closed issue for the {workflow.name} workflow")
+ workflow_runs = [run for run in workflow_runs if filter_workflow_runs(run, response[0])]
+
+ print(f"Number of workflow runs to consider: {len(workflow_runs)}")
+ success_rate = 1.0
+ if len(workflow_runs):
+ failed_runs = list(filter(lambda r: r.status == "failure", workflow_runs))
+ print(f"Number of failed workflow runs: {len(failed_runs)}")
+ success_rate -= (len(failed_runs) / len(workflow_runs))
+
+ print(f"Success rate: {success_rate}")
+ return True if success_rate < workflow.threshold else False
Review Comment:
Talked offline, I think we can move forward with making sure we have at least 3 runs (not failures)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #30424:
URL: https://github.com/apache/beam/pull/30424
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30424:
URL: https://github.com/apache/beam/pull/30424#issuecomment-1965280242
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @robertwb added as fallback since no labels match configuration
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix issue creation for grafana alerts [beam]
Posted by "andreydevyatkin (via GitHub)" <gi...@apache.org>.
andreydevyatkin commented on code in PR #30424:
URL: https://github.com/apache/beam/pull/30424#discussion_r1504676248
##########
.test-infra/tools/flaky_test_detection.py:
##########
@@ -65,11 +63,7 @@ def create_github_issue(repo, alert):
print(f"Body: {body}")
print(f"Labels: {labels}")
print("___")
-
- if READ_ONLY == "true":
- print("READ_ONLY is true, not creating issue")
- else:
- repo.create_issue(title=title, body=body, labels=labels)
+ repo.create_issue(title=title, body=body, labels=labels)
Review Comment:
Reverted back
##########
.test-infra/tools/flaky_test_detection.py:
##########
@@ -103,17 +97,23 @@ def main():
token = os.environ["GITHUB_TOKEN"]
auth = Auth.Token(token)
g = Github(auth=auth)
- repo = g.get_repo(f"{GIT_ORG}/{GIT_REPO}")
+ repo = g.get_repo(f"{GIT_ORG}/beam")
alerts = get_grafana_alerts()
open_issues = repo.get_issues(state="open", labels=["flaky_test"])
- workflow_ids = extract_workflow_id_from_issue_label(open_issues)
+ closed_issues = repo.get_issues(state="closed", labels=["flaky_test"])
+ workflow_open_issues = get_workflow_issues(open_issues)
+ workflow_closed_issues = get_workflow_issues(closed_issues)
for alert in alerts:
- if alert.workflow_id not in workflow_ids:
+ if alert.workflow_id in workflow_closed_issues.keys():
+ issue = workflow_closed_issues[alert.workflow_id]
+ if issue:
+ issue.edit(state="open")
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org