You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "volatilemolotov (via GitHub)" <gi...@apache.org> on 2024/02/15 12:58:43 UTC
[PR] Metrics prefetcher script [beam]
volatilemolotov opened a new pull request, #30327:
URL: https://github.com/apache/beam/pull/30327
Script changes and optimizations.
Most notable:
- changed the database schema to allow for alerting queries.
- removes the old sync script due to duplication
- changes the GA PostCommit Status dashboard with new query that uses crosstab to allow the dash to use the same tables without duplication
- Terraform to deploy the function
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1498368587
##########
.test-infra/metrics/grafana/dashboards/GA-Post-Commits_status_dashboard.json:
##########
@@ -935,7 +935,7 @@
"hide": false,
"metricColumn": "none",
"rawQuery": true,
- "rawSql": "with workflows as (\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'None' then 4 else 0 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'None' then 4 else 0 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'None' then 4 else 0 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'None' then 4 else 0 end as run_4,\n case when run5 like 'success' the
n 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'None' then 4 else 0 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'None' then 4 else 0 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when run7 like 'cancelled' then 3 when run7 like 'None' then 4 else 0 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'None' then 4 else 0 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' t
hen 2 when run9 like 'cancelled' then 3 when run9 like 'None' then 4 else 0 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'None' then 4 else 0 end as run_10\n from\n github_workflows\n where\n dashboard_category = 'core_infra'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
+ "rawSql": "with workflows as (\n\twith temp as (SELECT t1.workflow_id, t1.workflow_name, t1.workflow_filename, t1.dashboard_category, t1.run1, t2.run1id ,\nt1.run2, t2.run2id ,\nt1.run3, t2.run3id ,\nt1.run4, t2.run4id ,\nt1.run5, t2.run5id ,\nt1.run6, t2.run6id ,\nt1.run7, t2.run7id ,\nt1.run8, t2.run8id ,\nt1.run9, t2.run9id ,\nt1.run10, t2.run10id \nFROM (SELECT * FROM crosstab('SELECT github_workflows.workflow_id, name as workflow_name, filename AS workflow_name, dashboard_category, run_number, status \n FROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs_test.workflow_id = github_workflows.workflow_id\nORDER BY 1,5 DESC\n','SELECT m from generate_series(1,10) m')\nAS c1(workflow_id text, workflow_name text, workflow_filename text, dashboard_category text, run1 text, run2 text,run3 text, \n run4 text ,run5 text,run6 text, run7 text, \n run8 text ,run9 text,run10 text) \n\t ) AS t1\n JOIN (S
ELECT * FROM crosstab('SELECT name, status, github_workflow_runs_test.url AS run_url\nFROM github_workflow_runs_test\nINNER JOIN github_workflows ON github_workflow_runs_test.workflow_id = github_workflows.workflow_id\nORDER BY 1,3 DESC\n\n')\nAS c2(workflow_name text, run1id text, run2id text,run3id text, \n run4id text ,run5id text,run6id text, run7id text, \n run8id text ,run9id text,run10id text) \n\t\t\t ) AS t2\n ON t1.workflow_name = t2.workflow_name\n\t\t )\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'failure' then 0 else 4 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'failure' then 0 else 4 end as run_2,\n case when run3 like 'suc
cess' then 1 when run3 like 'in_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'failure' then 0 else 4 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'failure' then 0 else 4 end as run_4,\n case when run5 like 'success' then 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'failure' then 0 else 4 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'failure' then 0 else 4 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when
run7 like 'waiting' then 2 when run7 like 'cancelled' then 3 when run7 like 'failure' then 0 else 4 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'failure' then 0 else 4 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' then 2 when run9 like 'cancelled' then 3 when run9 like 'failure' then 0 else 4 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'failure' then 0 else 4 end as run_10\n from\n temp\n where\n dashboard_category = 'core_infra'\n)\nselect\n workflow_name,\n workflow_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n
run4Id,\n run_5,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
Review Comment:
Sorry for this, it keeps creeping in as i have 2 forks because of testing.
Should be fixed now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1493039161
##########
.test-infra/metrics/grafana/dashboards/GA-Post-Commits_status_dashboard.json:
##########
@@ -935,7 +935,7 @@
"hide": false,
"metricColumn": "none",
"rawQuery": true,
- "rawSql": "with workflows as (\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'None' then 4 else 0 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'None' then 4 else 0 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'None' then 4 else 0 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'None' then 4 else 0 end as run_4,\n case when run5 like 'success' the
n 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'None' then 4 else 0 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'None' then 4 else 0 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when run7 like 'cancelled' then 3 when run7 like 'None' then 4 else 0 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'None' then 4 else 0 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' t
hen 2 when run9 like 'cancelled' then 3 when run9 like 'None' then 4 else 0 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'None' then 4 else 0 end as run_10\n from\n github_workflows\n where\n dashboard_category = 'core_infra'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
+ "rawSql": "with workflows as (\n\twith temp as (SELECT t1.workflow_id, t1.job_name, t1.job_yml_filename, t1.dashboard_category, t1.run1, t2.run1id ,\nt1.run2, t2.run2id ,\nt1.run3, t2.run3id ,\nt1.run4, t2.run4id ,\nt1.run5, t2.run5id ,\nt1.run6, t2.run6id ,\nt1.run7, t2.run7id ,\nt1.run8, t2.run8id ,\nt1.run9, t2.run9id ,\nt1.run10, t2.run10id \nFROM (SELECT * FROM crosstab('SELECT github_workflows_test.workflow_id, name, filename, dashboard_category, run_number, status \n FROM github_workflow_runs_test\nINNER JOIN github_workflows_test ON github_workflow_runs_test.workflow_id = github_workflows_test.workflow_id\nORDER BY 1,5 DESC\n','SELECT m from generate_series(1,10) m')\nAS c1(workflow_id text, job_name text, job_yml_filename text, dashboard_category text, run1 text, run2 text,run3 text, \n run4 text ,run5 text,run6 text, run7 text, \n run8 text ,run9 text,run10 text) \n\t ) AS t1\n JOIN (SELECT * FROM crosstab('SEL
ECT name, status, github_workflow_runs_test.url AS run_url\nFROM github_workflow_runs_test\nINNER JOIN github_workflows_test ON github_workflow_runs_test.workflow_id = github_workflows_test.workflow_id\nORDER BY 1,3 DESC\n\n')\nAS c2(job_name text, run1id text, run2id text,run3id text, \n run4id text ,run5id text,run6id text, run7id text, \n run8id text ,run9id text,run10id text) \n\t\t\t ) AS t2\n ON t1.job_name = t2.job_name\n\t\t )\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'failure' then 0 else 4 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'failure' then 0 else 4 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in
_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'failure' then 0 else 4 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'failure' then 0 else 4 end as run_4,\n case when run5 like 'success' then 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'failure' then 0 else 4 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'failure' then 0 else 4 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when
run7 like 'cancelled' then 3 when run7 like 'failure' then 0 else 4 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'failure' then 0 else 4 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' then 2 when run9 like 'cancelled' then 3 when run9 like 'failure' then 0 else 4 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'failure' then 0 else 4 end as run_10\n from\n temp\n where\n dashboard_category = 'core_infra'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6
,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
Review Comment:
1.) yeah its a mistake ill fix it
2.) Think its the name. Will double check as i test again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1494404705
##########
.test-infra/metrics/grafana/dashboards/GA-Post-Commits_status_dashboard.json:
##########
@@ -935,7 +935,7 @@
"hide": false,
"metricColumn": "none",
"rawQuery": true,
- "rawSql": "with workflows as (\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'None' then 4 else 0 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'None' then 4 else 0 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'None' then 4 else 0 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'None' then 4 else 0 end as run_4,\n case when run5 like 'success' the
n 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'None' then 4 else 0 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'None' then 4 else 0 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when run7 like 'cancelled' then 3 when run7 like 'None' then 4 else 0 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'None' then 4 else 0 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' t
hen 2 when run9 like 'cancelled' then 3 when run9 like 'None' then 4 else 0 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'None' then 4 else 0 end as run_10\n from\n github_workflows\n where\n dashboard_category = 'core_infra'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
+ "rawSql": "with workflows as (\n\twith temp as (SELECT t1.workflow_id, t1.job_name, t1.job_yml_filename, t1.dashboard_category, t1.run1, t2.run1id ,\nt1.run2, t2.run2id ,\nt1.run3, t2.run3id ,\nt1.run4, t2.run4id ,\nt1.run5, t2.run5id ,\nt1.run6, t2.run6id ,\nt1.run7, t2.run7id ,\nt1.run8, t2.run8id ,\nt1.run9, t2.run9id ,\nt1.run10, t2.run10id \nFROM (SELECT * FROM crosstab('SELECT github_workflows_test.workflow_id, name, filename, dashboard_category, run_number, status \n FROM github_workflow_runs_test\nINNER JOIN github_workflows_test ON github_workflow_runs_test.workflow_id = github_workflows_test.workflow_id\nORDER BY 1,5 DESC\n','SELECT m from generate_series(1,10) m')\nAS c1(workflow_id text, job_name text, job_yml_filename text, dashboard_category text, run1 text, run2 text,run3 text, \n run4 text ,run5 text,run6 text, run7 text, \n run8 text ,run9 text,run10 text) \n\t ) AS t1\n JOIN (SELECT * FROM crosstab('SEL
ECT name, status, github_workflow_runs_test.url AS run_url\nFROM github_workflow_runs_test\nINNER JOIN github_workflows_test ON github_workflow_runs_test.workflow_id = github_workflows_test.workflow_id\nORDER BY 1,3 DESC\n\n')\nAS c2(job_name text, run1id text, run2id text,run3id text, \n run4id text ,run5id text,run6id text, run7id text, \n run8id text ,run9id text,run10id text) \n\t\t\t ) AS t2\n ON t1.job_name = t2.job_name\n\t\t )\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'failure' then 0 else 4 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'failure' then 0 else 4 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in
_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'failure' then 0 else 4 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'failure' then 0 else 4 end as run_4,\n case when run5 like 'success' then 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'failure' then 0 else 4 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'failure' then 0 else 4 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when
run7 like 'cancelled' then 3 when run7 like 'failure' then 0 else 4 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'failure' then 0 else 4 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' then 2 when run9 like 'cancelled' then 3 when run9 like 'failure' then 0 else 4 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'failure' then 0 else 4 end as run_10\n from\n temp\n where\n dashboard_category = 'core_infra'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6
,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
Review Comment:
Re 2:
Not sure which one you ask but in the first one most of the data is from `github_workflows_tests` so JOIN is needed.
Re 1: Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1497303671
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml:
##########
@@ -0,0 +1,331 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+categories:
+ - name: core_java
+ groupThreshold: 0.7
+ tests:
+ - "PreCommit SQL Java17"
+ - "PreCommit SQL Java11"
+ - "LoadTests Java GBK Smoke"
+ - "PreCommit Java Amazon-Web-Services IO Direct"
+ - "PreCommit Java Amqp IO Direct"
+ - "PreCommit Java Amazon-Web-Services2 IO Direct"
+ - "PreCommit Java"
+ - "PreCommit Java Cassandra IO Direct"
+ - "PreCommit Java Azure IO Direct"
+ - "PreCommit Java Cdap IO Direct"
+ - "PreCommit Java Clickhouse IO Direct"
+ - "PreCommit Java Csv IO Direct"
+ - "Java Tests"
+ - "PostCommit Java Avro Versions"
+ - "PreCommit Java Debezium IO Direct"
+ - "PreCommit Java File-schema-transform IO Direct"
+ - "PostCommit Java"
+ - "PreCommit Java GCP IO Direct"
+ - "PostCommit Java BigQueryEarlyRollout"
+ - "PreCommit Java Google-ads IO Direct"
+ - "PreCommit Java HBase IO Direct"
+ - "PreCommit Java ElasticSearch IO Direct"
+ - "PreCommit Java HCatalog IO Direct"
+ - "PreCommit Java Hadoop IO Direct"
+ - "PreCommit Java IOs Direct"
+ - "PostCommit Java Hadoop Versions"
+ - "PreCommit Java Jms IO Direct"
+ - "PostCommit Java ValidatesRunner Direct JavaVersions"
+ - "PreCommit Java Kafka IO Direct"
+ - "PostCommit Java Examples Direct"
+ - "PreCommit Java JDBC IO Direct"
+ - "PostCommit Java ValidatesRunner Samza"
+ - "PreCommit Java Mqtt IO Direct"
+ - "PreCommit Java Kinesis IO Direct"
+ - "PreCommit Java MongoDb IO Direct"
+ - "PostCommit Java IO Performance Tests"
+ - "PreCommit Java Kudu IO Direct"
+ - "PostCommit Java InfluxDbIO Integration Test"
+ - "PostCommit Java Jpms Direct Java21"
+ - "PostCommit Java ValidatesRunner Twister2"
+ - "PreCommit Java Neo4j IO Direct"
+ - "PostCommit Java Jpms Direct Java11"
+ - "PostCommit Javadoc"
+ - "PostCommit Java Jpms Direct Java17"
+ - "PreCommit Java Pulsar IO Direct"
+ - "PostCommit Java ValidatesRunner ULR"
+ - "PreCommit Java Parquet IO Direct"
+ - "PreCommit Java Redis IO Direct"
+ - "Java JMH"
+ - "PreCommit Java RabbitMq IO Direct"
+ - "PreCommit Java RequestResponse IO Direct"
+ - "PostCommit Java Nexmark Direct"
+ - "PreCommit Java Splunk IO Direct"
+ - "PreCommit Java Thrift IO Direct"
+ - "PreCommit Java Snowflake IO Direct"
+ - "PreCommit Java Solr IO Direct"
+ - "PostCommit Java PVR Samza"
+ - "PreCommit Java Tika IO Direct"
+ - "PostCommit Java SingleStoreIO IT"
+ - "PostCommit Java Sickbay"
+ - "PostCommit Java ValidatesRunner Direct"
+ - "PreCommit Java SingleStore IO Direct"
+ - "PreCommit Java InfluxDb IO Direct"
+ - "PreCommit Spotless"
+ - "PreCommit Kotlin Examples"
+ - name: dataflow_java
+ tests:
+ - "PostCommit XVR GoUsingJava Dataflow"
+ - "PostCommit XVR PythonUsingJavaSQL Dataflow"
+ - "PostCommit XVR JavaUsingPython Dataflow"
+ - "PostCommit XVR PythonUsingJava Dataflow"
+ - "PreCommit Java Examples Dataflow Java11"
+ - "PreCommit Java Examples Dataflow Java17"
+ - "PreCommit Java Examples Dataflow Java21"
+ - "PreCommit Java Examples Dataflow"
+ - "PostCommit Java ValidatesRunner Dataflow"
+ - "PostCommit Java Dataflow V1"
+ - "PostCommit Java ValidatesRunner Dataflow Streaming"
+ - "PostCommit Java Dataflow V2"
+ - "PostCommit Java ValidatesRunner Dataflow V2"
+ - "PostCommit Java Examples Dataflow"
+ - "PostCommit Java Examples Dataflow ARM"
+ - "PostCommit Java ValidatesRunner Dataflow V2 Streaming"
+ - "PostCommit Java ValidatesRunner Dataflow JavaVersions"
+ - "PostCommit Java Examples Dataflow Java"
+ - "PostCommit Java Examples Dataflow V2 Java"
+ - "PostCommit Java Jpms Dataflow Java11"
+ - "PostCommit Java Jpms Dataflow Java17"
+ - "PostCommit Java Nexmark Dataflow"
+ - "PostCommit Java Nexmark Dataflow V2"
+ - "PostCommit Java Nexmark Dataflow V2 Java"
+ - "PostCommit Java Tpcds Dataflow"
+ - "PostCommit Java Examples Dataflow V2"
+ - name: runners_java
+ tests:
+ - "PostCommit Java PVR Spark3 Streaming"
+ - "PostCommit Java ValidatesRunner Spark"
+ - "PostCommit Java Examples Spark"
+ - "PostCommit Java ValidatesRunner SparkStructuredStreaming"
+ - "PostCommit Java ValidatesRunner Spark Java11"
+ - "PostCommit Java PVR Spark Batch"
+ - "PreCommit Java Spark3 Versions"
+ - "PostCommit Java Tpcds Spark"
+ - "PostCommit Java Jpms Spark Java11"
+ - "PostCommit Java Nexmark Spark"
+ - "PostCommit Java Examples Flink"
+ - "PostCommit Java Tpcds Flink"
+ - "PostCommit Java PVR Flink Streaming"
+ - "PostCommit Java Jpms Flink Java11"
+ - "PreCommit Java PVR Flink Batch"
+ - "PostCommit Java Nexmark Flink"
+ - "PreCommit Java PVR Flink Docker"
+ - "PreCommit Java Flink Versions"
+ - "PostCommit Java ValidatesRunner Flink Java11"
+ - "PostCommit Java ValidatesRunner Flink"
+ - name: load_perf_java
+ tests:
+ - "LoadTests Java CoGBK Dataflow Batch"
+ - "LoadTests Java CoGBK Dataflow V2 Streaming JavaVersions"
+ - "LoadTests Java CoGBK Dataflow Streaming"
+ - "LoadTests Java Combine Dataflow Batch"
+ - "LoadTests Java Combine Dataflow Streaming"
+ - "LoadTests Java CoGBK Dataflow V2 Batch JavaVersions"
+ - "LoadTests Java GBK Dataflow Batch"
+ - "LoadTests Java GBK Dataflow Streaming"
+ - "LoadTests Java GBK Dataflow V2 Batch Java11"
+ - "LoadTests Java GBK Dataflow V2 Streaming Java11"
+ - "LoadTests Java GBK Dataflow V2 Batch Java17"
+ - "LoadTests Java GBK Dataflow V2 Streaming Java17"
+ - "LoadTests Java ParDo Dataflow Streaming"
+ - "LoadTests Java ParDo Dataflow V2 Streaming JavaVersions"
+ - "LoadTests Java ParDo Dataflow V2 Batch JavaVersions"
+ - "LoadTests Java ParDo Dataflow Batch"
+ - "LoadTests Java ParDo SparkStructuredStreaming Batch"
+ - "LoadTests Java CoGBK SparkStructuredStreaming Batch"
+ - "LoadTests Java Combine SparkStructuredStreaming Batch"
+ - "LoadTests Java GBK SparkStructuredStreaming Batch"
+ - "PerformanceTests BigQueryIO Batch Java Avro"
+ - "PerformanceTests BigQueryIO Streaming Java"
+ - "PerformanceTests BigQueryIO Batch Java Json"
+ - "PerformanceTests SQLBigQueryIO Batch Java"
+ - "PerformanceTests XmlIOIT"
+ - "PostCommit XVR Samza"
+ - "PerformanceTests ManyFiles TextIOIT"
+ - "PerformanceTests XmlIOIT HDFS"
+ - "PerformanceTests ParquetIOIT"
+ - "PerformanceTests ParquetIOIT HDFS"
+ - "PerformanceTests AvroIOIT"
+ - "PerformanceTests ManyFiles TextIOIT HDFS"
+ - "PerformanceTests TFRecordIOIT"
+ - "PerformanceTests Cdap"
+ - "PerformanceTests TextIOIT"
+ - "PerformanceTests AvroIOIT HDFS"
+ - "PerformanceTests SingleStoreIO"
+ - "PerformanceTests SparkReceiver IO"
+ - "PerformanceTests Compressed TextIOIT"
+ - "PerformanceTests TextIOIT HDFS"
+ - "PerformanceTests Compressed TextIOIT HDFS"
+ - "PerformanceTests HadoopFormat"
+ - "PerformanceTests JDBC"
+ - "PerformanceTests Kafka IO"
+ - name: core_python
+ tests:
+ - "Python Dependency Tests"
+ - "PreCommit Python Dataframes"
+ - "PreCommit Python Examples"
+ - "PreCommit Python Integration"
+ - "PostCommit Python ValidatesRunner Samza"
+ - "LoadTests Python Smoke"
+ - "Update Python Depedencies"
+ - "PreCommit Python Runners"
+ - "PreCommit Python Transforms"
+ - "PostCommit Python Xlang Gcp Direct"
+ - "Build python source distribution and wheels"
+ - "Python tests"
+ - "PostCommit Sickbay Python"
+ - "PostCommit Python"
+ - "PostCommit Python Arm"
+ - "PostCommit Python Examples Direct"
+ - "PreCommit Portable Python"
+ - "PreCommit Python Coverage"
+ - "PreCommit Python Docker"
+ - "PreCommit Python"
+ - "PostCommit Python MongoDBIO IT"
+ - "PreCommit Python Docs"
+ - "PreCommit Python Formatter"
+ - "PostCommit Python Nexmark Direct"
+ - "PreCommit Python Lint"
+ - name: runners_python
+ tests:
+ - "PostCommit Python ValidatesRunner Dataflow"
+ - "Python ValidatesContainer Dataflow ARM"
+ - "PostCommit Python Xlang Gcp Dataflow"
+ - "PostCommit Python Xlang IO Dataflow"
+ - "PostCommit Python Examples Dataflow"
+ - "PostCommit Python ValidatesContainer Dataflow"
+ - "PostCommit Python ValidatesContainer Dataflow With RC"
+ - "PostCommit Python ValidatesRunner Spark"
+ - "PostCommit Python Examples Spark"
+ - "PostCommit Python ValidatesRunner Flink"
+ - "PreCommit Python PVR Flink"
+ - "PostCommit Python Examples Flink"
+ - name: load_perf_python
+ tests:
+ - "PerformanceTests xlang KafkaIO Python"
+ - "LoadTests Python FnApiRunner Microbenchmark"
+ - "PerformanceTests SpannerIO Write 2GB Python Batch"
+ - "PerformanceTests SpannerIO Read 2GB Python"
+ - "PerformanceTests BiqQueryIO Read Python"
+ - "PerformanceTests BiqQueryIO Write Python Batch"
+ - "PerformanceTests TextIOIT Python"
+ - "PerformanceTests WordCountIT PythonVersions"
+ - "Performance alerting tool on Python load/performance/benchmark tests."
+ - "LoadTests Python SideInput Dataflow Batch"
+ - "LoadTests Python CoGBK Dataflow Batch"
+ - "LoadTests Python CoGBK Dataflow Streaming"
+ - "LoadTests Python Combine Dataflow Batch"
+ - "Inference Python Benchmarks Dataflow"
+ - "LoadTests Python Combine Dataflow Streaming"
+ - "LoadTests Python GBK Dataflow Batch"
+ - "LoadTests Python GBK Dataflow Streaming"
+ - "LoadTests Python GBK reiterate Dataflow Batch"
+ - "LoadTests Python GBK reiterate Dataflow Streaming"
+ - "LoadTests Python ParDo Dataflow Streaming"
+ - "CloudML Benchmarks Dataflow"
+ - "LoadTests Python ParDo Dataflow Batch"
+ - "LoadTests Python CoGBK Flink Batch"
+ - "LoadTests Python Combine Flink Batch"
+ - "LoadTests Python Combine Flink Streaming"
+ - "PerformanceTests PubsubIOIT Python Streaming"
+ - "LoadTests Python ParDo Flink Batch"
+ - "LoadTests Python ParDo Flink Streaming"
+ - name: go
+ tests:
+ - "PerformanceTests MongoDBIO IT"
+ - "PreCommit Go"
+ - "PreCommit GoPortable"
+ - "PreCommit GoPrism"
+ - "PostCommit Go VR Samza"
+ - "Go tests"
+ - "PostCommit Go"
+ - "PostCommit Go Dataflow ARM"
+ - "LoadTests Go CoGBK Dataflow Batch"
+ - "LoadTests Go Combine Dataflow Batch"
+ - "LoadTests Go GBK Dataflow Batch"
+ - "LoadTests Go ParDo Dataflow Batch"
+ - "LoadTests Go SideInput Dataflow Batch"
+ - "PostCommit Go VR Spark"
+ - "PostCommit Go VR Flink"
+ - "LoadTests Go CoGBK Flink Batch"
+ - "LoadTests Go Combine Flink Batch"
+ - "LoadTests Go GBK Flink Batch"
+ - "LoadTests Go ParDo Flink Batch"
+ - "LoadTests Go SideInput Flink Batch"
+ - name: core_infra
Review Comment:
I have put specific 0.8 to infra and rest should get 0.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "andreydevyatkin (via GitHub)" <gi...@apache.org>.
andreydevyatkin commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1497324625
##########
.test-infra/metrics/sync/github/sync_workflows.py:
##########
@@ -600,52 +319,86 @@ def append_workflow_runs(workflow, runs):
else:
number_of_runs_to_add =\
int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH) - len(workflow.runs)
- workflow.runs.extend([(0, 'None', 'None')] * number_of_runs_to_add)
+ workflow.runs.extend(
+ [WorkflowRun(0, "None", "None", workflow.id, "None")] * number_of_runs_to_add
+ )
if len(workflow.runs) >= int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH):
workflow_ids_to_fetch_extra_runs.pop(workflow_id, None)
print(f"Successfully fetched extra workflow runs for: {workflow.filename}")
page += 1
print("Successfully fetched workflow runs details")
for workflow in list(workflows.values()):
- runs = sorted(workflow.runs, key=lambda r: r[0], reverse=True)
+ runs = sorted(workflow.runs, key=lambda r: r.id, reverse=True)
workflow.runs = runs[:int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH)]
return list(workflows.values())
-def database_operations(connection, workflows):
+
+def save_workflows(workflows):
+ connection = init_db_connection()
# Create the table and update it with the latest workflow runs
if not workflows:
return
cursor = connection.cursor()
workflows_table_name = "github_workflows"
+ workflow_runs_table_name = "github_workflow_runs"
cursor.execute(f"DROP TABLE IF EXISTS {workflows_table_name};")
- create_table_query = f"""
- CREATE TABLE IF NOT EXISTS {workflows_table_name} (
- workflow_id integer NOT NULL PRIMARY KEY,
- job_name text NOT NULL,
- job_yml_filename text NOT NULL,
- dashboard_category text NOT NULL"""
- for i in range(int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH)):
- create_table_query += f""",
- run{i+1} text,
- run{i+1}Id text"""
- create_table_query += ")\n"
- cursor.execute(create_table_query)
- insert_query = f"INSERT INTO {workflows_table_name} VALUES "
+ cursor.execute(f"DROP TABLE IF EXISTS {workflow_runs_table_name};")
+ create_workflows_table_query = f"""
+ CREATE TABLE IF NOT EXISTS {workflows_table_name} (
+ workflow_id integer NOT NULL PRIMARY KEY,
+ name text NOT NULL,
+ filename text NOT NULL,
+ url text NOT NULL,
+ dashboard_category text NOT NULL,
+ threshold real NOT NULL)\n"""
+ create_workflow_runs_table_query = f"""
+ CREATE TABLE IF NOT EXISTS {workflow_runs_table_name} (
+ run_id text NOT NULL PRIMARY KEY,
+ status text NOT NULL,
+ url text NOT NULL,
+ workflow_id integer NOT NULL FOREIGN KEY,
+ started_at timestamp with time zone NOT NULL)\n"""
+ cursor.execute(create_workflows_table_query)
+ cursor.execute(create_workflow_runs_table_query)
+ insert_workflows_query = f"""
+ INSERT INTO {workflows_table_name} (workflow_id, name, filename, url, dashboard_category, threshold)
+ VALUES %s"""
+ insert_workflow_runs_query = f"""
+ INSERT INTO {workflow_runs_table_name} (run_id, status, url, workflow_id, started_at)
+ VALUES %s"""
+ insert_workflows = []
+ insert_workflow_runs = []
for workflow in workflows:
- category = get_dashboard_category(workflow.name)
- row_insert =\
- f"(\'{workflow.id}\',\'{workflow.name}\',\'{workflow.filename}\',\'{category}\'"
- for _, status, url in workflow.runs:
- row_insert += f",\'{status}\',\'{url}\'"
- insert_query += f"{row_insert}),"
- insert_query = insert_query[:-1] + ";"
- print(insert_query)
- cursor.execute(insert_query)
+ insert_workflows.append(
+ (
+ workflow.id,
+ workflow.name,
+ workflow.filename,
+ workflow.url,
+ workflow.category,
+ workflow.threshold
+ )
+ )
+ for run in workflow.runs:
+ if run.id != 0:
+ started_at = run.started_at.replace("T", " ")
+ insert_workflow_runs.append(
+ (
+ run.id,
+ run.status,
+ run.url,
+ workflow.id,
+ started_at
+ )
+ )
+ psycopg2.extras.execute_values(cursor, insert_workflows_query, insert_workflows)
+ psycopg2.extras.execute_values(cursor, insert_workflow_runs_query, insert_workflow_runs)
cursor.close()
connection.commit()
connection.close()
Review Comment:
@damccorm After analyzing the data and the Grafana dashboard schema we came to the conclusion that we can normalize the DB schema in order to get rid of data redundancy. In essence, we only need one GCP function/script to fetch the data and store it to the DB, and the grouping part for displaying test statistics and alerting can be done using SQL queries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on PR #30327:
URL: https://github.com/apache/beam/pull/30327#issuecomment-1948126216
@damccorm let me know what do you think on this PR. Once merged it will change the dashboard while the script needs to be deployed manually using terraform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1494423406
##########
.test-infra/metrics/sync/github/sync_workflows.py:
##########
@@ -600,52 +319,86 @@ def append_workflow_runs(workflow, runs):
else:
number_of_runs_to_add =\
int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH) - len(workflow.runs)
- workflow.runs.extend([(0, 'None', 'None')] * number_of_runs_to_add)
+ workflow.runs.extend(
+ [WorkflowRun(0, "None", "None", workflow.id, "None")] * number_of_runs_to_add
+ )
if len(workflow.runs) >= int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH):
workflow_ids_to_fetch_extra_runs.pop(workflow_id, None)
print(f"Successfully fetched extra workflow runs for: {workflow.filename}")
page += 1
print("Successfully fetched workflow runs details")
for workflow in list(workflows.values()):
- runs = sorted(workflow.runs, key=lambda r: r[0], reverse=True)
+ runs = sorted(workflow.runs, key=lambda r: r.id, reverse=True)
workflow.runs = runs[:int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH)]
return list(workflows.values())
-def database_operations(connection, workflows):
+
+def save_workflows(workflows):
+ connection = init_db_connection()
# Create the table and update it with the latest workflow runs
if not workflows:
return
cursor = connection.cursor()
workflows_table_name = "github_workflows"
+ workflow_runs_table_name = "github_workflow_runs"
cursor.execute(f"DROP TABLE IF EXISTS {workflows_table_name};")
- create_table_query = f"""
- CREATE TABLE IF NOT EXISTS {workflows_table_name} (
- workflow_id integer NOT NULL PRIMARY KEY,
- job_name text NOT NULL,
- job_yml_filename text NOT NULL,
- dashboard_category text NOT NULL"""
- for i in range(int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH)):
- create_table_query += f""",
- run{i+1} text,
- run{i+1}Id text"""
- create_table_query += ")\n"
- cursor.execute(create_table_query)
- insert_query = f"INSERT INTO {workflows_table_name} VALUES "
+ cursor.execute(f"DROP TABLE IF EXISTS {workflow_runs_table_name};")
+ create_workflows_table_query = f"""
+ CREATE TABLE IF NOT EXISTS {workflows_table_name} (
+ workflow_id integer NOT NULL PRIMARY KEY,
+ name text NOT NULL,
+ filename text NOT NULL,
+ url text NOT NULL,
+ dashboard_category text NOT NULL,
+ threshold real NOT NULL)\n"""
+ create_workflow_runs_table_query = f"""
+ CREATE TABLE IF NOT EXISTS {workflow_runs_table_name} (
+ run_id text NOT NULL PRIMARY KEY,
+ status text NOT NULL,
+ url text NOT NULL,
+ workflow_id integer NOT NULL FOREIGN KEY,
+ started_at timestamp with time zone NOT NULL)\n"""
+ cursor.execute(create_workflows_table_query)
+ cursor.execute(create_workflow_runs_table_query)
+ insert_workflows_query = f"""
+ INSERT INTO {workflows_table_name} (workflow_id, name, filename, url, dashboard_category, threshold)
+ VALUES %s"""
+ insert_workflow_runs_query = f"""
+ INSERT INTO {workflow_runs_table_name} (run_id, status, url, workflow_id, started_at)
+ VALUES %s"""
+ insert_workflows = []
+ insert_workflow_runs = []
for workflow in workflows:
- category = get_dashboard_category(workflow.name)
- row_insert =\
- f"(\'{workflow.id}\',\'{workflow.name}\',\'{workflow.filename}\',\'{category}\'"
- for _, status, url in workflow.runs:
- row_insert += f",\'{status}\',\'{url}\'"
- insert_query += f"{row_insert}),"
- insert_query = insert_query[:-1] + ";"
- print(insert_query)
- cursor.execute(insert_query)
+ insert_workflows.append(
+ (
+ workflow.id,
+ workflow.name,
+ workflow.filename,
+ workflow.url,
+ workflow.category,
+ workflow.threshold
+ )
+ )
+ for run in workflow.runs:
+ if run.id != 0:
+ started_at = run.started_at.replace("T", " ")
+ insert_workflow_runs.append(
+ (
+ run.id,
+ run.status,
+ run.url,
+ workflow.id,
+ started_at
+ )
+ )
+ psycopg2.extras.execute_values(cursor, insert_workflows_query, insert_workflows)
+ psycopg2.extras.execute_values(cursor, insert_workflow_runs_query, insert_workflow_runs)
cursor.close()
connection.commit()
connection.close()
Review Comment:
Circling back i see that you mention separate fetcher. We will modify the fetcher so that we will have only one fetcher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1493021576
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml:
##########
@@ -0,0 +1,331 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+categories:
+ - name: core_java
+ groupThreshold: 0.7
+ tests:
+ - "PreCommit SQL Java17"
+ - "PreCommit SQL Java11"
+ - "LoadTests Java GBK Smoke"
+ - "PreCommit Java Amazon-Web-Services IO Direct"
+ - "PreCommit Java Amqp IO Direct"
+ - "PreCommit Java Amazon-Web-Services2 IO Direct"
+ - "PreCommit Java"
+ - "PreCommit Java Cassandra IO Direct"
+ - "PreCommit Java Azure IO Direct"
+ - "PreCommit Java Cdap IO Direct"
+ - "PreCommit Java Clickhouse IO Direct"
+ - "PreCommit Java Csv IO Direct"
+ - "Java Tests"
+ - "PostCommit Java Avro Versions"
+ - "PreCommit Java Debezium IO Direct"
+ - "PreCommit Java File-schema-transform IO Direct"
+ - "PostCommit Java"
+ - "PreCommit Java GCP IO Direct"
+ - "PostCommit Java BigQueryEarlyRollout"
+ - "PreCommit Java Google-ads IO Direct"
+ - "PreCommit Java HBase IO Direct"
+ - "PreCommit Java ElasticSearch IO Direct"
+ - "PreCommit Java HCatalog IO Direct"
+ - "PreCommit Java Hadoop IO Direct"
+ - "PreCommit Java IOs Direct"
+ - "PostCommit Java Hadoop Versions"
+ - "PreCommit Java Jms IO Direct"
+ - "PostCommit Java ValidatesRunner Direct JavaVersions"
+ - "PreCommit Java Kafka IO Direct"
+ - "PostCommit Java Examples Direct"
+ - "PreCommit Java JDBC IO Direct"
+ - "PostCommit Java ValidatesRunner Samza"
+ - "PreCommit Java Mqtt IO Direct"
+ - "PreCommit Java Kinesis IO Direct"
+ - "PreCommit Java MongoDb IO Direct"
+ - "PostCommit Java IO Performance Tests"
+ - "PreCommit Java Kudu IO Direct"
+ - "PostCommit Java InfluxDbIO Integration Test"
+ - "PostCommit Java Jpms Direct Java21"
+ - "PostCommit Java ValidatesRunner Twister2"
+ - "PreCommit Java Neo4j IO Direct"
+ - "PostCommit Java Jpms Direct Java11"
+ - "PostCommit Javadoc"
+ - "PostCommit Java Jpms Direct Java17"
+ - "PreCommit Java Pulsar IO Direct"
+ - "PostCommit Java ValidatesRunner ULR"
+ - "PreCommit Java Parquet IO Direct"
+ - "PreCommit Java Redis IO Direct"
+ - "Java JMH"
+ - "PreCommit Java RabbitMq IO Direct"
+ - "PreCommit Java RequestResponse IO Direct"
+ - "PostCommit Java Nexmark Direct"
+ - "PreCommit Java Splunk IO Direct"
+ - "PreCommit Java Thrift IO Direct"
+ - "PreCommit Java Snowflake IO Direct"
+ - "PreCommit Java Solr IO Direct"
+ - "PostCommit Java PVR Samza"
+ - "PreCommit Java Tika IO Direct"
+ - "PostCommit Java SingleStoreIO IT"
+ - "PostCommit Java Sickbay"
+ - "PostCommit Java ValidatesRunner Direct"
+ - "PreCommit Java SingleStore IO Direct"
+ - "PreCommit Java InfluxDb IO Direct"
+ - "PreCommit Spotless"
+ - "PreCommit Kotlin Examples"
+ - name: dataflow_java
+ tests:
+ - "PostCommit XVR GoUsingJava Dataflow"
+ - "PostCommit XVR PythonUsingJavaSQL Dataflow"
+ - "PostCommit XVR JavaUsingPython Dataflow"
+ - "PostCommit XVR PythonUsingJava Dataflow"
+ - "PreCommit Java Examples Dataflow Java11"
+ - "PreCommit Java Examples Dataflow Java17"
+ - "PreCommit Java Examples Dataflow Java21"
+ - "PreCommit Java Examples Dataflow"
+ - "PostCommit Java ValidatesRunner Dataflow"
+ - "PostCommit Java Dataflow V1"
+ - "PostCommit Java ValidatesRunner Dataflow Streaming"
+ - "PostCommit Java Dataflow V2"
+ - "PostCommit Java ValidatesRunner Dataflow V2"
+ - "PostCommit Java Examples Dataflow"
+ - "PostCommit Java Examples Dataflow ARM"
+ - "PostCommit Java ValidatesRunner Dataflow V2 Streaming"
+ - "PostCommit Java ValidatesRunner Dataflow JavaVersions"
+ - "PostCommit Java Examples Dataflow Java"
+ - "PostCommit Java Examples Dataflow V2 Java"
+ - "PostCommit Java Jpms Dataflow Java11"
+ - "PostCommit Java Jpms Dataflow Java17"
+ - "PostCommit Java Nexmark Dataflow"
+ - "PostCommit Java Nexmark Dataflow V2"
+ - "PostCommit Java Nexmark Dataflow V2 Java"
+ - "PostCommit Java Tpcds Dataflow"
+ - "PostCommit Java Examples Dataflow V2"
+ - name: runners_java
+ tests:
+ - "PostCommit Java PVR Spark3 Streaming"
+ - "PostCommit Java ValidatesRunner Spark"
+ - "PostCommit Java Examples Spark"
+ - "PostCommit Java ValidatesRunner SparkStructuredStreaming"
+ - "PostCommit Java ValidatesRunner Spark Java11"
+ - "PostCommit Java PVR Spark Batch"
+ - "PreCommit Java Spark3 Versions"
+ - "PostCommit Java Tpcds Spark"
+ - "PostCommit Java Jpms Spark Java11"
+ - "PostCommit Java Nexmark Spark"
+ - "PostCommit Java Examples Flink"
+ - "PostCommit Java Tpcds Flink"
+ - "PostCommit Java PVR Flink Streaming"
+ - "PostCommit Java Jpms Flink Java11"
+ - "PreCommit Java PVR Flink Batch"
+ - "PostCommit Java Nexmark Flink"
+ - "PreCommit Java PVR Flink Docker"
+ - "PreCommit Java Flink Versions"
+ - "PostCommit Java ValidatesRunner Flink Java11"
+ - "PostCommit Java ValidatesRunner Flink"
+ - name: load_perf_java
+ tests:
+ - "LoadTests Java CoGBK Dataflow Batch"
+ - "LoadTests Java CoGBK Dataflow V2 Streaming JavaVersions"
+ - "LoadTests Java CoGBK Dataflow Streaming"
+ - "LoadTests Java Combine Dataflow Batch"
+ - "LoadTests Java Combine Dataflow Streaming"
+ - "LoadTests Java CoGBK Dataflow V2 Batch JavaVersions"
+ - "LoadTests Java GBK Dataflow Batch"
+ - "LoadTests Java GBK Dataflow Streaming"
+ - "LoadTests Java GBK Dataflow V2 Batch Java11"
+ - "LoadTests Java GBK Dataflow V2 Streaming Java11"
+ - "LoadTests Java GBK Dataflow V2 Batch Java17"
+ - "LoadTests Java GBK Dataflow V2 Streaming Java17"
+ - "LoadTests Java ParDo Dataflow Streaming"
+ - "LoadTests Java ParDo Dataflow V2 Streaming JavaVersions"
+ - "LoadTests Java ParDo Dataflow V2 Batch JavaVersions"
+ - "LoadTests Java ParDo Dataflow Batch"
+ - "LoadTests Java ParDo SparkStructuredStreaming Batch"
+ - "LoadTests Java CoGBK SparkStructuredStreaming Batch"
+ - "LoadTests Java Combine SparkStructuredStreaming Batch"
+ - "LoadTests Java GBK SparkStructuredStreaming Batch"
+ - "PerformanceTests BigQueryIO Batch Java Avro"
+ - "PerformanceTests BigQueryIO Streaming Java"
+ - "PerformanceTests BigQueryIO Batch Java Json"
+ - "PerformanceTests SQLBigQueryIO Batch Java"
+ - "PerformanceTests XmlIOIT"
+ - "PostCommit XVR Samza"
+ - "PerformanceTests ManyFiles TextIOIT"
+ - "PerformanceTests XmlIOIT HDFS"
+ - "PerformanceTests ParquetIOIT"
+ - "PerformanceTests ParquetIOIT HDFS"
+ - "PerformanceTests AvroIOIT"
+ - "PerformanceTests ManyFiles TextIOIT HDFS"
+ - "PerformanceTests TFRecordIOIT"
+ - "PerformanceTests Cdap"
+ - "PerformanceTests TextIOIT"
+ - "PerformanceTests AvroIOIT HDFS"
+ - "PerformanceTests SingleStoreIO"
+ - "PerformanceTests SparkReceiver IO"
+ - "PerformanceTests Compressed TextIOIT"
+ - "PerformanceTests TextIOIT HDFS"
+ - "PerformanceTests Compressed TextIOIT HDFS"
+ - "PerformanceTests HadoopFormat"
+ - "PerformanceTests JDBC"
+ - "PerformanceTests Kafka IO"
+ - name: core_python
+ tests:
+ - "Python Dependency Tests"
+ - "PreCommit Python Dataframes"
+ - "PreCommit Python Examples"
+ - "PreCommit Python Integration"
+ - "PostCommit Python ValidatesRunner Samza"
+ - "LoadTests Python Smoke"
+ - "Update Python Depedencies"
+ - "PreCommit Python Runners"
+ - "PreCommit Python Transforms"
+ - "PostCommit Python Xlang Gcp Direct"
+ - "Build python source distribution and wheels"
+ - "Python tests"
+ - "PostCommit Sickbay Python"
+ - "PostCommit Python"
+ - "PostCommit Python Arm"
+ - "PostCommit Python Examples Direct"
+ - "PreCommit Portable Python"
+ - "PreCommit Python Coverage"
+ - "PreCommit Python Docker"
+ - "PreCommit Python"
+ - "PostCommit Python MongoDBIO IT"
+ - "PreCommit Python Docs"
+ - "PreCommit Python Formatter"
+ - "PostCommit Python Nexmark Direct"
+ - "PreCommit Python Lint"
+ - name: runners_python
+ tests:
+ - "PostCommit Python ValidatesRunner Dataflow"
+ - "Python ValidatesContainer Dataflow ARM"
+ - "PostCommit Python Xlang Gcp Dataflow"
+ - "PostCommit Python Xlang IO Dataflow"
+ - "PostCommit Python Examples Dataflow"
+ - "PostCommit Python ValidatesContainer Dataflow"
+ - "PostCommit Python ValidatesContainer Dataflow With RC"
+ - "PostCommit Python ValidatesRunner Spark"
+ - "PostCommit Python Examples Spark"
+ - "PostCommit Python ValidatesRunner Flink"
+ - "PreCommit Python PVR Flink"
+ - "PostCommit Python Examples Flink"
+ - name: load_perf_python
+ tests:
+ - "PerformanceTests xlang KafkaIO Python"
+ - "LoadTests Python FnApiRunner Microbenchmark"
+ - "PerformanceTests SpannerIO Write 2GB Python Batch"
+ - "PerformanceTests SpannerIO Read 2GB Python"
+ - "PerformanceTests BiqQueryIO Read Python"
+ - "PerformanceTests BiqQueryIO Write Python Batch"
+ - "PerformanceTests TextIOIT Python"
+ - "PerformanceTests WordCountIT PythonVersions"
+ - "Performance alerting tool on Python load/performance/benchmark tests."
+ - "LoadTests Python SideInput Dataflow Batch"
+ - "LoadTests Python CoGBK Dataflow Batch"
+ - "LoadTests Python CoGBK Dataflow Streaming"
+ - "LoadTests Python Combine Dataflow Batch"
+ - "Inference Python Benchmarks Dataflow"
+ - "LoadTests Python Combine Dataflow Streaming"
+ - "LoadTests Python GBK Dataflow Batch"
+ - "LoadTests Python GBK Dataflow Streaming"
+ - "LoadTests Python GBK reiterate Dataflow Batch"
+ - "LoadTests Python GBK reiterate Dataflow Streaming"
+ - "LoadTests Python ParDo Dataflow Streaming"
+ - "CloudML Benchmarks Dataflow"
+ - "LoadTests Python ParDo Dataflow Batch"
+ - "LoadTests Python CoGBK Flink Batch"
+ - "LoadTests Python Combine Flink Batch"
+ - "LoadTests Python Combine Flink Streaming"
+ - "PerformanceTests PubsubIOIT Python Streaming"
+ - "LoadTests Python ParDo Flink Batch"
+ - "LoadTests Python ParDo Flink Streaming"
+ - name: go
+ tests:
+ - "PerformanceTests MongoDBIO IT"
+ - "PreCommit Go"
+ - "PreCommit GoPortable"
+ - "PreCommit GoPrism"
+ - "PostCommit Go VR Samza"
+ - "Go tests"
+ - "PostCommit Go"
+ - "PostCommit Go Dataflow ARM"
+ - "LoadTests Go CoGBK Dataflow Batch"
+ - "LoadTests Go Combine Dataflow Batch"
+ - "LoadTests Go GBK Dataflow Batch"
+ - "LoadTests Go ParDo Dataflow Batch"
+ - "LoadTests Go SideInput Dataflow Batch"
+ - "PostCommit Go VR Spark"
+ - "PostCommit Go VR Flink"
+ - "LoadTests Go CoGBK Flink Batch"
+ - "LoadTests Go Combine Flink Batch"
+ - "LoadTests Go GBK Flink Batch"
+ - "LoadTests Go ParDo Flink Batch"
+ - "LoadTests Go SideInput Flink Batch"
+ - name: core_infra
Review Comment:
This was actually the grouping that I wanted to have the higher threshold (not core_java), could you swap those? I actually think something like 0.8 or 0.9 is appropriate here
##########
.test-infra/metrics/sync/github/sync_workflows.py:
##########
@@ -600,52 +319,86 @@ def append_workflow_runs(workflow, runs):
else:
number_of_runs_to_add =\
int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH) - len(workflow.runs)
- workflow.runs.extend([(0, 'None', 'None')] * number_of_runs_to_add)
+ workflow.runs.extend(
+ [WorkflowRun(0, "None", "None", workflow.id, "None")] * number_of_runs_to_add
+ )
if len(workflow.runs) >= int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH):
workflow_ids_to_fetch_extra_runs.pop(workflow_id, None)
print(f"Successfully fetched extra workflow runs for: {workflow.filename}")
page += 1
print("Successfully fetched workflow runs details")
for workflow in list(workflows.values()):
- runs = sorted(workflow.runs, key=lambda r: r[0], reverse=True)
+ runs = sorted(workflow.runs, key=lambda r: r.id, reverse=True)
workflow.runs = runs[:int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH)]
return list(workflows.values())
-def database_operations(connection, workflows):
+
+def save_workflows(workflows):
+ connection = init_db_connection()
# Create the table and update it with the latest workflow runs
if not workflows:
return
cursor = connection.cursor()
workflows_table_name = "github_workflows"
+ workflow_runs_table_name = "github_workflow_runs"
cursor.execute(f"DROP TABLE IF EXISTS {workflows_table_name};")
- create_table_query = f"""
- CREATE TABLE IF NOT EXISTS {workflows_table_name} (
- workflow_id integer NOT NULL PRIMARY KEY,
- job_name text NOT NULL,
- job_yml_filename text NOT NULL,
- dashboard_category text NOT NULL"""
- for i in range(int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH)):
- create_table_query += f""",
- run{i+1} text,
- run{i+1}Id text"""
- create_table_query += ")\n"
- cursor.execute(create_table_query)
- insert_query = f"INSERT INTO {workflows_table_name} VALUES "
+ cursor.execute(f"DROP TABLE IF EXISTS {workflow_runs_table_name};")
+ create_workflows_table_query = f"""
+ CREATE TABLE IF NOT EXISTS {workflows_table_name} (
+ workflow_id integer NOT NULL PRIMARY KEY,
+ name text NOT NULL,
+ filename text NOT NULL,
+ url text NOT NULL,
+ dashboard_category text NOT NULL,
+ threshold real NOT NULL)\n"""
+ create_workflow_runs_table_query = f"""
+ CREATE TABLE IF NOT EXISTS {workflow_runs_table_name} (
+ run_id text NOT NULL PRIMARY KEY,
+ status text NOT NULL,
+ url text NOT NULL,
+ workflow_id integer NOT NULL FOREIGN KEY,
+ started_at timestamp with time zone NOT NULL)\n"""
+ cursor.execute(create_workflows_table_query)
+ cursor.execute(create_workflow_runs_table_query)
+ insert_workflows_query = f"""
+ INSERT INTO {workflows_table_name} (workflow_id, name, filename, url, dashboard_category, threshold)
+ VALUES %s"""
+ insert_workflow_runs_query = f"""
+ INSERT INTO {workflow_runs_table_name} (run_id, status, url, workflow_id, started_at)
+ VALUES %s"""
+ insert_workflows = []
+ insert_workflow_runs = []
for workflow in workflows:
- category = get_dashboard_category(workflow.name)
- row_insert =\
- f"(\'{workflow.id}\',\'{workflow.name}\',\'{workflow.filename}\',\'{category}\'"
- for _, status, url in workflow.runs:
- row_insert += f",\'{status}\',\'{url}\'"
- insert_query += f"{row_insert}),"
- insert_query = insert_query[:-1] + ";"
- print(insert_query)
- cursor.execute(insert_query)
+ insert_workflows.append(
+ (
+ workflow.id,
+ workflow.name,
+ workflow.filename,
+ workflow.url,
+ workflow.category,
+ workflow.threshold
+ )
+ )
+ for run in workflow.runs:
+ if run.id != 0:
+ started_at = run.started_at.replace("T", " ")
+ insert_workflow_runs.append(
+ (
+ run.id,
+ run.status,
+ run.url,
+ workflow.id,
+ started_at
+ )
+ )
+ psycopg2.extras.execute_values(cursor, insert_workflows_query, insert_workflows)
+ psycopg2.extras.execute_values(cursor, insert_workflow_runs_query, insert_workflow_runs)
cursor.close()
connection.commit()
connection.close()
Review Comment:
Can't remember if we discussed this, but do we still need this now that we have the separate fetcher? Can't we just populate everything at once?
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml:
##########
@@ -0,0 +1,331 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+categories:
Review Comment:
Thanks for moving this to config, wanted to do this and didn't have the time to go back and do the cleanup!
##########
.test-infra/metrics/grafana/dashboards/GA-Post-Commits_status_dashboard.json:
##########
@@ -935,7 +935,7 @@
"hide": false,
"metricColumn": "none",
"rawQuery": true,
- "rawSql": "with workflows as (\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'None' then 4 else 0 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'None' then 4 else 0 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'None' then 4 else 0 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'None' then 4 else 0 end as run_4,\n case when run5 like 'success' the
n 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'None' then 4 else 0 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'None' then 4 else 0 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when run7 like 'cancelled' then 3 when run7 like 'None' then 4 else 0 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'None' then 4 else 0 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' t
hen 2 when run9 like 'cancelled' then 3 when run9 like 'None' then 4 else 0 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'None' then 4 else 0 end as run_10\n from\n github_workflows\n where\n dashboard_category = 'core_infra'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
+ "rawSql": "with workflows as (\n\twith temp as (SELECT t1.workflow_id, t1.job_name, t1.job_yml_filename, t1.dashboard_category, t1.run1, t2.run1id ,\nt1.run2, t2.run2id ,\nt1.run3, t2.run3id ,\nt1.run4, t2.run4id ,\nt1.run5, t2.run5id ,\nt1.run6, t2.run6id ,\nt1.run7, t2.run7id ,\nt1.run8, t2.run8id ,\nt1.run9, t2.run9id ,\nt1.run10, t2.run10id \nFROM (SELECT * FROM crosstab('SELECT github_workflows_test.workflow_id, name, filename, dashboard_category, run_number, status \n FROM github_workflow_runs_test\nINNER JOIN github_workflows_test ON github_workflow_runs_test.workflow_id = github_workflows_test.workflow_id\nORDER BY 1,5 DESC\n','SELECT m from generate_series(1,10) m')\nAS c1(workflow_id text, job_name text, job_yml_filename text, dashboard_category text, run1 text, run2 text,run3 text, \n run4 text ,run5 text,run6 text, run7 text, \n run8 text ,run9 text,run10 text) \n\t ) AS t1\n JOIN (SELECT * FROM crosstab('SEL
ECT name, status, github_workflow_runs_test.url AS run_url\nFROM github_workflow_runs_test\nINNER JOIN github_workflows_test ON github_workflow_runs_test.workflow_id = github_workflows_test.workflow_id\nORDER BY 1,3 DESC\n\n')\nAS c2(job_name text, run1id text, run2id text,run3id text, \n run4id text ,run5id text,run6id text, run7id text, \n run8id text ,run9id text,run10id text) \n\t\t\t ) AS t2\n ON t1.job_name = t2.job_name\n\t\t )\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'failure' then 0 else 4 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'failure' then 0 else 4 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in
_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'failure' then 0 else 4 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'failure' then 0 else 4 end as run_4,\n case when run5 like 'success' then 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'failure' then 0 else 4 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'failure' then 0 else 4 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when
run7 like 'cancelled' then 3 when run7 like 'failure' then 0 else 4 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'failure' then 0 else 4 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' then 2 when run9 like 'cancelled' then 3 when run9 like 'failure' then 0 else 4 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'failure' then 0 else 4 end as run_10\n from\n temp\n where\n dashboard_category = 'core_infra'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6
,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
Review Comment:
A couple notes on the query:
1) I think you left a test postfix in by mistake (`github_workflows_test` instead of `github_workflows` and likewise for `github_workflow_runs_test`)
2) Why do we need the join in the first part of the query (`INNER JOIN github_workflows_test ON github_workflow_runs_test.workflow_id = github_workflows_test.workflow_id`) - isn't all of the data we need available in `github_workflow_runs`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1493036909
##########
.test-infra/metrics/sync/github/github_runs_prefetcher/code/config.yaml:
##########
@@ -0,0 +1,331 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+categories:
+ - name: core_java
+ groupThreshold: 0.7
+ tests:
+ - "PreCommit SQL Java17"
+ - "PreCommit SQL Java11"
+ - "LoadTests Java GBK Smoke"
+ - "PreCommit Java Amazon-Web-Services IO Direct"
+ - "PreCommit Java Amqp IO Direct"
+ - "PreCommit Java Amazon-Web-Services2 IO Direct"
+ - "PreCommit Java"
+ - "PreCommit Java Cassandra IO Direct"
+ - "PreCommit Java Azure IO Direct"
+ - "PreCommit Java Cdap IO Direct"
+ - "PreCommit Java Clickhouse IO Direct"
+ - "PreCommit Java Csv IO Direct"
+ - "Java Tests"
+ - "PostCommit Java Avro Versions"
+ - "PreCommit Java Debezium IO Direct"
+ - "PreCommit Java File-schema-transform IO Direct"
+ - "PostCommit Java"
+ - "PreCommit Java GCP IO Direct"
+ - "PostCommit Java BigQueryEarlyRollout"
+ - "PreCommit Java Google-ads IO Direct"
+ - "PreCommit Java HBase IO Direct"
+ - "PreCommit Java ElasticSearch IO Direct"
+ - "PreCommit Java HCatalog IO Direct"
+ - "PreCommit Java Hadoop IO Direct"
+ - "PreCommit Java IOs Direct"
+ - "PostCommit Java Hadoop Versions"
+ - "PreCommit Java Jms IO Direct"
+ - "PostCommit Java ValidatesRunner Direct JavaVersions"
+ - "PreCommit Java Kafka IO Direct"
+ - "PostCommit Java Examples Direct"
+ - "PreCommit Java JDBC IO Direct"
+ - "PostCommit Java ValidatesRunner Samza"
+ - "PreCommit Java Mqtt IO Direct"
+ - "PreCommit Java Kinesis IO Direct"
+ - "PreCommit Java MongoDb IO Direct"
+ - "PostCommit Java IO Performance Tests"
+ - "PreCommit Java Kudu IO Direct"
+ - "PostCommit Java InfluxDbIO Integration Test"
+ - "PostCommit Java Jpms Direct Java21"
+ - "PostCommit Java ValidatesRunner Twister2"
+ - "PreCommit Java Neo4j IO Direct"
+ - "PostCommit Java Jpms Direct Java11"
+ - "PostCommit Javadoc"
+ - "PostCommit Java Jpms Direct Java17"
+ - "PreCommit Java Pulsar IO Direct"
+ - "PostCommit Java ValidatesRunner ULR"
+ - "PreCommit Java Parquet IO Direct"
+ - "PreCommit Java Redis IO Direct"
+ - "Java JMH"
+ - "PreCommit Java RabbitMq IO Direct"
+ - "PreCommit Java RequestResponse IO Direct"
+ - "PostCommit Java Nexmark Direct"
+ - "PreCommit Java Splunk IO Direct"
+ - "PreCommit Java Thrift IO Direct"
+ - "PreCommit Java Snowflake IO Direct"
+ - "PreCommit Java Solr IO Direct"
+ - "PostCommit Java PVR Samza"
+ - "PreCommit Java Tika IO Direct"
+ - "PostCommit Java SingleStoreIO IT"
+ - "PostCommit Java Sickbay"
+ - "PostCommit Java ValidatesRunner Direct"
+ - "PreCommit Java SingleStore IO Direct"
+ - "PreCommit Java InfluxDb IO Direct"
+ - "PreCommit Spotless"
+ - "PreCommit Kotlin Examples"
+ - name: dataflow_java
+ tests:
+ - "PostCommit XVR GoUsingJava Dataflow"
+ - "PostCommit XVR PythonUsingJavaSQL Dataflow"
+ - "PostCommit XVR JavaUsingPython Dataflow"
+ - "PostCommit XVR PythonUsingJava Dataflow"
+ - "PreCommit Java Examples Dataflow Java11"
+ - "PreCommit Java Examples Dataflow Java17"
+ - "PreCommit Java Examples Dataflow Java21"
+ - "PreCommit Java Examples Dataflow"
+ - "PostCommit Java ValidatesRunner Dataflow"
+ - "PostCommit Java Dataflow V1"
+ - "PostCommit Java ValidatesRunner Dataflow Streaming"
+ - "PostCommit Java Dataflow V2"
+ - "PostCommit Java ValidatesRunner Dataflow V2"
+ - "PostCommit Java Examples Dataflow"
+ - "PostCommit Java Examples Dataflow ARM"
+ - "PostCommit Java ValidatesRunner Dataflow V2 Streaming"
+ - "PostCommit Java ValidatesRunner Dataflow JavaVersions"
+ - "PostCommit Java Examples Dataflow Java"
+ - "PostCommit Java Examples Dataflow V2 Java"
+ - "PostCommit Java Jpms Dataflow Java11"
+ - "PostCommit Java Jpms Dataflow Java17"
+ - "PostCommit Java Nexmark Dataflow"
+ - "PostCommit Java Nexmark Dataflow V2"
+ - "PostCommit Java Nexmark Dataflow V2 Java"
+ - "PostCommit Java Tpcds Dataflow"
+ - "PostCommit Java Examples Dataflow V2"
+ - name: runners_java
+ tests:
+ - "PostCommit Java PVR Spark3 Streaming"
+ - "PostCommit Java ValidatesRunner Spark"
+ - "PostCommit Java Examples Spark"
+ - "PostCommit Java ValidatesRunner SparkStructuredStreaming"
+ - "PostCommit Java ValidatesRunner Spark Java11"
+ - "PostCommit Java PVR Spark Batch"
+ - "PreCommit Java Spark3 Versions"
+ - "PostCommit Java Tpcds Spark"
+ - "PostCommit Java Jpms Spark Java11"
+ - "PostCommit Java Nexmark Spark"
+ - "PostCommit Java Examples Flink"
+ - "PostCommit Java Tpcds Flink"
+ - "PostCommit Java PVR Flink Streaming"
+ - "PostCommit Java Jpms Flink Java11"
+ - "PreCommit Java PVR Flink Batch"
+ - "PostCommit Java Nexmark Flink"
+ - "PreCommit Java PVR Flink Docker"
+ - "PreCommit Java Flink Versions"
+ - "PostCommit Java ValidatesRunner Flink Java11"
+ - "PostCommit Java ValidatesRunner Flink"
+ - name: load_perf_java
+ tests:
+ - "LoadTests Java CoGBK Dataflow Batch"
+ - "LoadTests Java CoGBK Dataflow V2 Streaming JavaVersions"
+ - "LoadTests Java CoGBK Dataflow Streaming"
+ - "LoadTests Java Combine Dataflow Batch"
+ - "LoadTests Java Combine Dataflow Streaming"
+ - "LoadTests Java CoGBK Dataflow V2 Batch JavaVersions"
+ - "LoadTests Java GBK Dataflow Batch"
+ - "LoadTests Java GBK Dataflow Streaming"
+ - "LoadTests Java GBK Dataflow V2 Batch Java11"
+ - "LoadTests Java GBK Dataflow V2 Streaming Java11"
+ - "LoadTests Java GBK Dataflow V2 Batch Java17"
+ - "LoadTests Java GBK Dataflow V2 Streaming Java17"
+ - "LoadTests Java ParDo Dataflow Streaming"
+ - "LoadTests Java ParDo Dataflow V2 Streaming JavaVersions"
+ - "LoadTests Java ParDo Dataflow V2 Batch JavaVersions"
+ - "LoadTests Java ParDo Dataflow Batch"
+ - "LoadTests Java ParDo SparkStructuredStreaming Batch"
+ - "LoadTests Java CoGBK SparkStructuredStreaming Batch"
+ - "LoadTests Java Combine SparkStructuredStreaming Batch"
+ - "LoadTests Java GBK SparkStructuredStreaming Batch"
+ - "PerformanceTests BigQueryIO Batch Java Avro"
+ - "PerformanceTests BigQueryIO Streaming Java"
+ - "PerformanceTests BigQueryIO Batch Java Json"
+ - "PerformanceTests SQLBigQueryIO Batch Java"
+ - "PerformanceTests XmlIOIT"
+ - "PostCommit XVR Samza"
+ - "PerformanceTests ManyFiles TextIOIT"
+ - "PerformanceTests XmlIOIT HDFS"
+ - "PerformanceTests ParquetIOIT"
+ - "PerformanceTests ParquetIOIT HDFS"
+ - "PerformanceTests AvroIOIT"
+ - "PerformanceTests ManyFiles TextIOIT HDFS"
+ - "PerformanceTests TFRecordIOIT"
+ - "PerformanceTests Cdap"
+ - "PerformanceTests TextIOIT"
+ - "PerformanceTests AvroIOIT HDFS"
+ - "PerformanceTests SingleStoreIO"
+ - "PerformanceTests SparkReceiver IO"
+ - "PerformanceTests Compressed TextIOIT"
+ - "PerformanceTests TextIOIT HDFS"
+ - "PerformanceTests Compressed TextIOIT HDFS"
+ - "PerformanceTests HadoopFormat"
+ - "PerformanceTests JDBC"
+ - "PerformanceTests Kafka IO"
+ - name: core_python
+ tests:
+ - "Python Dependency Tests"
+ - "PreCommit Python Dataframes"
+ - "PreCommit Python Examples"
+ - "PreCommit Python Integration"
+ - "PostCommit Python ValidatesRunner Samza"
+ - "LoadTests Python Smoke"
+ - "Update Python Depedencies"
+ - "PreCommit Python Runners"
+ - "PreCommit Python Transforms"
+ - "PostCommit Python Xlang Gcp Direct"
+ - "Build python source distribution and wheels"
+ - "Python tests"
+ - "PostCommit Sickbay Python"
+ - "PostCommit Python"
+ - "PostCommit Python Arm"
+ - "PostCommit Python Examples Direct"
+ - "PreCommit Portable Python"
+ - "PreCommit Python Coverage"
+ - "PreCommit Python Docker"
+ - "PreCommit Python"
+ - "PostCommit Python MongoDBIO IT"
+ - "PreCommit Python Docs"
+ - "PreCommit Python Formatter"
+ - "PostCommit Python Nexmark Direct"
+ - "PreCommit Python Lint"
+ - name: runners_python
+ tests:
+ - "PostCommit Python ValidatesRunner Dataflow"
+ - "Python ValidatesContainer Dataflow ARM"
+ - "PostCommit Python Xlang Gcp Dataflow"
+ - "PostCommit Python Xlang IO Dataflow"
+ - "PostCommit Python Examples Dataflow"
+ - "PostCommit Python ValidatesContainer Dataflow"
+ - "PostCommit Python ValidatesContainer Dataflow With RC"
+ - "PostCommit Python ValidatesRunner Spark"
+ - "PostCommit Python Examples Spark"
+ - "PostCommit Python ValidatesRunner Flink"
+ - "PreCommit Python PVR Flink"
+ - "PostCommit Python Examples Flink"
+ - name: load_perf_python
+ tests:
+ - "PerformanceTests xlang KafkaIO Python"
+ - "LoadTests Python FnApiRunner Microbenchmark"
+ - "PerformanceTests SpannerIO Write 2GB Python Batch"
+ - "PerformanceTests SpannerIO Read 2GB Python"
+ - "PerformanceTests BiqQueryIO Read Python"
+ - "PerformanceTests BiqQueryIO Write Python Batch"
+ - "PerformanceTests TextIOIT Python"
+ - "PerformanceTests WordCountIT PythonVersions"
+ - "Performance alerting tool on Python load/performance/benchmark tests."
+ - "LoadTests Python SideInput Dataflow Batch"
+ - "LoadTests Python CoGBK Dataflow Batch"
+ - "LoadTests Python CoGBK Dataflow Streaming"
+ - "LoadTests Python Combine Dataflow Batch"
+ - "Inference Python Benchmarks Dataflow"
+ - "LoadTests Python Combine Dataflow Streaming"
+ - "LoadTests Python GBK Dataflow Batch"
+ - "LoadTests Python GBK Dataflow Streaming"
+ - "LoadTests Python GBK reiterate Dataflow Batch"
+ - "LoadTests Python GBK reiterate Dataflow Streaming"
+ - "LoadTests Python ParDo Dataflow Streaming"
+ - "CloudML Benchmarks Dataflow"
+ - "LoadTests Python ParDo Dataflow Batch"
+ - "LoadTests Python CoGBK Flink Batch"
+ - "LoadTests Python Combine Flink Batch"
+ - "LoadTests Python Combine Flink Streaming"
+ - "PerformanceTests PubsubIOIT Python Streaming"
+ - "LoadTests Python ParDo Flink Batch"
+ - "LoadTests Python ParDo Flink Streaming"
+ - name: go
+ tests:
+ - "PerformanceTests MongoDBIO IT"
+ - "PreCommit Go"
+ - "PreCommit GoPortable"
+ - "PreCommit GoPrism"
+ - "PostCommit Go VR Samza"
+ - "Go tests"
+ - "PostCommit Go"
+ - "PostCommit Go Dataflow ARM"
+ - "LoadTests Go CoGBK Dataflow Batch"
+ - "LoadTests Go Combine Dataflow Batch"
+ - "LoadTests Go GBK Dataflow Batch"
+ - "LoadTests Go ParDo Dataflow Batch"
+ - "LoadTests Go SideInput Dataflow Batch"
+ - "PostCommit Go VR Spark"
+ - "PostCommit Go VR Flink"
+ - "LoadTests Go CoGBK Flink Batch"
+ - "LoadTests Go Combine Flink Batch"
+ - "LoadTests Go GBK Flink Batch"
+ - "LoadTests Go ParDo Flink Batch"
+ - "LoadTests Go SideInput Flink Batch"
+ - name: core_infra
Review Comment:
Oh those are just random when i was testing. sorry for not cleaning up.
Lets take this chance to figure out what we need specifically defined. Remember that if not defined 0.5 is default (its in code, not obvious from the config file )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on PR #30327:
URL: https://github.com/apache/beam/pull/30327#issuecomment-1952956558
Put into draft again as i need to cleanup the GA Post-Commits Status dashboard query just to make it more consistent with the naming and test it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #30327:
URL: https://github.com/apache/beam/pull/30327
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30327:
URL: https://github.com/apache/beam/pull/30327#issuecomment-1948188846
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damccorm for label build.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1497738540
##########
.test-infra/metrics/grafana/dashboards/GA-Post-Commits_status_dashboard.json:
##########
@@ -935,7 +935,7 @@
"hide": false,
"metricColumn": "none",
"rawQuery": true,
- "rawSql": "with workflows as (\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'None' then 4 else 0 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'None' then 4 else 0 end as run_2,\n case when run3 like 'success' then 1 when run3 like 'in_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'None' then 4 else 0 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'None' then 4 else 0 end as run_4,\n case when run5 like 'success' the
n 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'None' then 4 else 0 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'None' then 4 else 0 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when run7 like 'waiting' then 2 when run7 like 'cancelled' then 3 when run7 like 'None' then 4 else 0 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'None' then 4 else 0 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' t
hen 2 when run9 like 'cancelled' then 3 when run9 like 'None' then 4 else 0 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'None' then 4 else 0 end as run_10\n from\n github_workflows\n where\n dashboard_category = 'core_infra'\n)\nselect\n job_name,\n job_yml_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n run4Id,\n run_5,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
+ "rawSql": "with workflows as (\n\twith temp as (SELECT t1.workflow_id, t1.workflow_name, t1.workflow_filename, t1.dashboard_category, t1.run1, t2.run1id ,\nt1.run2, t2.run2id ,\nt1.run3, t2.run3id ,\nt1.run4, t2.run4id ,\nt1.run5, t2.run5id ,\nt1.run6, t2.run6id ,\nt1.run7, t2.run7id ,\nt1.run8, t2.run8id ,\nt1.run9, t2.run9id ,\nt1.run10, t2.run10id \nFROM (SELECT * FROM crosstab('SELECT github_workflows.workflow_id, name as workflow_name, filename AS workflow_name, dashboard_category, run_number, status \n FROM github_workflow_runs\nINNER JOIN github_workflows ON github_workflow_runs_test.workflow_id = github_workflows.workflow_id\nORDER BY 1,5 DESC\n','SELECT m from generate_series(1,10) m')\nAS c1(workflow_id text, workflow_name text, workflow_filename text, dashboard_category text, run1 text, run2 text,run3 text, \n run4 text ,run5 text,run6 text, run7 text, \n run8 text ,run9 text,run10 text) \n\t ) AS t1\n JOIN (S
ELECT * FROM crosstab('SELECT name, status, github_workflow_runs_test.url AS run_url\nFROM github_workflow_runs_test\nINNER JOIN github_workflows ON github_workflow_runs_test.workflow_id = github_workflows.workflow_id\nORDER BY 1,3 DESC\n\n')\nAS c2(workflow_name text, run1id text, run2id text,run3id text, \n run4id text ,run5id text,run6id text, run7id text, \n run8id text ,run9id text,run10id text) \n\t\t\t ) AS t2\n ON t1.workflow_name = t2.workflow_name\n\t\t )\n select\n *,\n case when run1 like 'success' then 1 when run1 like 'in_progress' then 2 when run1 like 'queued' then 2 when run1 like 'waiting' then 2 when run1 like 'cancelled' then 3 when run1 like 'failure' then 0 else 4 end as run_1,\n case when run2 like 'success' then 1 when run2 like 'in_progress' then 2 when run2 like 'queued' then 2 when run2 like 'waiting' then 2 when run2 like 'cancelled' then 3 when run2 like 'failure' then 0 else 4 end as run_2,\n case when run3 like 'suc
cess' then 1 when run3 like 'in_progress' then 2 when run3 like 'queued' then 2 when run3 like 'waiting' then 2 when run3 like 'cancelled' then 3 when run3 like 'failure' then 0 else 4 end as run_3,\n case when run4 like 'success' then 1 when run4 like 'in_progress' then 2 when run4 like 'queued' then 2 when run4 like 'waiting' then 2 when run4 like 'cancelled' then 3 when run4 like 'failure' then 0 else 4 end as run_4,\n case when run5 like 'success' then 1 when run5 like 'in_progress' then 2 when run5 like 'queued' then 2 when run5 like 'waiting' then 2 when run5 like 'cancelled' then 3 when run5 like 'failure' then 0 else 4 end as run_5,\n case when run6 like 'success' then 1 when run6 like 'in_progress' then 2 when run6 like 'queued' then 2 when run6 like 'waiting' then 2 when run6 like 'cancelled' then 3 when run6 like 'failure' then 0 else 4 end as run_6,\n case when run7 like 'success' then 1 when run7 like 'in_progress' then 2 when run7 like 'queued' then 2 when
run7 like 'waiting' then 2 when run7 like 'cancelled' then 3 when run7 like 'failure' then 0 else 4 end as run_7,\n case when run8 like 'success' then 1 when run8 like 'in_progress' then 2 when run8 like 'queued' then 2 when run8 like 'waiting' then 2 when run8 like 'cancelled' then 3 when run8 like 'failure' then 0 else 4 end as run_8,\n case when run9 like 'success' then 1 when run9 like 'in_progress' then 2 when run9 like 'queued' then 2 when run9 like 'waiting' then 2 when run9 like 'cancelled' then 3 when run9 like 'failure' then 0 else 4 end as run_9,\n case when run10 like 'success' then 1 when run10 like 'in_progress' then 2 when run10 like 'queued' then 2 when run10 like 'waiting' then 2 when run10 like 'cancelled' then 3 when run10 like 'failure' then 0 else 4 end as run_10\n from\n temp\n where\n dashboard_category = 'core_infra'\n)\nselect\n workflow_name,\n workflow_filename,\n run_1,\n run1Id,\n run_2,\n run2Id,\n run_3,\n run3Id,\n run_4,\n
run4Id,\n run_5,\n run5Id,\n run_6,\n run6Id,\n run_7,\n run7Id,\n run_8,\n run8Id,\n run_9,\n run9Id,\n run_10,\n run10Id\nfrom\n workflows;",
Review Comment:
This still references github_workflow_runs_test, should be github_workflow_runs I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Metrics prefetcher script [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on code in PR #30327:
URL: https://github.com/apache/beam/pull/30327#discussion_r1493042004
##########
.test-infra/metrics/sync/github/sync_workflows.py:
##########
@@ -600,52 +319,86 @@ def append_workflow_runs(workflow, runs):
else:
number_of_runs_to_add =\
int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH) - len(workflow.runs)
- workflow.runs.extend([(0, 'None', 'None')] * number_of_runs_to_add)
+ workflow.runs.extend(
+ [WorkflowRun(0, "None", "None", workflow.id, "None")] * number_of_runs_to_add
+ )
if len(workflow.runs) >= int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH):
workflow_ids_to_fetch_extra_runs.pop(workflow_id, None)
print(f"Successfully fetched extra workflow runs for: {workflow.filename}")
page += 1
print("Successfully fetched workflow runs details")
for workflow in list(workflows.values()):
- runs = sorted(workflow.runs, key=lambda r: r[0], reverse=True)
+ runs = sorted(workflow.runs, key=lambda r: r.id, reverse=True)
workflow.runs = runs[:int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH)]
return list(workflows.values())
-def database_operations(connection, workflows):
+
+def save_workflows(workflows):
+ connection = init_db_connection()
# Create the table and update it with the latest workflow runs
if not workflows:
return
cursor = connection.cursor()
workflows_table_name = "github_workflows"
+ workflow_runs_table_name = "github_workflow_runs"
cursor.execute(f"DROP TABLE IF EXISTS {workflows_table_name};")
- create_table_query = f"""
- CREATE TABLE IF NOT EXISTS {workflows_table_name} (
- workflow_id integer NOT NULL PRIMARY KEY,
- job_name text NOT NULL,
- job_yml_filename text NOT NULL,
- dashboard_category text NOT NULL"""
- for i in range(int(GH_NUMBER_OF_WORKFLOW_RUNS_TO_FETCH)):
- create_table_query += f""",
- run{i+1} text,
- run{i+1}Id text"""
- create_table_query += ")\n"
- cursor.execute(create_table_query)
- insert_query = f"INSERT INTO {workflows_table_name} VALUES "
+ cursor.execute(f"DROP TABLE IF EXISTS {workflow_runs_table_name};")
+ create_workflows_table_query = f"""
+ CREATE TABLE IF NOT EXISTS {workflows_table_name} (
+ workflow_id integer NOT NULL PRIMARY KEY,
+ name text NOT NULL,
+ filename text NOT NULL,
+ url text NOT NULL,
+ dashboard_category text NOT NULL,
+ threshold real NOT NULL)\n"""
+ create_workflow_runs_table_query = f"""
+ CREATE TABLE IF NOT EXISTS {workflow_runs_table_name} (
+ run_id text NOT NULL PRIMARY KEY,
+ status text NOT NULL,
+ url text NOT NULL,
+ workflow_id integer NOT NULL FOREIGN KEY,
+ started_at timestamp with time zone NOT NULL)\n"""
+ cursor.execute(create_workflows_table_query)
+ cursor.execute(create_workflow_runs_table_query)
+ insert_workflows_query = f"""
+ INSERT INTO {workflows_table_name} (workflow_id, name, filename, url, dashboard_category, threshold)
+ VALUES %s"""
+ insert_workflow_runs_query = f"""
+ INSERT INTO {workflow_runs_table_name} (run_id, status, url, workflow_id, started_at)
+ VALUES %s"""
+ insert_workflows = []
+ insert_workflow_runs = []
for workflow in workflows:
- category = get_dashboard_category(workflow.name)
- row_insert =\
- f"(\'{workflow.id}\',\'{workflow.name}\',\'{workflow.filename}\',\'{category}\'"
- for _, status, url in workflow.runs:
- row_insert += f",\'{status}\',\'{url}\'"
- insert_query += f"{row_insert}),"
- insert_query = insert_query[:-1] + ";"
- print(insert_query)
- cursor.execute(insert_query)
+ insert_workflows.append(
+ (
+ workflow.id,
+ workflow.name,
+ workflow.filename,
+ workflow.url,
+ workflow.category,
+ workflow.threshold
+ )
+ )
+ for run in workflow.runs:
+ if run.id != 0:
+ started_at = run.started_at.replace("T", " ")
+ insert_workflow_runs.append(
+ (
+ run.id,
+ run.status,
+ run.url,
+ workflow.id,
+ started_at
+ )
+ )
+ psycopg2.extras.execute_values(cursor, insert_workflows_query, insert_workflows)
+ psycopg2.extras.execute_values(cursor, insert_workflow_runs_query, insert_workflow_runs)
cursor.close()
connection.commit()
connection.close()
Review Comment:
Was working on multiple iterations of this trying to figure out different needs. At some point the data was split into worklfow table and a run table and it remained. Its connected by a key so i felt it can remain
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org