You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/09/26 16:18:52 UTC
[GitHub] [beam] AnandInguva opened a new pull request, #23371: RunInference Benchmarks UI
AnandInguva opened a new pull request, #23371:
URL: https://github.com/apache/beam/pull/23371
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23371:
URL: https://github.com/apache/beam/pull/23371#discussion_r980414683
##########
sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py:
##########
@@ -151,7 +151,7 @@ def get_all_distributions_by_type(dist, metric_id):
list of :class:`DistributionMetric` objects
"""
submit_timestamp = time.time()
- dist_types = ['count', 'max', 'min', 'sum']
+ dist_types = ['count', 'max', 'min', 'sum', 'mean']
Review Comment:
Looking back through history, it looks like this was intentionally omitted in favor of sum/count - https://github.com/apache/beam/commit/bef40cc02231945bf345065e0fb0703d42fce86c - are we able to just use sum/count instead of mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #23371:
URL: https://github.com/apache/beam/pull/23371#discussion_r980423505
##########
sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py:
##########
@@ -151,7 +151,7 @@ def get_all_distributions_by_type(dist, metric_id):
list of :class:`DistributionMetric` objects
"""
submit_timestamp = time.time()
- dist_types = ['count', 'max', 'min', 'sum']
+ dist_types = ['count', 'max', 'min', 'sum', 'mean']
Review Comment:
Yep. I went through that PR. I use the `LoadTest` which has helper functions coded already to publish the metrics to the InfluxDB.
I added `mean` because `DistributionResult` has mean() method and to avoid redundant calculation mean with sum/count
https://github.com/apache/beam/blob/3a4d57eb8976c5f503b32d478a80b1800490f66f/sdks/python/apache_beam/metrics/cells.py#L270
##########
sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py:
##########
@@ -151,7 +151,7 @@ def get_all_distributions_by_type(dist, metric_id):
list of :class:`DistributionMetric` objects
"""
submit_timestamp = time.time()
- dist_types = ['count', 'max', 'min', 'sum']
+ dist_types = ['count', 'max', 'min', 'sum', 'mean']
Review Comment:
Yep. I went through that PR. I use the `LoadTest` which has helper functions coded already to publish the metrics to the InfluxDB.
I added `mean` because `DistributionResult` has mean() method and to avoid redundant calculation of mean with sum/count
https://github.com/apache/beam/blob/3a4d57eb8976c5f503b32d478a80b1800490f66f/sdks/python/apache_beam/metrics/cells.py#L270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #23371:
URL: https://github.com/apache/beam/pull/23371#discussion_r980423505
##########
sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py:
##########
@@ -151,7 +151,7 @@ def get_all_distributions_by_type(dist, metric_id):
list of :class:`DistributionMetric` objects
"""
submit_timestamp = time.time()
- dist_types = ['count', 'max', 'min', 'sum']
+ dist_types = ['count', 'max', 'min', 'sum', 'mean']
Review Comment:
Yep. I went through that PR. I use the `LoadTest` which has helper functions coded already to publish the metrics to the InfluxDB.
I added it because `DistributionResult` has mean() method and to avoid redundant calculation mean with sum/count
https://github.com/apache/beam/blob/3a4d57eb8976c5f503b32d478a80b1800490f66f/sdks/python/apache_beam/metrics/cells.py#L270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23371:
URL: https://github.com/apache/beam/pull/23371#discussion_r980329369
##########
.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json:
##########
@@ -0,0 +1,175 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 16,
+ "links": [],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamInfluxDB",
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 3,
+ "interval": "6h",
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 2,
+ "nullPointMode": "connected",
+ "options": {
+ "dataLinks": []
+ },
+ "percentage": false,
+ "pointradius": 2,
+ "points": true,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "latency_milli_seconds",
+ "groupBy": [
+ {
+ "params": [
+ "$__interval"
+ ],
+ "type": "time"
+ },
+ {
+ "params": [
+ "null"
+ ],
+ "type": "fill"
+ }
+ ],
+ "orderByTime": "ASC",
+ "policy": "default",
+ "query": "SELECT mean(\"value\") FROM \"torch_inference_imagenet_resnet101\" WHERE metric = 'BeamML_PyTorch_pytorchruninference/beamml_runinference_count_load_model_latency_milli_secs' AND $timeFilter GROUP BY time($__interval), \"metric\"",
+ "rawQuery": true,
+ "refId": "A",
+ "resultFormat": "time_series",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "field"
+ },
+ {
+ "params": [],
+ "type": "mean"
+ }
+ ]
+ ],
+ "tags": []
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "RunInference Image Classification - ResNet 101",
Review Comment:
> I plan to include different graphs for each metric.
In that case, could we name this `Pytorch RunInference Image Classification - ResNet 101 Mean Model Load Time`?
> It's weird that the mean value is not getting published to InfluxDB. I will investigate it.
Cool - lmk what you find there. Other than that and the naming nit, this LGTM and should be ready to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23371:
URL: https://github.com/apache/beam/pull/23371#discussion_r980295684
##########
.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json:
##########
@@ -0,0 +1,175 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 16,
+ "links": [],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamInfluxDB",
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 3,
+ "interval": "6h",
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 2,
+ "nullPointMode": "connected",
+ "options": {
+ "dataLinks": []
+ },
+ "percentage": false,
+ "pointradius": 2,
+ "points": true,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "latency_milli_seconds",
+ "groupBy": [
+ {
+ "params": [
+ "$__interval"
+ ],
+ "type": "time"
+ },
+ {
+ "params": [
+ "null"
+ ],
+ "type": "fill"
+ }
+ ],
+ "orderByTime": "ASC",
+ "policy": "default",
+ "query": "SELECT mean(\"value\") FROM \"torch_inference_imagenet_resnet101\" WHERE metric = 'BeamML_PyTorch_pytorchruninference/beamml_runinference_count_load_model_latency_milli_secs' AND $timeFilter GROUP BY time($__interval), \"metric\"",
+ "rawQuery": true,
+ "refId": "A",
+ "resultFormat": "time_series",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "field"
+ },
+ {
+ "params": [],
+ "type": "mean"
+ }
+ ]
+ ],
+ "tags": []
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "RunInference Image Classification - ResNet 101",
Review Comment:
A couple questions/suggestions on this naming:
1) It would be good to make it clear this is a pytorch model
2) Are you planning on making different graphs per metric or overlaying different series on this graph? I'd probably vote for different graphs per metric since different metrics will have very different y-axis values, making them hard to visualize well at the same time. If we use different graphs, I'd recommend specifically calling out that this is measuring mean model load latency
##########
.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json:
##########
@@ -0,0 +1,175 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 16,
+ "links": [],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamInfluxDB",
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 3,
+ "interval": "6h",
Review Comment:
How often do we run the job that populates this graph?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm merged pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
damccorm merged PR #23371:
URL: https://github.com/apache/beam/pull/23371
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23371:
URL: https://github.com/apache/beam/pull/23371#issuecomment-1258405388
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #23371:
URL: https://github.com/apache/beam/pull/23371#discussion_r980408004
##########
sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py:
##########
@@ -151,7 +151,7 @@ def get_all_distributions_by_type(dist, metric_id):
list of :class:`DistributionMetric` objects
"""
submit_timestamp = time.time()
- dist_types = ['count', 'max', 'min', 'sum']
+ dist_types = ['count', 'max', 'min', 'sum', 'mean']
Review Comment:
@damccorm Added mean here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23371:
URL: https://github.com/apache/beam/pull/23371#discussion_r980460261
##########
sdks/python/apache_beam/testing/load_tests/load_test_metrics_utils.py:
##########
@@ -151,7 +151,7 @@ def get_all_distributions_by_type(dist, metric_id):
list of :class:`DistributionMetric` objects
"""
submit_timestamp = time.time()
- dist_types = ['count', 'max', 'min', 'sum']
+ dist_types = ['count', 'max', 'min', 'sum', 'mean']
Review Comment:
I'm good with that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #23371:
URL: https://github.com/apache/beam/pull/23371#issuecomment-1259518996
Run Python PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #23371:
URL: https://github.com/apache/beam/pull/23371#issuecomment-1258373434
>> We should definitely monitor and make sure this is doing the right thing once we do merge since its a tough change to review + an easy place to make mistakes
I agree.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23371:
URL: https://github.com/apache/beam/pull/23371#discussion_r980329369
##########
.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json:
##########
@@ -0,0 +1,175 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 16,
+ "links": [],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamInfluxDB",
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 3,
+ "interval": "6h",
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 2,
+ "nullPointMode": "connected",
+ "options": {
+ "dataLinks": []
+ },
+ "percentage": false,
+ "pointradius": 2,
+ "points": true,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "latency_milli_seconds",
+ "groupBy": [
+ {
+ "params": [
+ "$__interval"
+ ],
+ "type": "time"
+ },
+ {
+ "params": [
+ "null"
+ ],
+ "type": "fill"
+ }
+ ],
+ "orderByTime": "ASC",
+ "policy": "default",
+ "query": "SELECT mean(\"value\") FROM \"torch_inference_imagenet_resnet101\" WHERE metric = 'BeamML_PyTorch_pytorchruninference/beamml_runinference_count_load_model_latency_milli_secs' AND $timeFilter GROUP BY time($__interval), \"metric\"",
+ "rawQuery": true,
+ "refId": "A",
+ "resultFormat": "time_series",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "field"
+ },
+ {
+ "params": [],
+ "type": "mean"
+ }
+ ]
+ ],
+ "tags": []
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "RunInference Image Classification - ResNet 101",
Review Comment:
> I plan to include different graphs for each metric.
In that case, could we name this `Pytorch RunInference Image Classification - ResNet 101 Mean Model Load Time`?
> It's weird that the mean value is not getting published to InfluxDB. I will investigate it.
Cool - lmk what you find there. Other than that, this LGTM and should be ready to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #23371:
URL: https://github.com/apache/beam/pull/23371#issuecomment-1258299363
R: @damccorm
I added a single dashboard for now to test for now. I can add more in the coming PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #23371:
URL: https://github.com/apache/beam/pull/23371#discussion_r980316727
##########
.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json:
##########
@@ -0,0 +1,175 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 16,
+ "links": [],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamInfluxDB",
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 3,
+ "interval": "6h",
+ "legend": {
+ "avg": false,
+ "current": false,
+ "max": false,
+ "min": false,
+ "show": true,
+ "total": false,
+ "values": false
+ },
+ "lines": true,
+ "linewidth": 2,
+ "nullPointMode": "connected",
+ "options": {
+ "dataLinks": []
+ },
+ "percentage": false,
+ "pointradius": 2,
+ "points": true,
+ "renderer": "flot",
+ "seriesOverrides": [],
+ "spaceLength": 10,
+ "stack": false,
+ "steppedLine": false,
+ "targets": [
+ {
+ "alias": "latency_milli_seconds",
+ "groupBy": [
+ {
+ "params": [
+ "$__interval"
+ ],
+ "type": "time"
+ },
+ {
+ "params": [
+ "null"
+ ],
+ "type": "fill"
+ }
+ ],
+ "orderByTime": "ASC",
+ "policy": "default",
+ "query": "SELECT mean(\"value\") FROM \"torch_inference_imagenet_resnet101\" WHERE metric = 'BeamML_PyTorch_pytorchruninference/beamml_runinference_count_load_model_latency_milli_secs' AND $timeFilter GROUP BY time($__interval), \"metric\"",
+ "rawQuery": true,
+ "refId": "A",
+ "resultFormat": "time_series",
+ "select": [
+ [
+ {
+ "params": [
+ "value"
+ ],
+ "type": "field"
+ },
+ {
+ "params": [],
+ "type": "mean"
+ }
+ ]
+ ],
+ "tags": []
+ }
+ ],
+ "thresholds": [],
+ "timeFrom": null,
+ "timeRegions": [],
+ "timeShift": null,
+ "title": "RunInference Image Classification - ResNet 101",
Review Comment:
I plan to include different graphs for each metric. It's weird that the `mean` value is not getting published to InfluxDB. I will investigate it.
##########
.test-infra/metrics/grafana/dashboards/perftests_metrics/Python_ML_RunInference_Benchmark_Tests.json:
##########
@@ -0,0 +1,175 @@
+{
+ "annotations": {
+ "list": [
+ {
+ "builtIn": 1,
+ "datasource": "-- Grafana --",
+ "enable": true,
+ "hide": true,
+ "iconColor": "rgba(0, 211, 255, 1)",
+ "name": "Annotations & Alerts",
+ "type": "dashboard"
+ }
+ ]
+ },
+ "editable": true,
+ "gnetId": null,
+ "graphTooltip": 0,
+ "id": 16,
+ "links": [],
+ "panels": [
+ {
+ "aliasColors": {},
+ "bars": false,
+ "dashLength": 10,
+ "dashes": false,
+ "datasource": "BeamInfluxDB",
+ "fill": 1,
+ "fillGradient": 0,
+ "gridPos": {
+ "h": 9,
+ "w": 12,
+ "x": 12,
+ "y": 0
+ },
+ "hiddenSeries": false,
+ "id": 3,
+ "interval": "6h",
Review Comment:
Changed it to `24h`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #23371: RunInference Benchmarks UI
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #23371:
URL: https://github.com/apache/beam/pull/23371#issuecomment-1258529641
# [Codecov](https://codecov.io/gh/apache/beam/pull/23371?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#23371](https://codecov.io/gh/apache/beam/pull/23371?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f097de3) into [master](https://codecov.io/gh/apache/beam/commit/2f1f1a76419a032ea2d70671e3c6c9fe86b0626f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2f1f1a7) will **decrease** coverage by `0.00%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #23371 +/- ##
==========================================
- Coverage 73.42% 73.41% -0.01%
==========================================
Files 718 718
Lines 95622 95622
==========================================
- Hits 70208 70205 -3
- Misses 24103 24106 +3
Partials 1311 1311
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.19% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/23371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/python/apache\_beam/metrics/cells.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWV0cmljcy9jZWxscy5weQ==) | `82.11% <0.00%> (ø)` | |
| [...beam/testing/load\_tests/load\_test\_metrics\_utils.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9sb2FkX3Rlc3RzL2xvYWRfdGVzdF9tZXRyaWNzX3V0aWxzLnB5) | `34.07% <0.00%> (ø)` | |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `87.57% <0.00%> (-1.70%)` | :arrow_down: |
| [...python/apache\_beam/runners/worker/worker\_status.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3N0YXR1cy5weQ==) | `75.33% <0.00%> (-1.34%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/localfilesystem.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vbG9jYWxmaWxlc3lzdGVtLnB5) | `90.97% <0.00%> (-0.76%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.42% <0.00%> (-0.13%)` | :arrow_down: |
| [sdks/python/apache\_beam/transforms/util.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy91dGlsLnB5) | `96.22% <0.00%> (+0.15%)` | :arrow_up: |
| [...eam/runners/interactive/interactive\_environment.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9lbnZpcm9ubWVudC5weQ==) | `92.02% <0.00%> (+0.30%)` | :arrow_up: |
| [...eam/runners/portability/fn\_api\_runner/execution.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2V4ZWN1dGlvbi5weQ==) | `93.08% <0.00%> (+0.64%)` | :arrow_up: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/23371/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `97.56% <0.00%> (+2.43%)` | :arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org