You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/28 12:49:44 UTC
[GitHub] [beam] kamilwu opened a new pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
kamilwu opened a new pull request #11555:
URL: https://github.com/apache/beam/pull/11555
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624863732
This LGTM. I believe the only open comment is about adding a landing page, but otherwise I do not have additional comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624863286
> > Some different colors (Example: http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow?orgId=1 -- TextIOIT | 1 GB | GCS | "Many files" | GCS Copies is in blue color)
>
> It was a purposeful change. This is the only test within Java IO IT dashboard that reports a different kind of metric (not _read_time_ or _write_time_, but _copies_per_sec_). I can modify the color if you think all colors should be the same.
No, different colors make sense for different metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-622141777
Can be useful to add link instructions on how to access dashboards into PR description.
[CWIKI](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics), section Grafana UI
I took a brief look at dashboards, here are my ideas:
- I'd put graphs with similar arguments side-by-side, make graphs narrower. Goal is to see general sense of current status. Current layout is too sparse and it takes way too much time to scroll.
- Set default dashboard interval to last month. If you feel month is not enough, choose value you feel more relevant.
- Some space can be recovered by removing time scale.
- I think that similar metrics can be put into single graph, ex: TextIO 1Gb [GCS, HDFS].
- I would add a short description of what tests are executed. We all can access code, but it would be much more handy. This can be done either as a text field at the top of dashboard, or as a description hint on each of graphs, or both.
These are my initial thoughts.
R: @aaltay
Ahmet, I thought you might be interested as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624580219
> all graphs missing recent data, java | coGBK | 100B records with a single key missing spark data for longer.)
Tests are not publishing new metrics, this is work in progress: https://github.com/apache/beam/pull/11534, https://github.com/apache/beam/pull/11567 and https://github.com/apache/beam/pull/11577. I'm pretty sure this will be done by the end of this week.
As for spark data, spark tests were introduced a short time ago.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan edited a comment on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan edited a comment on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-622141777
Can be useful to add link instructions on how to access dashboards into PR description.
[CWIKI](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics), section Grafana UI
Summary:
docker-compose build
docker-compose up
http://localhost:3000
login with "admin":\<check password in docker-compose.yml\>
I took a brief look at dashboards, here are my ideas:
- I'd put graphs with similar arguments side-by-side, make graphs narrower. Goal is to see general sense of current status. Current layout is too sparse and it takes way too much time to scroll.
- Set default dashboard interval to last month. If you feel month is not enough, choose value you feel more relevant.
- Some space can be recovered by removing time scale.
- I think that similar metrics can be put into single graph, ex: TextIO 1Gb [GCS, HDFS].
- I would add a short description of what tests are executed. We all can access code, but it would be much more handy. This can be done either as a text field at the top of dashboard, or as a description hint on each of graphs, or both.
- Would be great if local docker-compose service can fetch some data. It will be much more easy to understand what would be shown on each of the graphs. This would also help a lot for debugging or modifying dashboards.
- Might not be a part of this PR, but I'd be interested in seeing some summary window that shows only anomalies. Not sure there's a convenient way to do this in Grafana though...
These are my initial thoughts.
R: @aaltay
Ahmet, I think you might be interested as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan edited a comment on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan edited a comment on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-622141777
Can be useful to add link instructions on how to access dashboards into PR description.
[CWIKI](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics), section Grafana UI
Summary:
docker-compose build
docker-compose up
http://localhost:3000
login with "admin":<check password in docker-compose.yml>
I took a brief look at dashboards, here are my ideas:
- I'd put graphs with similar arguments side-by-side, make graphs narrower. Goal is to see general sense of current status. Current layout is too sparse and it takes way too much time to scroll.
- Set default dashboard interval to last month. If you feel month is not enough, choose value you feel more relevant.
- Some space can be recovered by removing time scale.
- I think that similar metrics can be put into single graph, ex: TextIO 1Gb [GCS, HDFS].
- I would add a short description of what tests are executed. We all can access code, but it would be much more handy. This can be done either as a text field at the top of dashboard, or as a description hint on each of graphs, or both.
These are my initial thoughts.
R: @aaltay
Ahmet, I think you might be interested as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624265259
Hey Kamil,
can we also add a proper landing page for metrics site? People regularly can't navigate to dashboards they need. Adding landing page with intuitive navigation would help a lot. That should be a separate PR though. [BEAM-6710](https://issues.apache.org/jira/browse/BEAM-6710)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-622148223
Comments from Ahmet:
- It would be great if we can make data-points clickable with links to relevant job
- If we collect multiple datapoints per test, we can display those on the same graph
- Use 2 columns (or more) instead of 1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan edited a comment on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan edited a comment on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-622141777
Can be useful to add link instructions on how to access dashboards into PR description.
[CWIKI](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics), section Grafana UI
Summary:
docker-compose build
docker-compose up
http://localhost:3000
login with "admin":\<check password in docker-compose.yml\>
I took a brief look at dashboards, here are my ideas:
- I'd put graphs with similar arguments side-by-side, make graphs narrower. Goal is to see general sense of current status. Current layout is too sparse and it takes way too much time to scroll.
- Set default dashboard interval to last month. If you feel month is not enough, choose value you feel more relevant.
- Some space can be recovered by removing time scale.
- I think that similar metrics can be put into single graph, ex: TextIO 1Gb [GCS, HDFS].
- I would add a short description of what tests are executed. We all can access code, but it would be much more handy. This can be done either as a text field at the top of dashboard, or as a description hint on each of graphs, or both.
- Would be great if local docker-compose service can fetch some data. It will be much more easy to understand what would be shown on each of the graphs. This would also help a lot for debugging or modifying dashboards.
These are my initial thoughts.
R: @aaltay
Ahmet, I think you might be interested as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624591248
Thanks for all comments. I will merge this PR tomorrow if they are no further action points.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-625391338
> The landing page will be created in a separate PR. I think this one can be closed now. Thanks!
Thank you! Please cc me in the PR, I am interested in that change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624577907
> Some dashboards seem to miss data, but that's due to not all data migrated IIUC.
> go benchmarks are completely empty.
Some of the Python on Flink tests are currently turned off. Kafka IO dashboard is empty, because the job's been flaky for some time and Go benchmarks are empty, because Go tests aren't implemented yet. I think every other dashboard does not miss any data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624590542
> can we also add a proper landing page for metrics site?
Sure, I can take care of it. It's true the navigation is a bit complicated at the moment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624575065
> I believe we can get jenkins job ID via env.JOB_NAME
Didn't know about this. Thanks, this makes things much easier.
Added a jira to do this improvement in the future: https://issues.apache.org/jira/browse/BEAM-9892
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-621305792
R: @Ardagan Could you run docker-compose and take a look at new dashboards?
There is no data in the database yet, so charts will be empty. But I'm open to any suggestions regarding layout, naming, etc.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624173530
@aaltay
> It would be great if we can make data-points clickable with links to relevant job
Grafana has a feature called Data links [1] that could be use here. But the biggest challenge is to get Jenkins job id for specific data point. When Python or Java test sends their metrics to InfluxDB/BigQuery, they have no knowledge of Jenkins job that executes them.
Without a rework of sending metrics, this functionality will be difficult to implement.
@Ardagan Any thoughs?
[1] https://grafana.com/docs/grafana/latest/reference/datalinks/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624213555
> @aaltay
>
> > It would be great if we can make data-points clickable with links to relevant job
>
> Grafana has a feature called Data links [1] that could be use here. But the biggest challenge is to get Jenkins job id for specific data point. When Python or Java test sends their metrics to InfluxDB/BigQuery, they have no knowledge of Jenkins job that executes them.
>
> Without a rework of sending metrics, this functionality will be difficult to implement.
>
> @Ardagan Any thoughs?
>
> [1] https://grafana.com/docs/grafana/latest/reference/datalinks/
I believe we can get jenkins job ID via [env.JOB_NAME](https://stackoverflow.com/questions/8309383/how-to-get-the-jobname-from-jenkins). We can add jira to do this improvement in separate PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624586993
> Since all dashboards have python/java selectors, why Python IO IT Tests and Java IO IT Tests are different dashboards?
We have only few Python IO IT tests at the moment. If IO IT dashboards had the same python/java selectors as Load tests, most of charts would be empty.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624242630
> Done. I pushed modified version to the website (http://metrics.beam.apache.org)
I do not see the new dashboard here. How can I find it?
I see these three:
Code Velocity
Post-commit Test Reliability
Stability critical jobs status
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624248353
/cc @chamikaramj @tysonjh @kennknowles -- optional review request, if you would like to take a quick look at new benchmarks at http://metrics.beam.apache.org.
(Instructions from @Ardagan : To find dashboards: click at top-left on "Home" or "current dashboard name", this will open drop-down list with full set of dashboards.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624247136
Some comments:
- I do see missing data. (Example: http://metrics.beam.apache.org/d/fK0U4JqWz/cogbk-load-tests?orgId=1 -- all graphs missing recent data, java | coGBK | 100B records with a single key missing spark data for longer.)
- go benchmarks are completely empty.
- Some different colors (Example: http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow?orgId=1 -- TextIOIT | 1 GB | GCS | "Many files" | GCS Copies is in blue color)
- Since all dashboards have python/java selectors, why Python IO IT Tests and Java IO IT Tests are different dashboards?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-622148912
@tysonjh FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan edited a comment on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan edited a comment on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-622141777
Can be useful to add link instructions on how to access dashboards into PR description.
[CWIKI](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics), section Grafana UI
I took a brief look at dashboards, here are my ideas:
- I'd put graphs with similar arguments side-by-side, make graphs narrower. Goal is to see general sense of current status. Current layout is too sparse and it takes way too much time to scroll.
- Set default dashboard interval to last month. If you feel month is not enough, choose value you feel more relevant.
- Some space can be recovered by removing time scale.
- I think that similar metrics can be put into single graph, ex: TextIO 1Gb [GCS, HDFS].
- I would add a short description of what tests are executed. We all can access code, but it would be much more handy. This can be done either as a text field at the top of dashboard, or as a description hint on each of graphs, or both.
These are my initial thoughts.
R: @aaltay
Ahmet, I think you might be interested as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624088189
@Ardagan Thanks for your suggestions!
>
> I'd put graphs with similar arguments side-by-side, make graphs narrower. Goal is to see general sense of current status. Current layout is too sparse and it takes way too much time to scroll.
> Set default dashboard interval to last month. If you feel month is not enough, choose value you feel more relevant.
Done. I pushed modified version to the website (http://metrics.beam.apache.org) so that you can see what's changed. Also, I pushed some data to InfluxDB to make the review process easier.
> Some space can be recovered by removing time scale.
Do you mean X axis or Y axis?
> I think that similar metrics can be put into single graph, ex: TextIO 1Gb [GCS, HDFS].
I tried this out, but four different data series (read_time x2 and write_time x2) in one single chart were not very readable. I think it's better to keep them separated.
> I would add a short description of what tests are executed. We all can access code, but it would be much more handy.
What kind of description do you expect? We have some documentation on what tests are executed in Beam website [1] and cwiki [2]. If something needs to improved, let's improve website/wiki content. I prefer to avoid repetitions between website/wiki and descriptions in Grafana, because it'd be hard to keep them in sync.
[1] https://beam.apache.org/documentation/io/testing/#i-o-transform-integration-tests
[2] https://cwiki.apache.org/confluence/display/BEAM/Contribution+Testing+Guide#ContributionTestingGuide-TestsofCoreApacheBeamOperations
> Would be great if local docker-compose service can fetch some data. It will be much more easy to understand what would be shown on each of the graphs. This would also help a lot for debugging or modifying dashboards.
I agree. Created a JIRA ticket to track the effort: https://issues.apache.org/jira/browse/BEAM-9889
> Might not be a part of this PR, but I'd be interested in seeing some summary window that shows only anomalies.
This would be feasible if we introduced Kapacitor (a component responsible for detecting anomalies). We could write back Kapacitor alerts into InfluxDB and visualize them in a summary window in Grafana. This is not a part of this PR, but I have a plan to introduce anomaly detection this month.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan edited a comment on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan edited a comment on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-622141777
Can be useful to add link instructions on how to access dashboards into PR description.
[CWIKI](https://cwiki.apache.org/confluence/display/BEAM/Community+Metrics), section Grafana UI
Summary:
docker-compose build
docker-compose up
http://localhost:3000
login with "admin":\<check password in docker-compose.yml\>
I took a brief look at dashboards, here are my ideas:
- I'd put graphs with similar arguments side-by-side, make graphs narrower. Goal is to see general sense of current status. Current layout is too sparse and it takes way too much time to scroll.
- Set default dashboard interval to last month. If you feel month is not enough, choose value you feel more relevant.
- Some space can be recovered by removing time scale.
- I think that similar metrics can be put into single graph, ex: TextIO 1Gb [GCS, HDFS].
- I would add a short description of what tests are executed. We all can access code, but it would be much more handy. This can be done either as a text field at the top of dashboard, or as a description hint on each of graphs, or both.
These are my initial thoughts.
R: @aaltay
Ahmet, I think you might be interested as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kamilwu commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
kamilwu commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624585465
> Some different colors (Example: http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow?orgId=1 -- TextIOIT | 1 GB | GCS | "Many files" | GCS Copies is in blue color)
It was a purposeful change. This is the only test within Java IO IT dashboard that reports a different kind of metric (not _read_time_ or _write_time_, but _copies_per_sec_). I can modify the color if you think all colors should be the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan commented on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan commented on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624216529
Some dashboards seem to miss data, but that's due to not all data migrated IIUC.
LGTM otherwise.
@aaltay can you take a look as well please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay edited a comment on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
aaltay edited a comment on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624247136
Some comments:
- I do see missing data. (Example: http://metrics.beam.apache.org/d/fK0U4JqWz/cogbk-load-tests?orgId=1 -- all graphs missing recent data, java | coGBK | 100B records with a single key missing spark data for longer.)
- go benchmarks are completely empty.
- Some different colors (Example: http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow?orgId=1 -- TextIOIT | 1 GB | GCS | "Many files" | GCS Copies is in blue color)
- Since all dashboards have python/java selectors, why Python IO IT Tests and Java IO IT Tests are different dashboards?
I might be missing other issues as well. If they are easy to fix later, we can fix what is identified, merge and ask for feedback on dev@ list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Ardagan edited a comment on pull request #11555: [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
Posted by GitBox <gi...@apache.org>.
Ardagan edited a comment on pull request #11555:
URL: https://github.com/apache/beam/pull/11555#issuecomment-624213555
> @aaltay
>
> > It would be great if we can make data-points clickable with links to relevant job
>
> Grafana has a feature called Data links [1] that could be use here. But the biggest challenge is to get Jenkins job id for specific data point. When Python or Java test sends their metrics to InfluxDB/BigQuery, they have no knowledge of Jenkins job that executes them.
>
> Without a rework of sending metrics, this functionality will be difficult to implement.
>
> @Ardagan Any thoughs?
>
> [1] https://grafana.com/docs/grafana/latest/reference/datalinks/
I believe we can get jenkins job ID via [env.JOB_NAME](https://stackoverflow.com/questions/8309383/how-to-get-the-jobname-from-jenkins), but this will required update test metric report logic and DB update IIUC. We can add jira to do this improvement in separate PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org