[GitHub] [beam] kamilwu commented on a change in pull request #12499: [BEAM-10602] Fix load test metrics in Grafana dashboard

[GitBox <gi...@apache.org>]

kamilwu commented on a change in pull request #12499:
URL: https://github.com/apache/beam/pull/12499#discussion_r467907301



##########
File path: .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
##########
@@ -161,12 +164,13 @@ def streamingScenarios = { datasetName ->
       test           : 'apache_beam.testing.load_tests.pardo_test',
       runner         : CommonTestProperties.Runner.PORTABLE,
       pipelineOptions: [
-        job_name             : 'load-tests-python-flink-streaming-pardo-5-' + now,
+        job_name             : 'load-tests-python-flink-streaming-pardo-1-' + now,

Review comment:
       I have mixed feelings about it. This test is not `load-tests-python-flink-batch-pardo-1` but on streaming. There are more differences between them: batch-pardo-1 uses 10 iterations, this test uses 5 iterations. 0 counters in batch-pardo-1 vs. 3 counters right here. Because of that, I think we should stay with the previous job_name: `load-tests-python-flink-streaming-pardo-5`.
   
   The general idea behind load tests is that we run the same configuration on different runners, in different SDKs and in different mode (batch or streaming). Grafana dashboards for load tests were designed with that convention in mind. If you choose `java` and `streaming` from the list, Grafana will pull data from these measurements: `java_streaming_pardo_1`, `java_streaming_pardo_2` and so. Your streaming tests are a bit problematic, because they are not being run on Dataflow and batch. Also, they have no Java counterpart.
   
   That being said, I think about two solutions:
   1) Add more charts. We would end up with a total of six charts. The fifth and the sixth chart would be empty in most cases (for Java and for batch).
   2) Create a separate, more specific version of dashboard just for these two tests (streaming-pardo-5 and streaming-pardo-6). Leave "ParDo Load Tests" dashboard intact.
   
   @mxm What do you think?
   
   
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org