[GitHub] [beam] tvalentyn commented on a diff in pull request #25754: Onboard Performance tests using the Change point analysis tool.

["tvalentyn (via GitHub)" <gi...@apache.org>]

tvalentyn commented on code in PR #25754:
URL: https://github.com/apache/beam/pull/25754#discussion_r1137906112


##########
sdks/python/apache_beam/testing/analyzers/tests_config.yaml:
##########
@@ -16,22 +16,82 @@
 #
 
 test_1:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks_22
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet152
   project: apache-beam-testing
   metric_name: mean_load_model_latency_milli_secs
   labels:
     - run-inference
-  # Optional parameters.
-  min_runs_between_change_points: 3
-  num_runs_in_change_point_window: 30
 
 test_2:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet152
+  project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_3:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet101
+  project: apache-beam-testing
+  metric_name: mean_load_model_latency_milli_secs
+  labels:
+    - run-inference
+
+test_4:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet101
   project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_5:
+  test_name: TFT Criteo 10 GB no shuffle test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_cirteo_no_shuffle_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_6:
+  test_name: TFT Criteo 10 GB test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_criteo_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_7:
+  test_name: TFT Criteo 10 GB test with shuffle and fixed workers 50 on n1-standard-4 machine

Review Comment:
   50 is too many workers for this test, IMO. also, # of workers not indicated in other test cases, so maybe remove it here as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org