[GitHub] [beam] AnandInguva commented on a diff in pull request #25754: Onboard Performance tests using the Change point analysis tool.

["AnandInguva (via GitHub)" <gi...@apache.org>]

AnandInguva commented on code in PR #25754:
URL: https://github.com/apache/beam/pull/25754#discussion_r1138788032


##########
sdks/python/apache_beam/testing/analyzers/tests_config.yaml:
##########
@@ -16,22 +16,82 @@
 #
 
 test_1:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks_22
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet152
   project: apache-beam-testing
   metric_name: mean_load_model_latency_milli_secs
   labels:
     - run-inference
-  # Optional parameters.
-  min_runs_between_change_points: 3
-  num_runs_in_change_point_window: 30
 
 test_2:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet152
+  project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_3:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet101
+  project: apache-beam-testing
+  metric_name: mean_load_model_latency_milli_secs
+  labels:
+    - run-inference
+
+test_4:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet101
   project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_5:
+  test_name: TFT Criteo 10 GB no shuffle test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_cirteo_no_shuffle_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_6:
+  test_name: TFT Criteo 10 GB test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_criteo_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_7:
+  test_name: TFT Criteo 10 GB test with shuffle and fixed workers 50 on n1-standard-4 machine

Review Comment:
   This was the same configurations used in the tft criteo tests in google3. mentioning 50 workers indicates we use 50 machines rather than autoscaling. 
   
   if 50 is too many for this test, we can change that in a different PR and disable the config here. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org