You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by an...@apache.org on 2023/09/29 19:52:25 UTC
[beam] branch onboard_python_tests created (now 744ad3ccb0c)
This is an automated email from the ASF dual-hosted git repository.
anandinguva pushed a change to branch onboard_python_tests
in repository https://gitbox.apache.org/repos/asf/beam.git
at 744ad3ccb0c Add GBK and combine python batch load tests to perf alert tool
This branch includes the following new commits:
new 744ad3ccb0c Add GBK and combine python batch load tests to perf alert tool
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[beam] 01/01: Add GBK and combine python batch load tests to perf alert tool
Posted by an...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
anandinguva pushed a commit to branch onboard_python_tests
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 744ad3ccb0c2afa2e62782ac9484a7cd45b80824
Author: Anand Inguva <an...@gmail.com>
AuthorDate: Fri Sep 29 15:52:12 2023 -0400
Add GBK and combine python batch load tests to perf alert tool
---
.../testing/analyzers/tests_config.yaml | 222 +++++++++++++++------
1 file changed, 160 insertions(+), 62 deletions(-)
diff --git a/sdks/python/apache_beam/testing/analyzers/tests_config.yaml b/sdks/python/apache_beam/testing/analyzers/tests_config.yaml
index bc74f292c48..0350b7be439 100644
--- a/sdks/python/apache_beam/testing/analyzers/tests_config.yaml
+++ b/sdks/python/apache_beam/testing/analyzers/tests_config.yaml
@@ -18,86 +18,184 @@
# for the unique key to define a test, please use the following format:
# {test_name}-{metric_name}
-pytorch_image_classification_benchmarks-resnet152-mean_inference_batch_latency_micro_secs:
+# pytorch_image_classification_benchmarks-resnet152-mean_inference_batch_latency_micro_secs:
+# test_description:
+# Pytorch image classification on 50k images of size 224 x 224 with resnet 152.
+# Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L63
+# Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=2
+# test_target:
+# metrics_dataset: beam_run_inference
+# metrics_table: torch_inference_imagenet_results_resnet152
+# project: apache-beam-testing
+# metric_name: mean_inference_batch_latency_micro_secs
+
+# pytorch_image_classification_benchmarks-resnet101-mean_load_model_latency_milli_secs:
+# test_description:
+# Pytorch image classification on 50k images of size 224 x 224 with resnet 101.
+# Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L34
+# Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7
+# test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+# metrics_dataset: beam_run_inference
+# metrics_table: torch_inference_imagenet_results_resnet101
+# project: apache-beam-testing
+# metric_name: mean_load_model_latency_milli_secs
+
+# pytorch_image_classification_benchmarks-resnet101-mean_inference_batch_latency_micro_secs:
+# test_description:
+# Pytorch image classification on 50k images of size 224 x 224 with resnet 101.
+# Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L34
+# Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=2
+# test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+# metrics_dataset: beam_run_inference
+# metrics_table: torch_inference_imagenet_results_resnet101
+# project: apache-beam-testing
+# metric_name: mean_inference_batch_latency_micro_secs
+
+# pytorch_image_classification_benchmarks-resnet152-GPU-mean_inference_batch_latency_micro_secs:
+# test_description:
+# Pytorch image classification on 50k images of size 224 x 224 with resnet 152 with Tesla T4 GPU.
+# Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L151
+# Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7
+# test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+# metrics_dataset: beam_run_inference
+# metrics_table: torch_inference_imagenet_results_resnet101
+# project: apache-beam-testing
+# metric_name: mean_inference_batch_latency_micro_secs
+
+# pytorch_image_classification_benchmarks-resnet152-GPU-mean_load_model_latency_milli_secs:
+# test_description:
+# Pytorch image classification on 50k images of size 224 x 224 with resnet 152 with Tesla T4 GPU.
+# Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L151
+# Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7
+# test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+# metrics_dataset: beam_run_inference
+# metrics_table: torch_inference_imagenet_results_resnet152_tesla_t4
+# project: apache-beam-testing
+# metric_name: mean_load_model_latency_milli_secs
+
+# pytorch_image_classification_benchmarks-resnet152-GPU-mean_inference_batch_latency_micro_secs:
+# test_description:
+# Pytorch image classification on 50k images of size 224 x 224 with resnet 152 with Tesla T4 GPU.
+# Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L151).
+# Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?from=now-90d&to=now&viewPanel=2
+# test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+# metrics_dataset: beam_run_inference
+# metrics_table: torch_inference_imagenet_results_resnet152_tesla_t4
+# project: apache-beam-testing
+# metric_name: mean_inference_batch_latency_micro_secs
+
+# test_cloudml_benchmark_cirteo_no_shuffle_10GB-runtime_sec:
+# test_description:
+# TFT Criteo test on 10 GB data with no Reshuffle.
+# Test link - [Test link](https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/sdks/python/apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py#L82)
+# metrics_dataset: beam_cloudml
+# metrics_table: cloudml_benchmark_cirteo_no_shuffle_10GB
+# project: apache-beam-testing
+# metric_name: runtime_sec
+
+# test_cloudml_benchmark_criteo_10GB-runtime_sec:
+# test_description:
+# TFT Criteo test on 10 GB data.
+# Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/sdks/python/apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py#LL104C7-L104C41
+# metrics_dataset: beam_cloudml
+# metrics_table: cloudml_benchmark_criteo_10GB
+# project: apache-beam-testing
+# metric_name: runtime_sec
+
+
+
+# Python Combine load tests at http://35.193.202.176/d/WNzYt13Zk/combine-load-tests?orgId=1
+combine_python_batch_2gb_10_byte_records:
test_description:
- Pytorch image classification on 50k images of size 224 x 224 with resnet 152.
- Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L63
- Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=2
- test_target:
- metrics_dataset: beam_run_inference
- metrics_table: torch_inference_imagenet_results_resnet152
+ Combine Python Load Test 2 GB 10 byte records
+ Test link - https://github.com/apache/beam/blob/5e38decf9e723a385057131b01bbd33d8c60bda3/.test-infra/jenkins/job_LoadTests_Combine_Python.groovy#L76C24-L76C65
+ Test dashboard - http://35.193.202.176/d/WNzYt13Zk/combine-load-tests?orgId=1&from=now-90d&to=now&var-processingType=batch&var-sdk=python&viewPanel=2
+ test_target: apache_beam.testing.load_tests.combine_test
+ metrics_dataset: load_test
+ metrics_table: python_dataflow_batch_combine_1
+ metric_name: runtime
project: apache-beam-testing
- metric_name: mean_inference_batch_latency_micro_secs
-pytorch_image_classification_benchmarks-resnet101-mean_load_model_latency_milli_secs:
+combine_python_batch_2gb_fanout_4:
test_description:
- Pytorch image classification on 50k images of size 224 x 224 with resnet 101.
- Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L34
- Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7
- test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
- metrics_dataset: beam_run_inference
- metrics_table: torch_inference_imagenet_results_resnet101
+ Combine Python Load test - 2GB Fanout 4
+ Test link - https://github.com/apache/beam/blob/5e38decf9e723a385057131b01bbd33d8c60bda3/.test-infra/jenkins/job_LoadTests_Combine_Python.groovy#L52
+ Test Dashboard - http://35.193.202.176/d/WNzYt13Zk/combine-load-tests?orgId=1&from=now-90d&to=now&var-processingType=batch&var-sdk=python&viewPanel=4
+ test_target: apache_beam.testing.load_tests.combine_test
+ metrics_dataset: load_test
+ metrics_table: python_dataflow_batch_combine_4
+ metric_name: runtime
project: apache-beam-testing
- metric_name: mean_load_model_latency_milli_secs
-pytorch_image_classification_benchmarks-resnet101-mean_inference_batch_latency_micro_secs:
- test_description:
- Pytorch image classification on 50k images of size 224 x 224 with resnet 101.
- Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L34
- Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=2
- test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
- metrics_dataset: beam_run_inference
- metrics_table: torch_inference_imagenet_results_resnet101
+combine_python_batch_2gb_fanout_8:
+ test_description: Combine Python Load test - 2GB Fanout 8
+ test_target: apache_beam.testing.load_tests.combine_test
+ metrics_dataset: load_test
+ metrics_table: python_dataflow_batch_combine_5
+ metric_name: runtime
project: apache-beam-testing
- metric_name: mean_inference_batch_latency_micro_secs
-pytorch_image_classification_benchmarks-resnet152-GPU-mean_inference_batch_latency_micro_secs:
+# Python Batch GBK load tests at http://35.193.202.176/d/UYZ-oJ3Zk/gbk-load-tests?orgId=1&var-processingType=batch&var-sdk=python&from=now-90d&to=now
+gbk_python_batch_load_test_2gb_of_10B_records:
test_description:
- Pytorch image classification on 50k images of size 224 x 224 with resnet 152 with Tesla T4 GPU.
- Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L151
- Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7
- test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
- metrics_dataset: beam_run_inference
- metrics_table: torch_inference_imagenet_results_resnet101
+ GroupByKey Python Load test - 2GB of 10B records
+ python | GBK | Small records (10B)
+ Test Dashboard - http://35.193.202.176/d/UYZ-oJ3Zk/gbk-load-tests?orgId=1&var-processingType=batch&var-sdk=python&from=now-90d&to=now&viewPanel=2
+ Test link - https://github.com/apache/beam/blob/5e38decf9e723a385057131b01bbd33d8c60bda3/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy#L36C25-L36C72
+ test_target: apache_beam.testing.load_tests.group_by_key_test
+ metrics_table: python_dataflow_batch_gbk_1
+ metrics_dataset: load_test
+ metric_name: runtime
project: apache-beam-testing
- metric_name: mean_inference_batch_latency_micro_secs
-pytorch_image_classification_benchmarks-resnet152-GPU-mean_load_model_latency_milli_secs:
+gbk_python_batch_load_test_2gb_of_100B_records:
test_description:
- Pytorch image classification on 50k images of size 224 x 224 with resnet 152 with Tesla T4 GPU.
- Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L151
- Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?orgId=1&viewPanel=7
- test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
- metrics_dataset: beam_run_inference
- metrics_table: torch_inference_imagenet_results_resnet152_tesla_t4
+ GroupByKey Python Load test - 2GB of 100B records
+ python | GBK | Medium records (100B)
+ Test Dashboard - http://35.193.202.176/d/UYZ-oJ3Zk/gbk-load-tests?orgId=1&var-processingType=batch&var-sdk=python&from=now-90d&to=now&viewPanel=3
+ Test link - https://github.com/apache/beam/blob/5e38decf9e723a385057131b01bbd33d8c60bda3/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy#L60
+ test_target: apache_beam.testing.load_tests.group_by_key_test
+ metrics_table: python_dataflow_batch_gbk_2
+ metrics_dataset: load_test
+ metric_name: runtime
project: apache-beam-testing
- metric_name: mean_load_model_latency_milli_secs
-pytorch_image_classification_benchmarks-resnet152-GPU-mean_inference_batch_latency_micro_secs:
+gbk_python_batch_load_test_2gb_of_100KB_records:
test_description:
- Pytorch image classification on 50k images of size 224 x 224 with resnet 152 with Tesla T4 GPU.
- Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy#L151).
- Test dashboard - http://metrics.beam.apache.org/d/ZpS8Uf44z/python-ml-runinference-benchmarks?from=now-90d&to=now&viewPanel=2
- test_target: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
- metrics_dataset: beam_run_inference
- metrics_table: torch_inference_imagenet_results_resnet152_tesla_t4
+ GroupByKey Python Load test - 2GB of 100kB records
+ python | GBK | Large records (100kB)
+ Test Dashboard - http://35.193.202.176/d/UYZ-oJ3Zk/gbk-load-tests?orgId=1&var-processingType=batch&var-sdk=python&from=now-6M&to=now&viewPanel=4&inspect=4
+ Test link - https://github.com/apache/beam/blob/5e38decf9e723a385057131b01bbd33d8c60bda3/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy#L83
+ test_target: apache_beam.testing.load_tests.group_by_key_test
+ metrics_table: python_dataflow_batch_gbk_3
+ metrics_dataset: load_test
+ metric_name: runtime
project: apache-beam-testing
- metric_name: mean_inference_batch_latency_micro_secs
-test_cloudml_benchmark_cirteo_no_shuffle_10GB-runtime_sec:
+gbk_python_batch_load_test_fanout_4_times_witj_2GB_10byte_records_total:
+# this test looks little noisy. Remove this if it causes too many false
+# positives.
test_description:
- TFT Criteo test on 10 GB data with no Reshuffle.
- Test link - [Test link](https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/sdks/python/apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py#L82)
- metrics_dataset: beam_cloudml
- metrics_table: cloudml_benchmark_cirteo_no_shuffle_10GB
+ GroupByKey Python Load test - fanout 4 times with 2GB 10-byte records total
+ python | GBK | Fanout 4
+ Test Dashboard - http://35.193.202.176/d/UYZ-oJ3Zk/gbk-load-tests?orgId=1&var-processingType=batch&var-sdk=python&from=now-90d&to=now&viewPanel=5
+ Test link - https://github.com/apache/beam/blob/5e38decf9e723a385057131b01bbd33d8c60bda3/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy#L106
+ test_target: apache_beam.testing.load_tests.group_by_key_test
+ metrics_table: python_dataflow_batch_gbk_4
+ metrics_dataset: load_test
+ metric_name: runtime
project: apache-beam-testing
- metric_name: runtime_sec
-test_cloudml_benchmark_criteo_10GB-runtime_sec:
+
+gbk_python_batch_load_test_fanout_8_times_witj_2GB_10byte_records_total:
+# this test looks little noisy. Remove this if it causes too many false
+# positives.
test_description:
- TFT Criteo test on 10 GB data.
- Test link - https://github.com/apache/beam/blob/42d0a6e3564d8b9c5d912428a6de18fb22a13ac1/sdks/python/apache_beam/testing/benchmarks/cloudml/cloudml_benchmark_test.py#LL104C7-L104C41
- metrics_dataset: beam_cloudml
- metrics_table: cloudml_benchmark_criteo_10GB
- project: apache-beam-testing
- metric_name: runtime_sec
+ GroupByKey Python Load test - fanout 8 times with 2GB 10-byte records total
+ python | GBK | Fanout 8
+ Test Dashboard - http://35.193.202.176/d/UYZ-oJ3Zk/gbk-load-tests?orgId=1&var-processingType=batch&var-sdk=python&from=now-90d&to=now&viewPanel=6
+ Test link - https://github.com/apache/beam/blob/5e38decf9e723a385057131b01bbd33d8c60bda3/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy#L129
+ metrics_table: python_dataflow_batch_gbk_5
+ metrics_dataset: load_test
+ metric_name: runtime
+ project: apache-beam-testing
\ No newline at end of file