You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "AnandInguva (via GitHub)" <gi...@apache.org> on 2023/03/08 02:33:00 UTC

[GitHub] [beam] AnandInguva opened a new pull request, #25754: Cp analysis testing

AnandInguva opened a new pull request, #25754:
URL: https://github.com/apache/beam/pull/25754

   Don't merge until the service account has required big query permissions. Also, revert the deletion of files in .github/workflows
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #25754: Onboard Performance tests using the Change point analysis tool.

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25754:
URL: https://github.com/apache/beam/pull/25754#issuecomment-1468400934

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn merged pull request #25754: Onboard Performance tests using the Change point analysis tool.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn merged PR #25754:
URL: https://github.com/apache/beam/pull/25754


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #25754: Onboard Performance tests using the Change point analysis tool.

Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #25754:
URL: https://github.com/apache/beam/pull/25754#issuecomment-1468398137

   R: @tvalentyn 
   
   Note: We need to add perf-alert label to the repo.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #25754: Onboard Performance tests using the Change point analysis tool.

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #25754:
URL: https://github.com/apache/beam/pull/25754#issuecomment-1459219599

   # [Codecov](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#25754](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (216e43f) into [master](https://codecov.io/gh/apache/beam/commit/e4ddb861756444e9761695298598b264528e8647?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e4ddb86) will **decrease** coverage by `0.15%`.
   > The diff coverage is `66.66%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #25754      +/-   ##
   ==========================================
   - Coverage   72.82%   72.67%   -0.15%     
   ==========================================
     Files         775      750      -25     
     Lines      102936    99669    -3267     
   ==========================================
   - Hits        74958    72435    -2523     
   + Misses      26524    25879     -645     
   + Partials     1454     1355      -99     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...hon/apache\_beam/testing/analyzers/perf\_analysis.py](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9hbmFseXplcnMvcGVyZl9hbmFseXNpcy5weQ==) | `17.39% <0.00%> (ø)` | |
   | [.../python/apache\_beam/testing/analyzers/constants.py](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9hbmFseXplcnMvY29uc3RhbnRzLnB5) | `100.00% <100.00%> (ø)` | |
   | [...ache\_beam/testing/analyzers/github\_issues\_utils.py](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9hbmFseXplcnMvZ2l0aHViX2lzc3Vlc191dGlscy5weQ==) | `49.05% <100.00%> (ø)` | |
   | [sdks/go/pkg/beam/core/runtime/exec/plan.go](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9wbGFuLmdv) | `47.29% <0.00%> (-22.71%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/exec/datasource.go](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9kYXRhc291cmNlLmdv) | `62.09% <0.00%> (-6.85%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/exec/translate.go](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy90cmFuc2xhdGUuZ28=) | `17.19% <0.00%> (-6.55%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/exec/sdf\_invokers.go](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy9zZGZfaW52b2tlcnMuZ28=) | `70.18% <0.00%> (-6.13%)` | :arrow_down: |
   | [...ks/go/pkg/beam/runners/dataflow/dataflowlib/job.go](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL2RhdGFmbG93L2RhdGFmbG93bGliL2pvYi5nbw==) | `23.92% <0.00%> (-5.39%)` | :arrow_down: |
   | [sdks/go/pkg/beam/io/filesystem/gcs/gcs.go](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9pby9maWxlc3lzdGVtL2djcy9nY3MuZ28=) | `7.29% <0.00%> (-4.17%)` | :arrow_down: |
   | [sdks/go/pkg/beam/core/runtime/exec/userstate.go](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZXhlYy91c2Vyc3RhdGUuZ28=) | `5.46% <0.00%> (-1.92%)` | :arrow_down: |
   | ... and [47 more](https://codecov.io/gh/apache/beam/pull/25754?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on a diff in pull request #25754: Onboard Performance tests using the Change point analysis tool.

Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on code in PR #25754:
URL: https://github.com/apache/beam/pull/25754#discussion_r1140381816


##########
sdks/python/apache_beam/testing/analyzers/tests_config.yaml:
##########
@@ -16,22 +16,82 @@
 #
 
 test_1:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks_22
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet152
   project: apache-beam-testing
   metric_name: mean_load_model_latency_milli_secs
   labels:
     - run-inference
-  # Optional parameters.
-  min_runs_between_change_points: 3
-  num_runs_in_change_point_window: 30
 
 test_2:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet152
+  project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_3:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet101
+  project: apache-beam-testing
+  metric_name: mean_load_model_latency_milli_secs
+  labels:
+    - run-inference
+
+test_4:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet101
   project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_5:
+  test_name: TFT Criteo 10 GB no shuffle test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_cirteo_no_shuffle_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_6:
+  test_name: TFT Criteo 10 GB test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_criteo_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_7:
+  test_name: TFT Criteo 10 GB test with shuffle and fixed workers 50 on n1-standard-4 machine

Review Comment:
   removed the test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #25754: Onboard Performance tests using the Change point analysis tool.

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #25754:
URL: https://github.com/apache/beam/pull/25754#discussion_r1137906112


##########
sdks/python/apache_beam/testing/analyzers/tests_config.yaml:
##########
@@ -16,22 +16,82 @@
 #
 
 test_1:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks_22
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet152
   project: apache-beam-testing
   metric_name: mean_load_model_latency_milli_secs
   labels:
     - run-inference
-  # Optional parameters.
-  min_runs_between_change_points: 3
-  num_runs_in_change_point_window: 30
 
 test_2:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet152
+  project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_3:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet101
+  project: apache-beam-testing
+  metric_name: mean_load_model_latency_milli_secs
+  labels:
+    - run-inference
+
+test_4:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet101
   project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_5:
+  test_name: TFT Criteo 10 GB no shuffle test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_cirteo_no_shuffle_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_6:
+  test_name: TFT Criteo 10 GB test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_criteo_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_7:
+  test_name: TFT Criteo 10 GB test with shuffle and fixed workers 50 on n1-standard-4 machine

Review Comment:
   50 is too many workers for this test, IMO. also, # of workers not indicated in other test cases, so maybe remove it here as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #25754: Cp analysis testing

Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #25754:
URL: https://github.com/apache/beam/pull/25754#issuecomment-1459194502

   cc: @tvalentyn this is the branch I am using to run on my fork. I had to delete the files in workflows folder since my fork has only one GHA runner.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on pull request #25754: Onboard Performance tests using the Change point analysis tool.

Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on PR #25754:
URL: https://github.com/apache/beam/pull/25754#issuecomment-1474219648

   PTAL @tvalentyn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] AnandInguva commented on a diff in pull request #25754: Onboard Performance tests using the Change point analysis tool.

Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on code in PR #25754:
URL: https://github.com/apache/beam/pull/25754#discussion_r1138788032


##########
sdks/python/apache_beam/testing/analyzers/tests_config.yaml:
##########
@@ -16,22 +16,82 @@
 #
 
 test_1:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks_22
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet152
   project: apache-beam-testing
   metric_name: mean_load_model_latency_milli_secs
   labels:
     - run-inference
-  # Optional parameters.
-  min_runs_between_change_points: 3
-  num_runs_in_change_point_window: 30
 
 test_2:
-  test_name: apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 152
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet152
+  project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_3:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
+  metrics_dataset: beam_run_inference
+  metrics_table: torch_inference_imagenet_results_resnet101
+  project: apache-beam-testing
+  metric_name: mean_load_model_latency_milli_secs
+  labels:
+    - run-inference
+
+test_4:
+  test_name: Pytorch image classification on 50k images of size 224 x 224 with resnet 101
   metrics_dataset: beam_run_inference
   metrics_table: torch_inference_imagenet_results_resnet101
   project: apache-beam-testing
+  metric_name: mean_inference_batch_latency_micro_secs
+  labels:
+    - run-inference
+
+test_5:
+  test_name: TFT Criteo 10 GB no shuffle test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_cirteo_no_shuffle_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_6:
+  test_name: TFT Criteo 10 GB test
+  metrics_dataset: beam_cloudml
+  metrics_table: cloudml_benchmark_criteo_10GB
+  project: apache-beam-testing
+  metric_name: runtime_sec
+  labels:
+    - python_tft_criteo
+
+test_7:
+  test_name: TFT Criteo 10 GB test with shuffle and fixed workers 50 on n1-standard-4 machine

Review Comment:
   This was the same configurations used in the tft criteo tests in google3. mentioning 50 workers indicates we use 50 machines rather than autoscaling. 
   
   if 50 is too many for this test, we can change that in a different PR and disable the config here. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org