You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "pranavbhandari24 (via GitHub)" <gi...@apache.org> on 2023/10/31 20:15:21 UTC

[PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

pranavbhandari24 opened a new pull request, #29226:
URL: https://github.com/apache/beam/pull/29226

   This PR adds ability to run regression checks against Beam IO load tests.
   
   Below is the command to run this,
   
   ```
   python3 apache_beam/testing/analyzers/load_test_perf_analysis.py --config_file_path=apache_beam/testing/analyzers/io_tests_config.yaml
   ```
   
   We can automate runs for these regression checks by adding a GH workflow in a separate PR.
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29226:
URL: https://github.com/apache/beam/pull/29226#discussion_r1378125759


##########
sdks/python/apache_beam/testing/analyzers/load_test_perf_analysis.py:
##########
@@ -0,0 +1,64 @@
+import argparse
+import logging
+from apache_beam.testing.analyzers import constants
+from apache_beam.testing.analyzers import perf_analysis_utils
+from apache_beam.testing.analyzers import perf_analysis
+from apache_beam.testing.analyzers.perf_analysis_utils import MetricContainer, TestConfigContainer
+from google.cloud import bigquery
+
+
+class LoadTestMetricsFetcher(perf_analysis_utils.MetricsFetcher):
+    def fetch_metric_data(self, *, test_config: TestConfigContainer) -> MetricContainer:
+        test_name, pipeline_name = test_config.test_name.split(',')
+        query = f"""
+          SELECT timestamp, metric.value
+          FROM {test_config.project}.{test_config.metrics_dataset}.{test_config.metrics_table}
+          CROSS JOIN UNNEST(metrics) AS metric
+          WHERE test_name = "{test_name}" AND pipeline_name = "{pipeline_name}" AND metric.name = "{test_config.metric_name}"
+          ORDER BY timestamp DESC
+          LIMIT {constants._NUM_DATA_POINTS_TO_RUN_CHANGE_POINT_ANALYSIS}
+        """
+        logging.debug("Running query: %s" % query)
+        client = bigquery.Client()
+        query_job = client.query(query=query)
+        metric_data = query_job.result().to_dataframe()
+        if metric_data.empty:
+            logging.error("No results returned from BigQuery. Please check the query.")
+        return MetricContainer(
+            values=metric_data['value'].tolist(),
+            timestamps=metric_data['timestamp'].tolist(),
+        )
+
+
+if __name__ == '__main__':
+    logging.basicConfig(level=logging.INFO)
+    load_test_metrics_fetcher = LoadTestMetricsFetcher()
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        '--config_file_path',
+        required=True,
+        type=str,
+        help='Path to the config file that contains data to run the Change Point '
+             'Analysis.The default file will used will be '
+             'apache_beam/testing/analyzers/tests.config.yml. '
+             'If you would like to use the Change Point Analysis for finding '
+             'performance regression in the tests, '
+             'please provide an .yml file in the same structure as the above '
+             'mentioned file. ')
+    parser.add_argument(
+        '--save_alert_metadata',
+        action='store_true',
+        default=False,
+        help='Save perf alert/ GH Issue metadata to BigQuery table.')
+    known_args, unknown_args = parser.parse_known_args()
+
+    if unknown_args:
+        logging.warning('Discarding unknown arguments : %s ' % unknown_args)
+
+    perf_analysis.run(
+        big_query_metrics_fetcher=load_test_metrics_fetcher,
+        config_file_path=known_args.config_file_path,
+        # Set this to true while running in production.
+        save_alert_metadata=known_args.save_alert_metadata  # pylint: disable=line-too-long

Review Comment:
   seems no need "line-to-long"



##########
sdks/python/apache_beam/testing/analyzers/load_test_perf_analysis.py:
##########
@@ -0,0 +1,64 @@
+import argparse
+import logging
+from apache_beam.testing.analyzers import constants
+from apache_beam.testing.analyzers import perf_analysis_utils
+from apache_beam.testing.analyzers import perf_analysis
+from apache_beam.testing.analyzers.perf_analysis_utils import MetricContainer, TestConfigContainer
+from google.cloud import bigquery
+
+
+class LoadTestMetricsFetcher(perf_analysis_utils.MetricsFetcher):

Review Comment:
   a brief pydoc string is appreciate. e.g. if this is considered internal, then disclaim this is internal



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #29226:
URL: https://github.com/apache/beam/pull/29226#issuecomment-1789303168

   Merging because of the previous approvals 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #29226:
URL: https://github.com/apache/beam/pull/29226


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on PR #29226:
URL: https://github.com/apache/beam/pull/29226#issuecomment-1789306986

   Not a big deal, but in general we should wait for all checks to pass before merging


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

Posted by "pranavbhandari24 (via GitHub)" <gi...@apache.org>.
pranavbhandari24 commented on code in PR #29226:
URL: https://github.com/apache/beam/pull/29226#discussion_r1378147441


##########
sdks/python/apache_beam/testing/analyzers/load_test_perf_analysis.py:
##########
@@ -0,0 +1,64 @@
+import argparse
+import logging
+from apache_beam.testing.analyzers import constants
+from apache_beam.testing.analyzers import perf_analysis_utils
+from apache_beam.testing.analyzers import perf_analysis
+from apache_beam.testing.analyzers.perf_analysis_utils import MetricContainer, TestConfigContainer
+from google.cloud import bigquery

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #29226:
URL: https://github.com/apache/beam/pull/29226#issuecomment-1788046734

   ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/29226?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#29226](https://app.codecov.io/gh/apache/beam/pull/29226?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (b10369e) into [master](https://app.codecov.io/gh/apache/beam/commit/a036a18117059e3fbcc06b7e6fbc2f8c67c74d4e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (a036a18) will **decrease** coverage by `0.02%`.
   > Report is 2 commits behind head on master.
   > The diff coverage is `0.00%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #29226      +/-   ##
   ==========================================
   - Coverage   38.32%   38.31%   -0.02%     
   ==========================================
     Files         688      689       +1     
     Lines      101846   101890      +44     
   ==========================================
     Hits        39036    39036              
   - Misses      61230    61274      +44     
     Partials     1580     1580              
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/beam/pull/29226/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [python](https://app.codecov.io/gh/apache/beam/pull/29226/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `29.91% <0.00%> (-0.03%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files](https://app.codecov.io/gh/apache/beam/pull/29226?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...ache\_beam/testing/analyzers/perf\_analysis\_utils.py](https://app.codecov.io/gh/apache/beam/pull/29226?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9hbmFseXplcnMvcGVyZl9hbmFseXNpc191dGlscy5weQ==) | `0.00% <0.00%> (ø)` | |
   | [...\_beam/testing/analyzers/load\_test\_perf\_analysis.py](https://app.codecov.io/gh/apache/beam/pull/29226?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9hbmFseXplcnMvbG9hZF90ZXN0X3BlcmZfYW5hbHlzaXMucHk=) | `0.00% <0.00%> (ø)` | |
   
   ... and [1 file with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/29226/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29226:
URL: https://github.com/apache/beam/pull/29226#discussion_r1378125234


##########
sdks/python/apache_beam/testing/analyzers/load_test_perf_analysis.py:
##########
@@ -0,0 +1,64 @@
+import argparse
+import logging
+from apache_beam.testing.analyzers import constants
+from apache_beam.testing.analyzers import perf_analysis_utils
+from apache_beam.testing.analyzers import perf_analysis
+from apache_beam.testing.analyzers.perf_analysis_utils import MetricContainer, TestConfigContainer
+from google.cloud import bigquery

Review Comment:
   google.cloud dependencies are optional for beam python SDK. Use this pattern to avoid beam fail to import when `[gcp]` extension not installed: 
   
   https://github.com/apache/beam/blob/9e4668d725fdcc5683541499b0d9e45f9b0ef71d/sdks/python/apache_beam/io/gcp/bigtableio.py#L58-L59



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29226:
URL: https://github.com/apache/beam/pull/29226#issuecomment-1787970726

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Add ability to run performance regression checks on Beam IO Load tests. [beam]

Posted by "pranavbhandari24 (via GitHub)" <gi...@apache.org>.
pranavbhandari24 commented on PR #29226:
URL: https://github.com/apache/beam/pull/29226#issuecomment-1787968694

   R: @AnandInguva, @Abacn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org