You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Amar3tto (via GitHub)" <gi...@apache.org> on 2025/10/08 12:23:47 UTC
[PR] Add readme How to add a new ML benchmark pipeline [beam]
Amar3tto opened a new pull request, #36437:
URL: https://github.com/apache/beam/pull/36437
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add readme How to add a new ML benchmark pipeline [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36437:
URL: https://github.com/apache/beam/pull/36437#issuecomment-3381724721
Assigning reviewers:
R: @damccorm for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add readme How to add a new ML benchmark pipeline [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on code in PR #36437:
URL: https://github.com/apache/beam/pull/36437#discussion_r2413893166
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
+- Ensure the expected 'pcollection' parameter is passed to your builder. This parameter could be obtained from GCP Dataflow Jobs -> Your Job Page.
+- Keep naming consistent with other benchmarks.
+
+3. Add an options txt file
+
+- Location: .github/workflows/load-tests-pipeline-options/<pipeline_name>.txt
+- Include Dataflow and pipeline flags. Example:
+
+```
+--region=us-central1
+--machine_type=n1-standard-2
+--num_workers=75
+--disk_size_gb=50
+--autoscaling_algorithm=NONE
+--staging_location=gs://temp-storage-for-perf-tests/loadtests
+--temp_location=gs://temp-storage-for-perf-tests/loadtests
+--requirements_file=apache_beam/ml/inference/your-requirements-file.txt
+--publish_to_big_query=true
+--metrics_dataset=beam_run_inference
+--metrics_table=your_table
+--input_options={}
+--influx_measurement=your-measurement
+--device=CPU
+--runner=DataflowRunner
+```
+
+4. Wire it into the GitHub Action
+
+- Workflow: .github/workflows/beam_Inference_Python_Benchmarks_Dataflow.yml
+- Add your argument-file-path to the matrix.
+- Add a step that runs your <pipeline_name>_benchmarks.py with -PloadTest.args=$YOUR_ARGUMENTS.
+
+5. Test on your fork
+
+- Trigger the workflow manually.
+- Confirm the Dataflow job completes successfully.
+
+6. Verify metrics in BigQuery
+
+- Dataset: beam_run_inference. Table: your_table
+- Confirm new rows for your pipeline_name with recent timestamps.
+
+7. Update the website
+
+- Create: website/www/site/content/en/performance/<pipeline_name>/_index.md (short title/description).
+- Update: website/www/site/data/performance.yaml — add your pipeline and five chart entries with:
+- - looker_folder_id
+- - public_slug_id (from Looker, see below)
+
+8. Create Looker content (5 charts)
+
+- In Looker → Shared folders → run_inference: create a subfolder for your pipeline.
+- From an existing chart: Development mode → Explore from here → Go to LookML.
+- Point to your table/view and create 5 standard charts (latency/throughput/cost/etc.).
+- Save changes → Publish to production.
+- From Explore, open each, set fields/filters for your pipeline, Run, then Save as Look (in your folder).
+- Open each Look:
+- - Copy Look ID
+- - Add Look IDs to .test-infra/tools/refresh_looker_metrics.py.
+- - Exit Development mode → Edit Settings → Allow public access.
+- - Copy public_slug_id and paste into website/performance.yml.
+- - Run .test-infra/tools/refresh_looker_metrics.py script or manually download as PNG via the public slug and upload to GCS: gs://public_looker_explores_us_a3853f40/FOLDER_ID/<look_slug>.png
+
+9. Open a PR
Review Comment:
maybe share a PR example here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add readme How to add a new ML benchmark pipeline [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #36437:
URL: https://github.com/apache/beam/pull/36437#issuecomment-3381590902
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add readme How to add a new ML benchmark pipeline [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #36437:
URL: https://github.com/apache/beam/pull/36437#issuecomment-3381268801
## Summary of Changes
Hello @Amar3tto, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This PR updates the ML inference benchmarks README to include a detailed guide on how to add new ML benchmark pipelines. It also updates existing links and adds descriptions for new PyTorch and VLLM benchmarks.
### Highlights
* **Intent**: This pull request aims to provide comprehensive documentation on how to add new Machine Learning (ML) benchmark pipelines to the project. It clarifies the process for developers looking to contribute new performance tests for ML inference.
* **Changes**: The primary change is an update to the `sdks/python/apache_beam/testing/benchmarks/inference/README.md` file. This update includes:
1. **Updated Links:** Several outdated links related to performance dashboards and benchmark definitions have been corrected and updated.
2. **New Benchmark Descriptions:** Added detailed descriptions for two new ML benchmarks: 'PyTorch Sentiment Analysis DistilBERT base' and 'VLLM Gemma 2b Batch Performance on Tesla T4', including model, accelerator, host details, and links to their implementations.
3. **New Guide: 'How to add a new ML benchmark pipeline':** A detailed, step-by-step guide (9 steps) has been added. This guide covers the entire process from creating pipeline and benchmark implementations, configuring GitHub Actions, testing, verifying metrics in BigQuery, updating the website, to creating Looker content for performance visualization.
* **Reviewer Activity**: No specific reviewer activity has been recorded or provided in the context for this pull request.
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add readme How to add a new ML benchmark pipeline [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on code in PR #36437:
URL: https://github.com/apache/beam/pull/36437#discussion_r2414546765
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
+- Ensure the expected 'pcollection' parameter is passed to your builder. This parameter could be obtained from GCP Dataflow Jobs -> Your Job Page.
+- Keep naming consistent with other benchmarks.
+
+3. Add an options txt file
+
+- Location: .github/workflows/load-tests-pipeline-options/<pipeline_name>.txt
+- Include Dataflow and pipeline flags. Example:
+
+```
+--region=us-central1
+--machine_type=n1-standard-2
+--num_workers=75
+--disk_size_gb=50
+--autoscaling_algorithm=NONE
+--staging_location=gs://temp-storage-for-perf-tests/loadtests
+--temp_location=gs://temp-storage-for-perf-tests/loadtests
+--requirements_file=apache_beam/ml/inference/your-requirements-file.txt
Review Comment:
No, it is correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add readme How to add a new ML benchmark pipeline [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on code in PR #36437:
URL: https://github.com/apache/beam/pull/36437#discussion_r2414556561
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
+- Ensure the expected 'pcollection' parameter is passed to your builder. This parameter could be obtained from GCP Dataflow Jobs -> Your Job Page.
+- Keep naming consistent with other benchmarks.
+
+3. Add an options txt file
+
+- Location: .github/workflows/load-tests-pipeline-options/<pipeline_name>.txt
+- Include Dataflow and pipeline flags. Example:
+
+```
+--region=us-central1
+--machine_type=n1-standard-2
+--num_workers=75
+--disk_size_gb=50
+--autoscaling_algorithm=NONE
+--staging_location=gs://temp-storage-for-perf-tests/loadtests
+--temp_location=gs://temp-storage-for-perf-tests/loadtests
+--requirements_file=apache_beam/ml/inference/your-requirements-file.txt
+--publish_to_big_query=true
+--metrics_dataset=beam_run_inference
+--metrics_table=your_table
+--input_options={}
+--influx_measurement=your-measurement
+--device=CPU
+--runner=DataflowRunner
+```
+
+4. Wire it into the GitHub Action
+
+- Workflow: .github/workflows/beam_Inference_Python_Benchmarks_Dataflow.yml
+- Add your argument-file-path to the matrix.
+- Add a step that runs your <pipeline_name>_benchmarks.py with -PloadTest.args=$YOUR_ARGUMENTS.
+
+5. Test on your fork
+
+- Trigger the workflow manually.
+- Confirm the Dataflow job completes successfully.
+
+6. Verify metrics in BigQuery
+
+- Dataset: beam_run_inference. Table: your_table
+- Confirm new rows for your pipeline_name with recent timestamps.
+
+7. Update the website
+
+- Create: website/www/site/content/en/performance/<pipeline_name>/_index.md (short title/description).
+- Update: website/www/site/data/performance.yaml — add your pipeline and five chart entries with:
+- - looker_folder_id
+- - public_slug_id (from Looker, see below)
+
+8. Create Looker content (5 charts)
+
+- In Looker → Shared folders → run_inference: create a subfolder for your pipeline.
+- From an existing chart: Development mode → Explore from here → Go to LookML.
+- Point to your table/view and create 5 standard charts (latency/throughput/cost/etc.).
+- Save changes → Publish to production.
+- From Explore, open each, set fields/filters for your pipeline, Run, then Save as Look (in your folder).
+- Open each Look:
+- - Copy Look ID
+- - Add Look IDs to .test-infra/tools/refresh_looker_metrics.py.
+- - Exit Development mode → Edit Settings → Allow public access.
+- - Copy public_slug_id and paste into website/performance.yml.
+- - Run .test-infra/tools/refresh_looker_metrics.py script or manually download as PNG via the public slug and upload to GCS: gs://public_looker_explores_us_a3853f40/FOLDER_ID/<look_slug>.png
+
+9. Open a PR
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add readme How to add a new ML benchmark pipeline [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #36437:
URL: https://github.com/apache/beam/pull/36437#discussion_r2413899446
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
+- Ensure the expected 'pcollection' parameter is passed to your builder. This parameter could be obtained from GCP Dataflow Jobs -> Your Job Page.
+- Keep naming consistent with other benchmarks.
+
+3. Add an options txt file
+
+- Location: .github/workflows/load-tests-pipeline-options/<pipeline_name>.txt
+- Include Dataflow and pipeline flags. Example:
+
+```
+--region=us-central1
+--machine_type=n1-standard-2
+--num_workers=75
+--disk_size_gb=50
+--autoscaling_algorithm=NONE
+--staging_location=gs://temp-storage-for-perf-tests/loadtests
+--temp_location=gs://temp-storage-for-perf-tests/loadtests
+--requirements_file=apache_beam/ml/inference/your-requirements-file.txt
Review Comment:
The example path for `--requirements_file` appears to be missing the `sdks/python/` prefix. Assuming the benchmark is run from the repository root, this path would be incorrect and cause the test to fail. Adding the full path from the repository root will prevent potential confusion and errors.
```suggestion
--requirements_file=sdks/python/apache_beam/ml/inference/your-requirements-file.txt
```
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
Review Comment:
There's a minor typo with an extra space after `DataflowCostBenchmark`. Removing it will improve the document's polish.
```suggestion
- Inherit from DataflowCostBenchmark class.
```
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
+- Ensure the expected 'pcollection' parameter is passed to your builder. This parameter could be obtained from GCP Dataflow Jobs -> Your Job Page.
+- Keep naming consistent with other benchmarks.
+
+3. Add an options txt file
+
+- Location: .github/workflows/load-tests-pipeline-options/<pipeline_name>.txt
+- Include Dataflow and pipeline flags. Example:
+
+```
+--region=us-central1
+--machine_type=n1-standard-2
+--num_workers=75
+--disk_size_gb=50
+--autoscaling_algorithm=NONE
+--staging_location=gs://temp-storage-for-perf-tests/loadtests
+--temp_location=gs://temp-storage-for-perf-tests/loadtests
+--requirements_file=apache_beam/ml/inference/your-requirements-file.txt
+--publish_to_big_query=true
+--metrics_dataset=beam_run_inference
+--metrics_table=your_table
+--input_options={}
+--influx_measurement=your-measurement
+--device=CPU
+--runner=DataflowRunner
+```
+
+4. Wire it into the GitHub Action
+
+- Workflow: .github/workflows/beam_Inference_Python_Benchmarks_Dataflow.yml
+- Add your argument-file-path to the matrix.
+- Add a step that runs your <pipeline_name>_benchmarks.py with -PloadTest.args=$YOUR_ARGUMENTS.
+
+5. Test on your fork
+
+- Trigger the workflow manually.
+- Confirm the Dataflow job completes successfully.
+
+6. Verify metrics in BigQuery
+
+- Dataset: beam_run_inference. Table: your_table
+- Confirm new rows for your pipeline_name with recent timestamps.
+
+7. Update the website
+
+- Create: website/www/site/content/en/performance/<pipeline_name>/_index.md (short title/description).
+- Update: website/www/site/data/performance.yaml — add your pipeline and five chart entries with:
+- - looker_folder_id
+- - public_slug_id (from Looker, see below)
+
+8. Create Looker content (5 charts)
+
+- In Looker → Shared folders → run_inference: create a subfolder for your pipeline.
+- From an existing chart: Development mode → Explore from here → Go to LookML.
+- Point to your table/view and create 5 standard charts (latency/throughput/cost/etc.).
+- Save changes → Publish to production.
+- From Explore, open each, set fields/filters for your pipeline, Run, then Save as Look (in your folder).
+- Open each Look:
+- - Copy Look ID
+- - Add Look IDs to .test-infra/tools/refresh_looker_metrics.py.
+- - Exit Development mode → Edit Settings → Allow public access.
+- - Copy public_slug_id and paste into website/performance.yml.
+- - Run .test-infra/tools/refresh_looker_metrics.py script or manually download as PNG via the public slug and upload to GCS: gs://public_looker_explores_us_a3853f40/FOLDER_ID/<look_slug>.png
Review Comment:
The markdown for this nested list should use indentation for better rendering. Additionally, the last instruction is a bit dense. It can be clarified by separating the automated and manual steps and explaining the `FOLDER_ID` placeholder.
This suggestion improves both formatting and clarity.
```suggestion
- Copy Look ID
- Add Look IDs to .test-infra/tools/refresh_looker_metrics.py.
- Exit Development mode → Edit Settings → Allow public access.
- Copy public_slug_id and paste into website/performance.yml.
- Run `.test-infra/tools/refresh_looker_metrics.py` script. As a manual alternative, download as PNG via the public slug and upload to GCS: `gs://public_looker_explores_us_a3853f40/FOLDER_ID/<look_slug>.png` (where `FOLDER_ID` is from step 7).
```
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
+- Ensure the expected 'pcollection' parameter is passed to your builder. This parameter could be obtained from GCP Dataflow Jobs -> Your Job Page.
+- Keep naming consistent with other benchmarks.
+
+3. Add an options txt file
+
+- Location: .github/workflows/load-tests-pipeline-options/<pipeline_name>.txt
+- Include Dataflow and pipeline flags. Example:
+
+```
+--region=us-central1
+--machine_type=n1-standard-2
+--num_workers=75
+--disk_size_gb=50
+--autoscaling_algorithm=NONE
+--staging_location=gs://temp-storage-for-perf-tests/loadtests
+--temp_location=gs://temp-storage-for-perf-tests/loadtests
+--requirements_file=apache_beam/ml/inference/your-requirements-file.txt
+--publish_to_big_query=true
+--metrics_dataset=beam_run_inference
+--metrics_table=your_table
+--input_options={}
+--influx_measurement=your-measurement
+--device=CPU
+--runner=DataflowRunner
+```
+
+4. Wire it into the GitHub Action
+
+- Workflow: .github/workflows/beam_Inference_Python_Benchmarks_Dataflow.yml
+- Add your argument-file-path to the matrix.
+- Add a step that runs your <pipeline_name>_benchmarks.py with -PloadTest.args=$YOUR_ARGUMENTS.
Review Comment:
The placeholder `$YOUR_ARGUMENTS` is ambiguous. It would be helpful to clarify that this should be the content of the options file created in step 3. You might want to suggest a way to pass the file content, for example, using `$(cat <file_path>)` in a shell environment.
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
+- Ensure the expected 'pcollection' parameter is passed to your builder. This parameter could be obtained from GCP Dataflow Jobs -> Your Job Page.
+- Keep naming consistent with other benchmarks.
+
+3. Add an options txt file
+
+- Location: .github/workflows/load-tests-pipeline-options/<pipeline_name>.txt
+- Include Dataflow and pipeline flags. Example:
+
+```
+--region=us-central1
+--machine_type=n1-standard-2
+--num_workers=75
+--disk_size_gb=50
+--autoscaling_algorithm=NONE
+--staging_location=gs://temp-storage-for-perf-tests/loadtests
+--temp_location=gs://temp-storage-for-perf-tests/loadtests
+--requirements_file=apache_beam/ml/inference/your-requirements-file.txt
+--publish_to_big_query=true
+--metrics_dataset=beam_run_inference
+--metrics_table=your_table
+--input_options={}
+--influx_measurement=your-measurement
+--device=CPU
+--runner=DataflowRunner
+```
+
+4. Wire it into the GitHub Action
+
+- Workflow: .github/workflows/beam_Inference_Python_Benchmarks_Dataflow.yml
+- Add your argument-file-path to the matrix.
+- Add a step that runs your <pipeline_name>_benchmarks.py with -PloadTest.args=$YOUR_ARGUMENTS.
+
+5. Test on your fork
+
+- Trigger the workflow manually.
+- Confirm the Dataflow job completes successfully.
+
+6. Verify metrics in BigQuery
+
+- Dataset: beam_run_inference. Table: your_table
+- Confirm new rows for your pipeline_name with recent timestamps.
+
+7. Update the website
+
+- Create: website/www/site/content/en/performance/<pipeline_name>/_index.md (short title/description).
+- Update: website/www/site/data/performance.yaml — add your pipeline and five chart entries with:
+- - looker_folder_id
+- - public_slug_id (from Looker, see below)
Review Comment:
The markdown for this nested list uses `- -`, which may not render correctly in all markdown viewers. Using indentation for nested lists is more standard and ensures consistent rendering.
```suggestion
- looker_folder_id
- public_slug_id (from Looker, see below)
```
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
+- Ensure the expected 'pcollection' parameter is passed to your builder. This parameter could be obtained from GCP Dataflow Jobs -> Your Job Page.
Review Comment:
The terms 'builder' and the method for obtaining the 'pcollection' parameter are a bit vague. Clarifying these will help contributors understand what's needed more easily.
The 'builder' seems to be the `DataflowCostBenchmark` constructor, and the `pcollection` is the name of a `PCollection` in the pipeline graph used for metrics.
```suggestion
- Ensure the 'pcollection' parameter is passed to the `DataflowCostBenchmark` constructor. This is the name of the PCollection for which to measure throughput, and you can find this name in the Dataflow UI job graph.
```
##########
sdks/python/apache_beam/testing/benchmarks/inference/README.md:
##########
@@ -100,4 +102,95 @@ Approximate size of the models used in the tests
* bert-base-uncased: 417.7 MB
* bert-large-uncased: 1.2 GB
-All the performance tests are defined at [job_InferenceBenchmarkTests_Python.groovy](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy).
+## PyTorch Sentiment Analysis DistilBERT base
+
+**Model**: PyTorch Sentiment Analysis — DistilBERT (base-uncased)
+**Accelerator**: CPU only
+**Host**: 20 × n1-standard-2 (2 vCPUs, 7.5 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_sentiment_streaming.py).
+
+## VLLM Gemma 2b Batch Performance on Tesla T4
+
+**Model**: google/gemma-2b-it
+**Accelerator**: NVIDIA Tesla T4 GPU
+**Host**: 3 × n1-standard-8 (8 vCPUs, 30 GB RAM)
+
+Full pipeline implementation is available [here](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/vllm_gemma_batch.py).
+
+## How to add a new ML benchmark pipeline
+
+1. Create the pipeline implementation
+
+- Location: sdks/python/apache_beam/examples/inference (e.g., pytorch_sentiment.py)
+- Define CLI args and the logic
+- Keep parameter names consistent (e.g., --bq_project, --bq_dataset, --metrics_table).
+
+2. Create the benchmark implementation
+
+- Location: sdks/python/apache_beam/testing/benchmarks/inference (e.g., pytorch_sentiment_benchmarks.py)
+- Inherit from DataflowCostBenchmark class.
+- Ensure the expected 'pcollection' parameter is passed to your builder. This parameter could be obtained from GCP Dataflow Jobs -> Your Job Page.
+- Keep naming consistent with other benchmarks.
+
+3. Add an options txt file
+
+- Location: .github/workflows/load-tests-pipeline-options/<pipeline_name>.txt
+- Include Dataflow and pipeline flags. Example:
+
+```
+--region=us-central1
+--machine_type=n1-standard-2
+--num_workers=75
+--disk_size_gb=50
+--autoscaling_algorithm=NONE
+--staging_location=gs://temp-storage-for-perf-tests/loadtests
+--temp_location=gs://temp-storage-for-perf-tests/loadtests
+--requirements_file=apache_beam/ml/inference/your-requirements-file.txt
+--publish_to_big_query=true
+--metrics_dataset=beam_run_inference
+--metrics_table=your_table
+--input_options={}
Review Comment:
The `--input_options={}` flag is included in the example, but its purpose and expected values are not explained. Please add a brief description to clarify what this option is used for and provide an example of a valid value if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add readme How to add a new ML benchmark pipeline [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto merged PR #36437:
URL: https://github.com/apache/beam/pull/36437
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add readme How to add a new ML benchmark pipeline [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #36437:
URL: https://github.com/apache/beam/pull/36437#issuecomment-3381318706
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/36437?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 36.21%. Comparing base ([`c8df4da`](https://app.codecov.io/gh/apache/beam/commit/c8df4da229da49d533491857e1bb4ab5dbf4fd37?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f973a4e`](https://app.codecov.io/gh/apache/beam/commit/f973a4ed6c2dc82c739def11568e503abb931a58?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 2 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #36437 +/- ##
=============================================
- Coverage 55.03% 36.21% -18.82%
Complexity 1666 1666
=============================================
Files 1058 1058
Lines 165254 165291 +37
Branches 1190 1190
=============================================
- Hits 90941 59866 -31075
- Misses 72138 103250 +31112
Partials 2175 2175
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/36437/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/apache/beam/pull/36437/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.53% <ø> (-40.41%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/36437?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org