You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/10/10 11:41:05 UTC
[GitHub] [beam] shub-kris opened a new pull request, #23554: Benchmarking RunInference Example
shub-kris opened a new pull request, #23554:
URL: https://github.com/apache/beam/pull/23554
This PR aims to illustrates an example of printing and explaining different RunInference metrics using a pipeline that reads text and does classification from a transformer based model.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23554:
URL: https://github.com/apache/beam/pull/23554#discussion_r991276197
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
Review Comment:
```suggestion
We showcase different RunInference metrics when the pipeline is executed using the Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
```
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
Review Comment:
```suggestion
The main purpose of the example is to demonstrate and explain different metrics that are available when using the [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) transform to perform inference using a machine learning model. We use a pipeline that reads a list of sentences, tokenizes the text, and uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifying the texts into two different classes using `RunInference`.
```
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
+
+
+1. Dataflow with CPU: `python main.py --mode cloud --device CPU`
+2. Dataflow with GPU: `python main.py --mode cloud --device GPU`
+
+The pipeline can be broken down into few simple steps:
+1. Create a list of texts to use it as an input using `beam.Create`
Review Comment:
```suggestion
1. Create a list of texts to use as an input using `beam.Create`
```
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
Review Comment:
```suggestion
First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. To use GPUs, follow the setup instructions here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
You can then run your pipeline with the following commands:
```
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
+
+
+1. Dataflow with CPU: `python main.py --mode cloud --device CPU`
+2. Dataflow with GPU: `python main.py --mode cloud --device GPU`
+
+The pipeline can be broken down into few simple steps:
+1. Create a list of texts to use it as an input using `beam.Create`
+2. Tokenizing the text
+3. Using RunInference to do inference
+4. Postprocessing the output of RunInference
+
+The code snippet for the pipeline is:
+
+{{< highlight >}}
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ _ = (
+ pipeline
+ | "Create inputs" >> beam.Create(inputs)
+ | "Tokenize" >> beam.ParDo(Tokenize(cfg.TOKENIZER_NAME))
+ | "Inference" >>
+ RunInference(model_handler=KeyedModelHandler(model_handler))
+ | "Decode Predictions" >> beam.ParDo(PostProcessor()))
+{{< /highlight >}}
+
+
+## RunInference Metrics
+
+As mentioned above, we benchmarked the performance of RunInference using Dataflow on both CPU and GPU. These metrics can be seen in the GCP UI and can also be printed using
+{{< highlight >}}
+metrics = pipeline.result.metrics().query(beam.metrics.MetricsFilter())
+{{< /highlight >}}
+
+
+A snapshot of different metrics from GCP UI when using Dataflow on GPU:
+
+ 
+
+Some metrics commonly used for benchmarking are:
+
+* `num_inferences`: represents the total number of elements passed to `run_inference()`.
+
+* `inference_batch_latency_micro_secs_MEAN`: represents the average time taken to perform the inference across all batches of examples, measured in microseconds.
+
+* `inference_request_batch_size_COUNT`: represents the total number of samples across all batches of examples (created from `beam.BatchElements`) to be passed to run_inference()
+
+* `inference_request_batch_byte_size_MEAN`: represents the average size of all elements for all samples in all batches of examples (created from `beam.BatchElements`) to be passed to run_inference(). It is measured in bytes.
+
+* `model_byte_size_MEAN`: It represents the average memory consumed to load and initialize the model. It is measured in bytes.
+
+* `load_model_latency_milli_secs_MEAN`: represents the average time taken to load and initialize the model. It is measured in milliseconds.
Review Comment:
```suggestion
* `inference_request_batch_byte_size_MEAN`: represents the average size of all elements for all samples in all batches of examples (created from `beam.BatchElements`) to be passed to run_inference(). This is measured in bytes.
* `model_byte_size_MEAN`: Irepresents the average memory consumed to load and initialize the model. This is measured in bytes.
* `load_model_latency_milli_secs_MEAN`: represents the average time taken to load and initialize the model. This is measured in milliseconds.
```
##########
sdks/python/apache_beam/examples/inference/runinference_metrics/pipeline/transformations.py:
##########
@@ -0,0 +1,93 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""This file contains the transformations and utility functions for
+the pipeline."""
+import apache_beam as beam
+import torch
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerKeyedTensor
+from transformers import DistilBertForSequenceClassification
+from transformers import DistilBertTokenizer
+
+
+class CustomPytorchModelHandlerKeyedTensor(PytorchModelHandlerKeyedTensor):
+ """Wrapper around PytorchModelHandlerKeyedTensor to load a model on CPU."""
+ def load_model(self) -> torch.nn.Module:
+ """Loads and initializes a Pytorch model for processing."""
+ model = self._model_class(**self._model_params)
+ model.to(self._device)
+ file = FileSystems.open(self._state_dict_path, "rb")
+ model.load_state_dict(torch.load(file, map_location=self._device))
+ model.eval()
+ return model
+
+
+class HuggingFaceStripBatchingWrapper(DistilBertForSequenceClassification):
+ """Wrapper around HuggingFace model because RunInference requires a batch
+ as a list of dicts instead of a dict of lists. Another workaround
Review Comment:
Could you please create an issue to support this and then link to the issue in the code here? This is something we should probably be able to just infer/handle automatically
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
Review Comment:
```suggestion
`pipeline/transformations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
```
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
+
+
+1. Dataflow with CPU: `python main.py --mode cloud --device CPU`
+2. Dataflow with GPU: `python main.py --mode cloud --device GPU`
+
+The pipeline can be broken down into few simple steps:
+1. Create a list of texts to use it as an input using `beam.Create`
+2. Tokenizing the text
+3. Using RunInference to do inference
+4. Postprocessing the output of RunInference
Review Comment:
```suggestion
2. Tokenize the text
3. Use RunInference to do inference
4. Postprocess the output of RunInference
```
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
+
+
+1. Dataflow with CPU: `python main.py --mode cloud --device CPU`
+2. Dataflow with GPU: `python main.py --mode cloud --device GPU`
+
+The pipeline can be broken down into few simple steps:
+1. Create a list of texts to use it as an input using `beam.Create`
+2. Tokenizing the text
+3. Using RunInference to do inference
+4. Postprocessing the output of RunInference
+
+The code snippet for the pipeline is:
+
+{{< highlight >}}
Review Comment:
```suggestion
{{< highlight >}}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] shub-kris commented on a diff in pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
shub-kris commented on code in PR #23554:
URL: https://github.com/apache/beam/pull/23554#discussion_r991875043
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
+
+
+1. Dataflow with CPU: `python main.py --mode cloud --device CPU`
+2. Dataflow with GPU: `python main.py --mode cloud --device GPU`
+
+The pipeline can be broken down into few simple steps:
+1. Create a list of texts to use it as an input using `beam.Create`
+2. Tokenizing the text
+3. Using RunInference to do inference
+4. Postprocessing the output of RunInference
+
+The code snippet for the pipeline is:
+
+{{< highlight >}}
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ _ = (
+ pipeline
+ | "Create inputs" >> beam.Create(inputs)
+ | "Tokenize" >> beam.ParDo(Tokenize(cfg.TOKENIZER_NAME))
+ | "Inference" >>
+ RunInference(model_handler=KeyedModelHandler(model_handler))
+ | "Decode Predictions" >> beam.ParDo(PostProcessor()))
+{{< /highlight >}}
+
+
+## RunInference Metrics
+
+As mentioned above, we benchmarked the performance of RunInference using Dataflow on both CPU and GPU. These metrics can be seen in the GCP UI and can also be printed using
+{{< highlight >}}
+metrics = pipeline.result.metrics().query(beam.metrics.MetricsFilter())
+{{< /highlight >}}
+
+
+A snapshot of different metrics from GCP UI when using Dataflow on GPU:
+
+ 
Review Comment:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] shub-kris commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
shub-kris commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1288452706
@damccorm ready to merge as all tests passed except `codecov`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] shub-kris commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
shub-kris commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1276158962
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] shub-kris commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
shub-kris commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1273189134
@damccorm @andyxiexu please have a look and let me know what changes would you like to have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1273215855
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @y1chi for label python.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23554:
URL: https://github.com/apache/beam/pull/23554#discussion_r1000617549
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,102 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using the [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) transform to perform inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinzes the text, and uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifying the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using the Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transformations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP `PROJECT_ID`, `NUM_WORKERS` that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. To use GPUs, follow the setup instructions [here](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal).
+
+
+1. Dataflow with CPU: `python main.py --mode cloud --device CPU`
+2. Dataflow with GPU: `python main.py --mode cloud --device GPU`
+
+The pipeline can be broken down into few simple steps:
+1. Create a list of texts to use as an input using `beam.Create`
+2. Tokenize the text
+3. Use RunInference to do inference
+4. Postprocess the output of RunInference
+
+{{< highlight >}}
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ _ = (
+ pipeline
+ | "Create inputs" >> beam.Create(inputs)
+ | "Tokenize" >> beam.ParDo(Tokenize(cfg.TOKENIZER_NAME))
+ | "Inference" >>
+ RunInference(model_handler=KeyedModelHandler(model_handler))
+ | "Decode Predictions" >> beam.ParDo(PostProcessor()))
+{{< /highlight >}}
+
+
+## RunInference Metrics
+
+As mentioned above, we benchmarked the performance of RunInference using Dataflow on both CPU and GPU. These metrics can be seen in the GCP UI and can also be printed using
+
+{{< highlight >}}
+metrics = pipeline.result.metrics().query(beam.metrics.MetricsFilter())
+{{< /highlight >}}
+
+
+A snapshot of different metrics from GCP UI when using Dataflow on GPU:
+
+ 
Review Comment:
```suggestion
```
Right now, this doesn't render correctly - http://apache-beam-website-pull-requests.storage.googleapis.com/23554/documentation/ml/runinference-metrics/index.html#runinference-metrics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] shub-kris commented on a diff in pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
shub-kris commented on code in PR #23554:
URL: https://github.com/apache/beam/pull/23554#discussion_r991874721
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
+
+
+1. Dataflow with CPU: `python main.py --mode cloud --device CPU`
+2. Dataflow with GPU: `python main.py --mode cloud --device GPU`
+
+The pipeline can be broken down into few simple steps:
+1. Create a list of texts to use it as an input using `beam.Create`
+2. Tokenizing the text
+3. Using RunInference to do inference
+4. Postprocessing the output of RunInference
+
+The code snippet for the pipeline is:
+
+{{< highlight >}}
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ _ = (
+ pipeline
+ | "Create inputs" >> beam.Create(inputs)
+ | "Tokenize" >> beam.ParDo(Tokenize(cfg.TOKENIZER_NAME))
+ | "Inference" >>
+ RunInference(model_handler=KeyedModelHandler(model_handler))
+ | "Decode Predictions" >> beam.ParDo(PostProcessor()))
+{{< /highlight >}}
+
+
+## RunInference Metrics
+
+As mentioned above, we benchmarked the performance of RunInference using Dataflow on both CPU and GPU. These metrics can be seen in the GCP UI and can also be printed using
+{{< highlight >}}
+metrics = pipeline.result.metrics().query(beam.metrics.MetricsFilter())
+{{< /highlight >}}
+
+
+A snapshot of different metrics from GCP UI when using Dataflow on GPU:
+
+ 
Review Comment:
It's a snapshot from the GCP UI. Right now it is hosted on drive and shared to everyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yeandy commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
yeandy commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1278917635
That check is flaky most of the time. It should be fine to ignore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1277968585
Looking at other jobs, I think the precommit failures are flakiness. I'm going to try again, but also file an issue. This has even happened on our cron runs for master - https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] shub-kris commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
shub-kris commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1278527704
@damccorm now it only fails `codecov/patch`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1277814102
Run Website PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1273229131
# [Codecov](https://codecov.io/gh/apache/beam/pull/23554?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#23554](https://codecov.io/gh/apache/beam/pull/23554?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b8fcee7) into [master](https://codecov.io/gh/apache/beam/commit/b94cff209cc8d1ae61cc916ff6b0b68561dc34c8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b94cff2) will **decrease** coverage by `0.08%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #23554 +/- ##
==========================================
- Coverage 73.46% 73.37% -0.09%
==========================================
Files 718 723 +5
Lines 95884 95980 +96
==========================================
- Hits 70438 70423 -15
- Misses 24135 24246 +111
Partials 1311 1311
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.08% <0.00%> (-0.15%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/23554?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../examples/inference/runinference\_metrics/config.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3J1bmluZmVyZW5jZV9tZXRyaWNzL2NvbmZpZy5weQ==) | `0.00% <0.00%> (ø)` | |
| [...am/examples/inference/runinference\_metrics/main.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3J1bmluZmVyZW5jZV9tZXRyaWNzL21haW4ucHk=) | `0.00% <0.00%> (ø)` | |
| [...inference/runinference\_metrics/pipeline/options.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3J1bmluZmVyZW5jZV9tZXRyaWNzL3BpcGVsaW5lL29wdGlvbnMucHk=) | `0.00% <0.00%> (ø)` | |
| [...e/runinference\_metrics/pipeline/transformations.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3J1bmluZmVyZW5jZV9tZXRyaWNzL3BpcGVsaW5lL3RyYW5zZm9ybWF0aW9ucy5weQ==) | `0.00% <0.00%> (ø)` | |
| [...m/examples/inference/runinference\_metrics/setup.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3J1bmluZmVyZW5jZV9tZXRyaWNzL3NldHVwLnB5) | `0.00% <0.00%> (ø)` | |
| [.../python/apache\_beam/testing/test\_stream\_service.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy90ZXN0X3N0cmVhbV9zZXJ2aWNlLnB5) | `88.09% <0.00%> (-4.77%)` | :arrow_down: |
| [sdks/python/apache\_beam/utils/interactive\_utils.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdXRpbHMvaW50ZXJhY3RpdmVfdXRpbHMucHk=) | `95.12% <0.00%> (-2.44%)` | :arrow_down: |
| [...python/apache\_beam/runners/worker/worker\_status.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvd29ya2VyX3N0YXR1cy5weQ==) | `75.33% <0.00%> (-1.34%)` | :arrow_down: |
| [...che\_beam/runners/interactive/interactive\_runner.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9pbnRlcmFjdGl2ZS9pbnRlcmFjdGl2ZV9ydW5uZXIucHk=) | `90.50% <0.00%> (-1.27%)` | :arrow_down: |
| [...eam/runners/portability/fn\_api\_runner/execution.py](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9wb3J0YWJpbGl0eS9mbl9hcGlfcnVubmVyL2V4ZWN1dGlvbi5weQ==) | `92.44% <0.00%> (-0.65%)` | :arrow_down: |
| ... and [3 more](https://codecov.io/gh/apache/beam/pull/23554/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm merged pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
damccorm merged PR #23554:
URL: https://github.com/apache/beam/pull/23554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yeandy commented on a diff in pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
yeandy commented on code in PR #23554:
URL: https://github.com/apache/beam/pull/23554#discussion_r991410149
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
Review Comment:
```suggestion
### How to Run the Pipeline
```
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
Review Comment:
```suggestion
`config.py` defines some variables like GCP `PROJECT_ID`, `NUM_WORKERS` that are used multiple times
```
##########
sdks/python/apache_beam/examples/inference/runinference_metrics/setup.py:
##########
@@ -0,0 +1,41 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Setup.py module for the workflow's worker utilities.
+All the workflow related code is gathered in a package that will be built as a
+source distribution, staged in the staging area for the workflow being run and
+then installed in the workers when they start running.
+This behavior is triggered by specifying the --setup_file command line option
+when running the workflow for remote execution.
+"""
Review Comment:
```suggestion
"""Setup.py module for the workflow's worker utilities.
All the workflow related code is gathered in a package that will be built as a
source distribution, staged in the staging area for the workflow being run, and
then installed in the workers when they start running.
This behavior is triggered by specifying the --setup_file command line option
when running the workflow for remote execution.
"""
```
##########
sdks/python/apache_beam/examples/inference/runinference_metrics/pipeline/transformations.py:
##########
@@ -0,0 +1,93 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""This file contains the transformations and utility functions for
+the pipeline."""
+import apache_beam as beam
+import torch
+from apache_beam.io.filesystems import FileSystems
+from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerKeyedTensor
+from transformers import DistilBertForSequenceClassification
+from transformers import DistilBertTokenizer
+
+
+class CustomPytorchModelHandlerKeyedTensor(PytorchModelHandlerKeyedTensor):
+ """Wrapper around PytorchModelHandlerKeyedTensor to load a model on CPU."""
+ def load_model(self) -> torch.nn.Module:
+ """Loads and initializes a Pytorch model for processing."""
+ model = self._model_class(**self._model_params)
+ model.to(self._device)
+ file = FileSystems.open(self._state_dict_path, "rb")
+ model.load_state_dict(torch.load(file, map_location=self._device))
+ model.eval()
+ return model
+
+
+class HuggingFaceStripBatchingWrapper(DistilBertForSequenceClassification):
+ """Wrapper around HuggingFace model because RunInference requires a batch
+ as a list of dicts instead of a dict of lists. Another workaround
+ can be found here where they disable batching instead.
+ https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/inference/pytorch_language_modeling.py"""
+ def forward(self, **kwargs):
+ output = super().forward(**kwargs)
+ return [dict(zip(output, v)) for v in zip(*output.values())]
+
+
+class Tokenize(beam.DoFn):
+ """A DoFn for tokenizing texts"""
+ def __init__(self, model_name: str):
+ """Initialises a tokenizer based on the model_name"""
+ self._model_name = model_name
+
+ def setup(self):
+ """Loads the tokenizer"""
+ self._tokenizer = DistilBertTokenizer.from_pretrained(self._model_name)
+
+ def process(self, text_input: str):
+ """Prepocesses the text using the tokenizer"""
+ # We need to pad the tokens tensors to max length to make sure
+ # that all the tensors are of the same length and hence
+ # stack-able by the RunInference API, normally you would batch first
+ # and tokenize the batch after and pad each tensor
+ # the the max length in the batch.
+ tokens = self._tokenizer(
+ text_input, return_tensors="pt", padding="max_length", max_length=512)
+ # squeeze because tokenization add an extra dimension, which is empty
+ # in this case because we're tokenizing one element at a time.
+ tokens = {key: torch.squeeze(val) for key, val in tokens.items()}
+ return [(text_input, tokens)]
+
+
+class PostProcessor(beam.DoFn):
+ """Postprocess the RunInference output"""
+ def process(self, tuple_):
Review Comment:
What does `tuple_` have the underscore `_` as suffix here?
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
+
+
+1. Dataflow with CPU: `python main.py --mode cloud --device CPU`
+2. Dataflow with GPU: `python main.py --mode cloud --device GPU`
+
+The pipeline can be broken down into few simple steps:
+1. Create a list of texts to use it as an input using `beam.Create`
+2. Tokenizing the text
+3. Using RunInference to do inference
+4. Postprocessing the output of RunInference
+
+The code snippet for the pipeline is:
+
+{{< highlight >}}
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ _ = (
+ pipeline
+ | "Create inputs" >> beam.Create(inputs)
+ | "Tokenize" >> beam.ParDo(Tokenize(cfg.TOKENIZER_NAME))
+ | "Inference" >>
+ RunInference(model_handler=KeyedModelHandler(model_handler))
+ | "Decode Predictions" >> beam.ParDo(PostProcessor()))
+{{< /highlight >}}
+
+
+## RunInference Metrics
+
+As mentioned above, we benchmarked the performance of RunInference using Dataflow on both CPU and GPU. These metrics can be seen in the GCP UI and can also be printed using
+{{< highlight >}}
+metrics = pipeline.result.metrics().query(beam.metrics.MetricsFilter())
+{{< /highlight >}}
+
+
+A snapshot of different metrics from GCP UI when using Dataflow on GPU:
+
+ 
Review Comment:
What does this look like? Should we add the png locally?
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
Review Comment:
```suggestion
First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned [here](https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal).
```
##########
sdks/python/apache_beam/examples/inference/runinference_metrics/pipeline/options.py:
##########
@@ -0,0 +1,75 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""This file contains the pipeline options to configure
+the Dataflow pipeline."""
+
+from datetime import datetime
+from typing import Any
+
+import config as cfg
+from apache_beam.options.pipeline_options import PipelineOptions
+
+
+def get_pipeline_options(
+ project: str,
+ job_name: str,
+ mode: str,
+ device: str,
+ num_workers: int = cfg.NUM_WORKERS,
+ **kwargs: Any,
+) -> PipelineOptions:
+ """Function to retrieve the pipeline options.
+ Args:
+ project: GCP project to run on
+ mode: Indicator to run local, cloud or template
+ num_workers: Number of Workers for running the job parallely
+ Returns:
+ Dataflow pipeline options
+ """
+ job_name = f'{job_name}-{datetime.now().strftime("%Y%m%d%H%M%S")}'
+
+ staging_bucket = f"gs://{cfg.PROJECT_ID}-ml-examples"
+
+ # For a list of available options, check:
+ # https://cloud.google.com/dataflow/docs/guides/specifying-exec-params#setting-other-cloud-dataflow-pipeline-options
+ dataflow_options = {
+ "runner": "DirectRunner" if mode == "local" else "DataflowRunner",
+ "job_name": job_name,
+ "project": project,
+ "region": cfg.REGION,
+ "staging_location": f"{staging_bucket}/dflow-staging",
+ "temp_location": f"{staging_bucket}/dflow-temp",
+ "setup_file": "./setup.py",
+ }
+ flags = []
+ if device == "GPU":
+ flags = [
+ "--experiment=worker_accelerator=type:nvidia-tesla-p4;count:1;"\
+ "install-nvidia-driver",
+ "--experiment=use_runner_v2",
+ ]
+ dataflow_options.update({
+ "sdk_container_image": cfg.DOCKER_IMG,
+ "machine_type": "n1-standard-4",
+ })
+
+ # Optional parameters
+ if num_workers:
+ dataflow_options.update({"num_workers": num_workers})
+ print(dataflow_options)
Review Comment:
Same as above comment on print
##########
sdks/python/apache_beam/examples/inference/runinference_metrics/main.py:
##########
@@ -0,0 +1,127 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""This file contains the pipeline for loading a ML model, and exploring
+the different RunInference metrics."""
+import argparse
+import sys
+
+import apache_beam as beam
+import config as cfg
+from apache_beam.ml.inference import RunInference
+from apache_beam.ml.inference.base import KeyedModelHandler
+from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerKeyedTensor
+from pipeline.options import get_pipeline_options
+from pipeline.transformations import CustomPytorchModelHandlerKeyedTensor
+from pipeline.transformations import HuggingFaceStripBatchingWrapper
+from pipeline.transformations import PostProcessor
+from pipeline.transformations import Tokenize
+from transformers import DistilBertConfig
+
+
+def parse_arguments(argv):
+ """
+ Parses the arguments passed to the command line and
+ returns them as an object
+ Args:
+ argv: The arguments passed to the command line.
+ Returns:
+ The arguments that are being passed in.
+ """
+ parser = argparse.ArgumentParser(description="benchmark-runinference")
+
+ parser.add_argument(
+ "-m",
+ "--mode",
+ help="Mode to run pipeline in.",
+ choices=["local", "cloud"],
+ default="local",
+ )
+ parser.add_argument(
+ "-p",
+ "--project",
+ help="GCP project to run pipeline on.",
+ default=cfg.PROJECT_ID,
+ )
+ parser.add_argument(
+ "-d",
+ "--device",
+ help="Device to run the dataflow job on",
+ choices=["CPU", "GPU"],
+ default="CPU",
+ )
+
+ args, _ = parser.parse_known_args(args=argv)
+ return args
+
+
+def run():
+ """
+ Runs the pipeline that loads a transformer based text classification model
+ and does inference on a list of sentences.
+ At the end of pipeline, different metrics like latency,
+ throughput and others are printed.
+ """
+ args = parse_arguments(sys.argv)
+
+ inputs = [
+ "This is the worst food I have ever eaten",
+ "In my soul and in my heart, I’m convinced I’m wrong!",
+ "Be with me always—take any form—drive me mad!"\
+ "only do not leave me in this abyss, where I cannot find you!",
+ "Do I want to live? Would you like to live with your soul in the grave?",
+ "Honest people don’t hide their deeds.",
+ "Nelly, I am Heathcliff! He’s always,"\
+ "always in my mind: not as a pleasure,"\
+ "any more than I am always a pleasure to myself, but as my own being.",
+ ] * 1000
+
+ pipeline_options = get_pipeline_options(
+ job_name=cfg.JOB_NAME,
+ num_workers=cfg.NUM_WORKERS,
+ project=args.project,
+ mode=args.mode,
+ device=args.device,
+ )
+ model_handler_class = (
+ PytorchModelHandlerKeyedTensor
+ if args.device == "GPU" else CustomPytorchModelHandlerKeyedTensor)
+ device = "cuda:0" if args.device == "GPU" else args.device
+ model_handler = model_handler_class(
+ state_dict_path=cfg.MODEL_STATE_DICT_PATH,
+ model_class=HuggingFaceStripBatchingWrapper,
+ model_params={
+ "config": DistilBertConfig.from_pretrained(cfg.MODEL_CONFIG_PATH)
+ },
+ device=device,
+ )
+
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ _ = (
+ pipeline
+ | "Create inputs" >> beam.Create(inputs)
+ | "Tokenize" >> beam.ParDo(Tokenize(cfg.TOKENIZER_NAME))
+ | "Inference" >>
+ RunInference(model_handler=KeyedModelHandler(model_handler))
+ | "Decode Predictions" >> beam.ParDo(PostProcessor()))
+ metrics = pipeline.result.metrics().query(beam.metrics.MetricsFilter())
+ print("\n\n\n\n")
+ print(metrics)
Review Comment:
Should we be using `logging` instead of `print`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #23554:
URL: https://github.com/apache/beam/pull/23554#issuecomment-1277968741
Run Website PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #23554: Benchmarking RunInference Example
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #23554:
URL: https://github.com/apache/beam/pull/23554#discussion_r995829097
##########
sdks/python/apache_beam/examples/inference/runinference_metrics/main.py:
##########
@@ -0,0 +1,128 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""This file contains the pipeline for loading a ML model, and exploring
+the different RunInference metrics."""
+import argparse
+import logging
+import sys
+
+import apache_beam as beam
+import config as cfg
+from apache_beam.ml.inference import RunInference
+from apache_beam.ml.inference.base import KeyedModelHandler
+from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerKeyedTensor
+from pipeline.options import get_pipeline_options
+from pipeline.transformations import CustomPytorchModelHandlerKeyedTensor
+from pipeline.transformations import HuggingFaceStripBatchingWrapper
+from pipeline.transformations import PostProcessor
+from pipeline.transformations import Tokenize
+from transformers import DistilBertConfig
+
+
+def parse_arguments(argv):
+ """
+ Parses the arguments passed to the command line and
+ returns them as an object
+ Args:
+ argv: The arguments passed to the command line.
+ Returns:
+ The arguments that are being passed in.
+ """
+ parser = argparse.ArgumentParser(description="benchmark-runinference")
+
+ parser.add_argument(
+ "-m",
+ "--mode",
+ help="Mode to run pipeline in.",
+ choices=["local", "cloud"],
+ default="local",
+ )
+ parser.add_argument(
+ "-p",
+ "--project",
+ help="GCP project to run pipeline on.",
+ default=cfg.PROJECT_ID,
+ )
+ parser.add_argument(
+ "-d",
+ "--device",
+ help="Device to run the dataflow job on",
+ choices=["CPU", "GPU"],
+ default="CPU",
+ )
+
+ args, _ = parser.parse_known_args(args=argv)
+ return args
+
+
+def run():
+ """
+ Runs the pipeline that loads a transformer based text classification model
+ and does inference on a list of sentences.
+ At the end of pipeline, different metrics like latency,
+ throughput and others are printed.
+ """
+ args = parse_arguments(sys.argv)
+
+ inputs = [
+ "This is the worst food I have ever eaten",
+ "In my soul and in my heart, I’m convinced I’m wrong!",
+ "Be with me always—take any form—drive me mad!"\
+ "only do not leave me in this abyss, where I cannot find you!",
+ "Do I want to live? Would you like to live with your soul in the grave?",
+ "Honest people don’t hide their deeds.",
+ "Nelly, I am Heathcliff! He’s always,"\
+ "always in my mind: not as a pleasure,"\
+ "any more than I am always a pleasure to myself, but as my own being.",
+ ] * 1000
+
+ pipeline_options = get_pipeline_options(
+ job_name=cfg.JOB_NAME,
+ num_workers=cfg.NUM_WORKERS,
+ project=args.project,
+ mode=args.mode,
+ device=args.device,
+ )
+ model_handler_class = (
+ PytorchModelHandlerKeyedTensor
+ if args.device == "GPU" else CustomPytorchModelHandlerKeyedTensor)
+ device = "cuda:0" if args.device == "GPU" else args.device
+ model_handler = model_handler_class(
+ state_dict_path=cfg.MODEL_STATE_DICT_PATH,
+ model_class=HuggingFaceStripBatchingWrapper,
+ model_params={
+ "config": DistilBertConfig.from_pretrained(cfg.MODEL_CONFIG_PATH)
+ },
+ device=device,
+ )
+
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ _ = (
+ pipeline
+ | "Create inputs" >> beam.Create(inputs)
+ | "Tokenize" >> beam.ParDo(Tokenize(cfg.TOKENIZER_NAME))
+ | "Inference" >>
+ RunInference(model_handler=KeyedModelHandler(model_handler))
+ | "Decode Predictions" >> beam.ParDo(PostProcessor()))
+ metrics = pipeline.result.metrics().query(beam.metrics.MetricsFilter())
+ print("\n\n\n\n")
Review Comment:
```suggestion
```
##########
website/www/site/content/en/documentation/ml/runinference-metrics.md:
##########
@@ -0,0 +1,103 @@
+---
+title: "RunInference Metrics"
+---
+<!--
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# RunInference Metrics Example
+
+The main purpose of the example is to demonstrate and explain different metrics that are available when using [RunInference](https://beam.apache.org/documentation/transforms/python/elementwise/runinference/) for doing inference using a machine learning model. We use a pipeline that reads a list of sentences, tokeinze the text, uses a Transformer based model `distilbert-base-uncased-finetuned-sst-2-english` for classifies the texts into two different classes using `RunInference`.
+
+We showcase different RunInference metrics when the pipeline is executed using Dataflow Runner on CPU and GPU. The full example code can be found [here](https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples/inference/runinference_metrics/).
+
+
+The file structure for entire pipeline is:
+
+ runinference_metrics/
+ ├── pipeline/
+ │ ├── __init__.py
+ │ ├── options.py
+ │ └── transformations.py
+ ├── __init__.py
+ ├── config.py
+ ├── main.py
+ └── setup.py
+
+`pipeline/transormations.py` contains the code for `beam.DoFn` and additional functions that are used for pipeline
+
+`pipeline/options.py` contains the pipeline options to configure the Dataflow pipeline
+
+`config.py` defines some variables like GCP PROJECT_ID, NUM_WORKERS that are used multiple times
+
+`setup.py` defines the packages/requirements for the pipeline to run
+
+`main.py` contains the pipeline code and some additional functions used for running the pipeline
+
+
+### How to Run the Pipeline ?
+First, make sure you have installed the required packages. One should have access to a Google Cloud Project and then correctly configure the GCP variables like `PROJECT_ID`, `REGION`, and others in `config.py`. For using Dataflow with GPU, all the necessary setup instructions are mentioned here: https://github.com/GoogleCloudPlatform/python-docs-samples/tree/main/dataflow/gpu-examples/pytorch-minimal.
+
+
+1. Dataflow with CPU: `python main.py --mode cloud --device CPU`
+2. Dataflow with GPU: `python main.py --mode cloud --device GPU`
+
+The pipeline can be broken down into few simple steps:
+1. Create a list of texts to use it as an input using `beam.Create`
+2. Tokenizing the text
+3. Using RunInference to do inference
+4. Postprocessing the output of RunInference
+
+The code snippet for the pipeline is:
+
+{{< highlight >}}
+ with beam.Pipeline(options=pipeline_options) as pipeline:
+ _ = (
+ pipeline
+ | "Create inputs" >> beam.Create(inputs)
+ | "Tokenize" >> beam.ParDo(Tokenize(cfg.TOKENIZER_NAME))
+ | "Inference" >>
+ RunInference(model_handler=KeyedModelHandler(model_handler))
+ | "Decode Predictions" >> beam.ParDo(PostProcessor()))
+{{< /highlight >}}
+
+
+## RunInference Metrics
+
+As mentioned above, we benchmarked the performance of RunInference using Dataflow on both CPU and GPU. These metrics can be seen in the GCP UI and can also be printed using
+{{< highlight >}}
+metrics = pipeline.result.metrics().query(beam.metrics.MetricsFilter())
+{{< /highlight >}}
+
+
+A snapshot of different metrics from GCP UI when using Dataflow on GPU:
+
+ 
Review Comment:
> Should we add the png locally?
+1, lets check this in to the repo. That's less brittle (and less likely to be an IP issue since its technically not apache licensed in its current location)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org