You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/07 20:07:18 UTC
[GitHub] [beam] AnandInguva opened a new pull request, #21738: Inference benchmark tests
AnandInguva opened a new pull request, #21738:
URL: https://github.com/apache/beam/pull/21738
Benchmark test suite for RunInference benchmark tests. This PR includes only the benchmark test on ImageNet dataset. Others will be added in the future.
#21454
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Add a link to the appropriate issue in your description, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1160828155
> What is the next step on this PR?
Waiting on the licenses on the Datasets used for the performance tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1238219931
PTAL @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
aaltay commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1159352361
What is the next step on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1221556308
Run Python 3.9 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1238054955
I kicked off one here - https://ci-beam.apache.org/job/beam_SeedJob/10239/console
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1222542091
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1188343200
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1188397836
Inference benchmark test works. https://ci-beam.apache.org/job/beam_Inference_Python_Benchmarks_Dataflow_PR/9/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1235664536
Hm, it looks like that is now causing tests to fail though - would you mind taking a look? Other than that this LGTM and should be ready to merge once its resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ryanthompson591 commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
ryanthompson591 commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r904374991
##########
sdks/python/apache_beam/examples/inference/pytorch_image_classification.py:
##########
@@ -72,6 +73,7 @@ def parse_known_args(argv):
"""Parses args for the workflow."""
parser = argparse.ArgumentParser()
parser.add_argument(
+ '--input_file',
Review Comment:
does this mean it accepts either input_file or input as an argument?
##########
sdks/python/apache_beam/examples/inference/pytorch_image_classification.py:
##########
@@ -120,26 +128,34 @@ def run(argv=None, model_class=None, model_params=None, save_main_session=True):
model_class=model_class,
model_params=model_params))
- with beam.Pipeline(options=pipeline_options) as p:
- filename_value_pair = (
- p
- | 'ReadImageNames' >> beam.io.ReadFromText(
- known_args.input, skip_header_lines=1)
- | 'ReadImageData' >> beam.Map(
- lambda image_name: read_image(
- image_file_name=image_name, path_to_dir=known_args.images_dir))
- | 'PreprocessImages' >> beam.MapTuple(
- lambda file_name, data: (file_name, preprocess_image(data))))
- predictions = (
- filename_value_pair
- | 'PyTorchRunInference' >> RunInference(model_handler)
- | 'ProcessOutput' >> beam.ParDo(PostProcessor()))
-
- if known_args.output:
- predictions | "WriteOutputToGCS" >> beam.io.WriteToText( # pylint: disable=expression-not-assigned
- known_args.output,
- shard_name_template='',
- append_trailing_newlines=True)
+ if not test_pipeline:
+ p = beam.Pipeline(options=pipeline_options)
+ else:
+ p = test_pipeline
+
+ filename_value_pair = (
+ p
+ | 'ReadImageNames' >> beam.io.ReadFromText(
+ known_args.input, skip_header_lines=1)
+ | 'ReadImageData' >> beam.Map(
+ lambda image_name: read_image(
+ image_file_name=image_name, path_to_dir=known_args.images_dir))
+ | 'PreprocessImages' >> beam.MapTuple(
+ lambda file_name, data: (file_name, preprocess_image(data))))
+ predictions = (
+ filename_value_pair
+ | 'PyTorchRunInference' >> RunInference(model_handler).with_output_types(
Review Comment:
we shouldn't need to specify output_types anymore right? Since it is handled by the model_handler
##########
sdks/python/apache_beam/examples/inference/pytorch_image_classification.py:
##########
@@ -33,9 +33,10 @@
from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerTensor
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.runners.runner import PipelineResult
from PIL import Image
+from torchvision import models
from torchvision import transforms
-from torchvision.models.mobilenetv2 import MobileNetV2
Review Comment:
This sort of example seems like it should have a good explanation about what it is doing.
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_benchmarks.py:
##########
@@ -0,0 +1,66 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+# TODO: Decide which models we want to use
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
+ self.metrics_namespace = 'RunInferencePytorch'
+ super().__init__(metrics_namespace=self.metrics_namespace)
+
+ def test(self):
+ pretrained_model_name = self.pipeline.get_option('pretrained_model_name')
+ if not pretrained_model_name:
+ raise RuntimeError(
+ 'Please provide a pretrained torch model name.'
+ ' Model name must be from the module torchvision.models')
+ if pretrained_model_name == _PERF_TEST_MODELS[0]:
+ model_class = models.resnet50
+ elif pretrained_model_name == _PERF_TEST_MODELS[1]:
+ model_class = models.resnet101
+ elif pretrained_model_name == _PERF_TEST_MODELS[2]:
+ model_class = models.resnet152
+ else:
+ raise NotImplementedError
+
+ # model_params are same for all the models. But this may change if we add
+ # different models.
+ model_params = {'num_classes': 1000, 'pretrained': False}
+
+ self.result = pytorch_image_classification.run(
+ self.pipeline.get_full_options_as_args(),
+ model_class=model_class,
+ model_params=model_params,
+ test_pipeline=self.pipeline)
+
+ self.result.wait_until_finish(duration=self.timeout_ms)
Review Comment:
I also see a wait_until_finish in pytorch_image_classification.
Could that mean that this timeout won't work right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1238055179
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1167628301
Run BigQueryIO Read Performance Test Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1167622131
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1150380601
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1224677538
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ryanthompson591 commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
ryanthompson591 commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r955381674
##########
.test-infra/jenkins/LoadTestsBuilder.groovy:
##########
@@ -43,7 +43,7 @@ class LoadTestsBuilder {
static void loadTest(context, String title, Runner runner, SDK sdk, Map<String, ?> options,
- String mainClass, List<String> jobSpecificSwitches = null) {
+ String mainClass, List<String> jobSpecificSwitches = null, String requirementsFile = null) {
Review Comment:
is the requirements file the same as a requirements.txt file? Optional- call it requirementsTxtFile then and I think it should be clear to beam devs.
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
Review Comment:
is this a good tmp location? Maybe make a gs://temp-storage-for-perf-tests/tmp directory?
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet101',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet152',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ // Pytorch language modeling test using HuggingFace BERT models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-base-uncased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_base_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_base_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ bert_tokenizer : 'bert-base-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-base-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt',
+ ]
+ ],
+ [
+ title : 'Pytorch Langauge Modeling using Hugging Face bert-large-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-large-cased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_large_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_large_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ bert_tokenizer : 'bert-large-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-large-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ ]
+}
+
+def loadTestJob = { scope ->
+ List<Map> testScenarios = loadTestConfigurations()
+ for (Map testConfig: testScenarios){
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 180)
+ loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.pipelineOptions, testConfig.test, null, testConfig.pipelineOptions.requirements_file)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow',
+ 'Run Inference Benchmarks',
+ 'Inference benchmarks on Dataflow(\"Run Inference Benchmarks"\"")',
+ this
+ ) {
+ loadTestJob(delegate)
+ }
+
+// TODO(anandinguva): Change the cron job to run once a day
+CronJobBuilder.cronJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow', 'H 2 * * *',
Review Comment:
what is `H 2 * *`
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet101',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet152',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
Review Comment:
will this output conflict with the output from the other tests since it has the same name?
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
Review Comment:
If possible have an issue with the todo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r904440977
##########
sdks/python/apache_beam/examples/inference/pytorch_image_classification.py:
##########
@@ -72,6 +73,7 @@ def parse_known_args(argv):
"""Parses args for the workflow."""
parser = argparse.ArgumentParser()
parser.add_argument(
+ '--input_file',
Review Comment:
Yes. but you can only access both of them using args.input in the code
##########
sdks/python/apache_beam/examples/inference/pytorch_image_classification.py:
##########
@@ -33,9 +33,10 @@
from apache_beam.ml.inference.pytorch_inference import PytorchModelHandlerTensor
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.options.pipeline_options import SetupOptions
+from apache_beam.runners.runner import PipelineResult
from PIL import Image
+from torchvision import models
from torchvision import transforms
-from torchvision.models.mobilenetv2 import MobileNetV2
Review Comment:
We provide the documentation on what to expect on the `model_class`. I think that would be enough?
Mobilenet is just a placeholder if the user doesn't provide a model to run on this example
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1150443370
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r952232774
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -205,7 +205,9 @@ def __init__(self, flags=None, **kwargs):
# Build parser that will parse options recognized by the [sub]class of
# PipelineOptions whose object is being instantiated.
- parser = _BeamArgumentParser()
+ # set allow_abbrev=False to avoid prefix matching while parsing.
+ # https://docs.python.org/3/library/argparse.html#partial-parsing
+ parser = _BeamArgumentParser(allow_abbrev=False)
Review Comment:
One more way is to add to `--input` to the `LoadTestPipelineOptions`.
##########
sdks/python/apache_beam/examples/inference/pytorch_image_classification.py:
##########
@@ -122,10 +122,14 @@ def run(
model_class = models.mobilenet_v2
model_params = {'num_classes': 1000}
+ class PytorchModelHandlerTensorWithBatchSize(PytorchModelHandlerTensor):
Review Comment:
@yeandy I added batch size for this example. For language_modeling.py its already added as 1 due to `torch.stack'. I am leaving it as it is for now till we add the optional batching flag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1238220121
> I kicked off one here - https://ci-beam.apache.org/job/beam_SeedJob/10239/console
thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r950853667
##########
sdks/python/apache_beam/examples/inference/pytorch_image_classification.py:
##########
@@ -72,6 +73,7 @@ def parse_known_args(argv):
"""Parses args for the workflow."""
parser = argparse.ArgumentParser()
parser.add_argument(
+ '--input_file',
Review Comment:
Removed the --input_file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1221556285
Run Python 3.7 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1149113561
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yeandy commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
yeandy commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r955315368
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
+ self.metrics_namespace = 'RunInferencePytorch'
+ super().__init__(metrics_namespace=self.metrics_namespace)
+
+ def test(self):
+ pretrained_model_name = self.pipeline.get_option('pretrained_model_name')
+ if not pretrained_model_name:
+ raise RuntimeError(
+ 'Please provide a pretrained torch model name.'
+ ' Model name must be from the module torchvision.models')
+ if pretrained_model_name == _PERF_TEST_MODELS[0]:
+ model_class = models.resnet50
+ elif pretrained_model_name == _PERF_TEST_MODELS[1]:
+ model_class = models.resnet101
+ elif pretrained_model_name == _PERF_TEST_MODELS[2]:
+ model_class = models.resnet152
+ else:
+ raise NotImplementedError
+
+ # model_params are same for all the models. But this may change if we add
+ # different models.
+ model_params = {'num_classes': 1000, 'pretrained': False}
Review Comment:
The same `model_params` is being used for all model types. I was thinking we could attach a separate `model_params` to each model type, but that would be repeating a bunch of the same logic. I think what we have is fine.
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
Review Comment:
Got it, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1167619440
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1163773644
# [Codecov](https://codecov.io/gh/apache/beam/pull/21738?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#21738](https://codecov.io/gh/apache/beam/pull/21738?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (415000d) into [master](https://codecov.io/gh/apache/beam/commit/0ef5d3a185c1420da118208353ceb0b40b3a27c9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0ef5d3a) will **decrease** coverage by `0.02%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #21738 +/- ##
==========================================
- Coverage 73.97% 73.95% -0.03%
==========================================
Files 702 703 +1
Lines 92845 92880 +35
==========================================
+ Hits 68684 68689 +5
- Misses 22906 22936 +30
Partials 1255 1255
```
| Flag | Coverage Δ | |
|---|---|---|
| python | `83.54% <0.00%> (-0.04%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/21738?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...examples/inference/pytorch\_image\_classification.py](https://codecov.io/gh/apache/beam/pull/21738/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vZXhhbXBsZXMvaW5mZXJlbmNlL3B5dG9yY2hfaW1hZ2VfY2xhc3NpZmljYXRpb24ucHk=) | `0.00% <0.00%> (ø)` | |
| [...testing/benchmarks/inference/pytorch\_benchmarks.py](https://codecov.io/gh/apache/beam/pull/21738/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdGVzdGluZy9iZW5jaG1hcmtzL2luZmVyZW5jZS9weXRvcmNoX2JlbmNobWFya3MucHk=) | `0.00% <0.00%> (ø)` | |
| [...ython/apache\_beam/io/gcp/bigquery\_read\_internal.py](https://codecov.io/gh/apache/beam/pull/21738/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5X3JlYWRfaW50ZXJuYWwucHk=) | `53.92% <0.00%> (-0.50%)` | :arrow_down: |
| [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/21738/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.30% <0.00%> (-0.38%)` | :arrow_down: |
| [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/21738/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2JpZ3F1ZXJ5LnB5) | `69.85% <0.00%> (-0.23%)` | :arrow_down: |
| [...thon/apache\_beam/ml/inference/pytorch\_inference.py](https://codecov.io/gh/apache/beam/pull/21738/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL3B5dG9yY2hfaW5mZXJlbmNlLnB5) | `0.00% <0.00%> (ø)` | |
| [sdks/python/apache\_beam/ml/inference/base.py](https://codecov.io/gh/apache/beam/pull/21738/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vbWwvaW5mZXJlbmNlL2Jhc2UucHk=) | `95.37% <0.00%> (+0.08%)` | :arrow_up: |
| [sdks/python/apache\_beam/runners/common.py](https://codecov.io/gh/apache/beam/pull/21738/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9jb21tb24ucHk=) | `88.92% <0.00%> (+0.24%)` | :arrow_up: |
| [...ks/python/apache\_beam/runners/worker/data\_plane.py](https://codecov.io/gh/apache/beam/pull/21738/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvZGF0YV9wbGFuZS5weQ==) | `89.26% <0.00%> (+1.69%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/beam/pull/21738?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/beam/pull/21738?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0ef5d3a...415000d](https://codecov.io/gh/apache/beam/pull/21738?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm merged pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
damccorm merged PR #21738:
URL: https://github.com/apache/beam/pull/21738
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1188326031
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1160828157
[The Workflow run](https://github.com/apache/beam/actions/runs/2531393907) is cancelling this PR. It is an earlier duplicate of 1729654 run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r952124029
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
Review Comment:
Yep. Right now, I would need to instantiate a ModelHandler to get the namespace. For instantiation, I would need few parameters which are not necessarily available here in this .py file.
Making `get_metrics_namespace()` static would be ideal solution in this case
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
+ self.metrics_namespace = 'RunInferencePytorch'
+ super().__init__(metrics_namespace=self.metrics_namespace)
+
+ def test(self):
+ pretrained_model_name = self.pipeline.get_option('pretrained_model_name')
+ if not pretrained_model_name:
+ raise RuntimeError(
+ 'Please provide a pretrained torch model name.'
+ ' Model name must be from the module torchvision.models')
+ if pretrained_model_name == _PERF_TEST_MODELS[0]:
+ model_class = models.resnet50
+ elif pretrained_model_name == _PERF_TEST_MODELS[1]:
+ model_class = models.resnet101
+ elif pretrained_model_name == _PERF_TEST_MODELS[2]:
+ model_class = models.resnet152
+ else:
+ raise NotImplementedError
+
+ # model_params are same for all the models. But this may change if we add
+ # different models.
+ model_params = {'num_classes': 1000, 'pretrained': False}
Review Comment:
What do you mean by this?
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
Review Comment:
I see it too
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
Review Comment:
Right now, I planned to use one example from vision and one example from NLP. In the future, few more vision tests will be added.
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -205,7 +205,9 @@ def __init__(self, flags=None, **kwargs):
# Build parser that will parse options recognized by the [sub]class of
# PipelineOptions whose object is being instantiated.
- parser = _BeamArgumentParser()
+ # set allow_abbrev=False to avoid prefix matching while parsing.
+ # https://docs.python.org/3/library/argparse.html#partial-parsing
+ parser = _BeamArgumentParser(allow_abbrev=False)
Review Comment:
Yes, it is to address the two options. I am not sure about this breaking the users code(at least all the unit tests and IT tests passed in our case). @tvalentyn could add more perspective
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
+ self.metrics_namespace = 'RunInferencePytorch'
+ super().__init__(metrics_namespace=self.metrics_namespace)
+
+ def test(self):
+ pretrained_model_name = self.pipeline.get_option('pretrained_model_name')
+ if not pretrained_model_name:
+ raise RuntimeError(
+ 'Please provide a pretrained torch model name.'
+ ' Model name must be from the module torchvision.models')
+ if pretrained_model_name == _PERF_TEST_MODELS[0]:
+ model_class = models.resnet50
+ elif pretrained_model_name == _PERF_TEST_MODELS[1]:
+ model_class = models.resnet101
+ elif pretrained_model_name == _PERF_TEST_MODELS[2]:
+ model_class = models.resnet152
Review Comment:
SG. will change it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1150070217
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1167955254
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1227635406
@yeandy PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r960595623
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet101',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet152',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ // Pytorch language modeling test using HuggingFace BERT models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-base-uncased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_base_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_base_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ bert_tokenizer : 'bert-base-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-base-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt',
+ ]
+ ],
+ [
+ title : 'Pytorch Langauge Modeling using Hugging Face bert-large-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-large-cased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_large_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_large_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ bert_tokenizer : 'bert-large-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-large-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ ]
+}
+
+def loadTestJob = { scope ->
+ List<Map> testScenarios = loadTestConfigurations()
+ for (Map testConfig: testScenarios){
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 180)
+ loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.pipelineOptions, testConfig.test, null, testConfig.pipelineOptions.requirements_file)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow',
+ 'Run Inference Benchmarks',
+ 'Inference benchmarks on Dataflow(\"Run Inference Benchmarks"\"")',
+ this
+ ) {
+ loadTestJob(delegate)
+ }
+
+// TODO(anandinguva): Change the cron job to run once a day
+CronJobBuilder.cronJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow', 'H 2 * * *',
Review Comment:
Way of specifying cron jobs that runs every two hours in this case. I will change this to 24 hours.
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
Review Comment:
Removed TODO since its not necessary anymore
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet101',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet152',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
Review Comment:
Changed it. thanks
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
Review Comment:
This is the location for all the load tests. I think this gets deleted periodically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yeandy commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
yeandy commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r951479723
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet101',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet152',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ // pytorch language modeling test using HuggingFace bert models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
Review Comment:
```suggestion
title : 'Pytorch Lanugaue Modeling using Hugging Face bert-base-uncased model',
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1221596508
R: @yeandy @ryanthompson591
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r961679375
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet101',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_101' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet152',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_resnet152' + now + '.txt'
+ ]
+ ],
+ // Pytorch language modeling test using HuggingFace BERT models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-base-uncased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_base_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_base_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-base-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-base-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_base_uncased' + now + '.txt',
+ ]
+ ],
+ [
+ title : 'Pytorch Langauge Modeling using Hugging Face bert-large-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-large-cased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_large_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_large_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-large-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-large-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_large_uncased' + now + '.txt'
+ ]
+ ],
+ ]
+}
+
+def loadTestJob = { scope ->
+ List<Map> testScenarios = loadTestConfigurations()
+ for (Map testConfig: testScenarios){
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 180)
+ loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.pipelineOptions, testConfig.test, null, testConfig.pipelineOptions.requirements_file)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow',
+ 'Run Inference Benchmarks',
+ 'Inference benchmarks on Dataflow(\"Run Inference Benchmarks"\"")',
+ this
+ ) {
+ loadTestJob(delegate)
+ }
+
+// TODO(anandinguva): Change the cron job to run once a day
+CronJobBuilder.cronJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow', 'H 24 * * *',
Review Comment:
I changed this to 15 following the Big query perf tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1235512717
>> I'm also a little nervous about this change - could you help me understand why we need it?
The `LoadTestOptions` has an arg named `input_options`. RunInference perf tests use LoadTest as the base class since LoadTest class doc string instructs to use it for perf tests. RunInference perf tests need an option named `input` but if I provide that, then the `argparse` maps `input` to `input_options`.
More at https://docs.python.org/3/library/argparse.html#allow-abbrev
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1236124017
@damccorm Can you run a seed job on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r961818629
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet101',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_101' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet152',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_resnet152' + now + '.txt'
+ ]
+ ],
+ // Pytorch language modeling test using HuggingFace BERT models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-base-uncased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_base_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_base_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-base-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-base-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_base_uncased' + now + '.txt',
+ ]
+ ],
+ [
+ title : 'Pytorch Langauge Modeling using Hugging Face bert-large-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-large-cased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_large_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_large_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-large-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-large-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_large_uncased' + now + '.txt'
+ ]
+ ],
+ ]
+}
+
+def loadTestJob = { scope ->
+ List<Map> testScenarios = loadTestConfigurations()
+ for (Map testConfig: testScenarios){
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 180)
+ loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.pipelineOptions, testConfig.test, null, testConfig.pipelineOptions.requirements_file)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow',
+ 'Run Inference Benchmarks',
+ 'Inference benchmarks on Dataflow(\"Run Inference Benchmarks"\"")',
+ this
+ ) {
+ loadTestJob(delegate)
+ }
+
+// TODO(anandinguva): Change the cron job to run once a day
+CronJobBuilder.cronJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow', 'H 24 * * *',
Review Comment:
That should be fine here, though FWIW in general balancing out cron jobs is probably a better approach (so they're not competing for resources)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1235661821
> I removed the allow_abbrev and input to the LoadTestOptions so that we don't break anything
Thanks - I feel a lot better about that outcome
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1150263969
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1150290570
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ryanthompson591 commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
ryanthompson591 commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1149970014
Run Seed Job
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1150031545
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1188348148
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1168072702
Run Inference Benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1167625997
Run Python 3.7 PostCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] AnandInguva commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
AnandInguva commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r904441756
##########
sdks/python/apache_beam/examples/inference/pytorch_image_classification.py:
##########
@@ -120,26 +128,34 @@ def run(argv=None, model_class=None, model_params=None, save_main_session=True):
model_class=model_class,
model_params=model_params))
- with beam.Pipeline(options=pipeline_options) as p:
- filename_value_pair = (
- p
- | 'ReadImageNames' >> beam.io.ReadFromText(
- known_args.input, skip_header_lines=1)
- | 'ReadImageData' >> beam.Map(
- lambda image_name: read_image(
- image_file_name=image_name, path_to_dir=known_args.images_dir))
- | 'PreprocessImages' >> beam.MapTuple(
- lambda file_name, data: (file_name, preprocess_image(data))))
- predictions = (
- filename_value_pair
- | 'PyTorchRunInference' >> RunInference(model_handler)
- | 'ProcessOutput' >> beam.ParDo(PostProcessor()))
-
- if known_args.output:
- predictions | "WriteOutputToGCS" >> beam.io.WriteToText( # pylint: disable=expression-not-assigned
- known_args.output,
- shard_name_template='',
- append_trailing_newlines=True)
+ if not test_pipeline:
+ p = beam.Pipeline(options=pipeline_options)
+ else:
+ p = test_pipeline
+
+ filename_value_pair = (
+ p
+ | 'ReadImageNames' >> beam.io.ReadFromText(
+ known_args.input, skip_header_lines=1)
+ | 'ReadImageData' >> beam.Map(
+ lambda image_name: read_image(
+ image_file_name=image_name, path_to_dir=known_args.images_dir))
+ | 'PreprocessImages' >> beam.MapTuple(
+ lambda file_name, data: (file_name, preprocess_image(data))))
+ predictions = (
+ filename_value_pair
+ | 'PyTorchRunInference' >> RunInference(model_handler).with_output_types(
Review Comment:
Yep. That was from old code. removed it. thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1160828593
[The Workflow run](https://github.com/apache/beam/actions/runs/2531396454) is cancelling this PR. It is an earlier duplicate of 2083803 run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] yeandy commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
yeandy commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r951480517
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet101',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet152',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ // pytorch language modeling test using HuggingFace bert models
Review Comment:
```suggestion
// Pytorch language modeling test using HuggingFace BERT models
```
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
+ self.metrics_namespace = 'RunInferencePytorch'
+ super().__init__(metrics_namespace=self.metrics_namespace)
+
+ def test(self):
+ pretrained_model_name = self.pipeline.get_option('pretrained_model_name')
+ if not pretrained_model_name:
+ raise RuntimeError(
+ 'Please provide a pretrained torch model name.'
+ ' Model name must be from the module torchvision.models')
+ if pretrained_model_name == _PERF_TEST_MODELS[0]:
+ model_class = models.resnet50
+ elif pretrained_model_name == _PERF_TEST_MODELS[1]:
+ model_class = models.resnet101
+ elif pretrained_model_name == _PERF_TEST_MODELS[2]:
+ model_class = models.resnet152
Review Comment:
Maybe define `_PERF_TEST_MODELS` to be a dict mapping the string "resnet50" to the class? i.e. `_PERF_TEST_MODELS = {'resnet50': models.resnet50}` and do a retrieval from the lookup?
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet101',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet152',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ // pytorch language modeling test using HuggingFace bert models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
Review Comment:
```suggestion
title : 'Pytorch Language Modeling using Hugging Face bert-base-uncased model',
```
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
Review Comment:
Is it just me, or does anyone see these red-ish color blocks here?
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
Review Comment:
Do you mean `get_metrics_namespace()`? Let's add a GH issue to this.
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet101',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the performance test
+ pretrained_model_name : 'resnet152',
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt'
+ ]
+ ],
+ // pytorch language modeling test using HuggingFace bert models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-base-uncased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_base_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_base_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ // args defined in the example.
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ // TODO: make sure the model_state_dict_path weights are accurate.
+ bert_tokenizer : 'bert-base-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-base-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_' + now + '.txt',
+ ]
+ ],
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-large-uncased model',
Review Comment:
```suggestion
title : 'Pytorch Langauge Modeling using Hugging Face bert-large-uncased model',
```
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -205,7 +205,9 @@ def __init__(self, flags=None, **kwargs):
# Build parser that will parse options recognized by the [sub]class of
# PipelineOptions whose object is being instantiated.
- parser = _BeamArgumentParser()
+ # set allow_abbrev=False to avoid prefix matching while parsing.
+ # https://docs.python.org/3/library/argparse.html#partial-parsing
+ parser = _BeamArgumentParser(allow_abbrev=False)
Review Comment:
This is to address ambiguous options `input` and `input_options`? Will this break users who have previously written pipelines with other abbreviations, esp custom ones?
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ // (TODO): Add model name to experiments once decided on which models to use.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
Review Comment:
Should we be consistent with the naming of the `test` and `job_name`?
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
Review Comment:
Right now, this only runs `pytorch_image_classification`. Are we going to add `pytorch_image_segmentation` to this class as well?
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
+ self.metrics_namespace = 'RunInferencePytorch'
+ super().__init__(metrics_namespace=self.metrics_namespace)
+
+ def test(self):
+ pretrained_model_name = self.pipeline.get_option('pretrained_model_name')
+ if not pretrained_model_name:
+ raise RuntimeError(
+ 'Please provide a pretrained torch model name.'
+ ' Model name must be from the module torchvision.models')
+ if pretrained_model_name == _PERF_TEST_MODELS[0]:
+ model_class = models.resnet50
+ elif pretrained_model_name == _PERF_TEST_MODELS[1]:
+ model_class = models.resnet101
+ elif pretrained_model_name == _PERF_TEST_MODELS[2]:
+ model_class = models.resnet152
+ else:
+ raise NotImplementedError
+
+ # model_params are same for all the models. But this may change if we add
+ # different models.
+ model_params = {'num_classes': 1000, 'pretrained': False}
Review Comment:
Should we parametrize this now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #21738:
URL: https://github.com/apache/beam/pull/21738#issuecomment-1221596702
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on a diff in pull request #21738: Inference benchmark tests
Posted by GitBox <gi...@apache.org>.
damccorm commented on code in PR #21738:
URL: https://github.com/apache/beam/pull/21738#discussion_r960940764
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet101',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_101' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet152',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_resnet152' + now + '.txt'
+ ]
+ ],
+ // Pytorch language modeling test using HuggingFace BERT models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-base-uncased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_base_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_base_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-base-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-base-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_base_uncased' + now + '.txt',
+ ]
+ ],
+ [
+ title : 'Pytorch Langauge Modeling using Hugging Face bert-large-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-large-cased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_large_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_large_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-large-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-large-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_large_uncased' + now + '.txt'
+ ]
+ ],
+ ]
+}
+
+def loadTestJob = { scope ->
+ List<Map> testScenarios = loadTestConfigurations()
+ for (Map testConfig: testScenarios){
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 180)
+ loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.pipelineOptions, testConfig.test, null, testConfig.pipelineOptions.requirements_file)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow',
+ 'Run Inference Benchmarks',
+ 'Inference benchmarks on Dataflow(\"Run Inference Benchmarks"\"")',
+ this
+ ) {
+ loadTestJob(delegate)
+ }
+
+// TODO(anandinguva): Change the cron job to run once a day
+CronJobBuilder.cronJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow', 'H 24 * * *',
Review Comment:
Helpful cron validator - https://crontab.guru/#0_2_*_*_*
More info about H - https://stackoverflow.com/questions/47302607/meaning-of-h-5-in-cron-jenkins
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
Review Comment:
I think these come from weird space characters via copy paste? I've seen them elsewhere and they've been harmless
##########
sdks/python/apache_beam/options/pipeline_options.py:
##########
@@ -205,7 +205,9 @@ def __init__(self, flags=None, **kwargs):
# Build parser that will parse options recognized by the [sub]class of
# PipelineOptions whose object is being instantiated.
- parser = _BeamArgumentParser()
+ # set allow_abbrev=False to avoid prefix matching while parsing.
+ # https://docs.python.org/3/library/argparse.html#partial-parsing
+ parser = _BeamArgumentParser(allow_abbrev=False)
Review Comment:
I'm also a little nervous about this change - could you help me understand why we need it?
It seems like it would definitely have the potential to break existing pipelines - if we need to do that, it might be better to detect and warn that the behavior will be changed in the future.
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_language_modeling_benchmarks.py:
##########
@@ -0,0 +1,38 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_language_modeling
+from apache_beam.testing.load_tests.load_test import LoadTest
+
+
+class PytorchLanguageModelingBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
Review Comment:
Same comment above - please update to `get_metrics_namespace` and add an issue link instead of username in the TODO
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet101',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_101' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet152',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_resnet152' + now + '.txt'
+ ]
+ ],
+ // Pytorch language modeling test using HuggingFace BERT models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-base-uncased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_base_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_base_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-base-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-base-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_base_uncased' + now + '.txt',
+ ]
+ ],
+ [
+ title : 'Pytorch Langauge Modeling using Hugging Face bert-large-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-large-cased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_large_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_large_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-large-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-large-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_large_uncased' + now + '.txt'
+ ]
+ ],
+ ]
+}
+
+def loadTestJob = { scope ->
+ List<Map> testScenarios = loadTestConfigurations()
+ for (Map testConfig: testScenarios){
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 180)
+ loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.pipelineOptions, testConfig.test, null, testConfig.pipelineOptions.requirements_file)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow',
+ 'Run Inference Benchmarks',
+ 'Inference benchmarks on Dataflow(\"Run Inference Benchmarks"\"")',
+ this
+ ) {
+ loadTestJob(delegate)
+ }
+
+// TODO(anandinguva): Change the cron job to run once a day
Review Comment:
Nit - remove hanging TODO
##########
sdks/python/apache_beam/testing/benchmarks/inference/pytorch_image_classification_benchmarks.py:
##########
@@ -0,0 +1,63 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# pytype: skip-file
+
+import logging
+
+from apache_beam.examples.inference import pytorch_image_classification
+from apache_beam.testing.load_tests.load_test import LoadTest
+from torchvision import models
+
+_PERF_TEST_MODELS = ['resnet50', 'resnet101', 'resnet152']
+_PRETRAINED_MODEL_MODULE = 'torchvision.models'
+
+
+class PytorchVisionBenchmarkTest(LoadTest):
+ def __init__(self):
+ # TODO (anandinguva): make get_namespace() method in RunInference static
Review Comment:
Agreed - could you update to `get_metrics_namespace` and link to an issue in the TODO instead of just your username?
##########
.test-infra/jenkins/job_InferenceBenchmarkTests_Python.groovy:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import LoadTestsBuilder as loadTestsBuilder
+import PhraseTriggeringPostCommitBuilder
+import CronJobBuilder
+
+def now = new Date().format("MMddHHmmss", TimeZone.getTimeZone('UTC'))
+
+def loadTestConfigurations = {
+ ->
+ [
+ // Benchmark test config. Add multiple configs for multiple models.
+ [
+ title : 'Pytorch Vision Classification with Resnet 101',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet101',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet101',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet101',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet101.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_101' + now + '.txt'
+ ]
+ ],
+ [
+ title : 'Pytorch Imagenet Classification with Resnet 152',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_image_classification_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-imagenet-python' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_inference_imagenet_results_resnet152',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_inference_imagenet_resnet152',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ pretrained_model_name : 'resnet152',
+ input : 'gs://apache-beam-ml/testing/inputs/openimage_50k_benchmark.txt',
+ model_state_dict_path : 'gs://apache-beam-ml/models/torchvision.models.resnet152.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_resnet152' + now + '.txt'
+ ]
+ ],
+ // Pytorch language modeling test using HuggingFace BERT models
+ [
+ title : 'Pytorch Lanugaue Modeling using Hugging face bert-base-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-base-uncased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_base_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_base_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-base-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-base-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_base_uncased' + now + '.txt',
+ ]
+ ],
+ [
+ title : 'Pytorch Langauge Modeling using Hugging Face bert-large-uncased model',
+ test : 'apache_beam.testing.benchmarks.inference.pytorch_language_modeling_benchmarks',
+ runner : CommonTestProperties.Runner.DATAFLOW,
+ pipelineOptions: [
+ job_name : 'benchmark-tests-pytorch-language-modeling-bert-large-cased' + now,
+ project : 'apache-beam-testing',
+ region : 'us-central1',
+ staging_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ temp_location : 'gs://temp-storage-for-perf-tests/loadtests',
+ requirements_file : 'apache_beam/ml/inference/torch_tests_requirements.txt',
+ pickle_library : 'cloudpickle',
+ experiments : 'no_use_multiple_sdk_containers',
+ publish_to_big_query : true,
+ metrics_dataset : 'beam_run_inference',
+ metrics_table : 'torch_language_modeling_bert_large_uncased',
+ input_options : '{}', // this option is not required for RunInference tests.
+ influx_measurement : 'torch_language_modeling_bert_large_uncased',
+ influx_db_name : InfluxDBCredentialsHelper.InfluxDBDatabaseName,
+ influx_hostname : InfluxDBCredentialsHelper.InfluxDBHostUrl,
+ input : 'gs://apache-beam-ml/testing/inputs/sentences_50k.txt',
+ bert_tokenizer : 'bert-large-uncased',
+ model_state_dict_path : 'gs://apache-beam-ml/models/huggingface.BertForMaskedLM.bert-large-uncased.pth',
+ output : 'gs://temp-storage-for-end-to-end-tests/torch/result_bert_large_uncased' + now + '.txt'
+ ]
+ ],
+ ]
+}
+
+def loadTestJob = { scope ->
+ List<Map> testScenarios = loadTestConfigurations()
+ for (Map testConfig: testScenarios){
+ commonJobProperties.setTopLevelMainJobProperties(scope, 'master', 180)
+ loadTestsBuilder.loadTest(scope, testConfig.title, testConfig.runner, CommonTestProperties.SDK.PYTHON, testConfig.pipelineOptions, testConfig.test, null, testConfig.pipelineOptions.requirements_file)
+ }
+}
+
+PhraseTriggeringPostCommitBuilder.postCommitJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow',
+ 'Run Inference Benchmarks',
+ 'Inference benchmarks on Dataflow(\"Run Inference Benchmarks"\"")',
+ this
+ ) {
+ loadTestJob(delegate)
+ }
+
+// TODO(anandinguva): Change the cron job to run once a day
+CronJobBuilder.cronJob(
+ 'beam_Inference_Python_Benchmarks_Dataflow', 'H 24 * * *',
Review Comment:
```suggestion
'beam_Inference_Python_Benchmarks_Dataflow', 'H 2 * * *',
```
I think this is wrong. Cron syntax is `minute, hour, day of month, month, day of week`. So as written, this tries to run a job at the Hth minute (pseudorandom to avoid collisions) of the 24th hour every day of every week/month. Since the 24th hour doesn't exist, this is invalid syntax. Updating to the second hour should run it when most people are offline which is ideal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org