[GitHub] [beam] emilymye commented on a diff in pull request #17323: [BEAM-14219] Auto opt-in SDK container image prebuilding when eligibl…

[GitBox <gi...@apache.org>]

emilymye commented on code in PR #17323:
URL: https://github.com/apache/beam/pull/17323#discussion_r848894248


##########
sdks/python/apache_beam/runners/dataflow/dataflow_runner.py:
##########
@@ -393,6 +393,71 @@ def _adjust_pipeline_for_dataflow_v2(self, pipeline):
   def _check_for_unsupported_features_on_non_portable_worker(self, pipeline):
     pipeline.visit(self.combinefn_visitor())
 
+  def _enable_sdk_prebuild_if_applicable(self, options):
+    setup_options = options.view_as(SetupOptions)
+    # SDK container image pre-build explicitly enabled or disabled.
+    if (setup_options.prebuild_sdk_container_engine or
+        setup_options.disable_sdk_container_prebuild):
+      return
+    from apache_beam.runners.dataflow.internal import apiclient
+    # prebuild_sdk_container_engine only works with fnapi jobs.
+    if not apiclient._use_fnapi(options):
+      return
+    # if no additional dependencies, pre-building is not that useful.
+    if not any([setup_options.requirements_file,
+                setup_options.setup_file,
+                setup_options.sdk_location,
+                setup_options.extra_packages]):
+      return
+
+    google_cloud_options = options.view_as(GoogleCloudOptions)
+    from apache_beam.runners.dataflow.internal.clients import cloudbuild
+    from apache_beam.internal.gcp.auth import get_service_credentials
+    from apache_beam.internal.http_client import get_new_http
+    from apitools.base.py import exceptions
+
+    if google_cloud_options.no_auth:
+      credentials = None
+    else:
+      credentials = get_service_credentials()
+    cloudbuild_client = cloudbuild.CloudbuildV1(
+        credentials=credentials,
+        get_credentials=(not google_cloud_options.no_auth),
+        http=get_new_http(),
+        response_encoding='utf8')
+
+    # Check if the cloud build api is enabled on the project.
+    request = cloudbuild.CloudbuildProjectsBuildsListRequest(
+        projectId=google_cloud_options.project, pageSize=1)
+    try:
+      cloudbuild_client.projects_builds.List(request)
+    except exceptions.HttpForbiddenError as e:
+      if 'SERVICE_DISABLED' in str(e):
+        _LOGGER.info(
+            "Google Cloud Build API not enabled, consider enabling "
+            "it to utilize SDK container image pre-building workflow, "
+            "learn more Learn more about the workflow on "
+            "https://cloud.google.com/dataflow/docs/guides/"
+            "using-custom-containers#prebuild")
+      else:
+        _LOGGER.info(

Review Comment:
   should be info? or warning? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org