[GitHub] [beam] steveniemitz commented on a change in pull request #16229: [BEAM-13459] Cache artifacts across job runs on python+dataflow

[GitBox <gi...@apache.org>]

steveniemitz commented on a change in pull request #16229:
URL: https://github.com/apache/beam/pull/16229#discussion_r776508707



##########
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##########
@@ -556,9 +565,79 @@ def _get_sdk_image_overrides(self, pipeline_options):
     return (
         dict(s.split(',', 1) for s in sdk_overrides) if sdk_overrides else {})
 
+  @staticmethod
+  def _compute_sha256(file):
+    hasher = hashlib.sha256()
+    with open(file, 'rb') as f:
+      for chunk in iter(partial(f.read,
+                                DataflowApplicationClient._HASH_CHUNK_SIZE),
+                        b""):
+        hasher.update(chunk)
+    return hasher.hexdigest()
+
+  @staticmethod
+  def _split_gcs_path(path):
+    if not path.startswith('gs://'):
+      raise RuntimeError('Expected gs:// path, got %s', path)
+    return path[5:].split('/', 1)
+
+  def _cached_location(self, sha256):
+    sha_prefix = sha256[0:2]
+    return FileSystems.join(
+        self._root_staging_location,
+        DataflowApplicationClient._GCS_CACHE_PREFIX,
+        sha_prefix,
+        sha256)
+
+  @retry.with_exponential_backoff(
+      retry_filter=retry.retry_on_server_errors_and_timeout_filter)
+  def _gcs_object_exists(self, gcs_or_local_path):
+    if not gcs_or_local_path.startswith('gs://'):
+      return False
+    else:
+      bucket, name = self._split_gcs_path(gcs_or_local_path)
+      request = storage.StorageObjectsGetRequest(bucket=bucket, object=name)
+      try:
+        self._storage_client.objects.Get(request)
+        return True
+      except exceptions.HttpError as e:
+        return e.status_code not in (403, 404)
+
+  @retry.with_exponential_backoff(
+      retry_filter=retry.retry_on_server_errors_and_timeout_filter)
+  def _gcs_to_gcs_copy(self, from_gcs, to_gcs):

Review comment:
       I didn't try that (or even knew it existed), it seems like it'd indeed remove duplication here if it works correctly.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org