[GitHub] [incubator-mxnet] marcoabreu commented on a change in pull request #19654: [WIP] Migrate to use ECR as docker cache instead of dockerhub

[GitBox <gi...@apache.org>]

marcoabreu commented on a change in pull request #19654:
URL: https://github.com/apache/incubator-mxnet/pull/19654#discussion_r541164707



##########
File path: ci/build.py
##########
@@ -58,7 +60,11 @@ def get_docker_tag(platform: str, registry: str) -> str:
     platform = platform if any(x in platform for x in ['build.', 'publish.']) else 'build.{}'.format(platform)
     if not registry:
         registry = "mxnet_local"
-    return "{0}/{1}".format(registry, platform)
+    dockerfile_hash = 'unk'
+    with open(get_dockerfile(platform),"rb") as f:
+        bytes = f.read()
+        dockerfile_hash = hashlib.sha256(bytes).hexdigest()[:12]

Review comment:
       Just hashing the dockerfile is not sufficient IMO. It would cause a lot of collisions. Instead, I'd recommend to hash the entire Docker buildcontext directory. 
   
   The impact of these collisions would be that different jobs with actually different buildcontexts would constantly override the cache and thus basically invalidate it. 
   
   When you are creating the hash out of the entire context, please make sure to use a chunk/streaming based hash calculation instead of reading everything into the ram for efficiencys sake.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org