[GitHub] [incubator-mxnet] leezu opened a new issue #19605: Improve CI caching of docker multi-stage builds

[GitBox <gi...@apache.org>]

leezu opened a new issue #19605:
URL: https://github.com/apache/incubator-mxnet/issues/19605


   As per https://github.com/moby/moby/issues/34715#issuecomment-550567523 we should enable docker buildkit on CI to improve the caching behavior of multi-stage builds.
   
   On the MXNet side, the following patch may suffice
   
   ``` diff
   diff --git a/ci/Jenkinsfile_docker_cache b/ci/Jenkinsfile_docker_cache
   index 5f378b5d6..edad8aacd 100644
   --- a/ci/Jenkinsfile_docker_cache
   +++ b/ci/Jenkinsfile_docker_cache
   @@ -37,7 +37,7 @@ core_logic: {
          ws('workspace/docker_cache') {
            timeout(time: total_timeout, unit: 'MINUTES') {
              utils.init_git()
   -          sh "cd ci && python3 ./docker_login.py --secret-name ${env.DOCKERHUB_SECRET_NAME} && docker-compose -f docker/docker-compose.yml pull && docker-compose -f docker/docker-compose.yml build --parallel && COMPOSE_HTTP_TIMEOUT=600 docker-compose -f docker/docker-compose.yml push && docker logout"
   +          sh "cd ci && python3 ./docker_login.py --secret-name ${env.DOCKERHUB_SECRET_NAME} && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -f docker/docker-compose.yml pull && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -f docker/docker-compose.yml build --parallel && COMPOSE_HTTP_TIMEOUT=600 COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -f docker/docker-compose.yml push && docker logout"
            }
          }
        }
   diff --git a/ci/build.py b/ci/build.py
   index 1e9e23fad..57d1fa5b2 100755
   --- a/ci/build.py
   +++ b/ci/build.py
   @@ -72,6 +72,8 @@ def build_docker(platform: str, registry: str, num_retries: int, no_cache: bool,
    
        env = os.environ.copy()
        env["DOCKER_CACHE_REGISTRY"] = registry
   +    env["COMPOSE_DOCKER_CLI_BUILD"] = "1"
   +    env["DOCKER_BUILDKIT"] = "1"
    
        @retry(subprocess.CalledProcessError, tries=num_retries)
        def run_cmd(env=None):
   @@ -204,6 +206,8 @@ def load_docker_cache(platform, tag, docker_registry) -> None:
        if docker_registry:
            env = os.environ.copy()
            env["DOCKER_CACHE_REGISTRY"] = docker_registry
   +        env["COMPOSE_DOCKER_CLI_BUILD"] = "1"
   +        env["DOCKER_BUILDKIT"] = "1"
            cmd = ['docker-compose', '-f', 'docker/docker-compose.yml', 'pull', platform]
            logging.info("Running command: 'DOCKER_CACHE_REGISTRY=%s %s'", docker_registry, ' '.join(cmd))
            check_call(cmd, env=env)
   ```
   
   Prior to applying the patch on the mxnet side, https://github.com/apache/incubator-mxnet-ci/blob/master/tools/jenkins-slave-creation-unix/conf-ubuntu-cpu-c5/install.sh and https://github.com/apache/incubator-mxnet-ci/blob/master/tools/jenkins-slave-creation-unix/conf-ubuntu-gpu/install.sh need to be modified to install buildkit from https://github.com/moby/buildkit/releases and the CI AMI needs to be regenerated.
   
   The main issue with the existing CI cache setup is that on local developer machines, `ci/build.py [...] --cache-intermediate` won't work for multi-stage builds. In addition to that, CI will not re-use cached builds for the later stages of the build. Thus switching to buildkit may speed CI pipelines relying on multi-stage build by a few minutes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@mxnet.apache.org
For additional commands, e-mail: issues-help@mxnet.apache.org