[GitHub] [pulsar] lhotari edited a comment on pull request #9627: [Build] Reduce pulsar-test-latest-version docker image size

[GitBox <gi...@apache.org>]

lhotari edited a comment on pull request #9627:
URL: https://github.com/apache/pulsar/pull/9627#issuecomment-781991274


   > I wonder how to share the docker image between different jobs?
   
   @zymap The high level idea is described in the [draft PIP Changes to GitHub Actions based Pulsar CI](https://docs.google.com/document/d/1FNEWD3COdnNGMiryO9qBUW_83qtzAhqjDI5wwmPD-YE/edit?usp=sharing). It's possible by using the GitHub Actions Cache. There's [an action that adds a local HTTP proxy](https://github.com/marketplace/actions/http-cache-proxy) to the cache where you can POST and GET any key from the cache. I've been experimenting with it and it's really great and efficient. I'll share the results on the mailing list asap (next week).
   
   docker comes with `docker save` and `docker load` commands which allow to export an image to a stream of bytes. I've been experimenting with a solution where I pipe this to zstd compressor and with curl over http-cache-proxy to GitHub Actions Cache. It's pretty stable, however sometimes the upload or download fails. I'll add retries to address that issue.
   
   btw. The challenge with `docker save` is that it saves all layers. It also saves the `openjdk:8-jdk-slim` layers which could be pulled directly, however that's not a big concern since the size of `openjdk:8-jdk-slim` is 288MB uncompressed. This is a small portion of the total image size of `pulsar-test-latest-version` image. There are solutions to hack around the limitation, but they are fairly complex.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org