[GitHub] [airflow] potiuk commented on pull request #13730: Run CI (Build Image and committer PRs) on self-hosted runner

[GitBox <gi...@apache.org>]

potiuk commented on pull request #13730:
URL: https://github.com/apache/airflow/pull/13730#issuecomment-762489053


   Yes. Docker-composes of our still use a number of "official" images for integration and we seem to hit the "November" limits of DockerHub (we were protected from those so far apparently because GitHub public runners are exempted from thee limits).
   
   You. can see all of them here https://github.com/apache/airflow/tree/master/scripts/ci/docker-compose
   
   We already brought some of the images under "apache" organisation:
   * image: apache/airflow:krb5-kdc-server-2020.10.08
   * image: apache/airflow:openldap-2020.07.10-2.4.50
   * image: apache/airflow:presto-2020.10.08
   
   Those "apache" images are exempted from the rate limit introduced by DockerHub in November (Infra confirmed that in one of the discussions we had. The limits are described here: https://docs.docker.com/docker-hub/download-rate-limit/  and ASF has apparently a separate agreement with DockerHub about it).
   
   There are still quite a few images in our build that are just pulled from the "official" sources. They are used for "integration test". 
   
   * image: mysql:${MYSQL_VERSION}
   * image: postgres:${POSTGRES_VERSION}
   * image: cassandra:3.0
   * image: mongo:3
   * image: apachepinot/pinot:latest
   * image: rabbitmq:3.7
   * image: redis:5.0.1
   
   Each of those images might have different base images (busybox/alpine etc.) so it's quite expected to have a number of bae images pulled. As in Github Actions, all the images are run on a "fresh" machine, those images are pulled every time integration tests are run. And we have also a cleeanup stage that prunes the docker engine for every step, to save the space on the Github Runners (we used to have disk space problems, on those runners that we mitigated by 'system prune" for docker engine. Right now - basically ever time we run tests, those images are pulled (and this is anyhow needed for GitHub Runners because they are always "fresh") 
   
   The limits (for unauthenticated pulls) are rather severe and we seem to be hitting those when we use self-hosted runners. Those limits are per "public IP address" not per  user, and we might be sharing the IPs with others (or we can ourselves hit the  limits. The limit is 200 container image requests per 6 hours (https://www.docker.com/pricing) so it is rather easy to hit it .
   
   Those limits are not applicable for pulling images from GitHub public runners (they exempted their IP addresses from that), that's why we have not experienced it yet.
   
   There are few mitigations possible:
   
   1) skipping the "free space" step on self-hosted runners. That will make pulls to the DockerHub far less frequent, but this also means that the docker engine on our instances might grow uncontrollably and we need to do periodic pruning or restarting the instances (depending on where/how docker engine is running).  Possibly changing "docker system prune"  and removing "--all" there might help, but then we need to likely do some extra regular cleanup for those instances (GitHub Runners have the benefit that they are being maintained/started /reset by GitHub so we did not have to worry about it).
   
   2)  moving all the images we pull to "apache/airflow" DockerHub - similarly as we did with  openldap/presto/kerberos. Those images are exempt from he DockerHub limits and we can pull them as many times we want. This seems like the easiest solution. Rather than building those images, wee can simply pull->push them. We just need to figure out sensible naming convention because right now we can only store "apache/airflow" images, but we already have precendents with `apache/airflow:openldap-2020.07.10-2.4.50` and the like.
   
   3)  authenticating the requests from CI to DockerHub (we need a secret setup by INFRA for that).
   
   4)  moving all the images to our "GitHub Packages"  - the legacy container registry - there GITHUB_TOKEN might be used to authenticate (we already do that to pull our images that we share with all the jobs).
   
   5)  moving all the images we pull to "gchr.io" - the new registry enabled last week by INFRA has no limits (it is free for public images https://github.blog/2020-09-01-introducing-github-container-registry/) and it allows for public visibility of the images. Here are migrations steps: https://docs.github.com/en/packages/guides/migrating-to-github-container-registry-for-docker-images. We might simply pull/push the right images we are using and change the docker-compose images to point to it. The drawback is that currently the access management is manual (using the GitHub UI) -  but we have limited number of images so that might as well work.
   
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org