[GitHub] [airflow] potiuk edited a comment on issue #8400: Less frequent dockerhub builds

[GitBox <gi...@apache.org>]

potiuk edited a comment on issue #8400: Less frequent dockerhub builds
URL: https://github.com/apache/airflow/pull/8400#issuecomment-614783967
 
 
   @dimberman It always lags quite a bit as DockerHub is very slow just "check if the image needs to be rebuilt aka Docker build from cache" takes anything between 10 and 20 minutes.  Currently we build so many images (including the PROD one)  that when we have a few merges, the build queue grows faster than it is cleared and pretty much every day we end up in a situation that the build waits a long time (700mins = 12 hours is pretty normal if we have a series of master merges) + if we have a v1-10-stable push, that's even longer. 
   
   From CI point of view it's OK. The whole system is built in the way that if the image is old, it - at most - run CI a bit slower. Assuming that master/v1-10-stable are constantly moving, I do not think this is a big issue to add such "10 hours" skip. 
   
   I also opened INFRA ticket to see if we can push the image from Github Actions (see my comment below. But I doubt it will be OK from the security POV:
   > Currently we have quite a lot of builds happening in DockerHub (including production image). It takes a long time to build the images and they are queuing now. I have created an infra ticket to ask if we can push images from Github Actions (https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20124) but I have no high hopes for it - it requires storing secrets in GA. That's why I propose that build scripts will check if the image is <10 hours old and only actually build it if it is.
   
   Alternatively we might want to rebuild the images regularly - every night for example, but this is a bit problematic (I explained it to @turbaszek above) - we could generate a trigger URL and use it in an external service (there is no cron build support in DockerHub. But I am not sure it's safe.
   
   I realise it is not perfect, do you have any ideas how we can do it differently ?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services