[GitHub] potiuk commented on issue #4543: [AIRFLOW-3718] Multi-layered version of the docker image

[GitBox <gi...@apache.org>]

potiuk commented on issue #4543: [AIRFLOW-3718] Multi-layered version of the docker image
URL: https://github.com/apache/airflow/pull/4543#issuecomment-455242821
 
 
   @fokko - I am pushing an updated version. I know that famous quote, but I think in this case cache invalidation works on our favour. That quote really is about that you never know when to do the invalidation and in our case we will do very smart invalidation (as explained in detail in your question about implicit dependencies). PTAL and let me know if the strategy I explained makes sense to you.
   
   Actually we could even build in some mechanism to invalidate such cash automatically from time to time. My point is that it should not simply be done in the way that the whole image is always build from the scratch (which is the current behaviour) - there is totally no need for that and it has the nasty side effect for the users that it will pollute their docker lib directory with a lot of unused, frequently invalidated images.
   
   In this case the problem is with cache invalidation on the user side in fact. Docker does not know when an already downloaded image will not be needed so it will cache it until someone does 'docker system prune'. Otherwise the /var/lib/docker library will grow forever for someone who will regularly pull airflow images.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services