[DISCUSS][HELP-NEEDED] Production docker image WIP

[Jarek Potiuk <Ja...@polidea.com>]

Hello everyone,

I have FINALLY got to the point that I could create a WIP production docker
image PR.

It has all the support from Breeze (``breeze build-prod-image`` with
optional python version and extras). It is quite optimized for size but
likely I will be able to do some more optimizations soon.

If you are one of the people who are eagerly waiting for it - I am happy
to hear your comments, description of how you are using the image and your
requests/questions, etc.

Either here or in the #prod-docker-image channel on Slack. I am looking for
people who would like to test it in their environment. I have not yet fully
documented how to use it, but this is something that is rather
straightforward (I hope) and will mostly come from your stories on how you
are going to use it.

I already know about a few changes that I will have to implement to support
different python versions for 1.10 and I will be working on it now. Plus
testing with different databases and Kubernetes (using our CI environment).
And I am not 100% sure if webserver works in it.

I am looking forward to your input/comments in the PR, here or on Slack.
Note it's rather Alpha quality and I have not run a lot tests with it yet.

The PR is here: https://github.com/apache/airflow/pull/7832. When you check
it out you will be able to build it on your own - Breeze has there full
support for building the image locally (`breeze build-prod-image`) in a
nice way - with caching options,  ability to specify your own extras,
choosing python versions etc. BTW. For local development I recommend to use
`--use-local-cache` switch in Breeze.

The "base" images can be downloaded from my personal Dockerhub registry:

docker pull potiuk/airflow:master-python3.7
docker pull potiuk/airflow:master-python3.6

Note that the images are now built using those extras as default:


 async,azure_blob_storage,azure_cosmos,azure_container_instances,celery,crypto,

 elasticsearch,gcp,kubernetes,mysql,postgres,s3,emr,redis,slack,ssh,statsd,virtualenv

The images are for now only available for airflow 2.0 but I will be porting
them to 1.10.10.

Happy imaging!

J.

-- 

Jarek Potiuk
Polidea <https://www.polidea.com/> | Principal Software Engineer

M: +48 660 796 129 <+48660796129>
[image: Polidea] <https://www.polidea.com/>