Quick airflow setup with docker for training or local development

[Gerard Toonstra <gt...@gmail.com>]

Hey,

Ivan Sabik suggested to use docker-compose for my airflow ETL site, so that
it's
easier and quicker to get started. I managed to derive from the
puckel/docker-airflow repository
and made the etl airflow example uppable in 2 steps: The only assumption is
that you have a recent docker environment set up (1.6+ , not ubuntu's 1.5.2
for example).

1 run the docker-compose command
2 start the dag to set up connections, variables and pools.
3 wait
4 ready to rumble


I figured that this can be a useful approach for anyone to set up airflow
training or workshops or
if you need standard local dev environments that you can zap and restart
very easily (the latter really helps to get new joiners up and running in
minutes).

You can find the relevant docker compose here:

https://github.com/gtoonstra/etl-with-airflow/blob/master/docker-compose-LocalExecutor.yml

docs for running it:

https://gtoonstra.github.io/etl-with-airflow/etlexample.html#run-airflow-from-docker


Rgds,

Gerard