[GitHub] [airflow] gpongracz commented on pull request #8621: Docker Compose CeleryExecutor example

[GitBox <gi...@apache.org>]

gpongracz commented on pull request #8621:
URL: https://github.com/apache/airflow/pull/8621#issuecomment-623185001


   Looks like the entrypoint.sh is not doing an initdb or upgradedb prior to the command like is found in puckel
   https://github.com/puckel/docker-airflow/blob/master/script/entrypoint.sh#L110
   I think that the puckel entrypoint script is very elegant and may help inform design of this one...
   In the mean time I have written a docker-compose for a local executor definition that overcomes this by setting restart to on-failure and starting a container which runs an initdb which then stops after successfully completing - a bit of a cludge but it gets the current image working...
   This is certainly not a production script but it works with the current image
   I've commented out a couple of optional mounts for dags and in my case aws keys
   If a feature similar to the puckel entrypoint above is developed then the webserver and scheduler definitions in the following docker-compose could be collapsed into a single container
   
   **_docker-compose.yml (working example mitigating for entrypoint.sh lacking initdb)_**
   version: '3.7'
   services:
       postgres:
           image: postgres:10
           restart: on-failure
           environment:
               - POSTGRES_USER=airflow
               - POSTGRES_PASSWORD=airflow
               - POSTGRES_DB=airflow
           logging:
               options:
                   max-size: 10m
                   max-file: "3"
           ports:
               - "5432:5432"
       initdb:
           image: apache/airflow:latest
           restart: on-failure
           depends_on:
               - postgres
           environment:
               - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
               - AIRFLOW__CORE__EXECUTOR=LocalExecutor
           logging:
               options:
                   max-size: 10m
                   max-file: "3"
           command: initdb
       webserver:
           image: apache/airflow:latest
           restart: on-failure
           depends_on:
               - postgres
           environment:
               - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
               - AIRFLOW__CORE__LOAD_EXAMPLES=True
               - AIRFLOW__CORE__EXECUTOR=LocalExecutor
               - AIRFLOW__WEBSERVER__BASE_URL=http://localhost:8080
           logging:
               options:
                   max-size: 10m
                   max-file: "3"
   #        volumes:
   #            - ./dags:/opt/airflow/dags
   #            - ~/.aws:/home/airflow/.aws
           ports:
               - "8080:8080"
           command: webserver
       scheduler:
           image: apache/airflow:latest
           restart: on-failure
           depends_on:
               - postgres
           environment:
               - AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
               - AIRFLOW__CORE__LOAD_EXAMPLES=True
               - AIRFLOW__CORE__EXECUTOR=LocalExecutor
               - AIRFLOW__WEBSERVER__BASE_URL=http://localhost:8080
           logging:
               options:
                   max-size: 10m
                   max-file: "3"
   #        volumes:
   #            - ./dags:/opt/airflow/dags
   #            - ~/.aws:/home/airflow/.aws
           command: scheduler


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org