You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/06 12:45:26 UTC
[GitHub] [airflow] akihiro-inui opened a new issue #17470: Airflow scheduler does not start properly.
akihiro-inui opened a new issue #17470:
URL: https://github.com/apache/airflow/issues/17470
**Apache Airflow version**:
apache/airflow:2.1.2-python3.8
**Environment**:
Ubuntu 18.04
**What happened**:
When launching Airflow scheduler together with other Airflow services, it throws an error.
Error message
```
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
airflow-scheduler_1 | self.dialect.do_execute(
airflow-scheduler_1 | File "/home/airflow/.local/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 609, in do_execute
airflow-scheduler_1 | cursor.execute(statement, parameters)
airflow-scheduler_1 | psycopg2.errors.UndefinedColumn: column dag.concurrency does not exist
airflow-scheduler_1 | LINE 1: ..., dag.schedule_interval AS dag_schedule_interval, dag.concur...
```
**What you expected to happen**:
I expected Airflow scheduler to properly launch.
**How to reproduce it**:
I use docker-compose to launch Airflow services which use Dockerfile-Airflow to build the image.
Dockerfile-Airflow
```
FROM apache/airflow:2.1.2-python3.8
USER root
RUN apt-get update && apt-get install -y --no-install-recommends \
g++ \
unixodbc-dev \
unixodbc \
libpq-dev
RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN curl https://packages.microsoft.com/config/ubuntu/18.04/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update
RUN ACCEPT_EULA=Y apt-get install -y --allow-unauthenticated msodbcsql17
RUN ACCEPT_EULA=Y apt-get install -y --allow-unauthenticated mssql-tools
RUN usermod -u 50000 airflow
RUN groupmod -g 50000 airflow
USER 50000
COPY --chown=airflow project /opt/airflow/project
COPY --chown=airflow project/dags /opt/airflow/dags
COPY --chown=airflow airflow/logs /opt/airflow/logs
COPY --chown=airflow setup.cfg /setup.cfg
COPY --chown=airflow setup.py /setup.py
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bash_profile
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc
RUN echo 'export AIRFLOW_HOME="$PATH:/opt/airflow"' >> ~/.bash_profile
RUN echo 'export AIRFLOW_HOME="$PATH:/opt/airflow"' >> ~/.bashrc
RUN pip install -e .[all]
```
And this is my docker-compose.yml
```
version: '3'
x-project-common:
&project-common
build:
context: .
dockerfile: Dockerfile-airflow
environment:
&project-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: "postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres/${POSTGRES_DB}"
AIRFLOW__CELERY__RESULT_BACKEND: "db+postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres/${POSTGRES_DB}"
AIRFLOW__CELERY__BROKER_URL: redis://:@redis:6379/0
AIRFLOW__CORE__FERNET_KEY: ''
AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: 'true'
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
DB_HOST: ${DB_HOST}
DB_NAME: ${DB_NAME}
DB_PORT: ${DB_PORT}
DB_USER: ${DB_USER}
DB_PASSWORD: ${DB_PASSWORD}
DB_DRIVER: ${DB_DRIVER}
project_DB_SCHEMA: ${project_DB_SCHEMA}
DETECT_DB_SCHEMA: ${DETECT_DB_SCHEMA}
TEXTOTRON_HOST: ${TEXTOTRON_HOST}
TEXTOTRON_PORT: ${TEXTOTRON_PORT}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: ${POSTGRES_DB}
user: "${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-50000}"
depends_on:
redis:
condition: service_healthy
postgres:
condition: service_healthy
services:
postgres:
image: postgres:13
environment:
<<: *project-common-env
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ["CMD", "pg_isready", "-U", "airflow"]
interval: 5s
retries: 5
restart: always
redis:
image: redis:latest
ports:
- 6379:6379
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 30s
retries: 50
restart: always
airflow-webserver:
<<: *project-common
command: webserver
ports:
- 8080:8080
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-scheduler:
<<: *project-common
command: scheduler
healthcheck:
test: ["CMD-SHELL", 'airflow jobs check --job-type SchedulerJob --hostname "$${HOSTNAME}"']
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-worker:
<<: *project-common
command: celery worker
healthcheck:
test:
- "CMD-SHELL"
- 'celery --app airflow.executors.celery_executor.app inspect ping -d "celery@$${HOSTNAME}"'
interval: 10s
timeout: 10s
retries: 5
restart: always
airflow-init:
<<: *project-common
command: version
environment:
<<: *project-common-env
_AIRFLOW_DB_UPGRADE: 'true'
_AIRFLOW_WWW_USER_CREATE: 'true'
_AIRFLOW_WWW_USER_USERNAME: ${_AIRFLOW_WWW_USER_USERNAME:-airflow}
_AIRFLOW_WWW_USER_PASSWORD: ${_AIRFLOW_WWW_USER_PASSWORD:-airflow}
flower:
<<: *project-common
command: celery flower
ports:
- 5555:5555
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:5555/"]
interval: 10s
timeout: 10s
retries: 5
restart: always
volumes:
postgres-db-volume:
```
Then, use
docker-compose build
docker-compose up
should replicate the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17470: Airflow scheduler does not start properly.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17470:
URL: https://github.com/apache/airflow/issues/17470#issuecomment-894634389
You missed 'before you begin' and/or especially 'initialize environment' steps of the quick-start. https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html .
You likely did not initialize the database.
Please follow them precisely as they are written.
The quick start is written in the way that you have to follow it step-by-step. It's not production ready, it's not gonna work if you omit certain steps or modify them.
It's not a good idea to omit the steps from it if you do not know exactly what you are doing.
Clean up everything, including your docker environment 'docker compose down --volumes --remove-orphans' remove everything from the directory you work in and just follow the quick start precisely step-by-step.
Note that the docker compose is NOT production ready - things like database initialisation for example (which you apparently missed) is a separate step for example. If you want to make production-ready docker-compose you should modify it (for example the postgres db used for quick start uses Local docker volume which is likely not best solution for production db).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #17470: Airflow scheduler does not start properly.
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #17470:
URL: https://github.com/apache/airflow/issues/17470
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17470: Airflow scheduler does not start properly.
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17470:
URL: https://github.com/apache/airflow/issues/17470#issuecomment-894634389
You missed 'before you begin' and/or especially 'initialize environment' steps of the quick-start. https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html .
You likely did not initialize the database.
Please follow them precisely as they are written.
The quick start is written in the way that you have to follow it step-by-step. It's not production ready, it's not gonna work if you omit certain steps or modify them.
It's not a good idea to omit the steps from it if you do not know exactly what you are doing.
Clean up everything, including your docker environment 'docker compose down --volumes --remove-orphans' remove everything from the directory you work in and just follow the quick start precisely step-by-step.
Note that the docker compose is NOT production ready - things like database initialisation (which you apparently missed). If you want to make production-ready docker-compose you should modify it (for example the postgres db used for quick start uses Local docker volume which is likely not best solution for production db).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17470: Airflow scheduler does not start properly.
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17470:
URL: https://github.com/apache/airflow/issues/17470#issuecomment-894634389
You missed 'before you begin' and/or especially 'initialize environment' steps of the quick-start. https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html .
You likely did not initialize the database.
Please follow them precisely as they are written.
The quick start is written in the way that you have to follow it step-by-step. It's not production ready, it's not gonna work if you omit certain steps or modify them.
It's not a good idea to omit the steps from it if you do not know exactly what you are doing.
Clean up everything, including your docker environment 'docker compose down --volumes --remove-orphans' remove everything from the directory you work in and just follow the quick start precisely step-by-step.
Note that the docker compose is NOT production ready - things like database initialisation for example (which you apparently missed). If you want to make production-ready docker-compose you should modify it (for example the postgres db used for quick start uses Local docker volume which is likely not best solution for production db).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org