You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "V4kodin (via GitHub)" <gi...@apache.org> on 2025/02/27 14:19:21 UTC
[D] Airflow ci/cd pipeline to rebuild airflow [airflow]
GitHub user V4kodin created a discussion: Airflow ci/cd pipeline to rebuild airflow
I configured airflow pipeline, but have question about pipeline to restart airflow properly.
So this part of gitlab ci/cd pipeline should restart airflow after core changes in airflow configuration:
```
rebuild_project:
stage: rebuild
script:
- ssh ${DEPLOY_USER}@${DEPLOY_SERVER} "cd ${DEPLOY_PATH} && docker compose down && docker compose build --no-cache && docker compose up -d"
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
changes:
- requirements.txt
- start.sh
- docker-compose.yaml
- Dockerfile
- plugins/**/*
- config/**/*
```
But I had issue with `docker compose down`, once I used, it I got endless shutting down on worker, So I was forced to stop docker service and manually remove worker container.
Is there any proper way to rebuild Airflow via pipeline? Or I just should always do it manually?
It would be nice if pipeline would have an opportunity to pause all dags, when wait until all dags completed and restart after that (maybe unpause all pause dags after start).
GitHub link: https://github.com/apache/airflow/discussions/47149
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow ci/cd pipeline to rebuild airflow [airflow]
Posted by "enchant3dmango (via GitHub)" <gi...@apache.org>.
GitHub user enchant3dmango edited a comment on the discussion: Airflow ci/cd pipeline to rebuild airflow
If you want to pause DAGs in your CI/CD pipeline, you can use this [CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#pause).
It will be something like the following. You can use `docker compose up --build -d` instead of separating the build and restart command.
```yaml
rebuild_project:
stage: rebuild
script:
- ssh ${DEPLOY_USER}@${DEPLOY_SERVER} << 'EOF'
cd ${DEPLOY_PATH}
# Pause all DAGs
docker compose exec airflow-webserver airflow dags pause <regex-for-your dags> --treat-dag-id-as-regex
# Wait until all tasks are finished
while docker compose exec airflow-webserver airflow tasks list_instances | grep -q running; do
echo "Waiting for running tasks to finish..."
sleep 10
done
# Stop Airflow cleanly
docker compose down
# Rebuild and restart
docker compose up --build -d
# Unpause previously paused DAGs
docker compose exec airflow-webserver airflow dags unpause <regex-for-your dags> --treat-dag-id-as-regex
EOF
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
changes:
- requirements.txt
- start.sh
- docker-compose.yaml
- Dockerfile
- plugins/**/*
- config/**/*
```
GitHub link: https://github.com/apache/airflow/discussions/47149#discussioncomment-12340565
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow ci/cd pipeline to rebuild airflow [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
GitHub user potiuk added a comment to the discussion: Airflow ci/cd pipeline to rebuild airflow
I think you need to trace down what happens and what error you have - there is no "one way" - especially that docker-compose is not something we have recommended or have a guildeline to on how to productionize it, so any scripts you do, you need to make sure you test. I guess you mist have some error - say for example your docker compose down will fail (not sure how docker compose works) if there are no containers and that might break your whole command.
Not sure also what "endless shutdown means" and why - this is only you can see on what stops your compose from shutting down. That needs you to login to the server and see what's going on when you do "docker compose down" - there might be multiple reasons.
Re - pausing and unpausing dags - you could - potentially utilise CLI (as @enchant3dmango mentioned) or - better - REST API and python client we release, to iterate through and pause or maybe stop all the running tasks.
Again note that docker compose is not a perfect way of deploying airlfow as it lacks a lot of flexibility of things like K8S executor for example. Where you can shutdown and restart airflow without actually stopping all running pods (Airlfow has "pod adoption" mechanism in this case. So it's a bit of your chose to choose a platform (docker-compose) that has very little support for more "prodution" support you would like to see, and requiring from you to do more custom work to make it happen.
GitHub link: https://github.com/apache/airflow/discussions/47149#discussioncomment-12340604
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow ci/cd pipeline to rebuild airflow [airflow]
Posted by "enchant3dmango (via GitHub)" <gi...@apache.org>.
GitHub user enchant3dmango edited a comment on the discussion: Airflow ci/cd pipeline to rebuild airflow
If you want to pause DAGs in your CI/CD pipeline, you can use this [CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#pause).
It will be something like the following. You can use `docker compose up --build -d` instead of separating the build and restart command.
```yaml
rebuild_project:
stage: rebuild
script:
- ssh ${DEPLOY_USER}@${DEPLOY_SERVER} << 'EOF'
cd ${DEPLOY_PATH}
# Pause all DAGs
docker compose exec airflow-webserver airflow dags pause <regex-for-your-dags> --treat-dag-id-as-regex
# Wait until all tasks are finished
while docker compose exec airflow-webserver airflow tasks list_instances | grep -q running; do
echo "Waiting for running tasks to finish..."
sleep 10
done
# Stop Airflow cleanly
docker compose down
# Rebuild and restart
docker compose up --build -d
# Unpause previously paused DAGs
docker compose exec airflow-webserver airflow dags unpause <regex-for-your-dags> --treat-dag-id-as-regex
EOF
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
changes:
- requirements.txt
- start.sh
- docker-compose.yaml
- Dockerfile
- plugins/**/*
- config/**/*
```
GitHub link: https://github.com/apache/airflow/discussions/47149#discussioncomment-12340565
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org
Re: [D] Airflow ci/cd pipeline to rebuild airflow [airflow]
Posted by "enchant3dmango (via GitHub)" <gi...@apache.org>.
GitHub user enchant3dmango added a comment to the discussion: Airflow ci/cd pipeline to rebuild airflow
If you want to pause DAGs in your CI/CD pipeline, you can use this [CLI](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#pause).
It will be something like the following. You can use `docker compose up --build -d` instead of separating the build and restart command.
```yaml
rebuild_project:
stage: rebuild
script:
- ssh ${DEPLOY_USER}@${DEPLOY_SERVER} << 'EOF'
cd ${DEPLOY_PATH}
# Pause all DAGs
docker compose exec airflow-webserver airflow dags pause <regex-for-your dags> --treat-dag-id-as-regex
# Wait until all tasks are finished
while docker compose exec airflow-webserver airflow tasks list_instances | grep -q running; do
echo "Waiting for running tasks to finish..."
sleep 10
done
# Stop Airflow cleanly
docker compose down
# Rebuild and restart
docker compose up --build -d
# Unpause previously paused DAGs
docker compose exec airflow-webserver airflow dags unpause --all
EOF
rules:
- if: '$CI_COMMIT_BRANCH == "main"'
changes:
- requirements.txt
- start.sh
- docker-compose.yaml
- Dockerfile
- plugins/**/*
- config/**/*
```
GitHub link: https://github.com/apache/airflow/discussions/47149#discussioncomment-12340565
----
This is an automatically sent email for commits@airflow.apache.org.
To unsubscribe, please send an email to: commits-unsubscribe@airflow.apache.org