You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/08/20 00:07:09 UTC
[GitHub] [airflow] kaxil opened a new pull request #17745: Add triggerer to `docker-compose.yaml` file
kaxil opened a new pull request #17745:
URL: https://github.com/apache/airflow/pull/17745
Adds triggerer component added in #15389 (AIP-40) to the docker-compose.yaml file for quick start
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/main/UPDATING.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-905837509
Yep. Because we stopped using it. I am just about to delete `apache/airflow-ci` images because we switched to `ghcr.io` github images ~ 2 weeks ago and last week we disabled pushing the CI images to "airflow-ci" and I am just about to remove the whole "airflow-ci" . This is the last "checkbox" to complete here: https://github.com/apache/airflow/issues/16555 (but we waited for catching up with PRs based on earlier versions of Airflow).
You can find the latest CI images in `ghcr.io`:
See https://github.com/apache/airflow/blob/main/IMAGES.rst#naming-conventions
You should look at the : `ghcr.io/apache/airflow/main/ci/python3.9:latest` image :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-902986694
> Update the file to need a minimum of 2.2.0 to run this version. If there is another solution in mind, we can follow-up with a PR but until then this will help current testers to directly use this file or Helm chart to quickly and easily test out triggerer
Yep. I am fine with that :) - see my reasoning above :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-905837509
Yep. Because we stopped using it. I am just about to delete `apache/airflow-ci' images because we switched to `ghcr.io` github images ~ 2 weeks ago and last week we disabled pushing the CI images to "airflow-ci" and I am just about to remove the whole "airflow-ci" . This is the last "checkbox" to complete here: https://github.com/apache/airflow/issues/16555 (but we waited for catching up with PRs based on earlier versions of Airflow).
You can find the latest CI images in `ghcr.io`:
See https://github.com/apache/airflow/blob/main/IMAGES.rst#naming-conventions
You should look at the : `ghcr.io/apache/airflow/main/ci/python3.9:latest` image :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-902984636
Update the file to need a minimum of 2.2.0 to run this version. If there is another solution in mind, we can follow-up with a PR but until then this will help current testers to directly use this file or Helm chart to quickly and easily test out triggerer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil merged pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
kaxil merged pull request #17745:
URL: https://github.com/apache/airflow/pull/17745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil closed pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
kaxil closed pull request #17745:
URL: https://github.com/apache/airflow/pull/17745
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-906188877
FYI @jbsilva - > `airflow-ci` has been deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jbsilva edited a comment on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
jbsilva edited a comment on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-905808906
I read the 2.2.0 warning, but seems that the most recent image on Docker Hub (_apache/airflow-ci:main-python3.9_) still does not have the _triggerer_ command:
`airflow command error: argument GROUP_OR_COMMAND: invalid choice: 'triggerer' (choose from 'celery', 'cheat-sheet', 'config', 'connections', 'dags', 'db', 'info', 'jobs', 'kerberos', 'kubernetes', 'plugins', 'pools', 'providers', 'roles', 'rotate-fernet-key', 'scheduler', 'standalone', 'sync-perm', 'tasks', 'users', 'variables', 'version', 'webserver'), see help above.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-902985606
>
> The command to get docker-compose is https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html:
>
> ```
> curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.1.2/docker-compose.yaml'
> ```
I've been helping users with ~30 issues over the last few weeks. Some of them jus took the latest docker-compose and changed the version and run it. The problem is that you get to the "quick-start" page and download the file from there. And then they look at it, change the version to 2.0.2 ("why? Because this is the version that is installed in my production, I wanted to use the same"). They will not switch to earlier version of airlfow in the docs, they will not use the right command for the version. They just don't.
The file itself has no indication which version it supports, it's not named docker-compose-2.1.2 or anything like that, just docker compose. I simply do not want any more to close people's issues beginning with "I used your official docker compose and it does not work". In this case the error will be cryptic ant the user will not know what to do "the triggerrer command will fail".
The "quick-start" should be really "quick" and should give the users maximum guidance on how they should fix the problems, Almost by definition, the users of "quick-start" docker compose know nothing about airflow, and when they will see an error - which they won't understand, they will ... open an issue. I do not want this. I am currently on a spree of making sure we optimise our issue opening/handling experience. The best way to deal with issues is to make sure they will not get opened in the first place.
This is why few lines down I've added explicit "fail if the version is < 2.1.0 and tell them to use only newer version. Either we make this docker compose 2.2.0+ (and change the min version below so that user will get explicit "please upgrade message in case they use < 2.2.0) - which is fine, for me as the message will be clear. Or we introduce some kind of backwards compatibility. Either this or that, but "triggerer could not be started" instead of "You should upgrade airflow to 2.2.0" is a very bad message for first-time users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-902855896
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil edited a comment on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
kaxil edited a comment on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-902974705
> I am afraid this is going to fail if someone tries to run it with Airflow < 2.2. We should add some if's here as the docker-compose for quite some time will be used also for older versions of airflow.
Docker-compose does not support conditionals for spinning up services. The only workaround is exiting with `exit 0` in which case you lose health-check benefits. Plus docker-compose files are versioned with Airflow, users can use docker-compose file for that version .
The command to get docker-compose is https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html:
```
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.1.2/docker-compose.yaml'
```
The same reason why a user shouldn't be using older version of Airflow and looking into new docs should apply here.
Plus this is for local testing only, our Helm Chart already have this conditionals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jbsilva commented on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
jbsilva commented on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-905808906
I read the 2.2.0 warning, but seems that the most recent Docker Hub image (_apache/airflow-ci:main-python3.9_) still does not have the _triggerer_ command:
`airflow command error: argument GROUP_OR_COMMAND: invalid choice: 'triggerer' (choose from 'celery', 'cheat-sheet', 'config', 'connections', 'dags', 'db', 'info', 'jobs', 'kerberos', 'kubernetes', 'plugins', 'pools', 'providers', 'roles', 'rotate-fernet-key', 'scheduler', 'standalone', 'sync-perm', 'tasks', 'users', 'variables', 'version', 'webserver'), see help above.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-902985606
>
> The command to get docker-compose is https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html:
>
> ```
> curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.1.2/docker-compose.yaml'
> ```
I've been helping users with ~30 issues over the last few weeks. Some of them jus took the latest docker-compose and changed the version and run it. The problem is that you get to the "quick-start" page and download the file from there. And then they look at it, change the version to 2.0.2 ("why? Because this is the version that is installed in my production, I wanted to use the same"). They will not switch to earlier version of airlfow in the docs, they will not use the right command for the version. They just don't.
The file itself has no indication which version it supports, it's not named docker-compose-2.1.2 or anything like that, just docker compose. I simply do not want any more to close people's issues beginning with "I used your official docker compose and it does not work". In this case the error will be cryptic ant the user will not know what to do "the triggerrer command will fail". The "quick-start" should be really "quick" and should give the users maximum guidance on how they should fix the problems, Almost by definition, the users of "quick-start" docker compose know nothing about airflow, and when they will see an error - which they won't understand, they will ... open an issue. I do not want this. I am currently on a spree of making sure we optimise our issue opening/handling experience. The best way to deal with issues is to make sure they will not get opened in the first place.
This is why few lines down I've added explicit "fail if the version is < 2.1.0 and tell them to use only newer version. Either we make this docker compose 2.2.0+ (and change the min version below so that user will get explicit "please upgrade message in case they use < 2.2.0) - which is fine, for me as the message will be clear. Or we introduce some kind of backwards compatibility. Either this or that, but "triggerer could not be started" instead of "You should upgrade airflow to 2.2.0" is a very bad message for first-time users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kaxil commented on pull request #17745: Add triggerer to `docker-compose.yaml` file
Posted by GitBox <gi...@apache.org>.
kaxil commented on pull request #17745:
URL: https://github.com/apache/airflow/pull/17745#issuecomment-902974705
> I am afraid this is going to fail if someone tries to run it with Airflow < 2.2. We should add some if's here as the docker-compose for quite some time will be used also for older versions of airflow.
Why would someone use this
> I am afraid this is going to fail if someone tries to run it with Airflow < 2.2. We should add some if's here as the docker-compose for quite some time will be used also for older versions of airflow.
Docker-compose does not support conditionals for spinning up services. The only workaround is exiting with `exit 0` in which case you lose health-check benefits. Plus docker-compose files are versioned with Airflow, users can use docker-compose file for that version .
The command to get docker-compose is https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html:
```
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/2.1.2/docker-compose.yaml'
```
The same reason why a user shouldn't be using older version of Airflow and looking into new docs should apply here.
Plus this is for local testing only, our Helm Chart already have this conditionals.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org