You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/07/06 07:44:08 UTC
[GitHub] [airflow] Hartorn opened a new pull request #9684: Update some dependencies
Hartorn opened a new pull request #9684:
URL: https://github.com/apache/airflow/pull/9684
---
I had some trouble to install this along some latest version of packages, with Poetry as a dependency manager.
So here is what I needed to change :
- latest version of flask-swagger
- latest version of gunicorn
- latest version of jinja2
Not sure if you guys are concerned with this, but there is some great tools to help manage dependency in Github, such as Renovate.
Else, if I want to try and contribute a bit, where should I start ? :)
Make sure to mark the boxes below before creating PR: [x]
- [x] Description above provides context of the change
- [x] Unit tests coverage for changes (not needed for documentation changes)
- ~~ Target Github ISSUE in description if exists~~
- [x] Commits follow "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)"
- [x] Relevant documentation is updated including usage instructions.
- [ x I will engage committers as explained in [Contribution Workflow Example](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#contribution-workflow-example).
---
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in [UPDATING.md](https://github.com/apache/airflow/blob/master/UPDATING.md).
Read the [Pull Request Guidelines](https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#pull-request-guidelines) for more information.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654072262
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (flake8, pylint and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/master/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/master/docs/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/master/BREEZE.rst) for testing locally, itโs a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#coding-style-and-best-practices).
Apache Airflow is a community-driven project and together we are making it better ๐.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://apache-airflow-slack.herokuapp.com/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654165981
Awesome work, congrats on your first merged pull request!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Hartorn commented on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
Hartorn commented on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654256966
No worries, I will manage with a fork for now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Hartorn commented on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
Hartorn commented on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654147432
@potiuk I made the update.
Not sure why I had to do this though, isn't is possible to declare depencies in a pyproject.toml with python version ^3.6 ?
Wouldn'it do the same ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654163965
> @potiuk I made the update.
>
> Not sure why I had to do this though, isn't is possible to declare depencies in a pyproject.toml with python version ^3.6 ?
> Wouldn'it do the same ?
It is explained few paragraphs above: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies
Airflow is not a standard project - it's both an "application" (for people installing it) and "library" (for people writing DAGs). Usually applications have pinned dependencies and libraries have dependencies as opened as possible. This article is one of many that explains why: http://blog.chrisgorgolewski.org/2017/12/to-pin-or-not-to-pin-dependencies.html . Since we are a bit of both. We had to come up with a clever handling of deps:
* In setup.py we have as open deps as possible so that you can upgrade to newer version of depended library when it is released so that you can develop DAGs using latest compatible versions of libraries
* In "requirements.txt" we keep the pinned requirement file that you can use as --constraint when installing airflow as "application" (via pip install) (see INSTALL for recommended way of installing airflow).
The "Genereate requirements" process is our way to keep the pinned dependencies updated regularly and our test suite makes sure that when you generated a new dependencies, those are used during tests and we know that those new dependencies did not break airflow (if we find that they do, we either fix it or limit versions in setup.py - it happened many time in the past that we experienced this problem).
There are subtle differences between different python versions for those - you can compare those generated deps. - that's why have separate constraint file for each version of python.
I hope that explains why our process is a bit complex than with a regular library or application (yes, I'd love to use [poetry](https://python-poetry.org/) for that but poetry does not support this application <> library dualism we have).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654087241
You will need to regenerate requirements if you change those. See https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#generating-requirement-files (also for python 3.8)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk merged pull request #9684:
URL: https://github.com/apache/airflow/pull/9684
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654163965
> @potiuk I made the update.
>
> Not sure why I had to do this though, isn't is possible to declare depencies in a pyproject.toml with python version ^3.6 ?
> Wouldn'it do the same ?
It is explained few paragraphs above: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies
Airflow is not a standard project - it's both an "application" (for people installing it) and "library" (for people writing DAGs). Usually applications have pinned dependencies and libraries have dependencies as opened as possible. This article is one of many that explains why: http://blog.chrisgorgolewski.org/2017/12/to-pin-or-not-to-pin-dependencies.html . Since we are a bit of both, we had to come up with a clever handling of deps:
* In setup.py we have as open deps as possible so that you can upgrade to newer version of depended library when it is released so that you can develop DAGs using latest compatible versions of libraries
* In "requirements.txt" we keep the pinned requirement file that you can use as --constraint when installing airflow as "application" (via pip install) (see INSTALL for recommended way of installing airflow).
The "Genereate requirements" process is our way to keep the pinned dependencies updated regularly and our test suite makes sure that when you generated a new dependencies, those are used during tests and we know that those new dependencies did not break airflow (if we find that they do, we either fix it or limit versions in setup.py - it happened many time in the past that we experienced this problem).
There are subtle differences between different python versions for those - you can compare those generated deps. - that's why have separate constraint file for each version of python.
I hope that explains why our process is a bit complex than with a regular library or application (yes, I'd love to use [poetry](https://python-poetry.org/) for that but poetry does not support this application <> library dualism we have).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Hartorn commented on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
Hartorn commented on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654185398
@potiuk Thks for the great answer, I will check in depth both the links and the diff
Cheers
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654163965
> @potiuk I made the update.
>
> Not sure why I had to do this though, isn't is possible to declare depencies in a pyproject.toml with python version ^3.6 ?
> Wouldn'it do the same ?
It is explained few paragraphs above: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies
Airflow is not a standard project - it's bot an "application" (for people installing it) and "library" (for people writing DAGs). Usually applications have pinned dependencies and libraries have dependencies as opened as possible. This article is one of many that explains why: http://blog.chrisgorgolewski.org/2017/12/to-pin-or-not-to-pin-dependencies.html . Since we are a bit of both. We had to come up with a clever handling of deps:
* In setup.py we have as open deps as possible so that you can upgrade to newer version of depended library when it is released so that you can develop DAGs using latest compatible versions of libraries
* In "requirements.txt" we keep the pinned requirement file that you can use as --constraint when installing airflow as "application" (via pip install) (see INSTALL for recommended way of installing airflow).
The "Genereate requirements" process is our way to keep the pinned dependencies updated regularly and our test suite makes sure that when you generated a new dependencies, those are used during tests and we know that those new dependencies did not break airflow (if we find that they do, we either fix it or limit versions in setup.py - it happened many time in the past that we experienced this problem).
There are subtle differences between different python versions for those - you can compare those generated deps. - that's why have separate constraint file for each version of python.
I hope that explains why our process is a bit complex than with a regular library or application (yes, I'd love to use poetry for that but poetry does not support this application <> library dualism we have).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654229851
FYI @Hartorn -> we have to revert this one temporarily. We did not handle the case where tests were triggered after only the "setup.py" changed (#9690 ) so this one broke the master (via upgrade to one of azure dependencies) - for precisely the reason I described in my comment above :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654163965
> @potiuk I made the update.
>
> Not sure why I had to do this though, isn't is possible to declare depencies in a pyproject.toml with python version ^3.6 ?
> Wouldn'it do the same ?
It is explained few paragraphs above: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies
Airflow is not a standard project - it's bot an "application" (for people installing it) and "library" (for people writing DAGs). Usually applications have pinned dependencies and libraries have dependencies as opened as possible. This article is one of many that explains why: http://blog.chrisgorgolewski.org/2017/12/to-pin-or-not-to-pin-dependencies.html . Since we are a bit of both. We had to come up with a clever handling of deps:
* In setup.py we have as open deps as possible so that you can upgrade to newer version of depended library when it is released so that you can develop DAGs using latest compatible versions of libraries
* In "requirements.txt" we keep the pinned requirement file that you can use as --constraint when installing airflow as "application" (via pip install) (see INSTALL for recommended way of installing airflow).
The "Genereate requirements" process is our way to keep the pinned dependencies updated regularly and our test suite makes sure that when you generated a new dependencies, those are used during tests and we know that those new dependencies did not break airflow (if we find that they do, we either fix it or limit versions in setup.py - it happened many time in the past that we experienced this problem).
There are subtle differences between different python versions for those - you can compare those generated deps. - that's why have separate constraint file for each version of python.
I hope that explains why our process is a bit complex than with a regular library or application (yes, I'd love to use [poetry](https://python-poetry.org/) for that but poetry does not support this application <> library dualism we have).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654163965
> @potiuk I made the update.
>
> Not sure why I had to do this though, isn't is possible to declare depencies in a pyproject.toml with python version ^3.6 ?
> Wouldn'it do the same ?
It is explained few paragraphs above: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies
Airflow is not a standard project - it's both an "application" (for people installing it) and "library" (for people writing DAGs). Usually applications have pinned dependencies and libraries have dependencies as opened as possible. This article is one of many that explains why: http://blog.chrisgorgolewski.org/2017/12/to-pin-or-not-to-pin-dependencies.html . Since we are a bit of both, we had to come up with a clever handling of deps:
* In setup.py we have as open deps as possible so that you can upgrade to newer version of depended library when it is released so that you can develop DAGs using latest compatible versions of libraries
* In "requirements.txt" we keep the pinned requirement file that you can use as --constraint when installing airflow as "application" (via pip install) - see [INSTALL](INSTALL) for recommended way of installing airflow and some more options in CONTIBUTING.rst where you can get the constraint files directly from GitHub. This way you can be sure that you can have working airflow installed no matter if someone released a new version of their libraries that breaks it (happened many times in the past)
The "Genereate requirements" process is our way to keep the pinned dependencies updated regularly and our test suite makes sure that when you generated a new dependencies, those are used during tests and we know that those new dependencies did not break airflow (if we find that they do, we either fix it or limit versions in setup.py - it happened many time in the past that we experienced this problem).
There are subtle differences between different python versions for those - you can compare those generated deps. - that's why have separate constraint file for each version of python.
I hope that explains why our process is a bit complex than with a regular library or application (yes, I'd love to use [poetry](https://python-poetry.org/) for that but poetry does not support this application <> library dualism we have).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654229851
FYI @Hartorn -> we have to revert this one temporarily. We did not handle the case where tests were triggered after only the "setup.py" changed (#9690 ) so this one broke the master - for precisely the reason I described in my comment above :).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk commented on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654230602
Usually changes in deps were coupled with some code changes - that's why we have not seen this problem before.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on pull request #9684: Update some dependencies
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on pull request #9684:
URL: https://github.com/apache/airflow/pull/9684#issuecomment-654163965
> @potiuk I made the update.
>
> Not sure why I had to do this though, isn't is possible to declare depencies in a pyproject.toml with python version ^3.6 ?
> Wouldn'it do the same ?
It is explained few paragraphs above: https://github.com/apache/airflow/blob/master/CONTRIBUTING.rst#airflow-dependencies
Airflow is not a standard project - it's both an "application" (for people installing it) and "library" (for people writing DAGs). Usually applications have pinned dependencies and libraries have dependencies as opened as possible. This article is one of many that explains why: http://blog.chrisgorgolewski.org/2017/12/to-pin-or-not-to-pin-dependencies.html . Since we are a bit of both, we had to come up with a clever handling of deps:
* In setup.py we have as open deps as possible so that you can upgrade to newer version of depended library when it is released so that you can develop DAGs using latest compatible versions of libraries
* In "requirements.txt" we keep the pinned requirement file that you can use as --constraint when installing airflow as "application" (via pip install) - see [INSTALL](INSTALL) for recommended way of installing airflow and some more options in CONTIBUTING.rst where you can get the constraint files directly from GitHub.
The "Genereate requirements" process is our way to keep the pinned dependencies updated regularly and our test suite makes sure that when you generated a new dependencies, those are used during tests and we know that those new dependencies did not break airflow (if we find that they do, we either fix it or limit versions in setup.py - it happened many time in the past that we experienced this problem).
There are subtle differences between different python versions for those - you can compare those generated deps. - that's why have separate constraint file for each version of python.
I hope that explains why our process is a bit complex than with a regular library or application (yes, I'd love to use [poetry](https://python-poetry.org/) for that but poetry does not support this application <> library dualism we have).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org